Large language models present new opportunities for teaching and learning. The response accuracy of these models, however, is believed to depend on the prompt quality which can be a challenge for students. In this study, we aimed to explore how undergraduate students use ChatGPT for problem-solving, what prompting strategies they develop, the link between these strategies and the model’s response accuracy, the existence of individual prompting tendencies, and the impact of gender in this context. Our students used ChatGPT to solve five problems related to embedded systems and provided the solutions and the conversations with this model. We analyzed the conversations thematically to identify prompting strategies and applied different quantitative analyses to establish relationships between these strategies and the response accuracy and other factors. The findings indicate that students predominantly employ three types of prompting strategies: single copy-and-paste prompting (SCP), single reformulated prompting (SRP), and multiple-question prompting (MQP). ChatGPT’s response accuracy using SRP and MQP was significantly higher than using SCP, with effect sizes of -0.94 and -0.69, respectively. The student-by-student analysis revealed some tendencies. For example, 26 percent of the students consistently copied and pasted the questions into ChatGPT without any modification. Students who used MQP showed better performance in the final exam than those who did not use this prompting strategy. As for gender, female students tended to make extensive use of SCP, whereas male students tended to mix SCP and MQP. We conclude that students develop different prompting strategies that lead to different response qualities and learning. More research is needed to deepen our understanding and inform effective educational practices in the AI era.

🧠 Awesome ChatGPT Prompts [CSV dataset]

This is a Dataset Repository of Awesome ChatGPT Prompts View All Prompts on GitHub

  License

CC-0

Objective: Our objective is to evaluate the efficacy of ChatGPT 4 in accurately and effectively delivering genetic information, building on previous findings with ChatGPT 3.5. We focus on assessing the utility, limitations, and ethical implications of using ChatGPT in medical settings. Materials and Methods: A structured questionnaire, including the Brief User Survey (BUS-15) and custom questions, was developed to assess ChatGPT 4's clinical value. An expert panel of genetic counselors and clinical geneticists independently evaluated ChatGPT 4's responses to these questions. We also involved comparative analysis with ChatGPT 3.5, utilizing descriptive statistics and using R for data analysis. Results: ChatGPT 4 demonstrated improvements over 3.5 in context recognition, relevance, and informativeness. However, performance variability and concerns about the naturalness of the output were noted. No significant difference in accuracy was found between ChatGPT 3.5 and 4.0. Notably, the effic..., Study Design This study was conducted to evaluate the performance of ChatGPT 4 (March 23rd, 2023)  Model) in the context of genetic counseling and education. The evaluation involved a structured questionnaire, which included questions selected from the Brief User Survey (BUS-15) and additional custom questions designed to assess the clinical value of ChatGPT 4's responses. Questionnaire Development The questionnaire was built on Qualtrics, which comprised twelve questions: seven selected from the BUS-15 preceded by two additional questions that we designed. The initial questions focused on quality and answer relevancy: 1.    The overall quality of the Chatbot’s response is: (5-point Likert: Very poor to Very Good) 2.    The Chatbot delivered an answer that provided the relevant information you would include if asked the question. (5-point Likert: Strongly disagree to Strongly agree) The BUS-15 questions (7-point Likert: Strongly disagree to Strongly agree) focused on: 1.    Recogniti..., , # A comparative evaluation of ChatGPT 3.5 and ChatGPT 4 in responses to selected genetics questions - Full study data

https://doi.org/10.5061/dryad.s4mw6m9cv

This data was captured when evaluating the ability of ChatGPT to address questions patients may ask it about three genetic conditions (BRCA1, HFE, and MLH1). This data is associated with the JAMIA article of the similar name with the DOIÂ 10.1093/jamia/ocae128

Description of the data and file structure

1. Key: This tab contains the data structure, explaining the survey questions, and potential responses available.
2. Prompt Responses: This tab contains the prompts used for ChatGPT, and the response provided from each model (3.5 and 4)
3. GPT 4 Results: This tab provides the responses collected from the medical experts (genetic counselors and clinical geneticist) from the Qualtrics survey.
4. Accuracy (Qx_1): This tab contains the subset of results from both the Ch...

Data Availability Statement (for the paper) All dialogue logs and final responses collected in this study are publicly available in the PROSPECT repository on Zenodo (DOI: [to be assigned]). The repository contains PDF files of complete dialogue histories and Markdown files of final comprehensive analyses for all conditions and models used in this study, allowing for reproducibility and further analysis. ### README.md for Zenodo # PROSPECT: Professional Role Effects on Specialized Perspective Enhancement in Conversational Task ## OverviewThis repository (PROSPECT) contains the dataset associated with the paper:> "Empirical Investigation of Expertise, Multiperspectivity, and Abstraction Induction in Conversational AI Outputs through Professional Role Assignment to Both User and AI" This research analyzed changes in dialogue logs and final responses when professional roles were assigned to both user and AI sides across multiple Large Language Models (LLMs). This repository provides the complete dialogue logs (PDF format) and final responses (Markdown format) used in the analysis. ## Directory StructureThe repository structure under the top directory (PROSPECT/) is as follows: PROSPECT/├── dialogue/ # Dialogue histories (PDF)│ ├── none/│ ├── ai_only/│ ├── user_only/│ └── both/└── final_answers/ # Final responses (Markdown) ├── none/ ├── ai_only/ ├── user_only/ └── both/ - dialogue/ - Contains raw dialogue logs in PDF format. Subdirectories represent role assignment conditions: - none/: No roles assigned to either user or AI - ai_only/: Role assigned to AI only - user_only/: Role assigned to user only - both/: Roles assigned to both user and AI- final_answers/ - Contains final comprehensive analysis responses in Markdown format. Directory structure mirrors that of dialogue/. ## File Naming ConventionFiles in each directory follow this naming convention:[AI]_[conditionNumber]-[roleNumber].pdf[AI]_[conditionNumber]-[roleNumber].md- [AI]: AI model name used for dialogue (e.g., ChatGPT, ChatGPT-o1, Claude, Gemini)- [conditionNumber]: Number indicating role assignment condition - 0: none - 1: ai_only - 2: user_only - 3: both- [roleNumber]: Professional role number - 0: No role - 1: Detective - 2: Psychologist - 3: Artist - 4: Architect - 5: Natural Scientist ### Examples:- ChatGPT_3-1.pdf: Dialogue log with ChatGPT-4o model under "both" condition (3) with detective role (1)- Gemini_1-4.md: Final response from Gemini model under "ai_only" condition (1) with architect role (4) ## Role Number Reference| roleNumber | Professional Role ||-----------:|:-----------------|| 0 | No role || 1 | Detective || 2 | Psychologist || 3 | Artist || 4 | Architect || 5 | Natural Scientist| ## Data Description- Dialogue Histories (PDF format) Complete logs of questions and responses from each session, preserved as captured during the research. All dialogues were conducted in Japanese. While assistant version information is not included, implementation dates and model names are recorded within the files.- Final Responses (Markdown format) Excerpted responses to the final "comprehensive analysis request" as Markdown files, intended for text analysis and keyword extraction. All responses are in Japanese. *Note: This dataset contains dialogues and responses exclusively in Japanese. Researchers interested in lexical analysis or content analysis should consider this language specification. ## How to Use1. Please maintain the folder hierarchy after downloading.2. For meta-analysis or lexical analysis, refer to PDFs for complete dialogues and Markdown files for final responses.3. Utilize for research reproduction, secondary analysis, or meta-analysis. ## LicenseThis dataset is released under the CC BY 4.0 License.- Free to use and modify, but please cite this repository (DOI) and the associated paper when using the data. ## Related Publication ## Disclaimer- The dialogue logs contain no personal information or confidential data.- The provided logs and responses reflect the research timing; identical prompts may yield different responses due to AI model updates.- The creators assume no responsibility for any damages resulting from the use of this dataset. ## ContactFor questions or requests, please contact skeisuke@ibaraki-ct.ac.jp.

Summary of Artifacts

This is the replication package for the paper titled 'Can Developers Prompt? A Controlled Experiment for Code Documentation Generation' that is part of the 40th IEEE International Conference on Software Maintenance and Evolution (ICSME), from October 6 to 11, 2024, located in Flagstaff, AZ, USA.

Full Abstract

Large language models (LLMs) bear great potential for automating tedious development tasks such as creating and maintaining code documentation. However, it is unclear to what extent developers can effectively prompt LLMs to create concise and useful documentation. We report on a controlled experiment with 20 professionals and 30 computer science students tasked with code documentation generation for two Python functions. The experimental group freely entered ad-hoc prompts in a ChatGPT-like extension of Visual Studio Code, while the control group executed a predefined few-shot prompt. Our results reveal that professionals and students were unaware of or unable to apply prompt engineering techniques. Especially students perceived the documentation produced from ad-hoc prompts as significantly less readable, less concise, and less helpful than documentation from prepared prompts. Some professionals produced higher quality documentation by just including the keyword Docstring in their ad-hoc prompts. While students desired more support in formulating prompts, professionals appreciated the flexibility of ad-hoc prompting. Participants in both groups rarely assessed the output as perfect. Instead, they understood the tools as support to iteratively refine the documentation. Further research is needed to understand which prompting skills and preferences developers have and which support they need for certain tasks.

Author Information

Citation Information

@inproceedings{kruse-icsme-2024,
author={Kruse, Hans-Alexander and Puhlf{\"u}r{\ss}, Tim and Maalej, Walid},
booktitle={2022 IEEE International Conference on Software Maintenance and Evolution},
title={Can Developers Prompt? A Controlled Experiment for Code Documentation Generation},
year={2024},
doi={tba},
}

Artifacts Overview

1. Preprint

The file kruse-icsme-2024-preprint.pdf is the preprint version of the official paper. You should read the paper in detail to understand the study, especially its methodology and results.

2. Results

The folder results includes two subfolders, explained in the following.

Demographics RQ1 RQ2

The subfolder Demographics RQ1 RQ2 provides Jupyter Notebook file evaluation.ipynb for analyzing (1) the experiment participants' submissions of the digital survey and (2) the ad-hoc prompts that the experimental group entered into their tool. Hence, this file provides demographic information about the participants and results for the research questions 1 and 2. Please refer to the README file inside this subfolder for installation steps of the Jupyter Notebook file.

RQ2

The subfolder RQ2 contains further subfolders with Microsoft Excel files specific to the results of research question 2:

• The subfolder UEQ contains three times the official User Experience Questionnaire (UEQ) analysis Excel tool, with data entered from all participants/students/professionals.
• The subfolder Open Coding contains three Excel files with the open-coding results for the free-text answers that participants could enter at the end of the survey to state additional positive and negative comments about their experience during the experiment. The Consensus file provides the finalized version of the open coding process.

3. Extension

The folder extension contains the code of the Visual Studio Code (VS Code) extension developed in this study to generate code documentation with predefined prompts. Please refer to the README file inside the folder for installation steps. Alternatively, you can install the deployed version of this tool, called Code Docs AI, via the https://marketplace.visualstudio.com/items?itemName=re-devtools.code-docs-ai" href="https://marketplace.visualstudio.com/items?itemName=re-devtools.code-docs-ai">VS Code Marketplace.

You can install the tool to generate code documentation with ad-hoc prompts directly via the https://marketplace.visualstudio.com/items?itemName=zhang-renyang.chat-gpt" href="https://marketplace.visualstudio.com/items?itemName=zhang-renyang.chat-gpt">VS Code Marketplace. We did not include the code of this extension in this replication package due to license conflicts (GNUv3 vs. MIT).

4. Survey

The folder survey contains PDFs of the digital survey in two versions:

• The file Survey.pdf contains the rendered version of the survey (how it was presented to participants).
• The file SurveyOptions.pdf is an export of the LimeSurvey web platform. Its main purpose is to provide the technical answer codes, e.g., AO01 and AO02, that refer to the rendered answer texts, e.g., Yes and No. This can help you if you want to analyze the CSV files inside the results folder (instead of using the Jupyter Notebook file), as the CSVs contain the answer codes, not the answer texts. Please note that an export issue caused page 9 to be almost blank. However, this problem is negligible as the question on this page only contained one free-text answer field.

5. Appendix

The folder appendix provides additional material about the study:

• The subfolder tool_screenshots contains screenshots of both tools.
• The file few_shots.txt lists the few shots used for the predefined prompt tool.
• The file test_functions.py lists the functions used in the experiment.

Revisions

License

See LICENSE file.

Context

https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1842206%2Fd7e02bf38f4b08df2508d6b6e42f3066%2Fchatgpt2.png?generation=1700233710310045&alt=media" alt="">

Based on their wikipedia page

ChatGPT (Chat Generative Pre-trained Transformer) is a large language model-based chatbot developed by OpenAI and launched on November 30, 2022, that enables users to refine and steer a conversation towards a desired length, format, style, level of detail, and language. Successive prompts and replies, known as prompt engineering, are considered at each conversation stage as a context.

These reviews were extracted from Google Store App

Usage

This dataset should paint a good picture on what is the public's perception of the app over the years. Using this dataset, we can do the following

1. Extract sentiments and trends
2. Identify which version of the app had the most positive feedback, the worst.
3. Use topic modeling to identify the pain points of the application.

(AND MANY MORE!)

Note

Images generated using Bing Image Generator

Dataset Card for ShareGPT52K90K

  Dataset Summary

This dataset is a collection of approximately 52,00090,000 conversations scraped via the ShareGPT API before it was shut down. These conversations include both user prompts and responses from OpenAI's ChatGPT. This repository now contains the new 90K conversations version. The previous 52K may be found in the old/ directory.

  Supported Tasks and Leaderboards

text-generation

  Languages

This dataset is… See the full description on the dataset page: https://huggingface.co/datasets/RyokoAI/ShareGPT52K.