Using ChatGPT to Teach Scientific Reasoning Skills (in Psychology)

Summary: By getting the prompts right, ChatGPT can provide excellent 1 on 1 tutoring for students on how to apply scientific reasoning in social science (and probably all sciences).

I am currently teaching advanced undergraduate psychology and sociology students to use scientific reasoning in designing research studies and evaluating results of existing research. At the very beginning of the semester, students showed a strong tendency towards repetition- and memory-type knowledge, rather than employ the higher level cognitive skills (evaluation, analysis) that scientific reasoning requires. Once it became clear, however, that repetition-type responses were inadequate, the majority of students began rapidly developing their independent scientific reasoning skills. I measured a 10% improvement over the first eight weeks. 

For example, I might ask students to evaluate the following bit of evidence,

Mr. Rodriguez, a high school principal, reported having seen a therapist for eight months and concluded, “Therapists are useless. They didn’t help me one bit!”

I have designed this question by introducing a specific type of data (anecdote or testimonial) that has poor scientific reliability and validity, but I deliver it from a person who has social credibility. Can students look past this person’s position of power and evaluate the evidence objectively?

 

At the beginning of the semester, here is the kind of response that is common:

Student: Some therapists may not be helpful. 

Note how, in this example, the student hasn’t actually evaluated the evidence. Instead, they have given their own opinion, however vague it might be. After eight weeks, students reliably produced answers like this:

Student: The evidence is that Mr. Rodriguez saw a therapist for eight months and did not find them helpful. But this is testimonial evidence, which can’t be used to draw conclusions 

We can see that this time the student answered the question. Moreover, they have recognized a problem in the evidence that is not explicitly stated, and they have drawn an inference based on that problem. There is still room for further clarity in this response, but it demonstrates the beginning use of scientific reasoning.

The Problem

I have 25 students in my class, and they don’t all make the same mistakes. Approximately half of the students make the same mistake, I’ve found, and I can address this majority problem as a group and all at once. But the other half make mistakes that surprise me enough to think, "Now how did you get there?!" I have to go through the thought process of each student individually to see where they got tripped up. And even when I discover the mistake, that doesn’t mean one example of how to solve the problem from me will be sufficient in helping this student understand their mistake.

 

For example, I had an accomplished and confident student give an answer like this (all of these answers are pure inventions of mine, by the way; but they have been created with the purpose of demonstrating the mistakes I’ve seen):

Mr. Rodriguez is a high school principal who believes that therapy doesn’t work based on his experience. I think therapy is effective due to a lot of factors, even if Mr. Rodriguez the high school principal disagrees. Lots of factors and lack of control in the study make me say that Mr. Rodriguez’s evidence is not as good as it could otherwise be.

In analysis, we can see that this student is aware that mysterious factors and a “lack of control” will lead us to question the reliability of Mr. Rodriguez’s testimonial. But it isn’t clear to me that this student understands A) what sorts of factors (or controls) might impact the testimonial and B) how that impact might occur. In other words, this student has a lot of vague statements that fail to demonstrate their understanding of the words they’re using.

 

Now, I’ve often responded like this: “You’re being vague. You haven’t specified which controls need to be employed or why, so I’m not sure you understand what "control" means. Can you specify that for me?” But students, if they don't retreat into their turtle shells, still give a lot of vague answers. Add to this the self-reported problem of their being afraid to ask questions because they don’t want me or their classmates to think that they’re stupid.

How AI Can Help

This is where AI programs can step in. AI, such as ChatGPT, can help students develop their scientific reasoning skills, provided they (the students) are careful in what prompts they give AI. For example, ChatGPT can answer the question just fine, but this won’t help students learn which mistakes they (the students) are making.

 

I asked ChatGPT the same question as above.

Me: Evaluate the evidence: Mr. Rodriguez, a high school principal, reported having seen a therapist for eight months and concluded, “Therapists are useless. They didn’t help me one bit!”

ChatGPT: Mr. Rodriguez's conclusion that “therapists are useless” is not supported by strong scientific reasoning due to several key issues:

1. Anecdotal Evidence & Generalization

His statement is based solely on his personal experience, which does not represent the effectiveness of therapy as a whole. Therapy outcomes vary based on factors such as the therapist’s approach, the individual’s engagement, and the specific mental health condition being treated.

2. Lack of Consideration for Other Factors

Several factors could explain why therapy did not help him, including:

The type of therapy used (e.g., cognitive-behavioral therapy, psychodynamic therapy) may not have been the best fit for his needs.

The therapist-client relationship plays a crucial role in effectiveness; if there was poor rapport, therapy might not have been as beneficial.

Personal engagement and commitment—progress in therapy often depends on active participation, homework completion, and applying learned strategies outside of sessions.

Duration of therapy—some conditions require longer than eight months to see significant improvement.

This is an excellent response that goes a little beyond what I would expect nontherapist students to create on their own. While the response might help students get a sense of what “other factors” might look like in practice or how generalization plays a role in interpreting evidence, I don’t think it would do as good a job as helping a student come up with a good answer on their own.

Getting the AI Prompt Right

And so I played around the kinds of prompts I gave AI. First, I asked AI to improve the scientific reasoning employed in the previous (vague) answer:

Asking the AI for Help Improving Scientific Reasoning

Me: Help me improve my scientific reasoning in my answer to the same question: “Mr. Rodriguez is a high school principal who believes that therapy doesn’t work based on his experience. I think therapy is effective due to a lot of factors, even if Mr. Rodriguez the high school principal disagrees. Lots of factors and lack of control in the study make me say that Mr. Rodriguez’s evidence is not as good as it could otherwise be.”

ChatGPT: Here’s a revised version of your response with stronger scientific reasoning:

"Mr. Rodriguez’s conclusion that therapy is ineffective is based solely on his personal experience, which is a form of anecdotal evidence. This type of evidence lacks scientific rigor because it does not control for important variables that influence therapy outcomes. Factors such as the type of therapy, the therapist-client relationship, the individual’s level of engagement, and the duration of treatment all play a role in effectiveness. Without a systematic study comparing therapy outcomes across different individuals and conditions, his experience cannot be used to make a general conclusion about the usefulness of therapy. Research consistently shows that evidence-based therapy is effective for many mental health conditions, making his claim scientifically weak."

Analysis: This is another great answer, but seeing the answer doesn’t mean that students will understand how to get there. Indeed, it might just make them feel hopeless or encourage them to find some way to cheat.

I asked the AI To Help Me Understand How to Improve My Scientific Reasoning

Next, I adjusted the question I asked. That’s when the tutoring really began.

Me: Help me understand how to improve my scientific reasoning in my previous answer. 

ChatGPT: To improve your scientific reasoning, consider these key aspects:

1. Avoid Vague Language

Your original response includes phrases like “I think therapy is effective due to a lot of factors” and “Lots of factors and lack of control in the study”. These are too general and don’t specify which factors matter or why they affect the conclusion. Stronger reasoning requires being precise about what variables influence therapy outcomes.

Improvement: Instead of “a lot of factors”, specify relevant variables:

• Type of therapy used (e.g., CBT, psychodynamic)
• Quality of the therapist-client relationship
• Level of patient engagement
• Duration of therapy

This is essentially the kind of answer I have given to those students who have sought out my feedback, which is the minority of the class. But with the help of AI, students can continue down the path and ask follow-up questions, such as…

Me: Help me learn how to replace vague phrases with precise scientific explanations.

ChatGPT: Great question! Replacing vague phrases with precise scientific explanations improves the clarity and credibility of your argument. Here’s how you can do it:

---

1. Identify Vague Phrases

Vague phrases are often general, subjective, or unclear about what they mean. Look for words like:

• "a lot of factors" → What factors?
• "lack of control in the study" → What specific controls are missing?
• "Mr. Rodriguez's evidence is not as good as it could be" → Why is it weak?

---

2. Specify Key Variables or Concepts

Scientific reasoning involves identifying specific variables that affect an outcome. Instead of saying "a lot of factors," break it down:

Vague:
"Therapy is effective due to a lot of factors."

Precise:
"Therapy effectiveness depends on factors such as the type of therapy (e.g., CBT, psychodynamic), the therapist-client relationship, patient engagement, and treatment duration."

---

3. Explain Why the Evidence is Weak

If you’re rejecting someone’s claim, explain exactly why their evidence is flawed.

Vague:
"Mr. Rodriguez’s evidence is not as good as it could be."

Precise:
"Mr. Rodriguez’s claim is based on anecdotal evidence, which lacks scientific rigor because it does not control for variables such as individual differences, treatment type, or placebo effects."

---

4. Define Key Scientific Concepts

If you mention things like "lack of control," be specific about what that means in a scientific study.

Vague:
"There was a lack of control in the study."

Precise:
"The absence of a control group prevents us from determining whether therapy caused the improvement or if symptoms would have improved naturally."

---

5. Reference Empirical Research

When making a scientific argument, cite established research or principles.

Vague:
"Studies show therapy works."

Precise:
"Meta-analyses of randomized controlled trials indicate that cognitive-behavioral therapy (CBT) significantly reduces symptoms of depression and anxiety compared to placebo conditions."

---

Practice Exercise: Try Rewriting These Vague Statements

1. "Diet affects health in many ways."
2. "Technology changes people’s behavior."
3. "The study had some problems."

Give it a try, and I’ll help refine your responses! 😊

Conclusion

If I was working with a student on how to clarify their language, then I would have stopped after the first question. I would have asked, “You said ‘other factors.’” What factors do you have in mind?” Or I would have simply said, “You have to be more specific.” Unless I had an hour alone with the student and nothing else to do, I would never be able to come up with as many examples, sample problems, and bits of advice as ChatGPT provided in the space of about 5 minutes and four prompts.

I hope it’s clear that, with the right prompts, ChatGPT and most likely other AI programs can work as individual tutors for students. [It took this many questions until I lost access to the best model of ChatGPT—GPT-4o.]

For fun, I repeated the same series of prompts with the GPT-3.5 downgraded program. I didn’t spend as long looking at or evaluating the output, but it seemed to be just as helpful (i.e., significantly more helpful than I would be as a tutor).