Four Simple Design Strategies for Applied Research in Education (AR in E Part V of III)
I have deliberately waited to begin introducing research design strategies. This is because I believe that your questions and goals for your classroom should lead to your selection of a research design. The relationship should not be reversed.
Also, after looking at a few of my personal examples (specifically the examples in Chapters 3 and 4), you have seen how formal design guidelines are not always necessary. The more you practice applied research, the more formal your research will get. Not because you are learning to follow rules and be good researchers, but because your understanding of the mechanics of learning and relating in the classroom will have become so sophisticated that your research demands it. This all comes with practice.
What is a Design Strategy
A research design is sort of like a recipe for baking a cake. There are lots of different recipes for baking cake. Some have flour and some don’t. Some have sugar and others have a sugar substitute. Some have 11 layers and some have a single layer. And so on. But all cake recipes share a few common characteristics: they each call for something sweet, something fatty, and something that binds it all together.
Your design strategy will identify how you will define your key variables, and how you will record improvement or lack thereof.
Variables
Any research study has two types of variables: independent and dependent. The independent variable is the one that you are interested in playing around with in the hopes that it will lead to improvements. Dependent variables are the ones you look at after implementing your intervention. The dependent variables tell you if the intervention was successful or not.
If you are interested in improving students’ achievement on vocabulary quizzes, then you might look at the following variables.
Independent (classroom things you might change):
· Vocabulary study habits
· Vocabulary classroom activities
· Time spent on vocabulary
· Semantic mapping activity
· Group vocabulary activities
· Etc.
Dependent (outcomes you will monitor or measure)
· Vocabulary quiz scores
· Perceived student enjoyment
· Teacher enjoyment
· Number of class interruptions
· Student use of vocabulary terms without being asked
· Etc.
In my experience, I have found that it is helpful to state independent and dependent variables from the outset, but to always leave room and awareness for the role of variables that I had not anticipated. As I’ve shared before, I am sometimes surprised by the change that takes place—a change that is no doubt positive and beneficial, yet one that I could never have predicted. For example, during a semester where I designed a fully nondirectional advanced psychology course, I noticed how students felt more confident in their other classes. I had never imagined that such an outcome could occur, but I was pleased that it had!
Final note: it isn’t necessary to label your variables as “independent variable” and “dependent variable.” That isn’t really how human being speak. Instead, you might say something like, “I am going to try using small group semantic mapping exercises on all vocabulary terms for a week. I think that it will be fun, and that it will help students learn their vocabulary words better.” The methodological equivalent, which, again, is the way that nobody actually speaks in real life, would sound like this: “I am introducing a semantic mapping vocabulary activity as my independent variable, and will measure vocabulary quiz scores as my dependent variable.”
Table 6.1. Stating Your Variables in Ordinary and Methodological Forms
Spoken Like a Human Being (Goal) | “I am going to try using small group semantic mapping exercises on all vocabulary terms for a week. I think that it will be fun, and that it will help students learn their vocabulary words better.” |
Methodological Translation | “I am introducing a semantic mapping vocabulary activity as my independent variable, and will measure vocabulary quiz scores as my dependent variable.” |
Design Strategy #1: Test – Retest
The Test – Retest design strategy is the simplest and easiest to conduct. It is likely what you have already used if you have ever made an observation about student performance going up or down. In order to say something such as, “Students are improving!” you have to know where they are coming from. If the class average for this week’s vocabulary quiz is 90%, then how can we tell if that is an improvement? We have to go to the previous week or the previous weeks to see what those averages were. Only then can we get an idea which direction the test scores are moving.
The same sort of design can be used when looking at qualitative data. In order to say whether or not student morale has improved, you will have to collect some preliminary data about student morale. This means asking students for their opinion about vocabulary words and such before beginning your new intervention. Or, at the very least, making some preliminary observations, such as, “I always have at least four students complain about vocabulary words.” Then, after the intervention, you can count and see if the number of complaints goes up or down.
Table 6.2.
Week of the School Year | Vocab. Quiz Class Avg. | Average |
1 | 77 | 77 |
2 | 83 | |
3 | 71 |
Intervention
4 | 88 | 87.3 |
5 | 83 | |
6 | 91 |
In Table 6.2, I give an example of a table of pre-intervention data and post-intervention data. During weeks 1, 2, and 3 a hypothetical teacher continues teaching as usual. Or maybe they simply go into the record books to see what scores were like in the past. This is usually pretty easy data to collect if you have to keep a grading book. During weeks 4, 5, and 6 the new vocabulary activity is being used. Therefore, the quiz results from weeks 4, 5, and 6 can be compared with those of weeks 1, 2, and 3 to see if the scores went up or down.
It would be possible to use a single score to represent your pre-intervention data and a single score to represent your post-intervention data. But keep in mind that this data will be less reliable. It is best to collect as much pre-intervention and post-intervention data as possible. Notice in Table 6.2 how the scores in week 2 and week 5 are identical. If our hypothetical teacher only had these two data points, then they would probably conclude that the vocabulary activity had no effect on the classroom outcomes. But we can see that there probably was an effect, and the effect looks pretty big, too.
Advanced: Statistical Tests of Significance
If you want to take your applied research to a higher level—if, for example, you plan on writing an article about your intervention and want to publish the results—then you might wish to conduct a statistical test of significance (such as a t-test or analysis of variance). In order to do so, you will need more than six data points (represented above as class average vocabulary quiz scores from weeks 1-6). You will probably want to record the individual quiz scores from all students in class each week. If there are 20 students in class, then that will mean that you have about 20 data points from each week, or 60 data points for pre-test and 60 data points for post-test. The averages will still work out to 77 and 87.3, but now your data sets will capture all the variability in test scores. With these much larger data sets, you can use a statistical analytics software such as Microsoft Excel or SPSS to conduct a mean comparison test, to see if the 10.3% difference between 87.3 and 77 is statistically significant. If this sounds exciting to you, then I’m sure you will have no trouble tracking down the formulae you’ll need to make it happen.
Design Strategy #2: Quasi-Experimental
The next most common applied research design that you might want to use called quasi-experimental. The “quasi-” prefix just means that it doesn’t call for the same level of control as true experimental design.
In order for a study to qualify as an experiment, participants (which, in your case, will probably be your students) must 1) be selected from the population at random; 2) be randomly assigned to experimental and control conditions; and 3) have the option of choosing not to participate. It is unlikely that students in your class satisfy and of these conditions. That’s okay. Experimental conditions are not intended to resemble the real world where teachers and students learn together. Experimental conditions are intended to exert maximum control over variable influences.
Since you are not constrained by experimental conditions, you get to decide just how controlled you want to make your study. If this is your first time, then I recommend keeping it as simple as possible and forgetting about brainstorming all the possible confounding variables and designing strategies to control against them. Leave that for your subsequent research. (I will provide examples of control later for enterprising teachers.)
Version 1: Dividing Up Your Class
In the Test – Retest design, you were using the same group of students (your class) in the control (pre-intervention test) and experimental (post-intervention test) conditions. With quasi-experimental, you keep these groups separate. One group of students will experience your intervention, and another group of students will experience something different (or nothing at all). Then you compare the results from both groups of students.
For example, during week 1 you might divide your classroom in half. One half gets the semantic mapping worksheets along with the vocabulary words. The other half only gets the vocabulary words. Both groups are given 15 minutes to work on what they’ve been given. After the vocabulary quiz, you compare the scores between the two groups. Just like the Test – Retest design, it is best to continue collecting data over a few weeks in order to increase the reliability.
Version 2: Using a Control Class
It might just be easier to use another class as your control sample. That way you don’t have to design two separate activities for your single class. If you teach 4th Grade, for example, then you might find another 4th Grade teacher using the same vocabulary list who would be willing to let you use their scores as a comparison. In this example, you could have your whole class do the new vocabulary activity, and then compare your class’s score with the scores from the comparison class (i.e., the other 4th Grade class).
One summer, for example, I asked a colleague to administer a questionnaire to his pair of history courses. The questionnaire measured students’ perceived autonomy support from their instructor. I gave my students the same questionnaire. The only difference was that I had designed my summer course around providing high levels of autonomy support and my colleague hadn’t. I compared the two sets of results, and found that there was no statistically significant difference between the two groups (see Advanced: Statistical Tests of Significance, above). But I also compared total activities completed, feedback from students, and creativity exhibited. While my students did not register higher perceived autonomy support, they did seem to exhibit more creativity and self-directed interest, which told me that I was headed in the right direction even if I hadn’t quite nailed the delivery of autonomy support.
Qualitative
So far I have only given examples of quantitative research design. With these, you are counting and comparing numbers: test scores, attendance figures, questionnaire results, demerits or bonus points given, and so on. But, as a teacher, you are attuned to much more that is going on in your classrooms. You are sensitive to student attitudes, well-being, relationships, emotion, courage and worry, anxiety, pride, and on and on. You can feel when students are excited and when they are bored. You can feel when they need a little bit more time on a subject or when they’re ready to move on. This sensitivity marks the art of being an excellent teacher, and it grows with more experience—but only when you spend time reflecting on it and building this awareness. Qualitative research is an excellent way to do this.
There are many formal qualitative methods that you could follow in designing an applied study. But, as I have said throughout this manual, I think it is best to err on the side of too little rigor than too much. This is because, in the end, your goal is to develop your skills as a teacher. It won’t help if you burn yourself out going overboard on a carefully controlled study. If you can succeed in growing your understanding about teaching and learning and classroom mechanics just a tiny bit, then you will be successful. You can adopt more formal research methods once you feel like you have gotten the hang of the process.
Qualitative data is what you have before any sort of transformation or interpretation has occurred. A ten-page description of what happened during last week’s vocabulary quiz—right down to the three students who sat chewing the ends of their pencils as time ran out. This whole description, unanalyzed and unmodified, is your chunk of qualitative data. How you analyze it is up to you.
Quantitative data is pretty easy to analyze—even if you find yourself deep down into the statistical analysis rabbit hole. Quantitative data can be counted, added, subtracted, averaged, compared, and so on. What do you do with 10 pages of written description? Here’s what you do: you let your teacher-self take over. Read over (or listen to, or observe) what happened like the teacher that you are. What do you notice? What stands out to you? What seems significant? What seems important? What confuses you? What is obvious to you? And so on. You are the analytic tool.
Say, for example, that you are using the semantic mapping tool that I have discussed over and over again. You distribute this to your students and watch as they puzzle over what they are supposed to be doing. You can sense confusion, some interest and intrigue, and you might even see a few light bulbs go off. You answer their questions and see what you can do to help them. This goes on for a few weeks, and you can begin to feel the despair or gratitude that students have for the activity. If you have been paying attention, then you will probably also know how best to use the activity, and which steps/procedures to avoid. If you pay extra careful attention, then you might even learning something new about how students adopt new vocabulary words as their own, or the mistakes they make while doing so.
After this process, you will have a deep and personal understanding of how the vocabulary activity went. The next step is to clarify that understanding for yourself. Ask yourself, “What happened?” and “Was this beneficial?” “What would I do differently?” and so on.
This will seem strange at first. You have probably been taught to avoid reflecting on statements that begin with, “I feel that…” or “I believe that…”, when you’re trying to solve classroom problems or build understanding. But adopting an objective perspective—the view that you can see things perfectly for what they really are and unblemished by bias—ignores the personal knowledge that you have. For more on this subject, take a look at what Michael Polanyi has to say about knowledge (Polanyi 2009, 2022).
Mix-and-Mash (Combining Strategies)
Looking back at my applied research, almost all of it has been some combination of what I’ve described above. I try to collect quantitative and qualitative data (also called “mixed design”), and use all of it to best capture my understanding of what has happened in my classrooms. I have also found it helpful when other teachers, colleagues, and supervisors have questions about my work: some prefer to see the numbers while other prefer to see the descriptions. Personally, I find the personal reflections and analyses to be the most beneficial. With numbers, I always worry that the quantities don’t actually stand for what I hope they stand for. So I’m always taking a risk when using them.
As always, choose the strategy or strategies that make the most sense to you, as well as the one(s) that you are most comfortable using.
Comments
Post a Comment