Comparing Groups
Group comparisons are at the heart of many interesting questions addressed by psychologists, physicians, scientists, teachers, and engineers. For example, data scientists at OKCupid, an online dating site, examined whether frequent tweeters (users of Twitter) have shorter real-life relationships than others.1 In another example, researchers from Vanderbilt University asked the question of whether students who enter college with A.P. credits graduate with less student debt than their peers who do not have A.P. credits.2
Liao (2002) summed up group comparisons in the following manner: “The nature of doing science, be it natural or social, inevitably calls for comparison. Statistical methods are at the heart of such comparison, for they not only help us gain understanding of the world around us but often define how our research is to be carried out.”
Scientific Experiments: The Gold Standard of Group Comparisons
Questions about group differences are often studied through scientific experiments. When considering a scientific experiment to examine group differences, the design of the study plays a very important role. To help understand this, think about a researcher who is studying the efficacy of a new cold medication. Let’s say that the researcher has 100 people (each with a cold) who volunteer to be a part of her study. Let’s consider how she might design her study.
- Design 1: She gives the cold medicine to all 100 volunteers.
- Design 2: She gives the cold medicine to the first 50 volunteers (treatment group) and nothing to the other 50 volunteers (control group).
- Design 3: She randomly picks 50 of the volunteers to whom she gives the cold medicine (treatment group), and she gives nothing to the other 50 volunteers (control group).
All three designs have been used, and are still used, in research studies. There are pros and cons to each of the designs, and all are useful depending on what you want to know.
In Design 1, it is hard to judge the efficacy of the medication. For example, what if 60 of the volunteers had no cold symptoms after four days? Did the medication work? You might be thinking, “what would have happened if they hadn’t received any medication?” That is a great question. In this design, we don’t know.
Design 2 gives the researcher a comparison group. She can compare the number of volunteers in each group who have no cold symptoms after four days. This is a better design than Design 1 for examining efficacy. But, what if she found that after four days, 35 of the volunteers who got the medication had no symptoms, while only 25 of the volunteers who didn’t receive medication had no symptoms. Is this enough evidence for her to say the cold medication is effective? Probably not. Maybe most of the volunteers in the treatment group were already in later stages of their colds. Maybe they had more robust immune systems to begin with (e.g., due to differing exercise or nutrition habits) than the control group. You can imagine many such reasons that the treatment group would show quicker improvement than the control group.
Design 3 has the same comparison group advantage as Design 2. The big difference, however, is that the volunteers were put into the groups at random. By assigning participants at random, the researcher “equalizes” the treatment and control groups. What this means is that the groups have, on average, the SAME nutritional habits, the SAME exercise habits, and the SAME everything-else. That means that the only thing that is different between the two groups is that the treatment group got the cold medication and the control group didn’t. If the researcher uses this type of design, she can draw much stronger inferences about WHY the treatment group improved: it was because of the cold medication!
KEY COMPONENTS TO SCIENTIFIC EXPERIMENTS
The two key components of a scientific experiment are that there is a control and a treatment group , and participants are randomly asdsigned to either the treatment or control group. In the control group participants do not receive the treatment. This allows the control group to act as a reference group for our comparison. The random assignment makes the treatment and control groups statistically equivalent on all other factors so that we fan attribute any differences to the fact that the treatment group got the treatment and the control group did not.
Goals of Unit 3
In this unit, you will learn about experimental variation (the variation that we get just because of random assignment to groups). You will explore how to account for experimental variation and model that variation using a randomization test (a Monte Carlo method for evaluating whether an observed result in compatible with experimental variation from a hypothesized model). You will also learn how to carry out a randomization test using TinkerPlots. Finally, you will also learn about the importance of random assignment in drawing causal conclusions about the effect of a treatment in group comparisons.
References
The website OKTrends includes an answer to this question, as well as many others.↩︎
They found students with 10 or more AP credits have $1,000 less debt, on average, than those who don’t. Those students aslo tend to graduate earlier, are more likely to double major, and attend graduate school (see here).↩︎