In this chapter you will learn about using standardized effects for differences in proportions more interpretable.
There are several methods for computing standardizd effect sizes for differences in proportions. For example, Cohen (of Cohen’s d fame) developed a metric called Cohen’s h, which measures the distance between two proportions. This is based on an arcsine transformation of the proportions (remember your trigonometry??) and is computed as:
\[ h = 2 \arcsin(\sqrt{p_1}) – 2 \arcsin(\sqrt{p_2}) \]
where \(p_1\) and \(p_2\) are the sample proportions you want to measure the effect between. We can illustrate how to compute this standardized effect using the bachelor.csv data to find Cohen’s h to between the proportion of Bachelor and Bachelorettes that are still together with their chosen suitors. Below we import the data and compute summary values for the two groups.
library(educate)
library(ggformula)
library(mosaic)
library(mosaicCore)
library(tidyverse)
# Import data
bachelor <- read_csv("https://raw.githubusercontent.com/zief0002/epsy-5261/main/data/bachelor.csv")
# View data
bachelor# Compute proportions
df_stats(~still_together | show, data = bachelor, props)Warning: Excluding 2 rows due to missing data [df_stats()].Based on the proportion that are still together, \(\hat{p}_{\text{Bachelor}} = .111, ~\hat{p}_{\text{Bachelorette}} = .174\), the difference is .063—this is a raw effect of the difference. To compute Cohen’s h, the standardized effect, we can employ those proportions in to the formula above. Because arcsine transformations might not be at the tip of your brain, we use R to do this with the function asin().
# Compute Cohen's h
2 * asin(sqrt(.174)) - 2 * asin(sqrt(.111))[1] 0.1812569Cohen’s h value is .181. While the metric used for Cohen’s h is standardized, we typically don’t interpret the value; we just report it. We can, however, interpret the magnitude of Cohen’s h using the same guidelines as Cohen’s d:
\[ \begin{split} h &\approx 0.2 \qquad \text{Small Effect} \\[2ex] h &\approx 0.5 \qquad \text{Moderate Effect} \\[2ex] h &\approx 0.8 \qquad \text{Large Effect} \end{split} \] In this example the difference between the two proportions indicates that it is a small effect.
Cohen’s h can also be computed when comparing a single sample proportion to a hypothesized proportion (one-sample situation). For example, in Chapter 12 we evaluated whether the proportion of all Minnesota children under 6 years of age who are above the MDH reference level in 2018 was different than 0.029 (the proportion in 2012). Here the two inputs to Cohen’s h are the sample proportion (\(\hat{p}=.0139\)) and the value being tested in the null hypothesis (0.029)
# Compute Cohen's h
2 * asin(sqrt(.0139)) - 2 * asin(sqrt(.029))[1] -0.1059095Cohen’s h value is \(-0.11\), which corresponds to a small effect. (Note that the sign is irrelevant when considering the size of the effect.)
Import the fastx-reviews.csv data and compute the proportion of all critics’ “Fresh” reviews.
Use the nrow() function to compute the sample size. Just input the name of the data frame object into this function.
Use the formula (and R) to compute Cohen’s h comparing the sample proportion of “fresh” reviews to the hypothesized value of 0.5. Also indicate the magnitude of the effect based on Cohen’s guidelines.
The ci_cohens_h() function from the {educate} package can be used to find Cohen’s h and compute a confidence level for the population effect size. This function takes the following arguments:
p1: Proportion for Sample 1n1: Sample size for Sample 1n1: Proportion for Sample 2n2: Sample size for Sample 2conf.level: The confidence level for the interval (default is conf.level = 0.95)For example, to compute the 95% confidence interval for Cohen’s h in the population for our Bachelor/Bachelorette case study, we use the following syntax:
# Compute CI for Cohen's h
ci_cohens_h(
p1 = 0.174, #Proportion for Bachelorette
n1 = 25, #Sample size for Bachelorette
p2 = .111, #Proportion for Bachelor
n2 = 27, #Sample size for Bachelor
conf.level = 0.95
)
--------------------------------------------------
95% Confidence Interval
--------------------------------------------------
Cohen's h = 0.09062846
Lower Limit = -0.3627418
Upper Limit = 0.7252556
--------------------------------------------------The CI is \([-0.36,~0.73]\). Note that the lower limit is negative. This suggests that the proportion of Bachelors that stay together with their chosen suitor might be higher than the proportion of Bachelorettes that stay together with their chosen suitor. So, this CI is indicating that:
We can also use the ci_cohens_h() function to compare a single sample to a hypothesized value (e.g., finding the effect for a one-sample situation). To do this, whichever “sample” you use to input the hypothesized value (say p2), the corresponding sample size is inputted as Inf (i.e., n2=Inf). This ensures that only the actual sample summaries are used to compute the uncertainty in the CI.
Use the ci_cohens_h() function to compute the 95% confidence interval for the population effect in the Rotten Tomatoes example comparing to the hypothesized value of 0.5. For the sample size corresponding to the hypothesized value in the function, enter the value Inf.
Interpret the CI by commenting on the potential direction and magnitude of the effect.
A more commonly reported effect size is a risk ratio, also sometimes referred to as relative risk. This effect is most commonly used in medicine and health sciences, but is quite useful. A risk ratio measures the risk of a certain event happening in one group compared to the risk of the same event happening in another group. For example, a health research might compare the risk of COVID in low income communities to that in high income communities.
“Risk” is simply a proportion of something occurring in a group. In our Bachelor/Bachelorette example, “risk” might be staying together (or alternatively, breaking up). To compute the risk ratio, or relative risk, we are just computing the ratio of two proportions:
\[ \text{Risk Ratio} = \frac{p_1}{p_2} \] In our Bachelor/Bachelorette example the risk ratio of staying together is:
\[ \begin{split} \text{Risk Ratio} &= \frac{p_1}{p_2} \\[2ex] &= \frac{.174}{.111} \\[2ex] &= 1.56 \end{split} \] We can interpret this as Bachelorettes (numerator proportion) are 1.56 times as likely to stay together as Bachelors (denominator proportion).
The choice of which group goes in the numerator is irrelevant. However, interpretationally it is easier if you put the bigger proportion in the numerator.
We can also compute a risk ratio when we only have a single sample. Rather than computing the ratio of two sample proportions, we compute the ratio of the sample proportion and the hypothesized value of \(\pi\) tested in a hypothesis test.
As an example, consider the lead levels case study from Chapter 12. We evaluated whether the the proportion of all Minnesota children under 6 years of age who are above the MDH reference level in 2018 different than .029 (the proportion in 2012?), by testing the following hypotheses:
\[ \begin{split} H_0: \pi_{\text{Above MDH}} = .029 \\[1ex] H_A: \pi_{\text{Above MDH}} \neq .029 \end{split} \]
To compute a risk ratio, we import the data and compute numerical summaries.
# Import data
mn_lead <- read_csv("https://raw.githubusercontent.com/zief0002/epsy-5261/main/data/mn-lead.csv")Rows: 91706 Columns: 2
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (1): ebll
dbl (1): lead_level
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.# View data
mn_lead# Compute numerical summaries
df_stats(~ebll, data = mn_lead, props)To compute the risk ratio:
\[ \begin{split} \text{Risk Ratio} &= \frac{p}{\pi} \\[2ex] &= \frac{.014}{.029} \\[2ex] &= 0.48 \end{split} \]
We can interpret this as the proportion of all Minnesota children under 6 years of age who are above the MDH reference level in 2018 is less than half of that is 2012. Although typically we would put the larger number in the numerator, here we want to make the comparison back to the 2012 values, so we put that referent in the denominator. Remember, the choice of which value to put in the denominator is arbitrary—it just changes the interpretation.
We can compute a confidence interval for a risk ratio using the ci_risk_ratio() function from the {educate} package. This function takes the following arguments:
p1: Proportion for Sample 1n1: Sample size for Sample 1n1: Proportion for Sample 2n2: Sample size for Sample 2conf.level: The confidence level for the interval (default is conf.level = 0.95)For example, to compute the 95% confidence interval for the risk ratio in our Bachelor/Bachelorette case study, we use the following syntax:
# Compute CI for risk ratio
ci_risk_ratio(
p1 = 0.174, #Proportion for Bachelorette
n1 = 25, #Sample size for Bachelorette
p2 = .111, #Proportion for Bachelor
n2 = 27, #Sample size for Bachelor
conf.level = 0.95
)
--------------------------------------------------
95% Confidence Interval
--------------------------------------------------
Risk Ratio = 1.567568
Lower Limit = 0.7941019
Upper Limit = 3.094399
--------------------------------------------------The CI is \([0.79,~3.09]\), indicating that Bachelorettes (numerator proportion) are between 0.79 and 3.11 times as likely to stay together as Bachelors (denominator proportion). Note that the lower limit is below 1. This suggests that Bachelors are more likely than Bachelorettes to stay together with their chosen suitor. A risk ratio of 1 suggests that both groups are equally likely to stay together with their chosen suitor. So, this CI is indicating quite a lot of uncertainty:
conf_risk_ratio() With a Single SampleWe can also use the ci_risk_ratio() function to compare a single sample to a hypothesized value (e.g., in our MN lead exposure case study). To do this, whichever “sample” you use to input the hypothesized value (say p2), the corresponding sample size is inputted as Inf (i.e., n2=Inf). This ensures that only the actual sample summaries are used to compute the uncertainty in the CI.
For example, to compute the CI for the risk ratio we computed earlier for the MN lead exposure example, we would input:
# Compute CI for risk ratio
ci_risk_ratio(
p1 = 0.014, #Sample proportion
n1 = 91706, #Sample size
p2 = .029, #Hypothesized value
n2 = Inf, #Inf goes with the hypothesized value
conf.level = 0.95
)
--------------------------------------------------
95% Confidence Interval
--------------------------------------------------
Risk Ratio = 0.4827586
Lower Limit = 0.4698893
Upper Limit = 0.4959804
--------------------------------------------------The CI is \([0.470,~.496]\), indicating the proportion of all Minnesota children under 6 years of age who are above the MDH reference level in 2018 is between .470 and .496 times that in 2012. The high level of precision in the estimate is attributable to the large sample size, so we can say with some degree of certainty that the 2018 numbers are likely less than half of those in 2012.
Use the ci_risk_ratio() function to compute the risk ratio and the 99% CI for the population risk ratio that compares the proportion of “Fresh” reviews to the hypothesized value of 0.5.
Interpret the confidence interval by commenting on the plausible values in the interval.