Refereed Conference Proceedings
Balancing Student Success and Inferring Personalized Effects in Dynamic Experiments (2019). Proceedings of the 12th International Conference on Educational Data Mining. Hammad Shaikh, Arghavan Modiri, Joseph Jay Williams, and Anna Rafferty. [Poster] [Slides]
Randomized controlled trials (RCTs) can be embedded in educational technologies to evaluate how interventions affect student outcomes and how effectiveness varies with characteristics like prior knowledge. But RCTs often assign many students to ineffective conditions. Adaptive algorithms like contextual multi-armed bandits (MABs) could change how students are assigned to conditions over time, offering the potential to both evaluate effectiveness for subgroups of students and direct more students to interventions that are effective for them. We use simulations to compare contextual MABs to traditional RCTs and non-contextual MABs. Contextual MABs improve student outcomes for each subgroup; in contrast, non-contextual MABs may help one group of students, such as those with high prior knowledge, while hurting another. Because both MAB algorithms adaptively assign conditions based on prior students’ results, both recover biased estimates of condition effectiveness. However data collected from a contextual MAB is still nearly as good for inferring the optimal policy assignment as a traditional RCT.
Revisiting the Racial Achievement Gap: Evidence from Recent US Panel Data (2017)
This paper examines the racial gap in academic achievement in elementary and high school using recently released large-scale nationally representative panel data sets from the US. I find achievement disparities among white, black, Asian, and Hispanic students at kindergarten entry in mathematics, reading, science and working memory. Controlling for differences in household and school characteristics across race explains about 70% of the raw minority-white gaps, but moderate unexplained gaps still remain. Black-white and Hispanic-white math and reading achievement gaps widen as children progress through school, but achievement gaps in science and working memory remain constant. Significant racial achievement gaps exist in all key subjects in grade 9 and remain constant throughout high school. Bond and Lang (2013) argue that test scores are generally ordinal and show that certain results in the racial achievement gap literature depend on the cardinalization of the test score scale. To check the sensitivity of my primary results I use the methodology developed by Penney (2017) that provide estimates robust to the ordinality critique.
Research in Progress
The Impact of Information Disclosure on Student Course Selection (2018), with Robert McMillan and Linda Wang
University students may enrol in less challenging courses despite being eligible for more rigorous courses that may be more beneficial for them. This mismatch of easy courses and high-quality students discourages departments from offering challenging courses, and also limits the human capital potential of post-secondary students. We conduct a randomized experiment to examine the impact of informing high-achieving first-year economics students about their future course options. We find that providing an information session significantly increases the probability of eligible first-year students enrolling in the most rigorous second-year economics courses. Our result sheds light on the determinants of students’ course selection and suggests course enrolment strategies for departments and universities.