2027 Public Health PhD Research: Choosing a Comparison Group for a Policy Evaluation
Co-Founder and Chief Data Scientist
A public health policy can coincide with better outcomes without causing them. For a PhD researcher, the comparison group is central to separating a policy effect from changes that would have happened anyway. This guide is for research-oriented applicants and doctoral students planning evaluations of policies such as insurance expansions, tobacco regulations, or local prevention measures. It explains how to choose a comparison group, assess whether the comparison is credible, match it to data and statistical methods, and decide what research preparation a feasible dissertation requires.
Key Things You Should Know
- Choose the comparison group by asking what would likely have happened to the policy-exposed population without the policy - not merely which population is easiest to access.
- Check policy eligibility, exposure timing, baseline characteristics, prepolicy outcome trends, spillovers, and data comparability before selecting a design or model.
- Multiple prepolicy observations strengthen trend assessment, but they cannot prove that an unexposed group provides a valid counterfactual; report remaining uncertainty.
What Is a Comparison Group in Public Health Policy Evaluation?
A comparison group is a population, place, or time period used to estimate the outcome that policy-exposed units might have experienced without the policy. That unobserved outcome is the counterfactual. A credible comparison group resembles the exposed group in factors that shape the outcome, while remaining meaningfully unexposed to the policy.
Consider an illustrative evaluation of a city's smoke-free workplace ordinance. Workers in a city without the ordinance might be a comparison group for respiratory-related outcomes. They would be a poor choice, however, if a statewide rule reaches both cities or if their workplaces, health risks, and prepolicy trends differ substantially. The researcher must specify whether the question concerns the policy's effect on all eligible residents, people actually reached, or a narrower group; those are different effects.
Students entering this work from another discipline can use online public health bachelor's program features to identify introductory epidemiology and statistics preparation. That foundation helps, but it does not replace doctoral training in causal inference.
How Do You Choose an Appropriate Comparison Group?
Start with the policy rule, not an available dataset. Define who became eligible, when implementation began, whether uptake differed from eligibility, and the outcome and follow-up period of interest. Then identify populations that could plausibly have followed the exposed population's outcome trajectory without the policy.
Use this sequence to reject weak candidates before investing in a full analysis:
- Map exposure: Check eligibility boundaries, implementation dates, enforcement, neighboring policies, and likely spillovers into proposed comparison areas.
- Compare baseline conditions: Examine outcome levels and relevant determinants, including demographics, healthcare access, economic conditions, and earlier policy activity.
- Inspect prepolicy outcomes: Plot multiple observations where available; investigate divergent trends or changes that precede implementation.
- Check measurement: Confirm that both groups have comparable definitions, denominators, reporting practices, and observation periods.
- Document the trade-off: State why the chosen group is more credible than alternatives and what unmeasured differences remain.
Close geographic proximity is not automatically an advantage: neighboring jurisdictions may share a changing labor market or be affected by cross-border service use. Conversely, a more distant area may match prepolicy outcomes but differ in institutions that determine how the policy works. For students still building the quantitative foundation to make these judgments, low-cost public health bachelor's programs online are one possible starting point; compare actual methods coursework rather than assuming the degree label establishes research readiness.
Which Study Designs Use Comparison Groups in Policy Evaluation?
The right design depends on how the policy was assigned, how many units were exposed, and whether comparable prepolicy observations exist. The table contrasts designs by the comparison each makes - not by a claim that one design is always strongest.
| Design | Comparison | Most important credibility concern |
| Randomized or phased rollout | Units assigned to earlier versus later exposure | Whether assignment was truly random and groups stayed distinct |
| Difference-in-differences | Changes over time in exposed versus unexposed groups | Whether untreated outcome trends would have been comparable |
| Interrupted time series with a control | Changes in outcome level or trend after implementation, assessed against an unexposed series | Concurrent events and changes in measurement or trend |
| Synthetic control | An exposed jurisdiction versus a weighted combination of unexposed jurisdictions | Prepolicy fit, donor-pool suitability, and spillovers |
| Regression discontinuity | Units just above versus below an eligibility cutoff | Manipulation near the cutoff and limited generalizability beyond it |
A simple before-and-after study has a prepolicy reference period but no contemporaneous unexposed group. It may still describe change, yet it has less protection against unrelated events. When reviewing CEPH-accredited MPH schools and programs as a possible preparation route, look beyond general evaluation coverage for training in the assumptions behind the design you intend to use.
The best source captures the same outcome in exposed and comparison groups before and after implementation, with enough detail to define eligibility and detect changes in composition. Potential US sources include CDC surveillance systems such as the Behavioral Risk Factor Surveillance System, vital records, hospital discharge or claims files, and state or local administrative records. Policy texts, implementation records, and enforcement data are also needed to establish when and where exposure occurred.
Access and comparability matter as much as sample size. Survey questions, sampling frames, coding practices, geographic identifiers, and reporting rules can change; administrative data may omit uninsured people or those outside a participating system. Check whether estimates require survey weights and whether outcomes can be measured at the policy's actual level. Restricted records may require data-use agreements, security controls, and institutional review or a formal determination; a public dataset does not automatically settle every ethics question.
If a master's program is being considered to gain supervised data experience before doctoral research, cost-conscious online MPH program choices provide a starting point for comparing affordability. Confirm that the program also offers the relevant datasets, mentoring, and analytical coursework.
Write the outcome definition before inspecting postpolicy results. Specify the event or measure, numerator, denominator, geographic unit, observation interval, and follow-up window. For a policy intended to reduce preventable hospitalizations, a rate among the eligible population answers a different question from the number of admissions recorded by one hospital. Define whether the primary result is an absolute change, a rate difference, or another interpretable measure.
Use the same definitions in both groups and examine whether the policy itself could change who appears in the data. An insurance expansion, for example, may alter claims capture even if underlying health does not change. Consider outcomes that matter to the affected community, alongside possible unintended effects, and avoid choosing a favorable outcome only after seeing the results. Missing observations, small-area instability, and delayed effects should be addressed in the analysis plan.
Students weighing additional training against its expense can consult public health master's tuition and salary comparisons, but the relevant educational test here is whether a program teaches outcome construction and supervised analysis - not whether its graduates are assigned a particular earnings figure.
Confounding arises when another factor influences both policy exposure and the outcome. A jurisdiction adopting a prevention policy may also expand clinical services. Selection bias can arise when the people observed after implementation differ from those observed before it - for example, because the policy changes enrollment in the dataset. Neither problem disappears simply because a regression includes many variables.
Compare measured prepolicy characteristics and trends, identify plausible concurrent policies, and use subject-matter knowledge to select covariates measured before exposure. Depending on the design, matching, weighting, fixed effects, or restrictions may improve comparability. Test whether conclusions change under defensible alternative comparison groups and outcome definitions. Do not adjust away a variable caused by the policy when estimating its total effect, and acknowledge confounders that the data cannot measure.
Research ability can also be demonstrated through careful work rather than a flawless academic record. Applicants concerned about earlier grades can review guidance on MPH applications after uneven undergraduate grades while building stronger evidence of quantitative preparation; admissions competitiveness and the validity of an evaluation are separate questions.
A method is useful only if its estimate matches the policy question and its assumptions are credible. Difference-in-differences estimates the difference between the exposed group's outcome change and the comparison group's change; its causal interpretation requires, among other conditions, that their untreated trends would have been comparable. An event-study specification can display outcomes around implementation and probe prepolicy patterns, although an apparently flat pretrend does not prove the assumption.
For a single treated jurisdiction, synthetic control may construct a more convincing benchmark if suitable unexposed jurisdictions reproduce its prepolicy trajectory. Interrupted time-series models assess changes in level or trend but must account for underlying trends and time-related dependence. Regression-discontinuity estimates concern units near an eligibility threshold, not necessarily everyone affected by the policy. With staggered adoption, researchers should avoid treating already-exposed units as uncomplicated controls and choose an estimator suited to the timing and effect heterogeneity.
Report effect sizes with uncertainty intervals, examine model fit and sensitivity, and account for correlation within the units where policy is assigned. When only a small number of jurisdictions determine exposure, individual-level sample size alone does not make uncertainty estimates reliable. A student considering a methods-focused master's as preparation may encounter online public health master's programs without recommendation letters; a convenient application process is not evidence that the curriculum covers these methods at the depth a dissertation needs.
Public health PhD preparation may combine epidemiology, biostatistics, health policy, causal inference, and substantive work with government or community partners. Coursework introduces designs and assumptions; supervised research tests whether a student can define a defensible question, negotiate data access, write reproducible analysis, and explain limitations to policy audiences. A dissertation may require several related studies, but its structure and milestones depend on the program.
For this topic, compare potential programs on faculty expertise in quasi-experimental designs, access to policy and population-health data, opportunities to work with evaluation teams, and the availability of appropriate committee members. Ask who approves restricted-data use, whether an assistantship supports the proposed work, and what happens if a partner withdraws access. A PhD is appropriate for someone seeking sustained independent research training; a targeted methods course or supervised analyst role may be more direct for someone who only needs to evaluate one workplace policy.
Reviewing selectivity in public health doctoral study can help frame an application search, but apparent admissions accessibility should not outweigh research supervision, feasible data access, and clear funding terms.
Requirements vary by university and department. A relevant master's degree may help, but an MPH is not universally required. Programs may look for statistics or other quantitative preparation, research experience, a clear scholarly question, writing ability, and fit with available faculty. A publication, thesis, GRE score, or particular software package should not be treated as a universal prerequisite.
For a policy-evaluation application, a stronger research statement names a policy, target population, plausible comparison strategy, potential data sources, and a major identification threat. It should also distinguish data the applicant can realistically obtain from data that would require approval. Verify the program's current prerequisites, advisor-matching process, residence expectations, dissertation milestones, and whether funding is guaranteed or contingent on an assistantship or grant.
Applicants who need preparation before doctoral applications can explore provisional entry criteria for public health graduate study. Confirm any remaining prerequisites and deadlines with the specific master's program; provisional entry is not a substitute for the research preparation a PhD will demand.
Which Public Health PhD Careers Use Policy Evaluation Skills?
Policy-evaluation researchers work in universities, government agencies, research organizations, health systems, and nonprofits. Their responsibilities can include designing evaluations, negotiating data access, analyzing policy effects, assessing equity implications, and translating findings for decision-makers. A PhD can support independent research and teaching roles, but employers may also require a record of relevant studies, subject expertise, and collaborative experience.
Not every evaluation job requires a doctorate. An MPH or research-oriented MS plus strong methods and applied experience may be sufficient for some analyst positions; a PhD is more compelling when the goal is to lead a sustained research agenda or compete for roles explicitly requiring advanced research training. Before committing to doctoral study, compare actual job descriptions with the skills you can demonstrate and the funded research opportunities a program offers.
For compensation planning, salary planning for MPH graduates may offer context for an alternative pathway. Do not treat an MPH salary estimate - or an occupation-wide wage - as the expected earnings of a public health PhD policy evaluator.
Other Things You Should Know About Public Health
Yes. Separate plausible groups can test whether findings depend heavily on one choice. Each needs its own exposure and comparability assessment; agreement across groups is reassuring but does not eliminate shared bias.
Sometimes new evidence reveals that an original group was contaminated or measured inconsistently. Document the reason and report both planned and revised analyses when feasible, rather than presenting a results-driven change as the original plan.
Yes. Interviews or document review can explain implementation differences and possible spillovers that outcome data miss. Obtain appropriate permissions and describe how qualitative evidence informs, rather than independently proves, a causal claim.
Preregistration can clarify outcomes, comparison groups, and planned analyses before results are known. It is especially useful when researchers have many defensible analytical choices; deviations should be explained rather than hidden.
- Step 4 - Gather Credible Evidence | Program Evaluation https://www.cdc.gov/evaluation/php/evaluation-framework-action-guide/step-4-gather-credible-evidence.html
- Understanding and misunderstanding randomized controlled trials - PMC https://pmc.ncbi.nlm.nih.gov/articles/PMC6019115/
- A Guide to Understanding Public Health Research Study Designs | Johns Hopkins | Bloomberg School of Public Health https://publichealth.jhu.edu/2026/a-guide-to-understanding-public-health-research-study-designs
- Difference-in-Difference Estimation | Columbia University Mailman School of Public Health https://www.publichealth.columbia.edu/research/population-health-methods/difference-difference-estimation
- Detecting Spillover Effects: Design and Analysis of Multilevel Experiments | Institution for Social and Policy Studies https://isps.yale.edu/resource/detecting-spillover-effects-design-and-analysis-of-multilevel-experiments
- Types of Studies - Public Health Doctoral Studies (PhD and DrPH) - Research Guides at George Washington University https://guides.himmelfarb.gwu.edu/PhD/types-of-studies
- Designing experiments to measure spillover effects | World Bank Blogs https://blogs.worldbank.org/en/impactevaluations/designing-experiments-to-measure-spillover-effects
