2027 Writing a Reproducible Data Cleaning Log for MPH and MS Students
Co-Founder and Chief Data Scientist
A public health analysis can change substantially between receiving a dataset and reporting a result. If those changes exist only in a student's memory or scattered code comments, an advisor or project partner may be unable to check what happened.
This guide shows MPH and MS students how to build a reproducible data cleaning log: what to record, how to connect decisions to code, and how to protect restricted information. Use it to create an audit-ready course, thesis, or applied-project deliverable without turning routine documentation into a second research project.
Key Things You Should Know
- Keep an unchanged, access-controlled source dataset; run scripted cleaning steps; and record the decisions that explain those steps.
- For each consequential rule, document its rationale, affected variables or records, before-and-after checks, code location, and reviewer or approval when applicable.
- Counts alone are not enough: a reproducible log must explain why missing values, implausible measurements, and duplicate records were handled as they were.
What is a reproducible data cleaning log in public health research?
A reproducible data cleaning log is a dated record of how source data became the analysis-ready dataset. It identifies the inputs, transformations, decisions, quality checks, and outputs needed for an authorized person to inspect and rerun the process. The log explains decisions; executable code performs them. Neither substitutes for the other.
For example, a script might recode an unrecognized survey response as missing. The log should explain which response was affected, why it did not fit the documented categories, whether the source was checked, and what effect the rule had on usable records.
An MPH program-evaluation project may need a concise log that a community partner can understand; a research-oriented MS project may also need more detailed provenance for sensitivity analyses.
Students weighing those paths can use specialization choices in public health graduate study as context, but the logging standard should follow the project's risks and requirements, not its degree label.
Why should MPH and MS students document every data-cleaning decision?
"Every decision" means every decision that could change the data's meaning, the eligible population, an analysis result, or someone else's ability to reproduce the work - not a diary of every keystroke. In a public health dataset, excluding records with an impossible date could alter the study denominator.
Recoding a response category could change how a population is represented. A documented rationale lets a supervisor distinguish a justified correction from an unnoticed analytic choice.
Good logs also make revision less costly. When an advisor asks whether a different duplicate rule changes an estimate, the student can identify the original rule and rerun the workflow rather than reconstructing it from memory.
This discipline is useful in graduate research, but students do not need a doctorate to practice it; those considering further research training can separately assess online public health PhD tuition against their actual research goals.
What should a reproducible data cleaning log include?
Start with enough provenance to identify the permitted input without copying sensitive data into the log: source organization or system, extraction date, file version or approved identifier, data dictionary version, and the location of access instructions.
Record the software and script versions used to produce the output. A file checksum can help distinguish versions, provided storing or sharing it is allowed by the project's security rules.
Each substantive entry should capture the question and the decision, not merely the command. The following fields form a practical template for an entry:
- Scope: Dataset, variable or record group, and the point in the workflow where the issue arose.
- Rule and rationale: The exact condition applied, its source in a protocol or data dictionary, and any assumption that needs review.
- Impact: Before-and-after record counts, missingness or validation results, with an explanation of which denominator each count uses.
- Traceability: Script or notebook location, output version, date, author, and reviewer or approval where required.
- Uncertainty: Unresolved questions, exceptions, and planned sensitivity checks rather than a claim that the data are now error-free.
A student who learned to keep a smaller version of this record during shorter course terms for a public health BS can extend it for graduate work; speed of coursework does not remove the need to explain consequential changes.
Keep three connected artifacts: an unchanged source in its approved location, a script that creates cleaned outputs, and a human-readable decision log. Give each transformation a stable step identifier shared by the script and log. This separates "what the software did" from "why the team decided to do it."
A workable sequence prevents important checks from being recorded only after results are known:
- Define the intended analysis: State the population, unit of observation, relevant dates, and expected variables before selecting exclusions.
- Inventory the input: Record versions, variable definitions, permitted access, and baseline quality checks without altering the source.
- Write and run rules in order: Assign step identifiers; save diagnostic counts and flag questionable values before deleting or recoding them.
- Review decisions: Separate rules established by a protocol from judgment calls requiring an advisor, site supervisor, or data steward.
- Rebuild and reconcile: Rerun from approved inputs, confirm that outputs match expected checks, and archive permitted versions according to project policy.
For a modest class project, a plain-text log and script may be sufficient. Students considering cost-conscious online public health undergraduate study can use that same low-cost structure to establish the habit before taking on larger graduate datasets.
Choose tools that the student, supervisor, and authorized collaborators can actually maintain. A spreadsheet is convenient for a short decision register but can conceal accidental edits; a version-controlled text file makes changes easier to compare.
Neither makes a manually edited dataset reproducible. A script or recorded, rerunnable software workflow should generate the cleaned data.
R and Python can pair scripts with plain-text logs, automated checks, and package-environment records. Stata, SAS, and other statistical software can use documented command files and saved logs.
Notebooks can combine explanation and code, but students should test that they run from a fresh session in the intended order. Git can track code and documentation; restricted data generally belong only in approved storage, not an ordinary repository.
The cheapest viable setup is often the software already supported by the course or placement site. If the immediate gap is learning version control or scripted cleaning, a targeted course or supervised project may be more useful than choosing a new degree on that basis alone. Broader budget and pacing choices for public health students matter if additional education is genuinely needed.
These issues require separate decisions because the same automated "fix" can have different consequences. Record what was observed, the evidence for a rule, what was changed, and what remains uncertain. Preserve flags where feasible so a reviewer can assess alternatives.
Missing data: Distinguish a blank field, a documented "not asked" response, and a value made missing by a cleaning rule. Report the relevant denominator and missingness by variables important to the analysis. Do not assume that deletion or imputation is appropriate without considering why values may be absent.
Outliers: An unusual value may be a valid public health observation. Log the plausible range, its source, how the value was checked, and whether it was retained, corrected, flagged, or excluded. If a choice could change the conclusion, compare results under defensible alternatives rather than quietly removing inconvenient observations.
Duplicates: Define what counts as a duplicate: a repeated file import, multiple responses by one person, or legitimate repeated visits are not interchangeable. Document matching fields, tie-breaking rules, and effects on the unit of analysis. Avoid exposing identifying fields in a shareable log.
Students reviewing tuition across online MPH degrees should also check whether a methods course or supervised project teaches these judgments - not just software commands - before paying for additional training.
Use the same step identifier in the log and the command file. A log entry such as "STEP-04: flag dates outside the approved collection window" should point to the code that creates the flag, the documented window, and a check of affected records. The code should produce a summary that the log can reference without pasting row-level information into it.
Parameterize input and output locations, run steps in a defined order, and record software and dependency versions. Add checks that stop the workflow when a required variable is absent or a key expected to be unique is not unique.
Save intermediate data only when permitted and useful for review. A successful rerun demonstrates that the documented process works on the authorized inputs; it does not prove that every cleaning judgment was scientifically sound.
Students moving into larger research workflows may want to compare the supervision and computing support available in online public health doctoral programs. For an MPH or MS assignment, however, a clear, tested script and decision log are more valuable than an unnecessarily elaborate software system.
A log creates accountability for choices that may affect which communities appear in an analysis. For example, a rule that drops records lacking a geographic code could disproportionately remove people whose addresses were recorded differently. Document the rule, inspect its potential effect where authorized, and describe limitations without treating a group's absence as a data-quality nuisance.
Keep the log useful without including names, direct identifiers, free-text disclosures, or small-cell details that could reveal individuals. Access to public, deidentified, restricted, and identifiable data may be governed differently.
Follow the applicable data-use agreement, institution or site security policy, consent terms, and review requirements; do not assume that every classroom activity has the same oversight status or that HIPAA applies to every dataset. Obtain a determination from the responsible office when research or review requirements are unclear.
More advanced study does not itself grant data access or permission to publish restricted material. Students evaluating public health doctoral tuition and career comparisons should treat access, governance, and supervision as separate feasibility questions.
Look past a course title that mentions "data." Ask whether students write rerunnable code, maintain a data dictionary and cleaning log, test quality rules, receive feedback on missing-data decisions, and work within realistic access restrictions.
An MPH may emphasize explaining defensible decisions to practice partners; a research-oriented MS may offer deeper methods or thesis supervision. Actual curricula and projects vary by school.
Before enrolling, request a current syllabus or assignment description and ask who reviews reproducibility, what software support exists, and whether an approved applied project or thesis can include data-management work. If online delivery matters, verify live meetings, research supervision, and site-access expectations.
For accreditation questions, check the exact program's status and scope rather than assuming institutional recognition settles them; online MPH program accreditation comparisons can help frame that separate check.
How can reproducible data-cleaning skills support public health careers?
A well-documented project can show an employer how a student handles ambiguous records, checks denominators, protects confidential information, and explains decisions to colleagues. Those are relevant tasks in surveillance support, program evaluation, research assistance, and analyst roles.
Entry-level staff may work under an epidemiologist, evaluation lead, or data steward; a polished log is evidence of a skill, not proof of independent authority or eligibility for a particular job.
For a portfolio, use a public or synthetic dataset - or a partner-approved, carefully redacted description of the workflow. Show the question, decision rules, checks, and a safe example of the resulting log. Never upload restricted records, identifiable examples, or code containing credentials.
If comparing education with employment opportunities, treat average earnings after an MPH as broad career context, not a prediction of what this skill or a particular degree will pay. The next practical step is to have a supervisor try to follow the log and identify any decision they cannot verify.
Other Things You Should Know About Public Health
Update it when you adopt or change a consequential rule. Recording decisions during cleaning is more reliable than reconstructing them after the analysis is complete.
Yes, if your course or project accepts it. Share permitted code, rules, variable descriptions, and aggregate checks; explain that an authorized reviewer needs approved access to rerun the restricted-data workflow.
It can help draft a template or clarify wording if your institution permits its use. Verify every entry against the actual code and decisions, and do not enter restricted data into an unapproved tool.
Follow the project's approval process. An advisor may resolve an analytic question, while a data steward or site representative may need to address source-data meaning, access, or permitted use.
- Data Cleaning: Detecting, Diagnosing, and Editing... - PMC https://pmc.ncbi.nlm.nih.gov/articles/PMC1198040/
- Confidentiality and Privacy of Personal Data - NCBI - NIH https://www.ncbi.nlm.nih.gov/books/NBK236546/
- Quality Control Log Instructions | cdc.gov http://reach.cdc.gov/jobaid/quality-control-log-instructions
- Data Management Skills for Reproducible Science | ucdavis.edu https://laskowskilab.faculty.ucdavis.edu/2020/08/03/keeping-a-paper-trail-data-management-skills-for-reproducible-science/
- 410. Maintaining Data Confidentiality | Research Integrity & Security | University of Nevada, Reno https://www.unr.edu/research-integrity/program-areas/human-research/human-research-protection-policy-manual/410-maintaining-data-confidentiality
- Analysis Ready Datasets | harvard.edu https://datamanagement.hms.harvard.edu/collect-analyze/analysis-ready-datasets
- The Basic Reproducible Workflow Template · GitBook http://www.practicereproducibleresearch.org/core-chapters/3-basic.html
- What Research Data and Images Should You Preserve? | Manuel B. Garcia https://manuelgarcia.info/guides/research/guide/preserve-authentic-research-data-images
