From Jupyter notebook to reproducible scientific pipeline

The notebook runs. The plot exists. Then a colleague clones the repository and gets a different answer.

The usual causes are mundane: cell 17 was executed before cell 9, data_final_v3.csv contains a manual correction, a global variable survived from yesterday, the random seed was set in one branch, and the absolute path points to the author’s Downloads folder.

Jupyter did not create those failures. It made them easy to carry.

“Reproducible” can refer to several things:

• the same code and same input produce the same output;
• a clean machine can run the analysis;
• another person can understand each transformation;
• a new batch can be processed without editing code;
• a result can be traced to raw files, configuration and software version;
• the method can be changed without breaking unrelated stages.

Write the required level before restructuring anything.

For a one-off exploratory figure, a well-documented notebook with fixed inputs may be enough. A weekly instrument workflow with ten users needs stronger input validation, logging and ownership. A regulated or safety-related system sits outside the standard described here.

Adam Rule and colleagues treated notebooks as publishable computational narratives and recommended attention to cell order, dependencies, documentation, testing and reuse.^[a2-1] The operational test is simpler: can the complete result be produced from a clean process without remembering the author’s private sequence of actions?

Do not begin by translating every cell into a class hierarchy.

Keep notebooks for:

• inspecting a new file;
• trying transformations;
• plotting edge cases;
• comparing model behaviour;
• explaining a result;
• reviewing a small number of failures.

Move stable logic into importable functions and commands. The notebook should call those functions rather than contain a second implementation.

A useful split looks like this:

notebooks/
    01_inspect_inputs.ipynb
    02_review_quality_failures.ipynb
    03_result_figures.ipynb
src/project_name/
    ingest.py
    quality.py
    features.py
    models.py
    figures.py

The notebooks remain readable. The code path used for the final run lives in src.

A pipeline becomes reliable when it knows what an acceptable input looks like.

For a tabular instrument export, specify:

• required columns;
• column types;
• units;
• identifier format;
• timestamp format and timezone;
• allowed duplicate keys;
• missing-value rules;
• version or schema identifier;
• expected relationships between columns.

For images, include:

• file format;
• dimensions and axes;
• channel names;
• pixel size;
• bit depth;
• acquisition identifiers;
• plate, well or field mapping;
• mask and image naming relationship.

Pydantic, Pandera or explicit validation functions can enforce the contract. The library matters less than the error message.

Bad:

KeyError: 'sample_id'

Useful:

Input batch 2026-07-08 failed schema v1.3:
required column 'sample_id' is absent;
found possible replacement 'Sample ID';
no output was written.

Fail before partial results enter the output directory.

The pipeline reads raw inputs and writes derived outputs elsewhere.

Use a structure such as:

data/
    raw/          # read-only source material
    interim/      # parsed or aligned data
    processed/    # analysis-ready data
outputs/
    figures/
    tables/
    reports/
    manifests/

Do not overwrite the source export after correcting units or labels. Store the correction as code, configuration or a versioned mapping table.

A manual correction table can be legitimate:

record_id,field,old_value,new_value,reason,approved_by,date
R041,condition,control,treatment,source log correction,VT,2026-07-10

The problem is an unrecorded edit inside Excel.

A pipeline should obtain its behaviour from arguments or configuration, not from the order in which someone touched the code.

Move values such as these out of cells:

• sampling rate;
• channel selection;
• baseline interval;
• segmentation threshold;
• minimum object size;
• model hyperparameters;
• exclusion list;
• output resolution;
• random seed.

A YAML file is often enough:

study: pilot_a
sampling_rate_hz: 100
quality:
  flatline_seconds: 2.0
  max_missing_fraction: 0.05
analysis:
  baseline_seconds: [120, 420]
  random_seed: 20260712

Validate configuration too. A baseline ending before it begins should fail immediately.

The full workflow needs an entry point that starts from declared inputs.

For example:

amperie-run 
  --input data/raw/pilot_a 
  --config config/pilot_a.yaml 
  --output outputs/pilot_a_2026-07-12

That command may call Python functions directly. A more complex dependency graph may use Snakemake. Johannes Köster and Sven Rahmann designed Snakemake around explicit input-output rules that can scale from a workstation to a cluster without rewriting the workflow definition.^[a2-2]

Use a workflow engine when it solves a real dependency or execution problem. Twelve CSV files processed once per month do not automatically need Airflow, Kubernetes and a service account hierarchy.

Scientific pipeline tests should include more than mathematical unit tests.

Schema tests

• missing required column;
• wrong unit;
• duplicate identifier;
• malformed timestamp;
• unexpected channel;
• empty file.

Transformation tests

• known input gives known output;
• row order does not change the result where order is irrelevant;
• resampling preserves expected duration;
• label mapping handles all declared categories;
• image axes are interpreted correctly.

Invariant tests

• probabilities remain between zero and one;
• object count cannot be negative;
• retained duration cannot exceed recorded duration;
• an exclusion reason must accompany an excluded record;
• every output row links to a source identifier.

Regression tests

Use a small synthetic or approved example dataset. Store expected summary values or file hashes where exact determinism is appropriate. When the method changes intentionally, review and update the expectation with a reason.

Sandve and colleagues’ first rule for reproducible computational research was to track how every result was produced.^[a2-3] A test suite turns some of that memory into executable checks.

A scientific pipeline can be technically reproducible and methodologically wrong.

For predictive work, imputation, scaling, feature selection and learned preprocessing must be fitted on training data inside each fold. Scikit-learn’s Pipeline exists partly to keep those transformations attached to the estimator during cross-validation.^[a2-4]

The same issue appears outside conventional ML. A global normalization calculated across treatment and control may leak batch information. A segmentation threshold selected after reviewing all experimental groups may bias the measurement. Reproducibility preserves the error unless the scientific split is designed correctly.

Keep a separate validation test for this:

assert no participant_id appears in both train and test
assert scaler was fitted only on train indices
assert threshold-selection images are absent from final test images

Every material run should leave a small machine-readable record.

{
  "run_id": "pilot_a_2026-07-12T14-32-11Z",
  "git_commit": "7f9c2a1",
  "config_sha256": "...",
  "input_manifest_sha256": "...",
  "python": "3.12.4",
  "package_version": "0.4.0",
  "started_utc": "2026-07-12T14:32:11Z",
  "completed_utc": "2026-07-12T14:38:47Z",
  "status": "completed_with_warnings",
  "warnings": [
    "2 files excluded: duplicate session identifier"
  ]
}

For large data, the input manifest can store paths, sizes, modification times and checksums rather than copying files.

The FAIR principles emphasise findability, accessibility, interoperability and reuse for data and metadata.^[a2-5] A private pipeline does not need to publish every input, but it still benefits from persistent identifiers, clear metadata and explicit provenance.

An error stops the run because the result would be undefined or misleading. A warning records a known deviation that permits output.

Examples:

Errors

• duplicate primary key;
• unknown unit conversion;
• missing required channel;
• configuration inconsistent with the data;
• output directory already contains a different run.

Warnings

• optional metadata absent;
• one channel excluded under a declared rule;
• a small number of images failed focus QC;
• a deprecated input version was converted successfully.

Collect warnings into the report. Do not leave them only in terminal output.

Some decisions should remain manual:

• ambiguous segmentation boundaries;
• unusual signal morphology;
• protocol deviations;
• scientific exclusion decisions;
• interpretation of a negative result;
• acceptance of a new input variant.

Build a review queue with source links, reason codes and reviewer notes. A review_status column is better than a folder named check_these_maybe.

Before delivery, use a clean environment and a person who did not build the pipeline.

They should be able to:

1. install or start the environment;
2. run the example data;
3. understand where outputs were written;
4. locate an excluded record and its reason;
5. change one configuration value safely;
6. run the tests;
7. identify who owns future changes.

A Docker image can help with system dependencies. A lockfile can be enough for a pure Python package. The choice follows the client environment.

The handover package should contain:

• README with one working command near the top;
• architecture or data-flow diagram;
• environment specification;
• configuration reference;
• synthetic or approved example input;
• expected example output;
• test command;
• known limits;
• maintenance notes.

Do the work in this order:

1. run the notebook from a fresh kernel;
2. identify every external file and manual edit;
3. record the expected final outputs;
4. define input schemas;
5. move stable transformations into functions;
6. create one clean entry command;
7. add tests for known failures;
8. add configuration and a run manifest;
9. rerun on the original data;
10. ask another person to run the example package.

Do not spend the first week choosing an orchestration platform. Start by deleting the hidden step between the spreadsheet and cell 17.

For scoping, send the repository tree, a screen recording of the current run, two representative inputs and the output that must be reproduced.

References

Rule A, Birmingham A, Zuniga C, et al. Ten simple rules for writing and sharing computational analyses in Jupyter Notebooks. PLOS Computational Biology. 2019;15(7):e1007007.

Köster J, Rahmann S. Snakemake: a scalable bioinformatics workflow engine. Bioinformatics. 2012;28(19):2520–2522.

Sandve GK, Nekrutenko A, Taylor J, Hovig E. Ten Simple Rules for Reproducible Computational Research. PLOS Computational Biology. 2013;9(10):e1003285.

scikit-learn developers. Common pitfalls and recommended practices: data leakage.

Wilkinson MD, Dumontier M, Aalbersberg IJ, et al. The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data. 2016;3:160018.