Test whether the predictive signal survives contact with new data.

Keep the existing URL for continuity. Change the visible page name and navigation label.

A model trained on windows can learn the participant. A microscopy classifier can learn the plate edge. A sensor model can learn firmware version, collection site or missingness pattern. Accuracy does not identify which one happened.

We design the split around the real deployment boundary, then keep preprocessing, feature selection and tuning inside the training data.

• Does performance survive a new participant, session, device, batch, site or time period?
• Which baseline is the complex model actually beating?
• Are missingness, acquisition settings or protocol order carrying the target?
• How wide is the uncertainty around the reported metric?
• Which examples fail, and do they fail for the same reason?
• Does the result persist after an ablation or negative-control test?
• Would another cohort reduce uncertainty, or would it reproduce the same design fault?

If the intended use is prediction for a new person, the test set must contain people absent from training. If the intended use is transfer to a new instrument, hold out instruments. If tomorrow is the deployment target, random shuffling across time is the wrong experiment.

For repeated biological or sensor data, useful splitters may include:

• leave-one-participant-out;
• GroupKFold by participant, animal, plate, device or study site;
• StratifiedGroupKFold when class balance permits it;
• blocked or rolling time splits;
• external validation on a separately collected dataset.

The group field belongs to the scientific design; treating it as a convenience column after modelling breaks the claim.

Target and leakage map

We trace how labels were created, which records share a source, when preprocessing occurred and which information would exist at prediction time.

Baselines

A constant predictor, a transparent linear or tree model, and a simple domain baseline. A convolutional network that cannot beat logistic regression under the same split has not earned its maintenance cost.

Confounds and shortcuts

Participant identity, site, batch, device, duration, missingness, file naming, acquisition date and protocol order.

Uncertainty

Confidence intervals, bootstrap distributions, repeated grouped splits where appropriate, calibration and threshold sensitivity.

Failure slices

Performance by signal quality, subgroup, device, batch, event prevalence and recording condition. One pooled AUROC can conceal a model that works only on the cleanest third of the data.

Ablations and negative controls

Remove suspect channels, destroy temporal order, permute labels within the correct unit, or replace the target with a known nuisance variable. The control is chosen from the suspected failure.

• Written target and leakage map
• Reproducible validation split
• Baseline and challenger models
• Fold-safe preprocessing and feature selection
• Grouped performance with uncertainty
• Calibration and threshold analysis where relevant
• Confound, ablation and negative-control results
• Error table linked to source records
• Recommendation to proceed, collect specific data, revise the protocol or stop
• Code and technical review session

Production deployment, live monitoring, regulatory documentation, clinical performance claims and indefinite hyperparameter search. Those require a result worth carrying forward.

Can you improve our model?

Perhaps. First we reproduce the current result under the correct split. Improvement may come from labels, protocol, quality controls or simpler features rather than another architecture.

We have only 30 participants. Is ML pointless?

Thirty independent participants and 30,000 windows still provide 30 independent units for participant-level generalisation. Feasibility work can be useful, but the uncertainty belongs in the result.

Do you provide technical due diligence?

We review data, code and validation for the commissioning technical team. Reliance by investors, regulators or third parties needs separate contractual and insurance review.

Will you sign off the model?

No. We document the tests performed, the observed failures and the evidence available from the supplied data.

Show us the split before the model diagram.

Send the number of independent participants, devices, batches or assets; the number of repeated observations; the current target; and how the train and test sets were created.

Scope a validation sprint

• Why random windows inflate performance
• When ML should wait

How Services engagements are scoped