Why random windows inflate performance in physiological and sensor ML

A participant wears a sensor for 30 minutes. The recording is cut into 30-second windows with 50 per cent overlap. That produces 119 windows.

A random 80/20 split sends roughly 95 windows to training and 24 to testing. Both sets contain the same skin contact, sensor placement, baseline level, room, protocol, firmware and person-specific morphology. Adjacent windows can share 15 seconds of raw samples.

The test set looks unseen to scikit-learn. The participant has already been introduced 95 times.

Suppose the intended use is classification for a new participant.

Then the test question is:

Can a model trained on some participants predict the target for a participant absent from training?

Random window splitting answers another question:

Can a model label additional windows from participants it has already seen?

That second task may be legitimate for personal calibration. It cannot be reported as cross-person generalisation.

Sohrab Saeb and colleagues framed cross-validation around approximation of the intended clinical use case.^[a3-1] David Roberts and co-authors made the same argument for temporal, spatial and hierarchical ecological data: dependence in the observations should shape the validation block.^[a3-2]

Choose the deployment unit first:

A paper can report more than one setting. Each score needs the claim beside it.

With 50 per cent overlap, each interior sample belongs to two windows. With 90 per cent overlap, a sample can appear in about ten successive windows. Randomly splitting those windows can place near-duplicates on both sides.

Akbar Dehghani, Tristan Glatard and Emad Shihab compared subject-dependent and subject-wise evaluation in human activity recognition. They reported that ordinary k-fold validation increased performance by about 10 percentage points, rising to 16 points with overlapping windows in their experiments.^[a3-3]

Non-overlapping windows do not solve participant leakage. A person’s baseline physiology, movement style, device fit and collection environment persist across the session.

Physiological data contain stable individual structure:

• resting heart-rate range;
• pulse morphology;
• skin conductance or potential level;
• movement pattern;
• respiration rate;
• electrode impedance and placement;
• device-specific scaling;
• habitual response to the protocol.

If participant A contributes mostly positive labels and participant B mostly negative labels, a model can infer identity and recover the target indirectly.

Test this directly. Train a participant-identity classifier or measure how well simple features separate participants. Strong identity prediction does not invalidate the data, but it tells you what the target model can exploit.

Also inspect class balance per participant:

participant  positive_windows  negative_windows  positive_fraction
P001         87                12                0.879
P002         9                 91                0.090
P003         44                48                0.478

A window-level metric weights P001 as 99 examples. The scientific sample size for new-person generalisation still depends on the number and diversity of participants.

A safe sequence is:

1. assign participants, sessions, devices or batches to folds;
2. fit any learned preprocessing on the training fold;
3. create or select training windows;
4. create test windows only from held-out units;
5. tune hyperparameters inside the training data;
6. evaluate once on the outer test fold.

Windowing before assigning groups can be acceptable if every window retains the correct group identifier and no cross-boundary operation has used the full dataset. It is easier to audit when the groups are established first.

A grouped split in scikit-learn

from sklearn.model_selection import GroupKFold, cross_validate
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

model = Pipeline(
    steps=[
        ("scale", StandardScaler()),
        ("classifier", LogisticRegression(max_iter=2000)),
    ]
)

cv = GroupKFold(n_splits=5)

scores = cross_validate(
    model,
    X=features,
    y=labels,
    groups=participant_ids,
    cv=cv,
    scoring=["roc_auc", "balanced_accuracy"],
    return_estimator=True,
)

GroupKFold keeps each group out of the training set for the fold in which it is tested.^[a3-4] StratifiedGroupKFold attempts to preserve class proportions while maintaining non-overlapping groups, though small or highly imbalanced datasets may make perfect stratification impossible.

Check the generated folds. A splitter cannot repair a design with only two positive participants.

The group split may be correct while the preprocessing has already seen the test data.

Common examples:

• scaling all windows before cross-validation;
• selecting features on the full dataset;
• imputing from global medians;
• choosing a filter or threshold after inspecting held-out participants;
• learning a representation on all subjects;
• correcting batch effects with information from the test batch;
• oversampling before the split;
• choosing the window length from test performance.

Keep learned operations in the pipeline. Fit them inside each training fold.

The scikit-learn documentation uses Pipeline as the standard defence for preprocessing leakage during cross-validation.^[a3-5] Custom signal transforms can follow the same API. A baseline correction based only on each individual record may be deterministic and fold-independent; a global normalization learned from all participants is not.

Using the outer test folds to choose regularisation, architecture, feature family or threshold turns the test set into training information.

For small grouped datasets, use nested cross-validation:

• outer loop estimates performance on held-out groups;
• inner loop selects hyperparameters using only the outer training groups;
• the outer test group remains untouched until selection is complete.

With 40 participants and five outer folds, each outer test fold contains eight participants. The inner split operates on the remaining 32. That is expensive and often unstable. Report the instability instead of silently selecting the best seed.

Leave-one-participant-out testing can make sense when every individual result matters and the model can be fitted repeatedly. Its fold scores are highly correlated because the training sets overlap. Show participant-level predictions and confidence intervals; do not treat 40 folds as 40 independent experiments.

A window-level AUROC answers how windows rank. A participant-level decision may combine windows by mean probability, majority vote, maximum response or a prespecified temporal rule.

Do the aggregation inside the evaluation:

1. generate out-of-fold window predictions;
2. group them by the held-out participant;
3. apply the prespecified aggregation;
4. calculate the participant-level metric.

Avoid averaging fold AUROCs when fold sizes or class distributions differ sharply. Pool out-of-fold predictions when the design permits it, and retain fold-level results to show variability.

For repeated observations, report both:

• number of independent units;
• number of windows or measurements.

“120,000 samples” without “60 participants” is sales copy disguised as methods.

A random label permutation across all windows can break the group structure in an unrealistic way. Choose a control that preserves the nuisance you are testing.

Examples:

Participant shortcut

Permute labels between participants while preserving all windows within each participant. If performance survives, identity or participant-level imbalance is carrying the result.

Temporal shortcut

Shift labels within a session or use a target from the wrong time block.

Device shortcut

Predict device ID. Then test the target after holding out devices.

Missingness shortcut

Train a model using only missingness indicators and recording duration.

Protocol-order shortcut

Predict the target from trial number, acquisition date or task order.

A useful control is specific enough to embarrass the model.

A single mean can hide the operating shape.

For each held-out participant, report:

• number of eligible windows;
• class balance;
• signal-quality distribution;
• predicted probability distribution;
• participant-level error;
• device, session and protocol context.

Plot the errors against quality and coverage. A model may work on participants with 25 valid minutes and fail below ten. That threshold belongs in the result.

Check calibration too. A classifier can rank cases reasonably while producing probabilities that are unusable for a decision threshold.

Participant-held-out validation within one site estimates performance for new participants drawn from that site’s collection process. It says little about transfer to another hospital, device generation or protocol team.

Where the dataset permits it, compare:

• random windows;
• participant-held-out within site;
• leave-one-device-out;
• leave-one-site-out;
• temporal holdout on later collection.

The harder split will usually produce a lower and wider estimate. That is the estimate attached to the wider claim.

Sayash Kapoor and Arvind Narayanan reviewed leakage across 17 scientific fields and identified at least 294 affected papers in the published version of their analysis.^[a3-6] Their taxonomy includes obvious train-test contamination and harder failures where the prediction problem itself allows information that would be unavailable at deployment.

A clean train_test_split cannot answer the latter. The dataset design has to specify what the model is allowed to know and when.

A compact validation statement should include:

Independent units: 48 participants
Repeated observations: 5,712 windows
Deployment target: new participant from the same protocol
Outer validation: leave-one-participant-out
Inner selection: 5-fold GroupKFold on training participants
Preprocessing: fitted inside each inner and outer training fold
Primary metric: participant-level balanced accuracy
Secondary metrics: window AUROC, calibration slope, retained coverage
Negative controls: participant-label permutation; missingness-only model

Add the group distribution, failed participants and exact aggregation rule.

The first file I ask for in a validation review is often the table used to create the split. The model comes second.

References

Saeb S, Lonini L, Jayaraman A, Mohr DC, Kording KP. The need to approximate the use-case in clinical machine learning. GigaScience. 2017;6(5):1–9.

Roberts DR, Bahn V, Ciuti S, et al. Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecography. 2017;40(8):913–929.

Dehghani A, Glatard T, Shihab E. Subject Cross Validation in Human Activity Recognition. 2019.

scikit-learn developers. GroupKFold documentation.

scikit-learn developers. Common pitfalls and recommended practices: data leakage.

Kapoor S, Narayanan A. Leakage and the reproducibility crisis in machine-learning-based science. Patterns. 2023;4(9):100804.