Before modelling: how to determine whether a sensor dataset is usable

A 30-minute recording sampled at 100 Hz should contain 180,000 samples per channel. That count is the first useful model in the project.

Suppose one participant has 179,842 rows. Another has 180,000 rows but 614 duplicate timestamps. A third has the expected count, a flat PPG channel for six minutes and an event file written in laptop time while the wearable logged device time. All three files will load into pandas.

I start there because the expensive mistakes rarely announce themselves as exceptions.

Before inspecting frequency content or plotting a confusion matrix, define the independent unit and the intended use.

For a wearable study, the independent unit might be a participant. For an instrument-development test, it may be a device. For condition monitoring, it may be an asset or installation. A dataset with 60 participants and 2,000 windows per participant contains 60 independent participants, not 120,000 independent examples.

The intended use changes the audit. A heart-rate estimate for seated laboratory recordings has different quality requirements from an ambulatory endpoint during daily activity. The U.S. Food and Drug Administration’s 2023 guidance on digital health technologies separates verification of the measured parameter from validation in the proposed population and context of use.^[a1-1] A step-count method tested on healthy walking does not inherit validity for a Parkinsonian shuffling gait because the output column has the same name.

Write four lines:

Independent unit: participant
Deployment unit: new participant
Measurement: beat-to-beat interval from wearable PPG
Context: unsupervised daytime recording
Decision: retain or reject each 30-second interval before HRV analysis

Without those lines, “good signal quality” remains undefined.

The raw file is already downstream of hardware and firmware choices.

For each channel, record:

• sensor and model;
• physical placement;
• nominal and observed sampling rate;
• ADC range and resolution where available;
• analogue filtering;
• digital filtering or smoothing performed on the device;
• gain and reference configuration;
• units and conversion formula;
• firmware and app version;
• compression, packet aggregation or resampling;
• behaviour during buffer overflow or wireless loss.

A PPG exported at 64 Hz may have been sampled internally at another rate, filtered, decimated and packetised before export. An EDA channel labelled in microsiemens may be a derived conductance trace with undocumented smoothing. An accelerometer file may switch between signed integer counts and physical units after a firmware update.

The audit does not need a reverse-engineered circuit diagram. It needs enough information to identify what transformations have already happened and which claims remain testable.

Basic range checks

Plot the raw distribution per channel and per device. Record:

• minimum and maximum;
• proportion at each rail;
• repeated values;
• step size between representable values;
• long flat runs;
• abrupt changes in scale;
• values outside physical or configured limits.

A channel that uses less than one per cent of the available ADC range may still carry a usable signal, but the choice deserves inspection. A channel pinned at a rail has lost information. Filtering cannot recover clipped peaks.

Timestamps deserve their own analysis table.

For each stream:

1. parse the timestamp without silently coercing failures;
2. sort and test monotonicity;
3. count duplicates;
4. calculate every inter-sample interval;
5. compare observed duration with row count and nominal rate;
6. find gaps and bursts;
7. identify clock resets and timezone conversions;
8. compare shared events across devices.

A crystal error of 100 parts per million produces 0.18 seconds of drift over 30 minutes. Fifty parts per million produces 1.44 seconds over eight hours. That may be negligible for a five-minute average and destructive for a short stimulus-locked response.

Do not correct drift from the first and last timestamp alone when packets are missing. Fit the relationship between clocks using repeated shared events or reliable time exchanges. Plot residuals after correction. A linear correction will not fix a clock reset halfway through the session.

Event files need provenance

An event row should say which clock generated it and where the timestamp was captured.

These are different:

• the operator pressed a key;
• the application queued a sound;
• the operating system sent the audio buffer;
• the sound reached the participant;
• a microphone recorded the onset;
• the wearable received a marker over Bluetooth.

Calling all of them stimulus_time removes the latency model from the dataset.

For short-latency work, use a measured event channel where possible. For slower responses, document the expected latency and uncertainty. Preserve the original marker beside the corrected one.

A percentage of missing rows is too crude.

Build a missingness table by:

• participant or asset;
• channel;
• session phase;
• time of day;
• device and firmware;
• activity or operating condition;
• outcome or experimental group;
• consecutive gap length.

A five per cent loss spread as isolated packets is different from a five per cent loss concentrated in one treatment phase. Data may disappear when movement increases, the battery falls, skin contact weakens or the device leaves wireless range. Those mechanisms can correlate with the target.

FDA’s DHT guidance asks sponsors to consider storage, transmission, battery, poor-signal alerts and plans for missing data rather than treating loss as a final statistical nuisance.^[a1-1] The same discipline is useful in an early pilot.

Report retained coverage

For every analysis unit, show:

expected duration
recorded duration
valid timestamp duration
channel-available duration
quality-passed duration
analysis-eligible duration
number of eligible events

The denominator changes at each line. Keep it visible.

Christina Orphanidou and colleagues built signal-quality indices for wearable ECG and PPG around a specific question: can a reliable heart rate be obtained from the segment? Their manually labelled evaluation reported sensitivities and specificities of 94% and 97% for ECG, and 91% and 95% for PPG.^[a1-2] Those figures do not make the same index sufficient for ST-segment morphology, pulse transit time or subtle waveform-shape analysis.

Quality criteria should match the downstream measurement.

For ECG heart rate, plausible beat intervals and agreement between detectors may be useful. For PPG morphology, clipping and pulse-shape distortion matter even when a heart-rate estimate survives. For EDA, a rapid contact change may resemble a physiological response. Sara Taylor and colleagues published an artefact-identification approach for EDA because wrist motion produces failures that simple amplitude thresholds miss.^[a1-3]

A quality mask should carry a reason code:

00 valid
10 missing samples
20 flat line
30 saturation
40 abrupt contact step
50 high-frequency motion
60 timing uncertainty
70 protocol mismatch
80 reviewer uncertain

Reason codes permit different downstream rules. A timing-uncertain interval may remain useful for a daily mean and fail a stimulus-locked analysis.

These are three decisions.

Detection marks a suspected failure. Correction changes the values or time base. Exclusion decides whether the corrected or uncorrected segment enters a particular analysis.

Keep the original data immutable. Store corrections as derived outputs with parameters and version information.

Interpolation is particularly easy to abuse. Filling a two-sample packet gap may be reasonable for a low-frequency summary. Interpolating a 20-second contact loss creates a plausible line with no physiological evidence behind it. Mark the gap and let the analysis decide whether it can tolerate missingness.

The protocol supplies expected structure:

• phase start and end;
• stimulus duration;
• expected response window;
• rest or washout;
• operator intervention;
• device changes;
• participant actions;
• known interruptions.

Plot those events on raw channels before averaging.

If every participant shows a step at the start of a task, check whether the task also triggered a screen change, hand movement, sound playback or marker packet. A repeatable artefact is still an artefact.

Protocol order can become a confound. If every case was recorded in the morning and every control in the afternoon, the model may learn temperature, operator or device state. The audit table should expose that before feature extraction.

An IMU burst beside a PPG distortion supports a motion explanation. It does not prove that every concurrent physiological change is false. Exercise produces motion and genuine autonomic response at the same time.

Useful checks include:

• local versus spatially coherent changes;
• agreement between related channels;
• lag between movement and signal disturbance;
• recovery after contact is restored;
• repetition under a controlled manoeuvre;
• comparison with a reference instrument.

The strongest audit usually contains examples that remain ambiguous. Label them as such.

A compact matrix catches a large fraction of project risk:

Do not collapse this into a green tick. The row is part of the analysis result.

I would stop a modelling build and return to acquisition or protocol when:

• the target event cannot be aligned within the required response window;
• most positive events occur in a different device, operator or batch;
• the sensor transformation needed to interpret values is unknown;
• the retained independent units are too few for the intended split;
• the quality rule removes one class or condition disproportionately;
• clipping or bandwidth loss has destroyed the feature of interest;
• labels cannot be traced to a stable definition;
• a simple nuisance variable predicts the target as well as the signal.

That work is still useful. It identifies the next collection change and prevents a polished model from laundering an acquisition failure.

A serious signal-quality audit should leave the team with:

• acquisition inventory;
• timestamp and event-alignment report;
• missingness and retained-coverage tables;
• quality definitions and reason codes;
• raw examples of each failure;
• reproducible QC code;
• a list of analyses the data can support;
• a list of analyses blocked by the present evidence;
• a concrete collection or firmware change where one is needed.

For scoping, send one ordinary session, one known failure, the acquisition notes and the protocol event table. Do not clean the files first.

References

U.S. Food and Drug Administration. Digital Health Technologies for Remote Data Acquisition in Clinical Investigations. Final guidance, December 2023.

Orphanidou C, Bonnici T, Charlton P, Clifton D, Vallance D, Tarassenko L. Signal-Quality Indices for the Electrocardiogram and Photoplethysmogram: Derivation and Applications to Wireless Monitoring. IEEE Journal of Biomedical and Health Informatics. 2015;19(3):832–838.

Taylor S, Jaques N, Chen W, Fedor S, Sano A, Picard R. Automatic identification of artifacts in electrodermal activity data. 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. 2015:1934–1937.