How to validate cell segmentation when ground truth is limited

A segmentation model returns a Dice score of 0.91.

On the overlay, two touching nuclei have become one object. Five dim nuclei are missing. A bright fragment has been counted as a cell. The experiment uses object count and per-cell intensity.

The Dice score is technically correct and scientifically unhelpful.

Write the downstream measurement before choosing the validation metric.

Examples:

• count nuclei per field;
• estimate cell area;
• quantify mean fluorescence per cell;
• detect puncta inside each cell;
• measure nearest-neighbour distance;
• classify cell morphology;
• track cells through time;
• calculate treatment-level change in object count.

Each measurement has a different tolerance for segmentation error.

A one-pixel boundary shift may barely affect a count and materially alter membrane intensity. A merge of two cells corrupts count, area, intensity and neighbourhood structure. A missed dim object can create treatment bias if one condition systematically changes brightness.

Lena Maier-Hein and the Metrics Reloaded consortium built their validation framework around this problem: metric choice should follow the biomedical question, task type and failure costs.^[a4-1]

Two annotators can disagree while both follow reasonable biological interpretations.

Decide:

• which channel defines the boundary;
• whether touching cells must be separated;
• whether partial objects at the image edge count;
• how debris, apoptotic bodies or saturated objects are handled;
• whether holes belong inside a mask;
• how multinucleated cells are represented;
• minimum visible evidence for an object;
• whether the target is nucleus, cytoplasm, whole cell or colony.

Create an annotation guide with image examples. Include difficult cases and explicit exclusions.

For a small validation set, ask two annotators to label an overlapping subset. Their disagreement shows where “ground truth” is unstable. Do not average away a biological ambiguity and then blame the model for finding it.

Randomly selecting 20 fields from a large experiment often produces 20 ordinary fields.

Build a sampling frame from acquisition and image characteristics:

• treatment and control;
• plate, batch and acquisition date;
• instrument or objective;
• operator;
• centre and edge wells;
• sparse and dense fields;
• low and high intensity;
• in-focus and borderline focus;
• saturated and unsaturated images;
• common morphology and unusual morphology.

A practical first validation set might contain 30 to 60 fields, deliberately spread across those conditions. The exact number depends on object count, diversity and intended claim. Twenty fields with 500 nearly identical nuclei each do not provide the same evidence as 20 fields across five staining batches and three morphologies.

Keep one set for model or parameter selection and another for final evaluation. If the same difficult images guide every threshold adjustment, they have joined the training process.

Compare at least one simple method.

For fluorescence nuclei, that might be:

1. flat-field or background correction;
2. local or Otsu threshold;
3. morphological cleanup;
4. distance transform;
5. watershed separation;
6. connected-component filtering.

Then compare Cellpose, StarDist or a task-specific model under the same held-out images.

Cellpose was introduced as a generalist model trained on more than 70,000 segmented objects across varied image types.^[a4-2] StarDist predicts star-convex polygons or polyhedra, a useful representation for many densely packed nuclei.^[a4-3] Neither name answers whether the method works on this stain, magnification, density and boundary definition.

The classical baseline may win on a controlled assay. That is a good result when the method is stable, interpretable and cheap to maintain.

Pixel overlap alone does not describe instance errors.

Create a matching between predicted and reference objects, commonly using intersection over union (IoU). A predicted object can be matched to a reference object when IoU exceeds a declared threshold. The threshold should be reported, not hidden inside a library default.

From the matches, count:

• true detected objects;
• missed reference objects;
• false predicted objects;
• merges, where one prediction overlaps multiple references;
• splits, where multiple predictions overlap one reference;
• unmatched fragments;
• edge-case exclusions.

A one-to-one matching algorithm prevents one broad prediction from claiming success against several cells.

Error table

The dense field in row two needs a different intervention from the false detections in row three.

Counting

Use object-level precision, recall, F1 and count error. Report bias across fields or wells.

A model can achieve good pixel overlap and systematically undercount dense regions through merges.

Boundary-sensitive morphology

Use a boundary or surface metric beside overlap. Hausdorff distance is sensitive to extreme boundary errors; percentile variants can reduce domination by a single outlier. Surface Dice can be defined with a tolerance relevant to the pixel size and biological measurement.

Instance separation

Report merges and splits directly. Average precision across IoU thresholds can help, but retain the counts that describe the failure.

Fluorescence intensity

Compare the downstream intensity measurement produced by manual and automated masks. Plot the difference against the mean rather than relying on correlation. Bland and Altman’s method was developed because two measurement methods can correlate strongly while disagreeing systematically.^[a4-4]

Treatment effect

Run the final biological comparison under:

• reference masks;
• automated masks;
• plausible alternative segmentation parameters.

If the treatment conclusion changes across reasonable settings, segmentation uncertainty belongs in the result.

Pooled results reward the dominant easy condition.

Stratify by:

• object density;
• median intensity;
• focus score;
• batch;
• plate position;
• treatment;
• object size;
• instrument;
• acquisition date.

A model with 0.92 mean IoU may fall to 0.55 on low-intensity treated cells. The treatment group is exactly where the measurement matters.

Plot each field. Thirty dots reveal more than one decimal place.

Create an error-coded overlay:

• green: matched object;
• red: missed reference;
• blue: false detection;
• orange: merge;
• purple: split.

Link each row in the error table to the source image, reference mask and prediction. Review the worst fields, a random sample and fields near the acceptance threshold.

Keep the raw channel visible. A mask-only review hides whether the annotation or the model is implausible.

Once the first model is running, use it to choose the next annotations.

Prioritise:

• image conditions absent from the current set;
• fields with low model confidence;
• disagreement between Cellpose, StarDist and the classical baseline;
• unusually high object count;
• large count differences between methods;
• clusters of merge or split errors;
• images from a new batch or instrument.

Cellpose 2.0 reported that fine-tuning pretrained models with 500 to 1,000 annotated regions of interest approached models trained on much larger labelled sets in their experiments. Its human-in-the-loop workflow reduced the reported requirement to 100 to 200 regions of interest for the tested data.^[a4-5] Those numbers are evidence for targeted adaptation, not a guaranteed annotation budget for every microscopy problem.

Annotate the next set because it changes the error model, not because the dataset has reached a round number.

A pretrained model still has choices:

• expected object diameter;
• probability or object threshold;
• flow threshold;
• tile overlap;
• 2D versus 3D handling;
• channel selection;
• image normalization;
• minimum object size;
• post-processing.

Tuning these on the final test images spends the test set.

Record each run:

model: cellpose_cyto3
software_version: 3.x
diameter_px: 42
cellprob_threshold: -0.5
flow_threshold: 0.4
normalization: percentile_1_99
minimum_area_px: 120
selection_set: plates P01-P03
final_test_set: plate P04

If the software or pretrained weights change, rerun the locked test set and record the change.

The image masks are an intermediate product.

Check:

• every measurement row links to image and object ID;
• excluded objects carry a reason;
• channel units and bit depth are preserved;
• background correction is applied consistently;
• edge objects follow the declared rule;
• intensity is measured on the intended raw or corrected image;
• merged objects do not silently enter per-cell analyses;
• plate and treatment metadata join correctly.

A perfect mask connected to the wrong treatment label is a complete failure.

Random field splitting allows images from the same plate, staining run and acquisition session into training and test data.

For a method intended for future experiments, hold out the unit that will be new:

• plate;
• staining batch;
• acquisition day;
• instrument;
• laboratory;
• cell line or tissue type.

Use nested grouping when several dependencies exist. A model selected on plates 1 to 3 and tested on fields from those same plates has not demonstrated transfer to plate 4.

Examples:

Object recall >= 0.95 for fields below 100 nuclei
Merge rate <= 2% in dense fields
Median absolute count error <= 3 objects per field
No treatment group with object recall below 0.90
Mean fluorescence bias within predeclared tolerance
All failed fields exported to review queue

Set the tolerances from the scientific measurement and available manual process. A universal “Dice above 0.9” rule has no basis.

I would stop and revisit acquisition or annotation when:

• boundaries are invisible in the recorded channel;
• annotators cannot agree on the object definition;
• treatment changes appearance in a way absent from training;
• saturation removes the structure needed for separation;
• the downstream result is dominated by segmentation choice;
• batch identity predicts the mask errors;
• a new model version improves the mean and worsens the biologically important subgroup.

For scoping, send three ordinary fields, three failures, the channel definitions, pixel size, existing masks and the measurement the experiment needs. Include the dense field everyone avoids showing.

References

Maier-Hein L, Reinke A, Godau P, et al. Metrics Reloaded: recommendations for image analysis validation. Nature Methods. 2024;21:195–212.

Stringer C, Wang T, Michaelos M, Pachitariu M. Cellpose: a generalist algorithm for cellular segmentation. Nature Methods. 2021;18:100–106.

Weigert M, Schmidt U, Haase R, Sugawara K, Myers G. Star-convex Polyhedra for 3D Object Detection and Segmentation in Microscopy. WACV. 2020.

Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement90837-8). The Lancet. 1986;327(8476):307–310.

Pachitariu M, Stringer C. Cellpose 2.0: how to train your own model. Nature Methods. 2022;19:1634–1641.