Edit this page

NA-MIC Project Weeks

Back to Projects List

Robust Segmentation Experience in OHIF Viewer

Key Investigators

Project Description

OHIF Viewer is a widely used web-based medical image viewer built on Cornerstone3D. Segmentation is central to many clinical and research workflows, yet reliability and robustness in OHIF/Cornerstone still lag behind user expectations — with recurring errors and edge cases in everyday use (OHIF/Viewers#5453).

During Project Week 45, I would like to focus on improving the stability and user experience of segmentation in OHIF/Cornerstone. I am looking forward to collaborating with others interested in segmentation, digital pathology, and web-based imaging viewers.

This project targets three complementary areas:

Several related gaps remain in the broader stack: the WSI microscopy viewer (wsi-microscopy-viewer) does not yet support segmentation display, overlapping segmentations are not always handled reliably, and contour ↔ labelmap conversion is not yet exposed end-to-end in OHIF. This project aims to advance solutions within Cornerstone3D and OHIF toward a more dependable segmentation experience across radiology and pathology use cases.

For DICOM persistence, standards context, and recommended interchange with research formats (NIfTI), see Background and References.

Objective

  1. Improve segmentation reliability — Identify and address high-impact failure modes in OHIF/Cornerstone segmentation workflows (load, render, edit, persist) to reduce user-facing errors.
  2. Contour ↔ labelmap interoperability — Allow users and tools to convert between contour-based and labelmap-based segmentation representations in OHIF/Cornerstone3D, building on work such as Cornerstone3D PR #2170.
  3. Microscopy segmentation support — Enable segmentation rendering (labelmap and/or contour overlay) in the OHIF WSI microscopy viewer, progressing toward capabilities comparable to SLIM.
  4. DICOM persistence — Clarify recommended save/load paths for segmentations (DICOM SEG and emerging label-map encodings) and validate practical DICOM ↔ NIfTI interchange (see DICOM-SEG and format interchange).

Approach and Plan

  1. Audit current state — Review segmentation pain points in OHIF/Viewers and Cornerstone3D; compare microscopy segmentation support in wsi-microscopy-viewer vs. SLIM and document pipeline gaps.
  2. Reliability fixes — Triage and reproduce issues from community reports (e.g. #5453, #5849); prototype fixes or workarounds for the most impactful cases.
  3. Contour ↔ labelmap conversion — Review and test upstream conversion work; expose it in the OHIF UI so users need not convert representations manually.
  4. Microscopy segmentation MVP — Integrate Cornerstone3D segmentation rendering into the OHIF WSI microscopy viewport; target labelmap overlay as a first milestone.
  5. Overlapping segments — Investigate segment blending/ordering in Cornerstone3D so multiple overlapping segments render and interact correctly.
  6. DICOM persistence audit — Map save/load paths (classic SEG vs label-map objects); test against OHIF/Viewers PR #5806; document DICOM ↔ NIfTI assumptions (background notes).
  7. Integration testing — Validate changes with real DICOM SEG and RT-STRUCT datasets (radiology and pathology); record short demo screencasts.
  8. Documentation and follow-up — Open or update GitHub issues/PRs in cornerstonejs/cornerstone3D and OHIF/Viewers for work continuing beyond the week.

Progress and Next Steps

(To be filled in during and after the event)

  1. …
  2. …
  3. …

Illustrations

Background and References

OHIF and Cornerstone3D

Resource Notes
OHIF Viewer Web viewer
Cornerstone3D Rendering engine
Cornerstone3D PR #2170 Contour ↔ labelmap conversion
SLIM Viewer Microscopy segmentation reference
OHIF WSI Microscopy Viewer WSI viewer (limited segmentation support)
Item Relevance
#5453 Segmentation reliability
#5849 Labelmap interpolation (editing; affects exported SEG)
#4875 DICOM label-map support
PR #5806 SEG load/save via per-frame imageLoader; shared dcmjs writer
#2833 Interoperability between highdicom-authored SEGs and OHIF

DICOM-SEG and format interchange

Reference material for persistence, standards, tooling, and conversion with NIfTI. Intended as background for the DICOM persistence objective and approach step 6.

Goal Recommended approach Typical stack
Clinical / PACS / multi-site sharing DICOM Segmentation (SOP Class 1.2.840.10008.5.1.4.1.1.66.4) via DICOMweb alongside source images OHIF cornerstone-dicom-seg, @cornerstonejs/adapters, dcmjs; offline creation with highdicom or dcmqi
In-viewer editing in OHIF Cornerstone3D labelmap or contour during interaction; export to DICOM SEG on save from referenced volume metadata @cornerstonejs/adapters, dcmjs (PR #5806: shared dicomWriter.ts, preserved transfer syntax)
Research / ML pipelines NIfTI, NRRD, or Slicer .seg.nrrd internally; DICOM at archive boundaries SimpleITK, NiBabel, Slicer; highdicom or pydicom-seg for DICOM export
Many non-overlapping segments (100+) Label-map encodings when available; classic BINARY SEG remains valid but often large/slow highdicom LABELMAP prototypes; DCMTK/dcmqi; OHIF #5806 (labelmap RLE where supported)

OHIF loading (PR #5806): Replaces whole-object ArrayBuffer fetch with per-frame imageLoader decoding (Cornerstone3D fix/use-imageLoader-for-seg). Improves load/save for labelmap and compressed bitmap SEGs. Parser type (labelmap vs bitmap) follows SOP Class / pixel layout. Caveat: multi-frame SEGs on non–WADO-RS schemes (dicomfile:, wadouri:) may decode only one frame without error if /frames/N URLs are missing.

Standards landscape

Three related tracks:

  1. Segmentation IOD (existing) — Part 3 §C.8.20. Types include BINARY and FRACTIONAL. Multiple segments historically meant separate binary frame sets (supports overlap; inefficient for many non-overlapping labels). See PW38 DICOM SEG notes.

  2. Supplement 243 — Label Map Segmentation (Final Text, DICOM 2024d) — Dedicated IOD: pixel values encode segment membership in one array (no overlap per instance). NEMA Sup 243 · overview. Prototypes: highdicom (PW39); DCMTK/dcmqi/Slicer (PW40).

  3. LABELMAP Segmentation Type (classic SEG object) — Community encodings on the existing Segmentation IOD (RLE/JPEG-LS/JP2K). Example (TotalSegmentator, PW39): ~385 MB binary vs ~1.9–6.7 MB compressed labelmap.

OHIF: #4875 tracks first-class label-map support. Until broadly available, assume classic DICOM SEG for production interchange; experiment with label-map objects alongside #5806.

DICOM SEG vs NIfTI — why both persist

  DICOM SEG NIfTI / NRRD / seg.nrrd
Strengths Standard SOP Class; segment metadata; FoR / referenced series; DICOMweb / PACS Fast I/O; ML ecosystem; simple dense arrays
Weaknesses Size/speed for many-segment binary SEG; viewer variance Weak clinical metadata; no study linkage without sidecars

Hub-and-spoke pattern: NIfTI or in-memory labelmaps inside analysis; DICOM SEG at the boundary for VNA/PACS, OHIF, or Slicer.

DICOM ↔ NIfTI interchange

DICOM SEG → NIfTI

Mostly lossless for voxels if geometry and labels are preserved:

NIfTI → DICOM SEG

NIfTI does not carry DICOM identity or segment ontology. Required inputs:

Input If missing
Referenced series (Study/Series/SOP UIDs, Frame of Reference UID) Must be chosen explicitly — cannot derive from NIfTI alone
Spatial alignment with reference grid Resample NIfTI to reference geometry or document transform
Per-segment metadata (number, label, category codes) Sidecar JSON/CSV or defaults
Segmentation Type BINARY, LABELMAP, or Sup 243 object per overlap/tooling support
Overlap policy Overlapping labels in one array need separate binary segments or another representation

Suggested pipeline: load reference DICOM → resample labelmap → encode with highdicom or dcmqi → validate in OHIF via DICOMweb. Web-only: dcmjs adapters from Cornerstone labelmaps when viewport metadata includes the reference study.

DICOM tooling map

                    ┌─────────────────────────────────────────┐
                    │           Application layer            │
                    │  OHIF  │  3D Slicer  │  ML scripts     │
                    └────┬──────────┬──────────────┬──────────┘
                         │          │              │
           ┌─────────────┼──────────┼──────────────┼─────────────┐
           ▼             ▼          ▼              ▼             ▼
      dcmjs +      cornerstone-   dcmqi +     highdicom    pydicom-seg
      adapters     dicom-seg      Quant.       (+ pydicom)  (+ SimpleITK)
           │             Reporting      │
           └─────────────┬──────────────┘
                         ▼
              pydicom (Python)  │  dicomParser (JS, legacy)
                         ▼
              DCMTK / GDCM (C++) — dcmseg, DIMSE
Package Language Role
pydicom Python Low-level DICOM I/O
highdicom Python SEG/SR/PR builders; label-map prototypes
pydicom-seg Python Fast multiclass SEG → SimpleITK
dcmjs JavaScript Browser SEG (de)serialization
@cornerstonejs/adapters TypeScript Labelmap/contour ↔ DICOM SEG
DCMTK / dcmseg C++ Reference encoding
dcmqi C++ / CLI ITK/NIfTI ↔ DICOM SEG (Slicer)

Validate round-trips (highdicom → OHIF → dcmjs → Slicer) when changing write options (transfer syntax, fragmentMultiframe, labelmap vs bitmap).

DICOM standards and Project Week notes