Neuroimaging QC & mask editing
Inspect scans, pick failures and edit masks.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- FSLeyes · FreeView · MRIcroGL
- Class GUI-need label
G3· GUI-only
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R08.09.01.03-T01 Choose the best skull-strip among three candidates in FSLeyes
Original instruction
In FSLeyes load input/T1.nii.gz with the three candidate masks in input/masks/ as overlays. Inspect axial, coronal and sagittal views for over- and under-stripping, then write output/qc_result.json with chosen_mask, verdict (pass or fail) and a one-sentence rationale.
- Platforms
mac· macOSlin· Linux- Applications
- FSLeyes
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicknav· Pan/zoom/scrollvis· Visual judgment- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
chosen_mask and verdict must equal gold exactly; rationale keyword-checked for the deciding defect (temporal-lobe loss or neck inclusion). Mask Dice values are withheld and the masks' file names carry no hint. FSL command-line tools and nibabel are absent from the image, and FSLeyes' Python shell and --runscript are disabled.
Proposed partial credit
60% chosen mask; 25% verdict; 15% rationale names the defect.
R08.09.01.03-T02 Label eight fMRI runs pass or fail from mean and standard-deviation images
Original instruction
In FSLeyes open the 8 runs in input/func/ with their mean and tSD images. Judge each for signal dropout, ghosting, spiking and gross motion, and write output/qc.csv with run, verdict and the artifact type for failures.
- Platforms
mac· macOSlin· Linux- Applications
- FSLeyes
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentclick· Point & click- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Verdicts must match the expert labels for all 8 runs; artifact types must match the seeded defect per failing run (3 fail). No motion-parameter files are provided. FSL command-line tools and nibabel are absent from the image, and FSLeyes' Python shell and --runscript are disabled.
Proposed partial credit
70% verdicts (8.75 pts each); 30% artifact types.
R08.09.01.03-T03 Report the atlas label at a cluster peak in FSLeyes
Original instruction
In FSLeyes load input/MNI152_T1_2mm.nii.gz, overlay input/zstat1.nii.gz thresholded at 3.1, add the Harvard-Oxford cortical atlas, navigate to the peak voxel of the largest cluster, and write its MNI coordinates and the atlas label to output/answer.json.
- Platforms
mac· macOSlin· Linux- Applications
- FSLeyes
- Interfaces
desk· Desktop GUI- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clicknav· Pan/zoom/scrollvis· Visual judgment- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Coordinates within 4 mm of the gold peak and the label string equal to the Harvard-Oxford label at that location.
Proposed partial credit
50% peak location; 50% atlas label.
R08.09.01.03-T04 Edit a brain mask in FSLeyes to remove an included eye globe
Original instruction
Load input/T1.nii.gz and input/brain_mask.nii.gz in FSLeyes. The mask wrongly includes the right eye globe and part of the optic nerve. Use the edit tools (brush, fill, erase with a selected 3D region) to remove them without eroding cortex, and save output/brain_mask_fixed.nii.gz.
- Platforms
mac· macOSlin· Linux- Applications
- FSLeyes
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathclick· Point & clickvis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Dice ≥0.95 with the gold corrected mask; zero voxels remaining inside the gold eye-globe ROI; ≥99% of gold cortical mask voxels retained. FSL command-line tools and nibabel are absent from the image, and FSLeyes' Python shell and --runscript are disabled.
Proposed partial credit
40% eye removed; 35% Dice overall; 25% cortex preserved.
R08.09.01.03-T05 Fix a FreeSurfer white-matter surface error with control points in FreeView
Original instruction
In FreeView load subject input/sub-01 (T1, wm.mgz, surfaces). The white surface under-includes white matter in the left superior frontal region (shown in input/hint.png). Add control points there, save them as control.dat, rerun the pial/white regeneration step, and save output/lh.white plus output/control.dat.
- Platforms
lin· Linuxmac· macOS- Applications
- FreeView · FreeSurfer
- Interfaces
desk· Desktop GUI- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP10· Annotations- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Control points lie inside the gold defect region (≥80% within it, ≤5 outside); after rerun, surface-to-gold distance in the region drops below 0.7 mm mean and surfaces elsewhere change by <0.2 mm mean. recon-all is the only command-line step allowed; control.dat must be saved from FreeView's point-set tool.
Proposed partial credit
35% control points placed correctly; 40% surface corrected in region; 25% no change elsewhere.
R08.09.01.03-T06 Edit a wm.mgz to erase a misclassified voxel cluster in FreeView
Original instruction
In FreeView open input/sub-02/mri/T1.mgz and wm.mgz. A cluster of non-white-matter voxels near the left temporal pole is labelled 255. Erase exactly that cluster with the voxel-edit tool (leaving other 255 voxels untouched) and save output/wm_edited.mgz.
- Platforms
lin· Linuxmac· macOS- Applications
- FreeView
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathclick· Point & clickvis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
All gold erroneous voxels set below 110; ≥99.5% of other 255 voxels unchanged; no new 255 voxels added. The cluster is not separable by intensity threshold alone. FreeView's command console and -cmd files are disabled, and FSL, nibabel and FreeSurfer mri_* editors are absent from the image.
Proposed partial credit
50% cluster removed; 35% other voxels preserved; 15% no additions.
R08.09.01.03-T07 Judge EPI-to-T1 registration quality for four subjects in FSLeyes
Original instruction
In FSLeyes load the four registered EPI images in input/reg/ over their T1s and inspect edge alignment at the ventricles and cortical boundary. Write output/reg_qc.csv with subject, verdict and the direction of misregistration for failures (e.g., 'EPI shifted posteriorly').
- Platforms
mac· macOSlin· Linux- Applications
- FSLeyes
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentclick· Point & click- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Verdicts match gold for all four; for the two failures the stated direction must match the sign of the applied shift in the seeded transforms. Transform files are not provided. FSL command-line tools and nibabel are absent from the image, and FSLeyes' Python shell and --runscript are disabled.
Proposed partial credit
60% verdicts; 40% directions for failures.
R08.09.01.03-T08 Create a lesion mask by hand on a FLAIR image in FSLeyes
Original instruction
Load input/FLAIR.nii.gz in FSLeyes and create a new mask overlay. Draw the periventricular white-matter hyperintensity lesions visible across slices 40–70 with the brush, excluding normal septal and ependymal signal, and save output/lesion_mask.nii.gz plus the total lesion volume (mL) in output/answer.json.
- Platforms
mac· macOSlin· Linux- Applications
- FSLeyes
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathclick· Point & clickvis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP10· Annotations- Conditions
long· Long-horizon- Proposed difficulty
- 4 / 5
Proposed verifier
Dice ≥0.70 with the expert lesion mask, false-positive volume ≤25% of lesion volume, and volume within 20% of expert value. Intensity thresholding alone yields Dice ~0.5 on this scan (reference attempt). FSL command-line tools and nibabel are absent from the image, and FSLeyes' Python shell and --runscript are disabled.
Proposed partial credit
55% Dice; 25% false-positive control; 20% volume.
R08.09.01.03-T09 Compare Freesurfer aparc segmentation against the T1 and flag mislabelled regions
Original instruction
In FreeView load input/sub-03 with aparc+aseg.mgz over T1.mgz. Inspect the segmentation and list in output/errors.csv the three regions with visible label errors (region name, slice range, error type: over-inclusion, under-inclusion or wrong label), judged from anatomy, not statistics.
- Platforms
lin· Linuxmac· macOS- Applications
- FreeView
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmenthover· Hover & context- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Region set must equal the gold three (exact label names); error type must match per region; slice ranges must overlap the gold ranges. The subject's stats files are removed so outlier detection by numbers is unavailable. FreeView's command console is disabled and FreeSurfer stats tools (mri_segstats) are absent from the image.
Proposed partial credit
60% three regions (20 pts each); 25% error types; 15% slice ranges.
R08.09.01.03-T10 Render a 3D cortical surface with an overlay in MRIcroGL and export a figure
Original instruction
In MRIcroGL load input/mni152.nii.gz, add input/zstat.nii.gz as an overlay with threshold 2.3–6 and the 'red-yellow' colour scale, show a left-lateral 3D render (azimuth 270°, elevation 0°), add a colour bar, and save output/surface.png at 1600×1200.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- MRIcroGL
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clickdrag· Drag & dropvis· Visual judgment- Work product
WP03· Visual layout- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
PNG 1600×1200 with SSIM ≥0.85 against the reference render at that view, overlay voxels coloured only where z ≥2.3 (colour-mask check), and a colour bar present. MRIcroGL's Python scripting pane is disabled.
Proposed partial credit
30% overlay threshold and colours; 35% view; 20% image matches; 15% colour bar.
R08.09.01.03-T11 Pick the correct fieldmap-corrected EPI among three unwarping variants
Original instruction
In FSLeyes load input/epi_raw.nii.gz and the three unwarped versions in input/unwarped/ with the T1. Decide which used the correct phase-encoding direction by inspecting frontal and temporal distortion against the T1, and write the choice plus the direction error in the other two to output/answer.json.
- Platforms
mac· macOSlin· Linux- Applications
- FSLeyes
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentclick· Point & click- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Chosen file must equal gold; the stated direction error for each rejected file must match the seeded sign (over-unwarped or inverted). JSON sidecars with PhaseEncodingDirection are withheld. FSL command-line tools and nibabel are absent from the image, and FSLeyes' Python shell and --runscript are disabled.
Proposed partial credit
50% correct choice; 50% direction diagnosis for the two rejects.
R08.09.01.03-T12 Edit an aseg to correct a hippocampus boundary in FreeView
Original instruction
In FreeView load input/sub-04 T1.mgz and aseg.mgz. The left hippocampus label (17) extends into the amygdala across slices given in input/slices.txt. Recut the boundary with the voxel-edit tool so the labels follow the uncal recess, and save output/aseg_edited.mgz.
- Platforms
lin· Linuxmac· macOS- Applications
- FreeView
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathclick· Point & clickvis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP10· Annotations- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Left hippocampus Dice ≥0.90 and amygdala Dice ≥0.88 with the expert edit; voxels of other labels changed <0.5%; total intracranial label volume unchanged within 0.2%. FreeView's command console and -cmd files are disabled, and FSL, nibabel and FreeSurfer mri_* editors are absent from the image.
Proposed partial credit
40% hippocampus Dice; 35% amygdala Dice; 25% no collateral label changes.
R08.09.01.03-T13 Create a sphere ROI at given MNI coordinates in FSLeyes and report its mean value
Original instruction
In FSLeyes load input/MNI152_T1_2mm.nii.gz, create a 6 mm radius spherical mask centred at MNI (−42, 18, 24), save it as output/roi.nii.gz, then report the mean of input/cope1.nii.gz inside that ROI in output/answer.json.
- Platforms
mac· macOSlin· Linux- Applications
- FSLeyes
- Interfaces
desk· Desktop GUI- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clicktype· Text & formsnav· Pan/zoom/scroll- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
ROI centroid within 2 mm of the target, voxel count equal to the gold sphere ±5%, and the reported mean within 1% of the value computed from the saved ROI.
Proposed partial credit
40% ROI location; 30% size; 30% mean value.
R08.09.01.03-T14 Build a four-panel QC montage of ten subjects' registrations in FSLeyes
Original instruction
For the ten subjects in input/subs/, use FSLeyes to make a QC sheet: for each subject one axial, one coronal and one sagittal snapshot of the registered T1 with MNI edge overlay, assembled into output/qc_montage.png with subject labels, plus output/qc_notes.csv listing the two subjects whose registration fails.
- Platforms
mac· macOSlin· Linux- Applications
- FSLeyes
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicknav· Pan/zoom/scrollvis· Visual judgment- Work product
WP03· Visual layout- Conditions
long· Long-horizon- Proposed difficulty
- 3 / 5
Proposed verifier
Montage has 30 panels with readable subject labels (OCR) and edge overlays present; the failing-subject set equals gold exactly. FSL command-line tools and nibabel are absent from the image, and FSLeyes' Python shell and --runscript are disabled.
Proposed partial credit
40% complete montage; 20% labels and overlays; 40% failing subjects identified.
R08.09.01.03-T15 Identify the dropout-affected region of an EPI and name the likely cause
Original instruction
In FSLeyes load input/epi_mean.nii.gz with input/T1.nii.gz. Locate the region of marked signal dropout, report its anatomical label (from the Harvard-Oxford atlas at the dropout centre), its approximate extent in mm³, and the likely physical cause in output/answer.json.
- Platforms
mac· macOSlin· Linux- Applications
- FSLeyes
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentclick· Point & click- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Atlas label must equal gold; extent within a factor of 2 of the gold dropout volume; the stated cause must match the seeded one (air–tissue susceptibility at the sinus) by keyword. FSL command-line tools and nibabel are absent from the image, and FSLeyes' Python shell and --runscript are disabled.
Proposed partial credit
45% region label; 30% extent; 25% cause.
R08.09.01.03-T16 Control: pick the best skull-strip mask by computed overlap metrics
Original instruction
Control pair of T01. From the terminal, compute Dice and Jaccard of each mask in input/masks/ against input/reference_mask.nii.gz (provided here) with FSL or nibabel, and write chosen_mask and verdict to output/qc_result.json.
- Platforms
lin· Linuxmac· macOS- Applications
- FSL · nibabel
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Same chosen_mask and verdict as the T01 gold. Scored as the CLI control of T01; a reference mask is provided here precisely because the GUI task has none.
Proposed partial credit
60% chosen mask; 25% verdict; 15% metrics reported.
R08.09.01.03-T17 Detect and localize spike artifacts across time in an EPI series
Original instruction
In FSLeyes load the 4D series input/func.nii.gz and scrub through time (and use the time-series view at several voxels) to find volumes with spike artifacts. Write output/spikes.csv listing volume indices and the affected slice range, and the total count in output/answer.json.
- Platforms
mac· macOSlin· Linux- Applications
- FSLeyes
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentav· Audio/video over time- Work product
WP12· Answer/decision- Conditions
long· Long-horizon- Proposed difficulty
- 4 / 5
Proposed verifier
Volume index set F1 ≥0.85 against gold (4 spike volumes among 240) and slice ranges overlapping gold; motion-outlier tools (fsl_motion_outliers) are absent from the image. FSLeyes' Python shell and --runscript are disabled.
Proposed partial credit
60% spike volumes (F1 scaled); 25% slice ranges; 15% count.
R08.09.01.03-T18 Correct a tissue-segmentation leak at the skull in FSLeyes
Original instruction
input/gm_mask.nii.gz leaks into dura and skull along the superior convexity. In FSLeyes edit the mask to remove the leak while keeping true grey matter, including in the sulcal depths, and save output/gm_mask_fixed.nii.gz plus the voxel count removed in output/answer.json.
- Platforms
mac· macOSlin· Linux- Applications
- FSLeyes
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathclick· Point & clickvis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP10· Annotations- Conditions
long· Long-horizon- Proposed difficulty
- 4 / 5
Proposed verifier
Dice ≥0.93 with the expert mask; ≥90% of gold leak voxels removed; ≤2% of gold grey-matter voxels lost; reported count within 10% of actual change. FSL command-line tools and nibabel are absent from the image, and FSLeyes' Python shell and --runscript are disabled.
Proposed partial credit
35% leak removal; 35% Dice; 20% GM preservation; 10% count reported.
R08.09.01.03-T19 Draw a tumour segmentation across three modalities in MRIcroGL
Original instruction
In MRIcroGL load input/T1c.nii.gz, input/T2.nii.gz and input/FLAIR.nii.gz of the same patient. Draw a mask of the enhancing tumour core using T1c and a second mask of the peritumoural oedema using FLAIR, keeping them non-overlapping, and save output/core.nii.gz and output/edema.nii.gz plus volumes in output/answer.json.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- MRIcroGL
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathclick· Point & clickvis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP10· Annotations- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Core Dice ≥0.75 and oedema Dice ≥0.70 against expert masks; overlap between the two submitted masks <2% of core volume; volumes within 20% of expert values. MRIcroGL's Python scripting pane is disabled and FSL/nibabel are removed from the image.
Proposed partial credit
35% core; 35% oedema; 20% non-overlap; 10% volumes.
R08.09.01.03-T20 Triage a 12-subject structural QC batch and write a reviewer report
Original instruction
For the 12 T1s in input/batch/, inspect each in FSLeyes (motion, ringing, coverage, inhomogeneity, incidental findings) and produce output/qc_table.csv with per-subject verdict and artifact tags, plus output/report.md summarizing exclusions and the two subjects needing manual mask editing before reprocessing.
- Platforms
mac· macOSlin· Linux- Applications
- FSLeyes
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentclick· Point & click- Work product
WP01· Document- Conditions
long· Long-horizon- Proposed difficulty
- 4 / 5
Proposed verifier
Verdicts match expert labels for ≥11 of 12; artifact tags match the seeded defects per failing subject; the two edit-needed subjects must be exactly those in gold; report.md must state the exclusion count consistent with the table. FSL command-line tools and nibabel are absent from the image, and FSLeyes' Python shell and --runscript are disabled.
Proposed partial credit
45% verdicts; 25% artifact tags; 20% edit-needed subjects; 10% consistent report.
Mapped research references
Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.