Georeferencing & digitizing
Georeference scans and digitize features with clean topology.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- QGIS · ArcGIS Pro · GRASS GIS
- Class GUI-need label
G3· GUI-only
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R07.06.01.02-T01 Georeference a 1950 city map scan to Web Mercator with under 2 px RMSE
Original instruction
In QGIS 3.40 open the Georeferencer with input/city_1950.tif. Using road intersections matched against input/osm_basemap.gpkg, place at least 8 GCPs spread across the sheet, use a 1st-order polynomial, and write out/city_1950_3857.tif in EPSG:3857 (cubic resampling, LZW). Save the .points file. Mean GCP residual must be under 2 px.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- QGIS
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicknav· Pan/zoom/scrollvis· Visual judgment- Work product
WP04· 3D/spatial model- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
rasterio: CRS 3857; ≥8 GCPs in .points with convex hull covering ≥60% of sheet; RMSE <2 px; 20 hidden check intersections land within 6 m of truth. Python console/processing scripts disabled; GDAL CLI absent.
Proposed partial credit
20% GCP count/spread; 30% RMSE; 40% check-point accuracy; 10% output format
R07.06.01.02-T02 Digitize 25 building footprints from an orthophoto into a GeoPackage
Original instruction
In QGIS with input/ortho_2025.tif (10 cm, EPSG:25832) digitize the 25 buildings inside polygon 'AOI' as polygons in out/buildings.gpkg (layer buildings, field bldg_type from input/types_legend.pdf: house/garage/shed). Use snapping so shared walls coincide; no overlaps.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- QGIS
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathgeom· Precise geometryvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Match to reference by max IoU: mean IoU ≥0.80, count 25±1; no overlaps >0.05 m²; shared-wall vertices coincide ≤0.05 m; bldg_type accuracy ≥90%. QGIS Python console, auto-vectorization plugins (SAM, Deep Learning) and GDAL/OGR CLIs are disabled; the harness input log must show interactive digitizing.
Proposed partial credit
50% mean IoU (scaled 0.6→0.8); 15% count; 15% topology; 20% attributes
R07.06.01.02-T03 Clean a road layer of dangles and overshoots until topology checks pass
Original instruction
fixtures/roads.gpkg has 7 dangles (undershoots) and 4 overshoots at intersections. In QGIS, using snapping and vertex tools, fix them so that roads meet exactly at nodes, then run Topology Checker rules 'must not have dangles' (excluding the 3 legitimate dead ends listed in docs/dead_ends.csv) and 'must not have duplicates'. Save the layer.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- QGIS
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
geom· Precise geometryclick· Point & clicknav· Pan/zoom/scroll- Work product
WP04· 3D/spatial model- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader's topology check: 0 dangles except the 3 listed; no duplicates; no overshoot segments >0.1 m beyond nodes; total length within 0.5% of fixture minus overshoots; untouched features unchanged. QGIS Python console, Processing scripts and GDAL/OGR CLIs are disabled for the agent; the harness input log must show the features were digitized interactively.
Proposed partial credit
45% dangles fixed; 30% overshoots fixed; 25% no collateral edits
R07.06.01.02-T04 Georeference a scanned parcel sheet with a spline transform in ArcGIS Pro
Original instruction
In ArcGIS Pro 3.4 georeference input/parcel_sheet_12.tif (distorted paper scan) to fixtures/parcels.gdb/ControlCorners using 12 control points at the marked monument symbols, apply a Spline transformation, and save as out/parcel_sheet_12_geo.tif in EPSG:2263. Export the link table to out/links.txt.
- Platforms
win· Windows- Applications
- ArcGIS Pro
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicknav· Pan/zoom/scrollvis· Visual judgment- Work product
WP04· 3D/spatial model- Conditions
licensed· Licensed pro app- Proposed difficulty
- 3 / 5
Proposed verifier
links.txt has 12 links each target within 0.1 ft of the monument point; raster CRS 2263; transform spline (aux.xml); 15 hidden parcel corners within 1.5 ft. arcpy/Python window disabled.
Proposed partial credit
35% links accurate; 15% spline used; 40% hidden check corners; 10% output
R07.06.01.02-T05 Digitize parcels under geodatabase topology rules and validate to zero errors
Original instruction
In ArcGIS Pro, in fixtures/cadastre.gdb create feature dataset topology with rules Must Not Overlap and Must Not Have Gaps for Parcels. Digitize the 18 parcels from input/plat_georef.tif within the block outline (snapping to block edges), attribute PIN per plat labels, validate topology and fix all errors.
- Platforms
win· Windows- Applications
- ArcGIS Pro
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathgeom· Precise geometryvis· Visual judgment- Work product
WP04· 3D/spatial model- Conditions
licensed· Licensed pro app- Proposed difficulty
- 4 / 5
Proposed verifier
Topology exists with the 2 rules; validate yields 0 errors (excluding outer boundary gap exception marked); 18 parcels with PINs matching plat; vertex accuracy vs reference ≤0.3 m; union equals block outline. ArcGIS Pro Python window, Notebooks and standalone ArcPy are disabled for the agent; the harness input log must show the parcels were sketched interactively.
Proposed partial credit
20% topology config; 40% parcel geometry; 20% 0 errors; 20% PINs
R07.06.01.02-T06 Trace a river centerline from an orthophoto using QGIS tracing
Original instruction
In QGIS digitize the centerline of the river visible in input/ortho_river.tif from the upstream bridge to the weir (marked in input/marks.gpkg) as a single line in out/river.gpkg, with vertices no further than 2 m from the visible center.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- QGIS
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathnav· Pan/zoom/scrollvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Single LineString; endpoints within 3 m of marks; Hausdorff to reference centerline ≤4 m; length within 2%. QGIS Python console, Processing scripts and GDAL/OGR CLIs are disabled for the agent; the harness input log must show the features were digitized interactively.
Proposed partial credit
20% endpoints; 50% Hausdorff; 30% length
R07.06.01.02-T07 Digitize land-use polygons with shared boundaries and legend codes
Original instruction
From the georeferenced 1985 land-use map input/landuse_1985.tif digitize all polygons within AOI into out/landuse.gpkg with field lu_code per the map legend (12 classes). Use 'avoid overlap' so neighbouring polygons share edges; the union must exactly fill the AOI.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- QGIS
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathgeom· Precise geometryvis· Visual judgment- Work product
WP04· 3D/spatial model- Conditions
long· Long-horizon- Proposed difficulty
- 4 / 5
Proposed verifier
Area-weighted class agreement with reference raster ≥85%; no overlaps; gaps total <0.1% AOI; union equals AOI; polygon count within ±15%. QGIS Python console, Processing scripts and GDAL/OGR CLIs are disabled for the agent; the harness input log must show the features were digitized interactively.
Proposed partial credit
50% class agreement (scaled 0.6→0.85); 25% topology (no overlaps/gaps); 25% codes valid
R07.06.01.02-T08 Digitize contour lines from a topo scan in GRASS and attribute elevations
Original instruction
In GRASS GIS 8.4 (GUI vector digitizer) import georeferenced input/topo_sheet.tif into location EPSG:32633, digitize all 10 m contours between 300 and 450 m inside region 'aoi', assign attribute elev, and build a DEM with v.surf.rst to out/dem.tif.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- GRASS GIS
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathnav· Pan/zoom/scrollvis· Visual judgment- Work product
WP04· 3D/spatial model- Conditions
long· Long-horizon- Proposed difficulty
- 4 / 5
Proposed verifier
Contour vector: each elev value present; lines within 15 m (Hausdorff) of reference contours; DEM RMSE ≤3 m vs reference DEM at 200 points; GRASS GUI command console, Python shell and terminal are disabled (modules run from menu dialogs only); mapset history must show vector digitizer edits and no r.contour.
Proposed partial credit
45% contour geometry; 25% elevations; 30% DEM RMSE
R07.06.01.02-T09 Georeference a colonial-era map in its original projected CRS
Original instruction
input/survey_1912.tif uses a Cassini-Soldner grid shown in the map margin (origin and false easting/northing printed). In QGIS, define the custom CRS from the marginal notes, place GCPs on grid crossings in that CRS, georeference with a Helmert transform, then reproject to EPSG:32644 as out/survey_1912_utm.tif.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- QGIS
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & formsvis· Visual judgment- Work product
WP04· 3D/spatial model- Conditions
tutorial· Follow docs/tutorial- Proposed difficulty
- 4 / 5
Proposed verifier
Custom CRS PROJ string parameters match key; GCPs on grid crossings with residuals <1.5 px; 10 hidden check points in final raster within 15 m. QGIS Python console, Processing scripts and GDAL/OGR CLIs are disabled for the agent; writes to out/ by any process other than QGIS are flagged by a file audit log.
Proposed partial credit
30% CRS definition; 25% GCPs; 35% check accuracy; 10% output
R07.06.01.02-T10 Control pair of T01: georeference the map scan with GDAL commands
Original instruction
Control pair of T01. Using only GDAL CLI, georeference input/city_1950.tif with the GCP pixel/coordinate pairs you derive from input/osm_basemap.gpkg (write them to out/gcps.csv), polynomial order 1, and output out/city_1950_3857.tif in EPSG:3857.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- GDAL
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
code· Code & commands- Work product
WP04· 3D/spatial model- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Same checks as T01: RMSE, check-point accuracy, format. Used to measure GUI vs CLI gap.
Proposed partial credit
20% GCP count/spread; 30% RMSE; 40% check points; 10% format
R07.06.01.02-T11 Vectorize water mains from scanned utility maps into a connected network
Original instruction
In ArcGIS Pro digitize the water mains on the four georeferenced sheets input/wm_sheet_*.tif into fixtures/utility.gdb/WaterMain with attributes DIAMETER and MATERIAL from the sheet labels, and valves as points snapped to mains. The network must be connected (end-to-end snapping, no gaps >0.05 m).
- Platforms
win· Windows- Applications
- ArcGIS Pro
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathgeom· Precise geometryvis· Visual judgment- Work product
WP04· 3D/spatial model- Conditions
licensed· Licensed pro applong· Long-horizon- Proposed difficulty
- 4 / 5
Proposed verifier
Length-weighted match to reference mains (buffer 2 m) F1 ≥0.85; attribute accuracy ≥90%; 1 connected component; valves on lines ≤0.01 m; count ±5%. ArcGIS Pro Python window, Notebooks and standalone ArcPy are disabled for the agent; the harness input log must show the features were sketched interactively.
Proposed partial credit
40% geometry F1; 25% attributes; 20% connectivity; 15% valves
R07.06.01.02-T12 Map 60 trees from a drone orthomosaic with measured crown diameters
Original instruction
In QGIS digitize the 60 trees in input/park_ortho.tif inside AOI as points at crown centres in out/trees.gpkg, with field crown_m measured using the measure tool (crown diameter, nearest 0.5 m) and species_group (conifer/broadleaf) by appearance.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- QGIS
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgmentgeom· Precise geometry- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Hungarian matching to reference within 2 m: recall/precision ≥0.9; crown_m MAE ≤1.0 m; species accuracy ≥85%. QGIS Python console, Processing scripts and GDAL/OGR CLIs are disabled for the agent; the harness input log must show the features were digitized interactively.
Proposed partial credit
45% detection F1; 30% crown MAE; 25% species
R07.06.01.02-T13 Align a local-coordinate CAD site plan to a projected CRS with control points
Original instruction
input/site_plan_local.dxf is in a local grid. In QGIS import it, then using the four survey monuments (local and EPSG:2056 coordinates in input/monuments.csv) apply an affine/Helmert transformation to bring it into EPSG:2056, saving out/site_plan_2056.gpkg. Report residuals in out/residuals.csv.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- QGIS
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clickvis· Visual judgmentmulti· Cross-app- Work product
WP04· 3D/spatial model- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Transformed monuments within 0.05 m; 10 hidden building corners within 0.2 m; CRS 2056; residuals file matches grader fit ±0.01 m. QGIS Python console, Processing scripts and GDAL/OGR CLIs are disabled for the agent; writes to out/ by any process other than QGIS are flagged by a file audit log.
Proposed partial credit
30% transformation type & fit; 40% hidden corners; 20% residuals; 10% output
R07.06.01.02-T14 Georeference a metadata-less drone image to an ortho basemap in ArcGIS Pro
Original instruction
input/drone_frame_0417.jpg has no GPS tags. In ArcGIS Pro georeference it to fixtures/ortho2024.tif (EPSG:32618) with ≥10 control points on hard features, projective or 2nd-order polynomial, and save as out/drone_0417_geo.tif with RMS error reported in out/links.txt.
- Platforms
win· Windows- Applications
- ArcGIS Pro
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicknav· Pan/zoom/scrollvis· Visual judgment- Work product
WP04· 3D/spatial model- Conditions
licensed· Licensed pro app- Proposed difficulty
- 3 / 5
Proposed verifier
≥10 links; forward RMS < 0.5 m; 12 hidden check features within 0.8 m; CRS correct. ArcGIS Pro Python window, Notebooks and standalone ArcPy are disabled for the agent; writes to out/ by any process other than ArcGIS Pro are flagged by a file audit log.
Proposed partial credit
25% links; 25% RMS; 40% hidden checks; 10% output
R07.06.01.02-T15 Make a digitized street network routable: node splits, one-ways, no gaps
Original instruction
fixtures/streets_raw.gpkg (QGIS) was digitized without splits. Split lines at all true intersections (not at the 4 bridges/underpasses in docs/grade_sep.csv), close gaps ≤0.5 m, set oneway (FT/TF/no) from arrows on input/signage_map.tif, and save so QGIS Network Analysis shortest path A→B equals the reference route.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- QGIS
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
HY· Hybrid- Interaction primitives
click· Point & clickgeom· Precise geometryvis· Visual judgment- Work product
WP04· 3D/spatial model- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Node degree analysis: splits at intersections, none at grade separations; 0 gaps ≤0.5 m; oneway accuracy ≥95%; grader's 5 routing queries match reference routes. QGIS Python console, Processing scripts and GDAL/OGR CLIs are disabled for the agent; the harness input log must show the features were digitized interactively.
Proposed partial credit
30% splits; 15% gaps; 25% oneway; 30% routing queries
R07.06.01.02-T16 Digitize orthogonal building outlines with advanced digitizing constraints
Original instruction
Using QGIS Advanced Digitizing (perpendicular/parallel constraints), digitize the 15 buildings in AOI from input/ortho_village.tif into out/bldgs.gpkg so that every corner is 90°±1°, and add field levels from input/levels.csv by building number visible on input/cadastre_labels.pdf.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- QGIS
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
geom· Precise geometrypath· Continuous pathvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
All interior angles 90±1°; mean IoU ≥0.8; 15 features; levels attribute accuracy 100%. QGIS Python console, Processing scripts and GDAL/OGR CLIs are disabled for the agent; the harness input log must show the features were digitized interactively.
Proposed partial credit
30% orthogonality; 40% IoU; 15% count; 15% attributes
R07.06.01.02-T17 Digitize field boundaries from a satellite composite and attribute crop type
Original instruction
In ArcGIS Pro digitize the 30 agricultural fields in AOI from input/s2_composite.tif (10 m) into fixtures/farm.gdb/Fields as polygons sharing edges along hedgerows, with CROP from field notes input/survey_notes.xlsx (field IDs visible as labels on input/field_ids.pdf).
- Platforms
win· Windows- Applications
- ArcGIS Pro · Excel
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathvis· Visual judgmentmulti· Cross-app- Work product
WP10· Annotations- Conditions
licensed· Licensed pro appmultiapp· Multi-app- Proposed difficulty
- 3 / 5
Proposed verifier
Mean IoU ≥0.85; shared edges coincide (no overlaps >1 m²); CROP accuracy ≥95%; 30 features. ArcGIS Pro Python window, Notebooks and standalone ArcPy are disabled for the agent; the harness input log must show the features were sketched interactively.
Proposed partial credit
50% IoU; 20% topology; 30% crop attributes
R07.06.01.02-T18 Georeference and mosaic four adjacent map sheets clipped to their neatlines
Original instruction
input/sheet_NW.tif, NE, SW, SE are 1:25,000 scans with printed grid ticks (EPSG:31467). Georeference each in QGIS using ≥9 grid ticks, clip each to its neatline polygon (digitize neatlines), and build a seamless mosaic out/mosaic.tif plus out/neatlines.gpkg. Features crossing sheet edges must align within 5 m.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- QGIS
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicknav· Pan/zoom/scrollvis· Visual judgment- Work product
WP04· 3D/spatial model- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Per-sheet residual <1.5 px; neatline polygons within 10 m of reference; mosaic has no collar overlap/nodata seams; 20 edge-crossing features offset ≤5 m; CRS correct. QGIS Python console, Processing scripts and GDAL/OGR CLIs are disabled for the agent; writes to out/ by any process other than QGIS are flagged by a file audit log.
Proposed partial credit
30% four georeferences; 20% neatlines; 20% mosaic seamless; 30% edge matching
R07.06.01.02-T19 Rectify a lake map in GRASS and digitize the shoreline
Original instruction
In GRASS GIS georectify input/lake_map.png with i.group/i.rectify via the GCP Manager (≥6 GCPs, 1st order), then digitize the lake shoreline as a closed boundary/area and report area (ha) in out/lake.txt.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- GRASS GIS
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickpath· Continuous pathvis· Visual judgment- Work product
WP04· 3D/spatial model- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Rectified raster check points within 20 m; shoreline area within 3% of reference; IoU ≥0.9. GRASS GUI command console, Python shell and terminal are disabled (modules run from menu dialogs only); mapset history must show GCP Manager and vector digitizer use.
Proposed partial credit
40% rectification; 40% shoreline IoU; 20% area answer
R07.06.01.02-T20 Digitize an 1890 street network with names from a georeferenced historic map
Original instruction
From georeferenced input/city_1890.tif, digitize every street centreline in the old town AOI into out/streets_1890.gpkg (split at intersections, topology clean), with name transcribed from the map labels (historic spelling) and width_class (major/minor) from line weight.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- QGIS
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathvis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP04· 3D/spatial model- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Buffer-based (4 m) length F1 vs reference ≥0.85; 0 dangles except true dead ends; name accuracy (normalized Levenshtein ≥0.9) on ≥85% segments; width class accuracy ≥90%. QGIS Python console, Processing scripts and GDAL/OGR CLIs are disabled for the agent; the harness input log must show the features were digitized interactively.
Proposed partial credit
40% geometry F1; 15% topology; 30% names; 15% width class
Mapped research references
No benchmark reference is assigned to this class in the current mapping.