R08.09.01.02 · TASK CLASS

Survey instruments

Author surveys with logic, piping and quotas.

20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.

Typical applications: Qualtrics · LimeSurvey · REDCap surveys
Class GUI-need label: G2 · GUI-essential

Task proposals · 20

Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.

R08.09.01.02-T01 Author a ten-item Likert survey with display logic and block randomization

Original instruction

In the Qualtrics sandbox create a survey 'Wellbeing Pilot' with ten 5-point Likert items from input/items.csv in two blocks, show item 3 only if item 2 is answered 'Agree' or 'Strongly agree', randomize block order, force responses on all items, and export output/wellbeing.qsf.

Platforms: web · Any OS (web)
Applications: Qualtrics
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click type · Text & forms drag · Drag & drop
Work product: WP06 · Code/engineering
Conditions: None declared
Proposed difficulty: 2 / 5

Proposed verifier

QSF parsed: ten matrix/Likert items with the given labels, display logic on item 3 referencing item 2's two choices, a block randomizer over both blocks, force-response on every item. QSF import is disabled and no Qualtrics API token is issued to the sandbox account.

Proposed partial credit

30% items and scale; 30% display logic; 25% randomization; 15% force response.

R08.09.01.02-T02 Build a REDCap PHQ-9 instrument with auto-scoring and branching

Original instruction

In the REDCap sandbox project 'FC_Screening' create a PHQ-9 instrument from input/phq9.csv with 0–3 coded radio fields, a calculated total field, and a safety follow-up field shown only when item 9 > 0 or total ≥ 20. Enable it as a survey and export the data dictionary to output/dictionary.csv.

Platforms: web · Any OS (web)
Applications: REDCap
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click type · Text & forms
Work product: WP06 · Code/engineering
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

Dictionary: nine radio fields with 0–3 coding, a calc field whose expression sums exactly those nine, branching logic on the follow-up matching both conditions; a test record scores correctly and shows the follow-up only when the rule holds. The Data Import Tool, data-dictionary and project-XML uploads are disabled and no REDCap API token is issued to the sandbox user.

Proposed partial credit

25% items coded; 30% calculated total; 30% branching logic; 15% survey enabled.

R08.09.01.02-T03 Set gender quotas and a screen-out path in a Qualtrics survey

Original instruction

In the Qualtrics sandbox open 'Panel Study'. Add quotas of 50 completes per gender (male, female) based on Q2, set the survey to screen out respondents whose quota is full with a custom end-of-survey message, and ensure quota counts only completed responses. Export output/panel.qsf.

Platforms: web · Any OS (web)
Applications: Qualtrics
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click type · Text & forms
Work product: WP06 · Code/engineering
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

QSF holds two quotas of 50 with logic on Q2 and 'count only completed responses'; a quota-full branch routes to the custom end-of-survey element; simulated submissions beyond 50 in one group are screened out. QSF import is disabled and no Qualtrics API token is issued to the sandbox account.

Proposed partial credit

35% quotas defined; 25% completion-only counting; 30% screen-out routing; 10% message.

R08.09.01.02-T04 Build a LimeSurvey instrument with conditional subquestions and an export

Original instruction

In the LimeSurvey sandbox create survey 'Transport Diary': a multiple-choice mode-of-transport question, then per selected mode a follow-up duration question shown by condition, plus a mandatory consent question gating the rest. Activate it, submit one test response, and export the structure to output/transport.lss and the response to output/response.csv.

Platforms: web · Any OS (web)
Applications: LimeSurvey
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click type · Text & forms
Work product: WP06 · Code/engineering
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

LSS: consent question mandatory with relevance equations gating later groups; one duration question per mode with relevance referencing that mode's answer; the test response shows only the selected modes' follow-ups. LSS/LSQ import is disabled and LimeSurvey's RemoteControl API is off for the sandbox account.

Proposed partial credit

25% question structure; 35% relevance conditions; 25% consent gate; 15% activation and export.

R08.09.01.02-T05 Pipe earlier answers into later question text in Qualtrics

Original instruction

In the Qualtrics sandbox survey 'Pipe Test', pipe the respondent's name from Q1 and their selected city from Q2 into the text of Q5, and pipe the Q3 slider value into the Q6 scale anchors. Use embedded data for the recruitment source from the URL parameter 'src' and show it on the last page. Export output/pipe.qsf.

Platforms: web · Any OS (web)
Applications: Qualtrics
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click type · Text & forms
Work product: WP06 · Code/engineering
Conditions: None declared
Proposed difficulty: 2 / 5

Proposed verifier

QSF contains piped-text references to the correct question IDs in Q5 and Q6 and an embedded-data field 'src'; a simulated response with ?src=mail renders name, city, slider value and source correctly in the preview HTML. QSF import is disabled and no Qualtrics API token is issued to the sandbox account.

Proposed partial credit

35% question piping; 30% anchor piping; 25% embedded data; 10% display on last page.

R08.09.01.02-T06 Build a REDCap longitudinal project with three events and repeating instruments

Original instruction

In the REDCap sandbox create project 'FC_Longitudinal' with arm 1 events baseline, month 3 and month 6, assign the demographics instrument to baseline only and the symptom diary to all events as a repeating instrument, define the event offsets (0, 90, 180 days), and export output/project.xml (project XML with metadata).

Platforms: web · Any OS (web)
Applications: REDCap
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click type · Text & forms drag · Drag & drop
Work product: WP06 · Code/engineering
Conditions: long · Long-horizon
Proposed difficulty: 4 / 5

Proposed verifier

Project XML: three events with the stated offsets, instrument-event mappings exactly as specified, symptom diary flagged repeating; a test record can store two diary instances at month 3 and none at a non-mapped event. The Data Import Tool, data-dictionary and project-XML uploads are disabled and no REDCap API token is issued to the sandbox user.

Proposed partial credit

30% events and offsets; 30% instrument mapping; 25% repeating configuration; 15% test record behaviour.

R08.09.01.02-T07 Add randomization to two survey arms with a balanced allocation in Qualtrics

Original instruction

In the Qualtrics sandbox survey 'Vignette Study', add a randomizer that presents one of three vignette blocks with evenly presented elements, store which vignette was shown in embedded data, and add attention-check logic that flags respondents failing the check. Export output/vignette.qsf.

Platforms: web · Any OS (web)
Applications: Qualtrics
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click drag · Drag & drop type · Text & forms
Work product: WP06 · Code/engineering
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

QSF: randomizer over the three blocks with 'evenly present elements' on and 1 element shown; embedded-data assignment inside each block; attention-check branch setting a flag field. A 300-response simulation gives allocation within ±5% of equal. QSF import is disabled and no Qualtrics API token is issued to the sandbox account.

Proposed partial credit

35% randomizer settings; 25% allocation recorded; 30% attention-check flag; 10% balance in simulation.

R08.09.01.02-T08 Configure REDCap data-quality rules and field validation

Original instruction

In the REDCap sandbox project 'FC_Screening' set validation on date fields (date_ymd with a range 2020-01-01 to today), numeric ranges on age (18–99) and BMI (10–70), required fields for the core variables listed in input/required.txt, and three custom data-quality rules from input/dq_rules.txt. Export output/dictionary.csv and output/dq_rules_export.csv.

Platforms: web · Any OS (web)
Applications: REDCap
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click type · Text & forms
Work product: WP08 · App/system state
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

Dictionary shows the stated validation types, ranges and required flags; the custom rules exist with equivalent logic; a seeded test record with out-of-range values is caught by each rule when data quality is executed. The Data Import Tool, data-dictionary and project-XML uploads are disabled and no REDCap API token is issued to the sandbox user.

Proposed partial credit

30% field validation; 25% required fields; 35% custom rules fire; 10% exports.

R08.09.01.02-T09 Build a conjoint-style choice task with randomized attribute levels in Qualtrics

Original instruction

In the Qualtrics sandbox create 'Choice Task': 8 screens, each showing two product profiles built from four attributes (levels in input/attributes.csv) using loop-and-merge with randomized level assignment, forcing a choice per screen and recording the shown levels in embedded data. Export output/conjoint.qsf.

Platforms: web · Any OS (web)
Applications: Qualtrics
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click type · Text & forms drag · Drag & drop
Work product: WP06 · Code/engineering
Conditions: long · Long-horizon
Proposed difficulty: 5 / 5

Proposed verifier

QSF: loop-and-merge with 8 iterations and fields for all four attributes per profile; randomized assignment implemented (randomizer or JS with recorded levels); a 200-response simulation shows each level appearing within ±10% of expected and both profiles never identical. QSF import is disabled and no Qualtrics API token is issued to the sandbox account.

Proposed partial credit

25% loop structure; 35% randomized level assignment; 25% levels recorded; 15% no duplicate profiles.

R08.09.01.02-T10 Fix a Qualtrics survey whose display logic hides a required question

Original instruction

In the Qualtrics sandbox survey 'Broken Flow', respondents are stuck because Q7 is force-response but hidden by display logic when Q4 is skipped. Diagnose it in the preview, fix the logic so the survey can always be completed without losing the intended skip pattern, document the change in output/fix_notes.md and export output/fixed.qsf.

Platforms: web · Any OS (web)
Applications: Qualtrics
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click vis · Visual judgment type · Text & forms
Work product: WP06 · Code/engineering
Conditions: hidden · Hidden/stale state
Proposed difficulty: 3 / 5

Proposed verifier

Simulated respondents covering all Q4 answer paths (including skip) complete the survey; the intended skip pattern (Q7 asked only to the target subgroup) still holds in the response matrix; fix_notes.md names the force-response/display-logic conflict. QSF import is disabled and no Qualtrics API token is issued to the sandbox account.

Proposed partial credit

45% all paths completable; 30% skip pattern preserved; 25% cause documented.

R08.09.01.02-T11 Set up a REDCap survey queue with invitations and a reminder schedule

Original instruction

In the REDCap sandbox project 'FC_Screening', enable three surveys in a queue with conditions (second survey only when consent = yes), configure automated invitations to send on enrolment and remind after 3 and 7 days, using the test email domain only, and export the queue and invitation settings to output/queue_settings.json. Do not send to any external address.

Platforms: web · Any OS (web)
Applications: REDCap
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click type · Text & forms
Work product: WP08 · App/system state
Conditions: risk · Irreversible/risky
Proposed difficulty: 3 / 5

Proposed verifier

Settings show the three queue entries with the consent condition, automated invitations with the two reminders, and recipients restricted to the sandbox domain; the mail log shows no message to a non-test domain. The Data Import Tool, data-dictionary and project-XML uploads are disabled and no REDCap API token is issued to the sandbox user.

Proposed partial credit

30% queue conditions; 35% invitation schedule; 25% reminders; 10% no external sends.

R08.09.01.02-T12 Add accessible, mobile-friendly formatting to a long LimeSurvey instrument

Original instruction

In the LimeSurvey sandbox survey 'Long Form' (60 questions), split it into pages of at most 8 questions, add a progress bar, set keyboard-accessible radio lists instead of dropdowns for the 5-point scales, add question-level help text from input/help.csv, and verify on a 390 px viewport that no horizontal scrolling occurs. Export output/longform.lss and output/mobile.png.

Platforms: web · Any OS (web)
Applications: LimeSurvey
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click type · Text & forms nav · Pan/zoom/scroll
Work product: WP06 · Code/engineering
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

LSS shows ≤8 questions per group, progress bar on, list-radio question types for the scales, help text present for each listed question; the 390 px screenshot has no horizontal overflow (page width ≤ viewport) and tab order reaches every input. LSS/LSQ import is disabled and LimeSurvey's RemoteControl API is off for the sandbox account.

Proposed partial credit

25% pagination and progress bar; 25% accessible controls; 25% help text; 25% mobile check.

R08.09.01.02-T13 Create a Google Forms screening survey with section branching and a response sheet

Original instruction

Signed in to the harness's test Google Workspace account, in Google Forms create 'Study Screening' with eligibility questions from input/screening.csv, branch ineligible respondents straight to a thank-you section, require an email, limit to one response, and link responses to a new sheet. Submit two test responses (one eligible, one not) and export output/responses.csv plus the form link in output/link.txt.

Platforms: web · Any OS (web)
Applications: Google Forms
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click type · Text & forms
Work product: WP06 · Code/engineering
Conditions: None declared
Proposed difficulty: 2 / 5

Proposed verifier

Form structure: section branching per answer as specified, email required, one-response limit; responses.csv shows the two test submissions with the ineligible one lacking the later-section columns. Apps Script is disabled and the test account holds no OAuth credentials for the Forms API, so the form must be built in the browser.

Proposed partial credit

35% branching; 25% settings; 25% test responses correct; 15% sheet export.

R08.09.01.02-T14 Build a Qualtrics survey with a randomized-order matrix and reverse-scored items

Original instruction

In the Qualtrics sandbox create 'Scale Pilot' with a 12-item matrix from input/scale.csv, randomize statement order, reverse-score the five flagged items via recode values so the total is directionally consistent, add a scoring category summing all items, and export output/scale.qsf and output/scoring.json.

Platforms: web · Any OS (web)
Applications: Qualtrics
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click type · Text & forms
Work product: WP06 · Code/engineering
Conditions: None declared
Proposed difficulty: 4 / 5

Proposed verifier

QSF: statement randomization on, recode values reversed exactly for the five flagged items, scoring category including all 12 items; a simulated all-'Agree' response yields the gold total, which differs if reversal is wrong. QSF import is disabled and no Qualtrics API token is issued to the sandbox account.

Proposed partial credit

25% matrix and randomization; 40% reverse coding; 25% scoring category; 10% exports.

R08.09.01.02-T15 Migrate a paper questionnaire into REDCap with matching coding and skip patterns

Original instruction

input/paper_form.pdf is a 3-page clinical questionnaire with skip instructions ('if no, go to Q12'). Build it in the REDCap sandbox with identical variable coding to input/codebook.csv, implement every skip as branching logic, add a matrix for the symptom grid, and export output/dictionary.csv. Note in output/queries.md anything the paper form leaves ambiguous.

Platforms: web · Any OS (web)
Applications: REDCap
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click type · Text & forms vis · Visual judgment
Work product: WP06 · Code/engineering
Conditions: long · Long-horizon clarify · Ask when unclear
Proposed difficulty: 5 / 5

Proposed verifier

Dictionary field names, types and choice codings equal the codebook; every skip instruction present as branching logic (checked against a gold rule list); the matrix grid present; queries.md must list the two genuinely ambiguous items (seeded) and no more than four items total. The Data Import Tool, data-dictionary and project-XML uploads are disabled and no REDCap API token is issued to the sandbox user.

Proposed partial credit

30% fields and coding; 35% branching logic; 20% matrix; 15% ambiguities raised.

R08.09.01.02-T16 Control: build the Likert survey structure through the Qualtrics API

Original instruction

Control pair of T01. Using a separate API-enabled Qualtrics sandbox account from the terminal, create the same ten-item Likert survey with the display logic on item 3, block randomization and force response, then download the QSF to output/wellbeing.qsf.

Platforms: lin · Linux mac · macOS
Applications: Qualtrics API · curl
Interfaces: term · Terminal api · API/SDK
GUI need · task setting: AP · API-sufficient
GUI need · ordinary access: AP · API-sufficient
Interaction primitives: code · Code & commands
Work product: WP06 · Code/engineering
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

Same QSF checks as T01 (items, display logic, randomizer, force response). Scored as the API control of T01.

Proposed partial credit

30% items; 30% display logic; 25% randomization; 15% force response.

R08.09.01.02-T17 Set up a REDCap double-data-entry project and resolve mismatches

Original instruction

In the REDCap sandbox create project 'FC_DDE' with double data entry enabled for the instrument in input/instrument.csv, enter the two sets of records from input/entry1.csv and input/entry2.csv as entry persons 1 and 2, then use the data-comparison tool to merge and resolve the 6 discrepancies according to input/source_truth.csv. Export output/merged.csv.

Platforms: web · Any OS (web)
Applications: REDCap
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click type · Text & forms vis · Visual judgment
Work product: WP08 · App/system state
Conditions: long · Long-horizon
Proposed difficulty: 4 / 5

Proposed verifier

Merged records match source_truth.csv on all fields; the comparison log shows 6 discrepancies reviewed; DDE mode was enabled before entry (project settings history). The Data Import Tool, data-dictionary and project-XML uploads are disabled and no REDCap API token is issued to the sandbox user.

Proposed partial credit

25% DDE configured; 30% both entries recorded; 35% discrepancies resolved to truth; 10% export.

R08.09.01.02-T18 Add JavaScript-based response-time capture to a Qualtrics item

Original instruction

In the Qualtrics sandbox survey 'RT Pilot', add timing capture to Q3 so the time from page display to first click and the total page time are stored in embedded data, add a timer-based auto-advance after 20 s, and make sure the values appear in the response export. Submit two test responses and export output/rt.csv plus output/rt.qsf.

Platforms: web · Any OS (web)
Applications: Qualtrics
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click type · Text & forms code · Code & commands
Work product: WP06 · Code/engineering
Conditions: None declared
Proposed difficulty: 4 / 5

Proposed verifier

Export contains both timing fields with plausible values for the two test responses (first-click ≤ page time, both >0); the QSF has a timing question or JS plus embedded-data fields and an auto-advance at 20 s. QSF import is disabled and no Qualtrics API token is issued to the sandbox account.

Proposed partial credit

35% timing fields captured; 25% values plausible; 25% auto-advance; 15% export.

R08.09.01.02-T19 Clean and codebook a messy survey export into an analysis-ready table

Original instruction

input/raw_export.csv is a Qualtrics export with two header rows, incomplete responses, straightliners and preview responses. In the sandbox project settings plus a spreadsheet, produce output/analysis.csv with one header row, only finished non-preview responses, numeric coded values, reverse-scored items per input/codebook.csv and a straightliner flag, and write output/codebook_final.csv describing every column.

Platforms: web · Any OS (web)
Applications: Qualtrics · Microsoft Excel
Interfaces: brow · Browser desk · Desktop GUI
GUI need · task setting: HY · Hybrid
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click type · Text & forms multi · Cross-app
Work product: WP02 · Records/table
Conditions: multiapp · Multi-app
Proposed difficulty: 3 / 5

Proposed verifier

Row set equals gold (finished, non-preview); coded values and reversals match the codebook; straightliner flag matches the gold rule (zero variance across the 12 scale items); codebook_final.csv documents every output column.

Proposed partial credit

30% filtering; 30% coding and reversal; 25% straightliner flag; 15% codebook.

R08.09.01.02-T20 Deploy a multi-language survey with a language selector in LimeSurvey

Original instruction

In the LimeSurvey sandbox take survey 'Transport Diary' and add French and Spanish translations from input/translations.csv for all questions, answer options and the welcome text, enable a language switcher, set English as the base, and verify each language renders fully. Export output/multilang.lss and screenshots output/fr.png and output/es.png.

Platforms: web · Any OS (web)
Applications: LimeSurvey
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click type · Text & forms nav · Pan/zoom/scroll
Work product: WP06 · Code/engineering
Conditions: long · Long-horizon
Proposed difficulty: 4 / 5

Proposed verifier

LSS holds complete fr and es strings for every question, subquestion and answer option (no untranslated fallbacks); screenshots show translated text with no English leakage in the question area (text extraction); switcher enabled. LSS/LSQ import is disabled and LimeSurvey's RemoteControl API is off for the sandbox account.

Proposed partial credit

45% complete translations; 25% answer options and welcome text; 20% switcher and rendering; 10% exports.

Mapped research references

No benchmark reference is assigned to this class in the current mapping.

Survey instruments

Task proposals · 20

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Mapped research references

In this workflow family