Mobile app build & simulator testing
Build, run and fix an iOS/Android app in Xcode/Android Studio and verify behavior in the simulator.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- Xcode · Android Studio · iOS Simulator
- Class GUI-need label
HY· Hybrid
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R02.05.01.01-T01 Fix Swift 6 strict-concurrency build errors in an iOS app
Original instruction
Xcode 16 project fixtures/Pantry.xcodeproj fails to build after switching the language mode to Swift 6 (14 concurrency errors). Fix the code without lowering the language mode or adding @preconcurrency/nonisolated(unsafe) escapes, then run the app on the iPhone 16 Simulator and confirm the pantry list loads.
- Platforms
mac· macOSios· iOS/iPadOS- Applications
- Xcode · iOS Simulator
- Interfaces
desk· Desktop GUImob· Mobile touch- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandsclick· Point & clickvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader runs xcodebuild for iOS Simulator with SWIFT_VERSION=6: zero errors and zero concurrency warnings; grep forbids @preconcurrency and nonisolated(unsafe) additions. Hidden XCUITest launches the app and asserts 12 pantry rows render.
Proposed partial credit
40% clean Swift 6 build; 20% no escape hatches; 40% hidden UI test passes
R02.05.01.01-T02 Keep a login button above the soft keyboard on Android
Original instruction
In Android Studio, project fixtures/Shopper (Compose, minSdk 26) hides the 'Sign in' button behind the keyboard on the login screen. Fix inset handling so the button stays visible with the IME open on a Pixel 7 API 34 emulator in portrait and landscape, without making the screen scroll-only.
- Platforms
mac· macOSwin· Windowsand· Android- Applications
- Android Studio · Android Emulator
- Interfaces
desk· Desktop GUImob· Mobile touch- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
code· Code & commandstouch· Touch gesturesvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Hidden Compose UI test focuses the email field, waits for IME visible, and asserts the button's bounds lie above the IME inset top in both orientations. Gradle build must pass lint with no new warnings. Screenshot diff on the closed-keyboard state SSIM ≥0.98 vs baseline.
Proposed partial credit
50% button visible with IME (portrait 25, landscape 25); 30% closed-keyboard layout unchanged; 20% build and lint clean
R02.05.01.01-T03 Add App Group and push capabilities to two Xcode targets
Original instruction
In fixtures/Weather.xcodeproj, use Signing & Capabilities to add Push Notifications and App Groups (group.com.example.weather) to both the app and the WeatherWidget extension, with automatic signing under the provided test team. Build both targets for the Simulator.
- Platforms
mac· macOS- Applications
- Xcode
- Interfaces
desk· Desktop GUI- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Grader parses project.pbxproj and both .entitlements files: aps-environment present for app, application-groups contains the exact id for both targets, SystemCapabilities entries consistent. xcodebuild for both schemes succeeds.
Proposed partial credit
40% app entitlements; 40% widget entitlements; 20% builds succeed
R02.05.01.01-T04 Ship a dark theme with screenshot goldens on Android
Original instruction
Add a dark color scheme to fixtures/NotesApp (Material 3 Compose) following input/dark_palette.json, toggled by system setting. Verify on the emulator with dark mode on, then record Paparazzi screenshot goldens for the 3 main screens in both themes.
- Platforms
mac· macOSwin· Windowsand· Android- Applications
- Android Studio · Android Emulator
- Interfaces
desk· Desktop GUImob· Mobile touch- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandsvis· Visual judgmentclick· Point & click- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Hidden Paparazzi run compares agent goldens with reference renders: per-screen SSIM ≥0.95 and key surface colors ΔE2000 ≤2 vs palette. Instrumented test toggles uiMode night via adb and asserts background color of the main screen.
Proposed partial credit
40% palette applied correctly; 30% emulator night-mode switch works; 30% goldens match reference
R02.05.01.01-T05 Implement and test a custom-scheme deep link on iOS
Original instruction
In fixtures/Orders iOS app, make myshop://order/<id> open OrderDetailView for that id from cold and warm start; invalid ids show an 'Order not found' alert. Test it in the Simulator with xcrun simctl openurl and add an XCUITest covering both cases.
- Platforms
mac· macOSios· iOS/iPadOS- Applications
- Xcode · iOS Simulator
- Interfaces
desk· Desktop GUImob· Mobile touchterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandsclick· Point & click- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Grader builds and runs hidden XCUITests: simctl openurl myshop://order/1042 on cold and warm launch shows the detail title 'Order #1042'; myshop://order/zzz shows the alert. Agent's own UI test target must exist and pass.
Proposed partial credit
40% cold+warm deep link; 25% invalid id alert; 20% Info.plist scheme; 15% agent UI test passes
R02.05.01.01-T06 Fix state loss on rotation in an Android form
Original instruction
In fixtures/Survey (Android Views + ViewModel), answers on page 2 of the survey disappear when the emulator rotates and the step indicator resets. Reproduce on a Pixel 7 API 34 emulator, fix it so state survives rotation and process death, and keep the existing tests green.
- Platforms
mac· macOSwin· Windowsand· Android- Applications
- Android Studio · Android Emulator
- Interfaces
desk· Desktop GUImob· Mobile touch- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandstouch· Touch gestures- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Hidden Espresso test fills page 2, rotates, asserts fields and step '2/4'; second test uses ActivityScenario.recreate plus adb 'am kill' with saved-state check. All existing unit tests pass.
Proposed partial credit
40% survives rotation; 40% survives process death; 20% existing tests green
R02.05.01.01-T07 Find and fix a retain cycle with Instruments
Original instruction
fixtures/Gallery iOS app grows memory each time the photo detail screen is opened and closed. Use Instruments (Leaks/Allocations) on the Simulator to find the retain cycle, fix it, and save the trace before and after as output/before.trace and output/after.trace.
- Platforms
mac· macOSios· iOS/iPadOS- Applications
- Xcode · Instruments · iOS Simulator
- Interfaces
desk· Desktop GUImob· Mobile touch- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clickvis· Visual judgmentcode· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Grader runs a scripted XCUITest opening/closing detail 20 times, then 'leaks' on the process: 0 leaked PhotoDetailViewModel instances and persistent memory growth <2 MB. Both .trace files must exist and contain a Leaks track (xctrace export).
Proposed partial credit
50% leak gone; 20% memory growth under threshold; 30% before/after traces recorded
R02.05.01.01-T08 Cut scroll jank in a Compose list below 16 ms frames
Original instruction
The product feed in fixtures/Feed (Compose) janks while flinging. Profile it in Android Studio on the API 34 emulator, fix the causes (unstable lambdas, image decode on main thread, missing keys) and add a Macrobenchmark that flings the feed 5 times.
- Platforms
mac· macOSwin· Windowsand· Android- Applications
- Android Studio · Android Emulator
- Interfaces
desk· Desktop GUImob· Mobile touch- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandsvis· Visual judgmenttouch· Touch gestures- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Grader runs the agent's and a hidden Macrobenchmark on a fixed emulator profile: frameDurationCpuMs P90 <16 ms and frameOverrunMs P95 <0. Visual check: feed screenshot SSIM ≥0.97 vs baseline (no UI regression).
Proposed partial credit
20% benchmark exists; 50% P90 frame time target; 20% P95 overrun; 10% no visual regression
R02.05.01.01-T09 Test a subscription purchase with StoreKit configuration
Original instruction
In fixtures/ProNotes iOS app, create a local StoreKit configuration file with an auto-renewable 'pro_monthly' ($4.99, 1-week free trial) and make the paywall unlock Pro after purchase and on restore. Exercise it in the Simulator, then expire the subscription with Xcode's Transaction Manager and confirm Pro locks again.
- Platforms
mac· macOSios· iOS/iPadOS- Applications
- Xcode · iOS Simulator · StoreKit Test
- Interfaces
desk· Desktop GUImob· Mobile touch- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clickcode· Code & commandstouch· Touch gestures- Work product
WP06· Code/engineering- Conditions
tutorial· Follow docs/tutorial- Proposed difficulty
- 4 / 5
Proposed verifier
Hidden SKTestSession-based UI tests: purchase unlocks Pro, restore on reinstall unlocks, expireSubscription locks. .storekit file parsed for product id, price, trial. No real App Store account is used.
Proposed partial credit
25% StoreKit config; 30% purchase unlocks; 20% restore; 25% expiry relocks
R02.05.01.01-T10 Localize an iOS app into German and Japanese without truncation
Original instruction
Add German and Japanese to fixtures/Habit using a String Catalog; translations are in input/strings_de.csv and input/strings_ja.csv. Check the onboarding and settings screens on the iPhone SE (3rd gen) Simulator in both languages and fix any truncated labels.
- Platforms
mac· macOSios· iOS/iPadOS- Applications
- Xcode · iOS Simulator
- Interfaces
desk· Desktop GUImob· Mobile touch- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
HY· Hybrid- Interaction primitives
click· Point & clickvis· Visual judgmentcode· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader parses Localizable.xcstrings: all 86 keys translated in de and ja and equal to the CSVs. Hidden XCUITest in -AppleLanguages de/ja on iPhone SE checks no label's intrinsic width exceeds its frame and no '…' in label text.
Proposed partial credit
40% catalog complete and exact; 40% no truncation (de 20, ja 20); 20% build clean
R02.05.01.01-T11 Fix RenderFlex overflows in a Flutter app on small screens
Original instruction
fixtures/trip_planner (Flutter 3.24) shows yellow overflow stripes on the booking screen on the iPhone SE Simulator and a 320 dp-wide Android emulator. Fix the layout so no overflow occurs at text scale 1.0 and 1.3 on both, without hiding content.
- Platforms
mac· macOSios· iOS/iPadOSand· Android- Applications
- Flutter · iOS Simulator · Android Emulator
- Interfaces
desk· Desktop GUImob· Mobile touch- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandsvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Hidden widget tests pump the booking screen at 320x568 and 375x667 logical px with textScaler 1.0/1.3: zero FlutterError overflow reports; all 7 form fields and the CTA findable and hit-testable. Golden test at 390 px SSIM ≥0.95 vs reference.
Proposed partial credit
50% no overflow across 4 configs; 30% all content reachable; 20% golden
R02.05.01.01-T12 Upgrade an Expo app by two SDK versions and run on both platforms
Original instruction
Upgrade fixtures/RecipeBox from Expo SDK 49 to 51, fix breaking changes (camera and file-system APIs), then prebuild and run on the iOS Simulator and an Android emulator. The Detox smoke suite in e2e/ must pass on both.
- Platforms
mac· macOSios· iOS/iPadOSand· Android- Applications
- Expo · React Native · iOS Simulator · Android Emulator · Detox
- Interfaces
desk· Desktop GUImob· Mobile touchterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandstouch· Touch gestures- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
package.json expo ~51; expo-doctor reports 0 issues. Grader runs Detox e2e on iOS and Android builds: all 6 specs pass on each. Camera screen opens without redbox in both.
Proposed partial credit
20% dependencies upgraded; 20% expo-doctor clean; 30% iOS e2e; 30% Android e2e
R02.05.01.01-T13 Build free and paid flavors installed side by side
Original instruction
In fixtures/Metronome (Android), add product flavors free and paid: applicationId suffixes .free/.paid, app names 'Metronome Free'/'Metronome Pro', separate launcher icons from input/icons/, and BuildConfig.SHOW_ADS true only in free. Install both debug builds on the emulator at the same time.
- Platforms
mac· macOSwin· Windowsand· Android- Applications
- Android Studio · Android Emulator
- Interfaces
desk· Desktop GUImob· Mobile touch- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clickcode· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Grader builds both flavors and inspects APKs (aapt2 dump badging): ids, labels, icon hashes match inputs; BuildConfig field values correct. On the emulator, pm list packages shows both installed simultaneously.
Proposed partial credit
40% flavors and ids; 30% labels and icons; 15% BuildConfig flag; 15% both installed
R02.05.01.01-T14 Resolve Auto Layout conflicts across three device sizes
Original instruction
The storyboard of fixtures/Checkout logs 'Unable to simultaneously satisfy constraints' on the payment screen and misplaces the total label on iPad. In Interface Builder, fix the constraints so the layout matches input/payment_spec.pdf on iPhone SE, iPhone 16 Pro Max and iPad Air in portrait.
- Platforms
mac· macOSios· iOS/iPadOS- Applications
- Xcode · Interface Builder · iOS Simulator
- Interfaces
desk· Desktop GUImob· Mobile touch- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
HY· Hybrid- Interaction primitives
geom· Precise geometryclick· Point & clickvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Hidden XCUITest on the 3 simulators: no constraint-conflict log lines; element frames match spec margins ±2 pt; ambiguity check (hasAmbiguousLayout) false for all views. Storyboard must still compile; programmatic constraint code in the view controller is rejected by diff. Writes to the .storyboard by any process other than Xcode are detected via Endpoint Security file events and fail.
Proposed partial credit
30% no conflicts; 45% frames per device (15 each); 25% no ambiguous layout
R02.05.01.01-T15 Fix a geofence alert using a simulated GPX route
Original instruction
fixtures/Commute should notify when the user enters a 150 m region around Work, but never fires. Simulate input/route.gpx in the iOS Simulator (Features > Location), find the bug in region monitoring, fix it, and confirm the local notification appears once on arrival.
- Platforms
mac· macOSios· iOS/iPadOS- Applications
- Xcode · iOS Simulator
- Interfaces
desk· Desktop GUImob· Mobile touch- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
HY· Hybrid- Interaction primitives
click· Point & clickcode· Code & commandsvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader replays the GPX via simctl location, then reads the delivered notifications through a hidden test hook: exactly one 'Arrived at Work' notification within 60 s of crossing the region boundary; region radius 150 m in CLCircularRegion. Unit tests stay green.
Proposed partial credit
30% bug identified (radius/authorization); 50% single notification on arrival; 20% tests green
R02.05.01.01-T16 Request notification permission correctly on Android 14
Original instruction
fixtures/Reminders never shows notifications on Android 13+ because POST_NOTIFICATIONS is never requested. Add the runtime request with a rationale dialog on second ask, and verify on an API 34 emulator that a reminder posts after granting.
- Platforms
mac· macOSwin· Windowsand· Android- Applications
- Android Studio · Android Emulator
- Interfaces
desk· Desktop GUImob· Mobile touch- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandstouch· Touch gestures- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Hidden UiAutomator test: fresh install shows the system permission dialog on first reminder; after Allow, dumpsys notification lists the reminder in channel 'reminders'; after Deny then retry, the rationale dialog appears. Manifest declares the permission.
Proposed partial credit
25% manifest; 35% grant path posts notification; 25% rationale on re-ask; 15% channel id
R02.05.01.01-T17 Add a WidgetKit widget showing the next event
Original instruction
Add a WidgetKit extension to fixtures/Agenda that shows the next calendar event (title, start time) in small and medium sizes, sharing data through the existing App Group. Add it to the Simulator home screen and confirm it updates when a new event is created in the app.
- Platforms
mac· macOSios· iOS/iPadOS- Applications
- Xcode · iOS Simulator
- Interfaces
desk· Desktop GUImob· Mobile touch- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
HY· Hybrid- Interaction primitives
code· Code & commandsclick· Point & clicktouch· Touch gesturesvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Grader builds the extension, renders widget snapshots via a hidden preview test for both families, and OCRs title/time vs expected. After inserting an event through the app UI test, WidgetCenter timeline reload yields the new event in the snapshot within 1 refresh.
Proposed partial credit
20% extension builds; 30% small+medium render correct data; 30% updates after new event; 20% app group shared store
R02.05.01.01-T18 Migrate to Kotlin DSL and AGP 8 with a minified release
Original instruction
Migrate fixtures/Ledger from Groovy Gradle and AGP 7.4 to Kotlin DSL, a version catalog and AGP 8.5, enable R8 minification for release, and fix the resulting crashes (Gson models, reflection in the plugin loader). The release build must run on the emulator through the full instrumentation suite.
- Platforms
mac· macOSwin· Windowsand· Android- Applications
- Android Studio · Gradle · Android Emulator
- Interfaces
desk· Desktop GUImob· Mobile touchterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandsclick· Point & click- Work product
WP06· Code/engineering- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
No *.gradle Groovy files remain; libs.versions.toml exists; AGP 8.5 resolved. Grader builds a release-signed (test key) minified APK and runs the 22 instrumentation tests against it: all pass, no ClassNotFound in logcat. APK size ≤ 70% of the unminified build.
Proposed partial credit
20% Kotlin DSL + catalog; 15% AGP 8.5 builds; 45% minified instrumentation pass rate; 20% size target
R02.05.01.01-T19 Reproduce a coupon total bug from a screen recording and fix it
Original instruction
input/bug.mp4 shows the checkout total in fixtures/Cart (SwiftUI) going wrong after applying coupon SAVE10 and then changing quantity. Reproduce the exact sequence in the Simulator, fix the bug, and add an XCUITest that replays it.
- Platforms
mac· macOSios· iOS/iPadOS- Applications
- Xcode · iOS Simulator
- Interfaces
desk· Desktop GUImob· Mobile touch- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
HY· Hybrid- Interaction primitives
av· Audio/video over timetouch· Touch gesturescode· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Hidden XCUITest replays the sequence from the video and asserts the total equals the reference for 6 quantity/coupon combinations. Agent's test must fail on the original commit and pass on the fix (grader runs both).
Proposed partial credit
20% repro steps recorded in output/repro.md; 50% hidden combos correct; 30% agent test fails-before/passes-after
R02.05.01.01-T20 Add an offline banner to a Kotlin Multiplatform app on iOS and Android
Original instruction
fixtures/NewsKMP (shared Kotlin, SwiftUI and Compose UIs) must show a red 'Offline — showing saved stories' banner within 2 s of losing connectivity and hide it on reconnect, keeping cached stories readable. Implement it on both platforms, test by toggling network on the iOS Simulator and Android emulator, and add UI tests for each.
- Platforms
mac· macOSios· iOS/iPadOSand· Android- Applications
- Android Studio · Xcode · iOS Simulator · Android Emulator
- Interfaces
desk· Desktop GUImob· Mobile touchterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandstouch· Touch gesturesvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Grader toggles network (emulator console 'network status off', simulator via Network Link Conditioner profile) and runs hidden UI tests: banner text exact, visible ≤2 s, hidden ≤2 s after reconnect, 10 cached stories still open. Shared connectivity logic must live in commonMain.
Proposed partial credit
15% shared logic in commonMain; 35% Android behavior; 35% iOS behavior; 15% agent UI tests pass
Mapped research references
No benchmark reference is assigned to this class in the current mapping.