Public task library / WorkArena / WorkArena++
BENCHMARK SOURCE · PINNED EVIDENCE

WorkArena / WorkArena++

33 atomic L1 templates plus 682 L2/L3 variants of 341 shared WorkArena++ workflows on ServiceNow. Every pair retains a shared workflow identifier.

0fixed definitions
715templates / level variants
715records with verifier references
0executed by this collection

Environment and counting scope

L2 provides explicit steps; L3 presents a ticket and company protocols and adds task closure. The 682 variants are 341 paired workflow classes, not 682 unrelated workflows. Seeds and generated configurations do not multiply collection counts. The preserved L1 pools contain 19,833 raw JSON rows, distinct from the README’s reported 19,912 unique instances. ServiceNow instances require separate provisioning; the upstream instance pool is gated. No environment or grader was executed.

What is checked?

L1 validators inspect browser state, ServiceNow records or assistant answers. Compositions validate designated subtasks in order; helper steps can be excluded. The shared validator remembers passed checkpoints instead of rechecking them later, and some workflows override it. Infeasible-task variants inspect an infeasibility response. These are source observations, not execution results or an accuracy assessment.

Source preservation does not establish evaluator accuracy or end-to-end reproducibility.

Source revision e885130962f61fb64ac305497996a46cfc7d18f0
Upstream license: Apache-2.0 · Platforms: Web

Upstream benchmark authors: Alexandre Drouin · Maxime Gasse · Massimo Caccia · Issam H. Laradji · Manuel Del Verme · Tom Marty · David Vazquez · Nicolas Chapados · Alexandre Lacoste · Léo Boisvert · Megh Thakkar · Thibault Le Sellier De Chezelles · Quentin Cappart · Paper attribution

One workflow, different information access

341 shared workflow classes × two levels = 682 registered variants. Expand a record below to compare its paired version; sampled runs and seeds are not additional templates.

Independent registry and curriculum audit · JSON

Source categories

These are upstream/importer labels; the records are not yet mapped to the taxonomy’s 430 task classes.

contextual_understanding_infeasible_tasks 104dashboard 4data_driven_decision_making_and_reasoning 312form 5information_retrieval 78knowledge 1list-filter 6list-sort 6menu 2planning_and_problem_solving 132service catalog 9sophisticated_memory 56

Inspect every collected definition

Expand a record for its original instruction and verifier links. Task text remains in English. Use the interactive browser above for full-text filtering.

workarena.servicenow.all-menuDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 1,000 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get task configuration
        self.module = (
            self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        )

        # When menu tasks do not need to be validated, the URL can be omitted from their config
        self.final_url = self.instance.snow_url + self.module.get("url", "")

        # Generate goal
        goal = f'Navigate to the "{self.module["module"]}" module of the "{self.module["application"]}" application.'
        info = {}

        return goal, info
workarena.servicenow.create-change-requestDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 1,000 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get the task configuration
        assert self.all_configs is not None, "No configuration available for the task."
        config = self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        # If fixed_config is not None we already set the required attributes in the constructor
        if self.fixed_config is None:
            self._set_required_config_attributes(config)
        self.protected_fields = self.task_fields
        # Generate the goal
        goal = (
            f"Create a new {self.table_label} with "
            + prettyprint_enum(
                [
                    f'a value of "{self.template_record[f]}"'
                    + f' for field "{config["fields"][f]}"'
                    for f in self.task_fields
                ]
            )
            + "."
        )
        info = {}

        return goal, info
workarena.servicenow.create-hardware-assetDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 1,000 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get the task configuration
        assert self.all_configs is not None, "No configuration available for the task."
        config = self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        # If fixed_config is not None we already set the required attributes in the constructor
        if self.fixed_config is None:
            self._set_required_config_attributes(config)
        self.protected_fields = self.task_fields
        # Generate the goal
        goal = (
            f"Create a new {self.table_label} with "
            + prettyprint_enum(
                [
                    f'a value of "{self.template_record[f]}"'
                    + f' for field "{config["fields"][f]}"'
                    for f in self.task_fields
                ]
            )
            + "."
        )
        info = {}

        return goal, info
workarena.servicenow.create-incidentDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 1,000 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get the task configuration
        assert self.all_configs is not None, "No configuration available for the task."
        config = self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        # If fixed_config is not None we already set the required attributes in the constructor
        if self.fixed_config is None:
            self._set_required_config_attributes(config)
        self.protected_fields = self.task_fields
        # Generate the goal
        goal = (
            f"Create a new {self.table_label} with "
            + prettyprint_enum(
                [
                    f'a value of "{self.template_record[f]}"'
                    + f' for field "{config["fields"][f]}"'
                    for f in self.task_fields
                ]
            )
            + "."
        )
        info = {}

        return goal, info
workarena.servicenow.create-problemDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 1,000 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get the task configuration
        assert self.all_configs is not None, "No configuration available for the task."
        config = self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        # If fixed_config is not None we already set the required attributes in the constructor
        if self.fixed_config is None:
            self._set_required_config_attributes(config)
        self.protected_fields = self.task_fields
        # Generate the goal
        goal = (
            f"Create a new {self.table_label} with "
            + prettyprint_enum(
                [
                    f'a value of "{self.template_record[f]}"'
                    + f' for field "{config["fields"][f]}"'
                    for f in self.task_fields
                ]
            )
            + "."
        )
        info = {}

        return goal, info
workarena.servicenow.create-userDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 1,000 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get the task configuration
        assert self.all_configs is not None, "No configuration available for the task."
        config = self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        # If fixed_config is not None we already set the required attributes in the constructor
        if self.fixed_config is None:
            self._set_required_config_attributes(config)
        self.protected_fields = self.task_fields
        # Generate the goal
        goal = (
            f"Create a new {self.table_label} with "
            + prettyprint_enum(
                [
                    f'a value of "{self.template_record[f]}"'
                    + f' for field "{config["fields"][f]}"'
                    for f in self.task_fields
                ]
            )
            + "."
        )
        info = {}

        return goal, info
workarena.servicenow.filter-asset-listDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 1,000 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get the task configuration
        config = self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        self.filter_columns = config["filter_columns"]
        self.filter_values = config["filter_values"]
        # Base filter configs do not have filter_operands, so we default to "is"
        self.filter_operators = config.get("filter_operators", ["is" for _ in self.filter_columns])
        self.filter_kind = config["filter_kind"]
        list_info = config.get("list_info")
        if list_info is None:
            list_info = {"columns": table_column_info(self.instance, self.table_name)}
        self.list_info = list_info
        self.filter_len = len(self.filter_columns)

        # Generate goal
        goal = self.get_pretty_printed_description(goal=True)
        info = {}

        return goal, info
workarena.servicenow.filter-change-request-listDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 1,000 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get the task configuration
        config = self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        self.filter_columns = config["filter_columns"]
        self.filter_values = config["filter_values"]
        # Base filter configs do not have filter_operands, so we default to "is"
        self.filter_operators = config.get("filter_operators", ["is" for _ in self.filter_columns])
        self.filter_kind = config["filter_kind"]
        list_info = config.get("list_info")
        if list_info is None:
            list_info = {"columns": table_column_info(self.instance, self.table_name)}
        self.list_info = list_info
        self.filter_len = len(self.filter_columns)

        # Generate goal
        goal = self.get_pretty_printed_description(goal=True)
        info = {}

        return goal, info
workarena.servicenow.filter-hardware-listDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 1,000 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get the task configuration
        config = self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        self.filter_columns = config["filter_columns"]
        self.filter_values = config["filter_values"]
        # Base filter configs do not have filter_operands, so we default to "is"
        self.filter_operators = config.get("filter_operators", ["is" for _ in self.filter_columns])
        self.filter_kind = config["filter_kind"]
        list_info = config.get("list_info")
        if list_info is None:
            list_info = {"columns": table_column_info(self.instance, self.table_name)}
        self.list_info = list_info
        self.filter_len = len(self.filter_columns)

        # Generate goal
        goal = self.get_pretty_printed_description(goal=True)
        info = {}

        return goal, info
workarena.servicenow.filter-incident-listDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 1,000 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get the task configuration
        config = self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        self.filter_columns = config["filter_columns"]
        self.filter_values = config["filter_values"]
        # Base filter configs do not have filter_operands, so we default to "is"
        self.filter_operators = config.get("filter_operators", ["is" for _ in self.filter_columns])
        self.filter_kind = config["filter_kind"]
        list_info = config.get("list_info")
        if list_info is None:
            list_info = {"columns": table_column_info(self.instance, self.table_name)}
        self.list_info = list_info
        self.filter_len = len(self.filter_columns)

        # Generate goal
        goal = self.get_pretty_printed_description(goal=True)
        info = {}

        return goal, info
workarena.servicenow.filter-service-catalog-item-listDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 1,000 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get the task configuration
        config = self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        self.filter_columns = config["filter_columns"]
        self.filter_values = config["filter_values"]
        # Base filter configs do not have filter_operands, so we default to "is"
        self.filter_operators = config.get("filter_operators", ["is" for _ in self.filter_columns])
        self.filter_kind = config["filter_kind"]
        list_info = config.get("list_info")
        if list_info is None:
            list_info = {"columns": table_column_info(self.instance, self.table_name)}
        self.list_info = list_info
        self.filter_len = len(self.filter_columns)

        # Generate goal
        goal = self.get_pretty_printed_description(goal=True)
        info = {}

        return goal, info
workarena.servicenow.filter-user-listDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 1,000 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get the task configuration
        config = self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        self.filter_columns = config["filter_columns"]
        self.filter_values = config["filter_values"]
        # Base filter configs do not have filter_operands, so we default to "is"
        self.filter_operators = config.get("filter_operators", ["is" for _ in self.filter_columns])
        self.filter_kind = config["filter_kind"]
        list_info = config.get("list_info")
        if list_info is None:
            list_info = {"columns": table_column_info(self.instance, self.table_name)}
        self.list_info = list_info
        self.filter_len = len(self.filter_columns)

        # Generate goal
        goal = self.get_pretty_printed_description(goal=True)
        info = {}

        return goal, info
workarena.servicenow.impersonationDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 600 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get task configuration
        self.user_full_name = (
            self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        )
        assert self.user_full_name in self.all_configs

        # Generate goal
        goal = f"Impersonate the user {self.user_full_name}."
        info = {}

        return goal, info
workarena.servicenow.knowledge-base-searchDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 1,000 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get task configuration
        config = self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        self.item = config["item"]
        self.answer = config["value"]
        self.alternative_answers = config["alternative_answers"]
        self.question = config["question"]
        if self.search_by_title:
            self.kb_article_title = config["kb_article_title"]

        # Generate goal
        if self.is_only_navigating:
            goal = f'Navigate to a relevant article in the knowledge base by searching for: "{self.item}" and open the article: "{self.kb_article_title}"'
        else:
            goal = f'Answer the following question using the knowledge base: "{self.question}"'
        info = {}

        return goal, info
workarena.servicenow.order-apple-mac-book-pro15Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 10 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get the task configuration
        assert self.all_configs is not None, "No configuration available for the task."
        self.config = (
            self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        )
        self.requested_item = self.config["item"]
        self.short_description = self.config["description"]
        self.quantity = self.config["quantity"]
        self.requested_configuration = self.config["configuration"]

        # Generate goal
        if self.config_only_in_desc:
            goal = self.get_pretty_printed_description()
        else:
            goal = f'Go to the hardware store and order {self.quantity} "{self.requested_item}"'
        if len(self.requested_configuration) > 0:
            goal += f" with configuration {dict((k, v[1]) for k, v in self.requested_configuration.items())}"
        info = {}

        # Used to keep track of the sysid of the request for validation
        self.request_sysid = None

        return goal, info
workarena.servicenow.order-apple-watchDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 10 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get the task configuration
        assert self.all_configs is not None, "No configuration available for the task."
        self.config = (
            self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        )
        self.requested_item = self.config["item"]
        self.short_description = self.config["description"]
        self.quantity = self.config["quantity"]
        self.requested_configuration = self.config["configuration"]

        # Generate goal
        if self.config_only_in_desc:
            goal = self.get_pretty_printed_description()
        else:
            goal = f'Go to the hardware store and order {self.quantity} "{self.requested_item}"'
        if len(self.requested_configuration) > 0:
            goal += f" with configuration {dict((k, v[1]) for k, v in self.requested_configuration.items())}"
        info = {}

        # Used to keep track of the sysid of the request for validation
        self.request_sysid = None

        return goal, info
workarena.servicenow.order-developer-laptopDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 1,000 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get the task configuration
        assert self.all_configs is not None, "No configuration available for the task."
        self.config = (
            self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        )
        self.requested_item = self.config["item"]
        self.short_description = self.config["description"]
        self.quantity = self.config["quantity"]
        self.requested_configuration = self.config["configuration"]

        # Generate goal
        if self.config_only_in_desc:
            goal = self.get_pretty_printed_description()
        else:
            goal = f'Go to the hardware store and order {self.quantity} "{self.requested_item}"'
        if len(self.requested_configuration) > 0:
            goal += f" with configuration {dict((k, v[1]) for k, v in self.requested_configuration.items())}"
        info = {}

        # Used to keep track of the sysid of the request for validation
        self.request_sysid = None

        return goal, info
workarena.servicenow.order-development-laptop-p-cDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 40 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get the task configuration
        assert self.all_configs is not None, "No configuration available for the task."
        self.config = (
            self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        )
        self.requested_item = self.config["item"]
        self.short_description = self.config["description"]
        self.quantity = self.config["quantity"]
        self.requested_configuration = self.config["configuration"]

        # Generate goal
        if self.config_only_in_desc:
            goal = self.get_pretty_printed_description()
        else:
            goal = f'Go to the hardware store and order {self.quantity} "{self.requested_item}"'
        if len(self.requested_configuration) > 0:
            goal += f" with configuration {dict((k, v[1]) for k, v in self.requested_configuration.items())}"
        info = {}

        # Used to keep track of the sysid of the request for validation
        self.request_sysid = None

        return goal, info
workarena.servicenow.order-ipad-miniDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 80 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get the task configuration
        assert self.all_configs is not None, "No configuration available for the task."
        self.config = (
            self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        )
        self.requested_item = self.config["item"]
        self.short_description = self.config["description"]
        self.quantity = self.config["quantity"]
        self.requested_configuration = self.config["configuration"]

        # Generate goal
        if self.config_only_in_desc:
            goal = self.get_pretty_printed_description()
        else:
            goal = f'Go to the hardware store and order {self.quantity} "{self.requested_item}"'
        if len(self.requested_configuration) > 0:
            goal += f" with configuration {dict((k, v[1]) for k, v in self.requested_configuration.items())}"
        info = {}

        # Used to keep track of the sysid of the request for validation
        self.request_sysid = None

        return goal, info
workarena.servicenow.order-ipad-proDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 60 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get the task configuration
        assert self.all_configs is not None, "No configuration available for the task."
        self.config = (
            self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        )
        self.requested_item = self.config["item"]
        self.short_description = self.config["description"]
        self.quantity = self.config["quantity"]
        self.requested_configuration = self.config["configuration"]

        # Generate goal
        if self.config_only_in_desc:
            goal = self.get_pretty_printed_description()
        else:
            goal = f'Go to the hardware store and order {self.quantity} "{self.requested_item}"'
        if len(self.requested_configuration) > 0:
            goal += f" with configuration {dict((k, v[1]) for k, v in self.requested_configuration.items())}"
        info = {}

        # Used to keep track of the sysid of the request for validation
        self.request_sysid = None

        return goal, info
workarena.servicenow.order-loaner-laptopDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 350 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get the task configuration
        assert self.all_configs is not None, "No configuration available for the task."
        self.config = (
            self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        )
        self.requested_item = self.config["item"]
        self.short_description = self.config["description"]
        self.quantity = self.config["quantity"]
        self.requested_configuration = self.config["configuration"]

        # Generate goal
        if self.config_only_in_desc:
            goal = self.get_pretty_printed_description()
        else:
            goal = f'Go to the hardware store and order {self.quantity} "{self.requested_item}"'
        if len(self.requested_configuration) > 0:
            goal += f" with configuration {dict((k, v[1]) for k, v in self.requested_configuration.items())}"
        info = {}

        # Used to keep track of the sysid of the request for validation
        self.request_sysid = None

        return goal, info
workarena.servicenow.order-sales-laptopDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 1,000 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get the task configuration
        assert self.all_configs is not None, "No configuration available for the task."
        self.config = (
            self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        )
        self.requested_item = self.config["item"]
        self.short_description = self.config["description"]
        self.quantity = self.config["quantity"]
        self.requested_configuration = self.config["configuration"]

        # Generate goal
        if self.config_only_in_desc:
            goal = self.get_pretty_printed_description()
        else:
            goal = f'Go to the hardware store and order {self.quantity} "{self.requested_item}"'
        if len(self.requested_configuration) > 0:
            goal += f" with configuration {dict((k, v[1]) for k, v in self.requested_configuration.items())}"
        info = {}

        # Used to keep track of the sysid of the request for validation
        self.request_sysid = None

        return goal, info
workarena.servicenow.order-standard-laptopDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 1,000 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get the task configuration
        assert self.all_configs is not None, "No configuration available for the task."
        self.config = (
            self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        )
        self.requested_item = self.config["item"]
        self.short_description = self.config["description"]
        self.quantity = self.config["quantity"]
        self.requested_configuration = self.config["configuration"]

        # Generate goal
        if self.config_only_in_desc:
            goal = self.get_pretty_printed_description()
        else:
            goal = f'Go to the hardware store and order {self.quantity} "{self.requested_item}"'
        if len(self.requested_configuration) > 0:
            goal += f" with configuration {dict((k, v[1]) for k, v in self.requested_configuration.items())}"
        info = {}

        # Used to keep track of the sysid of the request for validation
        self.request_sysid = None

        return goal, info
workarena.servicenow.sort-asset-listDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 150 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get the task configuration
        self.config = (
            self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        )
        self.sort_fields = self.config["sort_fields"]
        self.sort_dirs = self.config["sort_dirs"]

        # Get the task goal
        goal = self.config["goal"]
        info = {}

        return goal, info
workarena.servicenow.sort-change-request-listDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 150 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get the task configuration
        self.config = (
            self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        )
        self.sort_fields = self.config["sort_fields"]
        self.sort_dirs = self.config["sort_dirs"]

        # Get the task goal
        goal = self.config["goal"]
        info = {}

        return goal, info
workarena.servicenow.sort-hardware-listDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 150 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get the task configuration
        self.config = (
            self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        )
        self.sort_fields = self.config["sort_fields"]
        self.sort_dirs = self.config["sort_dirs"]

        # Get the task goal
        goal = self.config["goal"]
        info = {}

        return goal, info
workarena.servicenow.sort-incident-listDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 150 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get the task configuration
        self.config = (
            self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        )
        self.sort_fields = self.config["sort_fields"]
        self.sort_dirs = self.config["sort_dirs"]

        # Get the task goal
        goal = self.config["goal"]
        info = {}

        return goal, info
workarena.servicenow.sort-service-catalog-item-listDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 150 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get the task configuration
        self.config = (
            self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        )
        self.sort_fields = self.config["sort_fields"]
        self.sort_dirs = self.config["sort_dirs"]

        # Get the task goal
        goal = self.config["goal"]
        info = {}

        return goal, info
workarena.servicenow.sort-user-listDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 150 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        super().setup_goal(page=page)

        # Get the task configuration
        self.config = (
            self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
        )
        self.sort_fields = self.config["sort_fields"]
        self.sort_dirs = self.config["sort_dirs"]

        # Get the task goal
        goal = self.config["goal"]
        info = {}

        return goal, info
workarena.servicenow.multi-chart-min-max-retrievalDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 56 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: playwright.sync_api.Page) -> Tuple[str | dict]:
        super().setup_goal(page=page)

        # Get the instance report filter config
        REPORT_DATE_FILTER, REPORT_TIME_FILTER = self._get_filter_config()

        # Configure task
        # ... sample a configuration
        self.config = (
            self.fixed_config if self.fixed_config else self.random.choice(self.all_configs())
        )
        # ... set start URL based on config
        # ...... some of the reports have need a date filter to be applied so we do this by patching a placeholder in the URL
        self.start_url = self.instance.snow_url + self.config["url"].replace(
            "REPORT_DATE_FILTER", REPORT_DATE_FILTER
        ).replace("REPORT_TIME_FILTER", REPORT_TIME_FILTER)

        # Produce goal string based on question type
        chart_locator = (
            f"the \"{self.config['chart_series']}\" series of "
            if self.config["chart_series"]
            else ""
        ) + (
            f"the \"{self.config['chart_title']}\" chart"
            if self.config["chart_title"]
            else "the chart"
        )
        if self.config["question"].startswith("value"):
            q_info = [x.strip() for x in self.config["question"].split(";")]
            goal = f'What is the value of "{q_info[2]}" in {chart_locator} (in {q_info[1]})?'
        elif self.config["question"] == "max":
            goal = f"What is the maximum value in {chart_locator}? Give me both the label and the count. If there are many, pick one."
        elif self.config["question"] == "min":
            goal = f"What is the minimum value in {chart_locator}? Give me both the label and the count. If there are many, pick one."
        elif self.config["question"] == "mean":
            goal = f"What is the average value in {chart_locator}? Round off to the next highest integer."
        elif self.config["question"] == "median":
            goal = f"What is the median value in {chart_locator}?"
        elif self.config["question"] == "mode":
            goal = f"What is the mode value in {chart_locator}?"
        else:
            raise NotImplementedError(f"Question type {self.config['question']} not supported")

        return goal, {}
workarena.servicenow.multi-chart-value-retrievalDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 381 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: playwright.sync_api.Page) -> Tuple[str | dict]:
        super().setup_goal(page=page)

        # Get the instance report filter config
        REPORT_DATE_FILTER, REPORT_TIME_FILTER = self._get_filter_config()

        # Configure task
        # ... sample a configuration
        self.config = (
            self.fixed_config if self.fixed_config else self.random.choice(self.all_configs())
        )
        # ... set start URL based on config
        # ...... some of the reports have need a date filter to be applied so we do this by patching a placeholder in the URL
        self.start_url = self.instance.snow_url + self.config["url"].replace(
            "REPORT_DATE_FILTER", REPORT_DATE_FILTER
        ).replace("REPORT_TIME_FILTER", REPORT_TIME_FILTER)

        # Produce goal string based on question type
        chart_locator = (
            f"the \"{self.config['chart_series']}\" series of "
            if self.config["chart_series"]
            else ""
        ) + (
            f"the \"{self.config['chart_title']}\" chart"
            if self.config["chart_title"]
            else "the chart"
        )
        if self.config["question"].startswith("value"):
            q_info = [x.strip() for x in self.config["question"].split(";")]
            goal = f'What is the value of "{q_info[2]}" in {chart_locator} (in {q_info[1]})?'
        elif self.config["question"] == "max":
            goal = f"What is the maximum value in {chart_locator}? Give me both the label and the count. If there are many, pick one."
        elif self.config["question"] == "min":
            goal = f"What is the minimum value in {chart_locator}? Give me both the label and the count. If there are many, pick one."
        elif self.config["question"] == "mean":
            goal = f"What is the average value in {chart_locator}? Round off to the next highest integer."
        elif self.config["question"] == "median":
            goal = f"What is the median value in {chart_locator}?"
        elif self.config["question"] == "mode":
            goal = f"What is the mode value in {chart_locator}?"
        else:
            raise NotImplementedError(f"Question type {self.config['question']} not supported")

        return goal, {}
workarena.servicenow.single-chart-value-retrievalDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 1,000 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: playwright.sync_api.Page) -> Tuple[str | dict]:
        super().setup_goal(page=page)

        # Get the instance report filter config
        REPORT_DATE_FILTER, REPORT_TIME_FILTER = self._get_filter_config()

        # Configure task
        # ... sample a configuration
        self.config = (
            self.fixed_config if self.fixed_config else self.random.choice(self.all_configs())
        )
        # ... set start URL based on config
        # ...... some of the reports have need a date filter to be applied so we do this by patching a placeholder in the URL
        self.start_url = self.instance.snow_url + self.config["url"].replace(
            "REPORT_DATE_FILTER", REPORT_DATE_FILTER
        ).replace("REPORT_TIME_FILTER", REPORT_TIME_FILTER)

        # Produce goal string based on question type
        chart_locator = (
            f"the \"{self.config['chart_series']}\" series of "
            if self.config["chart_series"]
            else ""
        ) + (
            f"the \"{self.config['chart_title']}\" chart"
            if self.config["chart_title"]
            else "the chart"
        )
        if self.config["question"].startswith("value"):
            q_info = [x.strip() for x in self.config["question"].split(";")]
            goal = f'What is the value of "{q_info[2]}" in {chart_locator} (in {q_info[1]})?'
        elif self.config["question"] == "max":
            goal = f"What is the maximum value in {chart_locator}? Give me both the label and the count. If there are many, pick one."
        elif self.config["question"] == "min":
            goal = f"What is the minimum value in {chart_locator}? Give me both the label and the count. If there are many, pick one."
        elif self.config["question"] == "mean":
            goal = f"What is the average value in {chart_locator}? Round off to the next highest integer."
        elif self.config["question"] == "median":
            goal = f"What is the median value in {chart_locator}?"
        elif self.config["question"] == "mode":
            goal = f"What is the mode value in {chart_locator}?"
        else:
            raise NotImplementedError(f"Question type {self.config['question']} not supported")

        return goal, {}
workarena.servicenow.single-chart-min-max-retrievalDynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L1 · 346 saved configuration rows; this template counts once.

Original configuration pool ↗

Inspect the original goal generator
def setup_goal(self, page: playwright.sync_api.Page) -> Tuple[str | dict]:
        super().setup_goal(page=page)

        # Get the instance report filter config
        REPORT_DATE_FILTER, REPORT_TIME_FILTER = self._get_filter_config()

        # Configure task
        # ... sample a configuration
        self.config = (
            self.fixed_config if self.fixed_config else self.random.choice(self.all_configs())
        )
        # ... set start URL based on config
        # ...... some of the reports have need a date filter to be applied so we do this by patching a placeholder in the URL
        self.start_url = self.instance.snow_url + self.config["url"].replace(
            "REPORT_DATE_FILTER", REPORT_DATE_FILTER
        ).replace("REPORT_TIME_FILTER", REPORT_TIME_FILTER)

        # Produce goal string based on question type
        chart_locator = (
            f"the \"{self.config['chart_series']}\" series of "
            if self.config["chart_series"]
            else ""
        ) + (
            f"the \"{self.config['chart_title']}\" chart"
            if self.config["chart_title"]
            else "the chart"
        )
        if self.config["question"].startswith("value"):
            q_info = [x.strip() for x in self.config["question"].split(";")]
            goal = f'What is the value of "{q_info[2]}" in {chart_locator} (in {q_info[1]})?'
        elif self.config["question"] == "max":
            goal = f"What is the maximum value in {chart_locator}? Give me both the label and the count. If there are many, pick one."
        elif self.config["question"] == "min":
            goal = f"What is the minimum value in {chart_locator}? Give me both the label and the count. If there are many, pick one."
        elif self.config["question"] == "mean":
            goal = f"What is the average value in {chart_locator}? Round off to the next highest integer."
        elif self.config["question"] == "median":
            goal = f"What is the median value in {chart_locator}?"
        elif self.config["question"] == "mode":
            goal = f"What is the mode value in {chart_locator}?"
        else:
            raise NotImplementedError(f"Question type {self.config['question']} not supported")

        return goal, {}
workarena.servicenow.basic-filter-problems-and-mark-duplicates-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.basic-filter-problems-and-mark-duplicates-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.basic-filter-problems-and-mark-duplicates-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.basic-filter-problems-and-mark-duplicates-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.basic-filter-problems-and-mark-duplicates-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.basic-filter-problems-and-mark-duplicates-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.priority-filter-problems-and-mark-duplicates-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.priority-filter-problems-and-mark-duplicates-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.priority-filter-problems-and-mark-duplicates-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.priority-filter-problems-and-mark-duplicates-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.priority-filter-problems-and-mark-duplicates-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.priority-filter-problems-and-mark-duplicates-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.high-priority-filter-problems-and-mark-duplicates-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.high-priority-filter-problems-and-mark-duplicates-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.high-priority-filter-problems-and-mark-duplicates-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.high-priority-filter-problems-and-mark-duplicates-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.high-priority-filter-problems-and-mark-duplicates-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.high-priority-filter-problems-and-mark-duplicates-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.workload-balancing-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        num_users = self.random.randint(self.min_users, self.max_users)
        max_problems = self.random.randint(*self.max_problem_range)
        min_problems = self.random.randint(*self.min_problem_range)

        # Create users, create problems and assign problems to users
        for i in range(num_users):
            if i == 0:
                num_problems = max_problems
            elif i == num_users - 1:
                num_problems = min_problems
            else:
                num_problems = self.random.randint(*self.mid_problem_range)
            first_name = fake.first_name() + "-" + fake.first_name()
            last_name = fake.last_name() + "-" + fake.last_name()
            user_full_name = first_name + " " + last_name
            _, _, user_sys_id = create_user(
                instance=self.instance,
                first_name=first_name,
                last_name=last_name,
                user_roles=[
                    "admin",
                    "problem_admin",
                    "problem_manager",
                    "problem_coordinator",
                    "problem_task_analyst",
                ],
                random=self.random,
            )
            self.user_sys_ids.append(user_sys_id)

            if i == 0:
                self.user_with_most_problems = user_full_name
            elif i == num_users - 1:
                self.user_with_least_problems = user_full_name

            # Create problems assigned to current user
            for j in range(num_problems):
                # Assign a priority to the problem; 1 being highest priority and 5 being lowest
                # the use of j % 5 is to ensure that the priority is between 1 and 5 and that there is
                # only one problem with the lowest priority
                priority = (j % 5) + 1
                self.lowest_priority = max(self.lowest_priority, priority)
                problem_sys_id, problem_number = create_problem(
                    instance=self.instance,
                    user_sys_id=user_sys_id,
                    priority=priority,
                    problem_hashtag=self.problem_hashtag,
                    return_number=True,
                )
                # The last problem created is the one to re-assign as it will be the one with the lowest priority (highest priority value)
                # and the first user will be the one with the most problems assigned
                if i == 0 and j == num_problems - 1:
                    self.problem_to_edit_sys_id = problem_sys_id
                    self.problem_to_edit_number = problem_number
                self.problem_sys_ids.append(problem_sys_id)

        # Create a report for problems of the current category
        self.report_sys_id, plot_title = create_report(
            instance=self.instance,
            table="problem",
            filter_hashtag=self.problem_hashtag,
            field="assigned_to",
            plot_title=f"Problems for with hashtag {self.problem_hashtag}",
            random=self.random,
        )
        self.plot_title = plot_title

        # Sample a configuration
        config = self._get_config()
        # Get the task description
        self.short_description = (
            f"Balance the workload for problems with hashtag {self.problem_hashtag}"
        )
        self.task_description = f"Referring to company protocol '{self.protocol_name}' (located in the \"Company Protocols\" knowledge base) re-distribute the problems with hashtag={self.problem_hashtag}."

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.workload-balancing-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        num_users = self.random.randint(self.min_users, self.max_users)
        max_problems = self.random.randint(*self.max_problem_range)
        min_problems = self.random.randint(*self.min_problem_range)

        # Create users, create problems and assign problems to users
        for i in range(num_users):
            if i == 0:
                num_problems = max_problems
            elif i == num_users - 1:
                num_problems = min_problems
            else:
                num_problems = self.random.randint(*self.mid_problem_range)
            first_name = fake.first_name() + "-" + fake.first_name()
            last_name = fake.last_name() + "-" + fake.last_name()
            user_full_name = first_name + " " + last_name
            _, _, user_sys_id = create_user(
                instance=self.instance,
                first_name=first_name,
                last_name=last_name,
                user_roles=[
                    "admin",
                    "problem_admin",
                    "problem_manager",
                    "problem_coordinator",
                    "problem_task_analyst",
                ],
                random=self.random,
            )
            self.user_sys_ids.append(user_sys_id)

            if i == 0:
                self.user_with_most_problems = user_full_name
            elif i == num_users - 1:
                self.user_with_least_problems = user_full_name

            # Create problems assigned to current user
            for j in range(num_problems):
                # Assign a priority to the problem; 1 being highest priority and 5 being lowest
                # the use of j % 5 is to ensure that the priority is between 1 and 5 and that there is
                # only one problem with the lowest priority
                priority = (j % 5) + 1
                self.lowest_priority = max(self.lowest_priority, priority)
                problem_sys_id, problem_number = create_problem(
                    instance=self.instance,
                    user_sys_id=user_sys_id,
                    priority=priority,
                    problem_hashtag=self.problem_hashtag,
                    return_number=True,
                )
                # The last problem created is the one to re-assign as it will be the one with the lowest priority (highest priority value)
                # and the first user will be the one with the most problems assigned
                if i == 0 and j == num_problems - 1:
                    self.problem_to_edit_sys_id = problem_sys_id
                    self.problem_to_edit_number = problem_number
                self.problem_sys_ids.append(problem_sys_id)

        # Create a report for problems of the current category
        self.report_sys_id, plot_title = create_report(
            instance=self.instance,
            table="problem",
            filter_hashtag=self.problem_hashtag,
            field="assigned_to",
            plot_title=f"Problems for with hashtag {self.problem_hashtag}",
            random=self.random,
        )
        self.plot_title = plot_title

        # Sample a configuration
        config = self._get_config()
        # Get the task description
        self.short_description = (
            f"Balance the workload for problems with hashtag {self.problem_hashtag}"
        )
        self.task_description = f"Referring to company protocol '{self.protocol_name}' (located in the \"Company Protocols\" knowledge base) re-distribute the problems with hashtag={self.problem_hashtag}."

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.workload-balancing-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        num_users = self.random.randint(self.min_users, self.max_users)
        max_problems = self.random.randint(*self.max_problem_range)
        min_problems = self.random.randint(*self.min_problem_range)

        # Create users, create problems and assign problems to users
        for i in range(num_users):
            if i == 0:
                num_problems = max_problems
            elif i == num_users - 1:
                num_problems = min_problems
            else:
                num_problems = self.random.randint(*self.mid_problem_range)
            first_name = fake.first_name() + "-" + fake.first_name()
            last_name = fake.last_name() + "-" + fake.last_name()
            user_full_name = first_name + " " + last_name
            _, _, user_sys_id = create_user(
                instance=self.instance,
                first_name=first_name,
                last_name=last_name,
                user_roles=[
                    "admin",
                    "problem_admin",
                    "problem_manager",
                    "problem_coordinator",
                    "problem_task_analyst",
                ],
                random=self.random,
            )
            self.user_sys_ids.append(user_sys_id)

            if i == 0:
                self.user_with_most_problems = user_full_name
            elif i == num_users - 1:
                self.user_with_least_problems = user_full_name

            # Create problems assigned to current user
            for j in range(num_problems):
                # Assign a priority to the problem; 1 being highest priority and 5 being lowest
                # the use of j % 5 is to ensure that the priority is between 1 and 5 and that there is
                # only one problem with the lowest priority
                priority = (j % 5) + 1
                self.lowest_priority = max(self.lowest_priority, priority)
                problem_sys_id, problem_number = create_problem(
                    instance=self.instance,
                    user_sys_id=user_sys_id,
                    priority=priority,
                    problem_hashtag=self.problem_hashtag,
                    return_number=True,
                )
                # The last problem created is the one to re-assign as it will be the one with the lowest priority (highest priority value)
                # and the first user will be the one with the most problems assigned
                if i == 0 and j == num_problems - 1:
                    self.problem_to_edit_sys_id = problem_sys_id
                    self.problem_to_edit_number = problem_number
                self.problem_sys_ids.append(problem_sys_id)

        # Create a report for problems of the current category
        self.report_sys_id, plot_title = create_report(
            instance=self.instance,
            table="problem",
            filter_hashtag=self.problem_hashtag,
            field="assigned_to",
            plot_title=f"Problems for with hashtag {self.problem_hashtag}",
            random=self.random,
        )
        self.plot_title = plot_title

        # Sample a configuration
        config = self._get_config()
        # Get the task description
        self.short_description = (
            f"Balance the workload for problems with hashtag {self.problem_hashtag}"
        )
        self.task_description = f"Referring to company protocol '{self.protocol_name}' (located in the \"Company Protocols\" knowledge base) re-distribute the problems with hashtag={self.problem_hashtag}."

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.workload-balancing-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        num_users = self.random.randint(self.min_users, self.max_users)
        max_problems = self.random.randint(*self.max_problem_range)
        min_problems = self.random.randint(*self.min_problem_range)

        # Create users, create problems and assign problems to users
        for i in range(num_users):
            if i == 0:
                num_problems = max_problems
            elif i == num_users - 1:
                num_problems = min_problems
            else:
                num_problems = self.random.randint(*self.mid_problem_range)
            first_name = fake.first_name() + "-" + fake.first_name()
            last_name = fake.last_name() + "-" + fake.last_name()
            user_full_name = first_name + " " + last_name
            _, _, user_sys_id = create_user(
                instance=self.instance,
                first_name=first_name,
                last_name=last_name,
                user_roles=[
                    "admin",
                    "problem_admin",
                    "problem_manager",
                    "problem_coordinator",
                    "problem_task_analyst",
                ],
                random=self.random,
            )
            self.user_sys_ids.append(user_sys_id)

            if i == 0:
                self.user_with_most_problems = user_full_name
            elif i == num_users - 1:
                self.user_with_least_problems = user_full_name

            # Create problems assigned to current user
            for j in range(num_problems):
                # Assign a priority to the problem; 1 being highest priority and 5 being lowest
                # the use of j % 5 is to ensure that the priority is between 1 and 5 and that there is
                # only one problem with the lowest priority
                priority = (j % 5) + 1
                self.lowest_priority = max(self.lowest_priority, priority)
                problem_sys_id, problem_number = create_problem(
                    instance=self.instance,
                    user_sys_id=user_sys_id,
                    priority=priority,
                    problem_hashtag=self.problem_hashtag,
                    return_number=True,
                )
                # The last problem created is the one to re-assign as it will be the one with the lowest priority (highest priority value)
                # and the first user will be the one with the most problems assigned
                if i == 0 and j == num_problems - 1:
                    self.problem_to_edit_sys_id = problem_sys_id
                    self.problem_to_edit_number = problem_number
                self.problem_sys_ids.append(problem_sys_id)

        # Create a report for problems of the current category
        self.report_sys_id, plot_title = create_report(
            instance=self.instance,
            table="problem",
            filter_hashtag=self.problem_hashtag,
            field="assigned_to",
            plot_title=f"Problems for with hashtag {self.problem_hashtag}",
            random=self.random,
        )
        self.plot_title = plot_title

        # Sample a configuration
        config = self._get_config()
        # Get the task description
        self.short_description = (
            f"Balance the workload for problems with hashtag {self.problem_hashtag}"
        )
        self.task_description = f"Referring to company protocol '{self.protocol_name}' (located in the \"Company Protocols\" knowledge base) re-distribute the problems with hashtag={self.problem_hashtag}."

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.workload-balancing-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        num_users = self.random.randint(self.min_users, self.max_users)
        max_problems = self.random.randint(*self.max_problem_range)
        min_problems = self.random.randint(*self.min_problem_range)

        # Create users, create problems and assign problems to users
        for i in range(num_users):
            if i == 0:
                num_problems = max_problems
            elif i == num_users - 1:
                num_problems = min_problems
            else:
                num_problems = self.random.randint(*self.mid_problem_range)
            first_name = fake.first_name() + "-" + fake.first_name()
            last_name = fake.last_name() + "-" + fake.last_name()
            user_full_name = first_name + " " + last_name
            _, _, user_sys_id = create_user(
                instance=self.instance,
                first_name=first_name,
                last_name=last_name,
                user_roles=[
                    "admin",
                    "problem_admin",
                    "problem_manager",
                    "problem_coordinator",
                    "problem_task_analyst",
                ],
                random=self.random,
            )
            self.user_sys_ids.append(user_sys_id)

            if i == 0:
                self.user_with_most_problems = user_full_name
            elif i == num_users - 1:
                self.user_with_least_problems = user_full_name

            # Create problems assigned to current user
            for j in range(num_problems):
                # Assign a priority to the problem; 1 being highest priority and 5 being lowest
                # the use of j % 5 is to ensure that the priority is between 1 and 5 and that there is
                # only one problem with the lowest priority
                priority = (j % 5) + 1
                self.lowest_priority = max(self.lowest_priority, priority)
                problem_sys_id, problem_number = create_problem(
                    instance=self.instance,
                    user_sys_id=user_sys_id,
                    priority=priority,
                    problem_hashtag=self.problem_hashtag,
                    return_number=True,
                )
                # The last problem created is the one to re-assign as it will be the one with the lowest priority (highest priority value)
                # and the first user will be the one with the most problems assigned
                if i == 0 and j == num_problems - 1:
                    self.problem_to_edit_sys_id = problem_sys_id
                    self.problem_to_edit_number = problem_number
                self.problem_sys_ids.append(problem_sys_id)

        # Create a report for problems of the current category
        self.report_sys_id, plot_title = create_report(
            instance=self.instance,
            table="problem",
            filter_hashtag=self.problem_hashtag,
            field="assigned_to",
            plot_title=f"Problems for with hashtag {self.problem_hashtag}",
            random=self.random,
        )
        self.plot_title = plot_title

        # Sample a configuration
        config = self._get_config()
        # Get the task description
        self.short_description = (
            f"Balance the workload for problems with hashtag {self.problem_hashtag}"
        )
        self.task_description = f"Referring to company protocol '{self.protocol_name}' (located in the \"Company Protocols\" knowledge base) re-distribute the problems with hashtag={self.problem_hashtag}."

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.workload-balancing-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        num_users = self.random.randint(self.min_users, self.max_users)
        max_problems = self.random.randint(*self.max_problem_range)
        min_problems = self.random.randint(*self.min_problem_range)

        # Create users, create problems and assign problems to users
        for i in range(num_users):
            if i == 0:
                num_problems = max_problems
            elif i == num_users - 1:
                num_problems = min_problems
            else:
                num_problems = self.random.randint(*self.mid_problem_range)
            first_name = fake.first_name() + "-" + fake.first_name()
            last_name = fake.last_name() + "-" + fake.last_name()
            user_full_name = first_name + " " + last_name
            _, _, user_sys_id = create_user(
                instance=self.instance,
                first_name=first_name,
                last_name=last_name,
                user_roles=[
                    "admin",
                    "problem_admin",
                    "problem_manager",
                    "problem_coordinator",
                    "problem_task_analyst",
                ],
                random=self.random,
            )
            self.user_sys_ids.append(user_sys_id)

            if i == 0:
                self.user_with_most_problems = user_full_name
            elif i == num_users - 1:
                self.user_with_least_problems = user_full_name

            # Create problems assigned to current user
            for j in range(num_problems):
                # Assign a priority to the problem; 1 being highest priority and 5 being lowest
                # the use of j % 5 is to ensure that the priority is between 1 and 5 and that there is
                # only one problem with the lowest priority
                priority = (j % 5) + 1
                self.lowest_priority = max(self.lowest_priority, priority)
                problem_sys_id, problem_number = create_problem(
                    instance=self.instance,
                    user_sys_id=user_sys_id,
                    priority=priority,
                    problem_hashtag=self.problem_hashtag,
                    return_number=True,
                )
                # The last problem created is the one to re-assign as it will be the one with the lowest priority (highest priority value)
                # and the first user will be the one with the most problems assigned
                if i == 0 and j == num_problems - 1:
                    self.problem_to_edit_sys_id = problem_sys_id
                    self.problem_to_edit_number = problem_number
                self.problem_sys_ids.append(problem_sys_id)

        # Create a report for problems of the current category
        self.report_sys_id, plot_title = create_report(
            instance=self.instance,
            table="problem",
            filter_hashtag=self.problem_hashtag,
            field="assigned_to",
            plot_title=f"Problems for with hashtag {self.problem_hashtag}",
            random=self.random,
        )
        self.plot_title = plot_title

        # Sample a configuration
        config = self._get_config()
        # Get the task description
        self.short_description = (
            f"Balance the workload for problems with hashtag {self.problem_hashtag}"
        )
        self.task_description = f"Referring to company protocol '{self.protocol_name}' (located in the \"Company Protocols\" knowledge base) re-distribute the problems with hashtag={self.problem_hashtag}."

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.work-assignment-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.incident_configs = []
        number_assignments = self.random.randint(self.min_assignments, self.max_assignments)

        all_existing_incidents = table_api_call(
            instance=self.instance, table="incident", method="GET"
        )["result"]
        all_incident_numbers = [incident["number"] for incident in all_existing_incidents]
        new_incident_numbers = []
        for _ in range(number_assignments):
            incident_number = (
                self.prefix + str(id(self) % (10**8)).zfill(8)[:4] + str(random.randint(100, 999))
            )
            while (
                incident_number in all_incident_numbers or incident_number in new_incident_numbers
            ):
                incident_number = (
                    self.prefix
                    + str(id(self) % (10**8)).zfill(8)[:4]
                    + str(random.randint(100, 999))
                )
            new_incident_numbers.append(incident_number)

        self.active_categories = self.random.choice(
            ["hardware", "software", "network", "database"], self.num_categories, replace=False
        )
        for incident_number in new_incident_numbers:
            ### We can reduce the categories here if the setup takes too long
            category = self.random.choice(self.active_categories)
            incident_response = create_incident(
                instance=self.instance,
                incident_number=incident_number,
                caller_sys_id=self._base_user_sysid,
                category=category,
                priority=4,
                impact=2,  # priority is calculated as some combination of impact and urgency
                urgency=3,
            )
            self.incident_configs.append(incident_response)

        self.experts = dict({category: [] for category in self.active_categories})
        for _ in range(self.max_experts_per_category):
            for category in self.active_categories:
                self.experts[category].append(
                    create_user(
                        instance=self.instance,
                        first_name=f"{fake.first_name()}-{fake.first_name()}",
                        last_name=f"{fake.last_name()}-{fake.last_name()}",
                        return_full_response=True,
                        user_roles=["itil"],
                        random=self.random,
                    )
                )
        expert_string = ""
        for category in self.active_categories:
            category_experts = ", ".join(
                expert["first_name"] + " " + expert["last_name"]
                for expert in self.experts[category]
            )
            expert_string += f"{category.capitalize()} agents: {category_experts} \n"
        incident_numbers = ", ".join(new_incident_numbers)

        # Get the task description
        self.task_description = (
            f'Referring to company protocol "{self.protocol_name}" (located in the "Company Protocols" knowledge base) assign work to the agents with the following information: \n'
            + f"Incidents to assign: {incident_numbers} \n\n"
            + f"{expert_string}"
        )

        # Sample a configuration
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Service Desk > Incidents. \n"
                + f"\n2. You have to assign the following incidents to relevant agents: {incident_numbers}. You can filter the list using each incident number and use the 'Assigned to' field to assign an incident.\n"
                + f"\n3. You have to ensure that each incident is assigned to a relevant agent based on the category of the incident.\n"
                + f"\nThe category wise agents are as follows. You can assign an incident to ANY agent from the category:\n"
                + f"{expert_string}"
            )

        return goal, info
workarena.servicenow.work-assignment-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.incident_configs = []
        number_assignments = self.random.randint(self.min_assignments, self.max_assignments)

        all_existing_incidents = table_api_call(
            instance=self.instance, table="incident", method="GET"
        )["result"]
        all_incident_numbers = [incident["number"] for incident in all_existing_incidents]
        new_incident_numbers = []
        for _ in range(number_assignments):
            incident_number = (
                self.prefix + str(id(self) % (10**8)).zfill(8)[:4] + str(random.randint(100, 999))
            )
            while (
                incident_number in all_incident_numbers or incident_number in new_incident_numbers
            ):
                incident_number = (
                    self.prefix
                    + str(id(self) % (10**8)).zfill(8)[:4]
                    + str(random.randint(100, 999))
                )
            new_incident_numbers.append(incident_number)

        self.active_categories = self.random.choice(
            ["hardware", "software", "network", "database"], self.num_categories, replace=False
        )
        for incident_number in new_incident_numbers:
            ### We can reduce the categories here if the setup takes too long
            category = self.random.choice(self.active_categories)
            incident_response = create_incident(
                instance=self.instance,
                incident_number=incident_number,
                caller_sys_id=self._base_user_sysid,
                category=category,
                priority=4,
                impact=2,  # priority is calculated as some combination of impact and urgency
                urgency=3,
            )
            self.incident_configs.append(incident_response)

        self.experts = dict({category: [] for category in self.active_categories})
        for _ in range(self.max_experts_per_category):
            for category in self.active_categories:
                self.experts[category].append(
                    create_user(
                        instance=self.instance,
                        first_name=f"{fake.first_name()}-{fake.first_name()}",
                        last_name=f"{fake.last_name()}-{fake.last_name()}",
                        return_full_response=True,
                        user_roles=["itil"],
                        random=self.random,
                    )
                )
        expert_string = ""
        for category in self.active_categories:
            category_experts = ", ".join(
                expert["first_name"] + " " + expert["last_name"]
                for expert in self.experts[category]
            )
            expert_string += f"{category.capitalize()} agents: {category_experts} \n"
        incident_numbers = ", ".join(new_incident_numbers)

        # Get the task description
        self.task_description = (
            f'Referring to company protocol "{self.protocol_name}" (located in the "Company Protocols" knowledge base) assign work to the agents with the following information: \n'
            + f"Incidents to assign: {incident_numbers} \n\n"
            + f"{expert_string}"
        )

        # Sample a configuration
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Service Desk > Incidents. \n"
                + f"\n2. You have to assign the following incidents to relevant agents: {incident_numbers}. You can filter the list using each incident number and use the 'Assigned to' field to assign an incident.\n"
                + f"\n3. You have to ensure that each incident is assigned to a relevant agent based on the category of the incident.\n"
                + f"\nThe category wise agents are as follows. You can assign an incident to ANY agent from the category:\n"
                + f"{expert_string}"
            )

        return goal, info
workarena.servicenow.work-assignment-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.incident_configs = []
        number_assignments = self.random.randint(self.min_assignments, self.max_assignments)

        all_existing_incidents = table_api_call(
            instance=self.instance, table="incident", method="GET"
        )["result"]
        all_incident_numbers = [incident["number"] for incident in all_existing_incidents]
        new_incident_numbers = []
        for _ in range(number_assignments):
            incident_number = (
                self.prefix + str(id(self) % (10**8)).zfill(8)[:4] + str(random.randint(100, 999))
            )
            while (
                incident_number in all_incident_numbers or incident_number in new_incident_numbers
            ):
                incident_number = (
                    self.prefix
                    + str(id(self) % (10**8)).zfill(8)[:4]
                    + str(random.randint(100, 999))
                )
            new_incident_numbers.append(incident_number)

        self.active_categories = self.random.choice(
            ["hardware", "software", "network", "database"], self.num_categories, replace=False
        )
        for incident_number in new_incident_numbers:
            ### We can reduce the categories here if the setup takes too long
            category = self.random.choice(self.active_categories)
            incident_response = create_incident(
                instance=self.instance,
                incident_number=incident_number,
                caller_sys_id=self._base_user_sysid,
                category=category,
                priority=4,
                impact=2,  # priority is calculated as some combination of impact and urgency
                urgency=3,
            )
            self.incident_configs.append(incident_response)

        self.experts = dict({category: [] for category in self.active_categories})
        for _ in range(self.max_experts_per_category):
            for category in self.active_categories:
                self.experts[category].append(
                    create_user(
                        instance=self.instance,
                        first_name=f"{fake.first_name()}-{fake.first_name()}",
                        last_name=f"{fake.last_name()}-{fake.last_name()}",
                        return_full_response=True,
                        user_roles=["itil"],
                        random=self.random,
                    )
                )
        expert_string = ""
        for category in self.active_categories:
            category_experts = ", ".join(
                expert["first_name"] + " " + expert["last_name"]
                for expert in self.experts[category]
            )
            expert_string += f"{category.capitalize()} agents: {category_experts} \n"
        incident_numbers = ", ".join(new_incident_numbers)

        # Get the task description
        self.task_description = (
            f'Referring to company protocol "{self.protocol_name}" (located in the "Company Protocols" knowledge base) assign work to the agents with the following information: \n'
            + f"Incidents to assign: {incident_numbers} \n\n"
            + f"{expert_string}"
        )

        # Sample a configuration
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Service Desk > Incidents. \n"
                + f"\n2. You have to assign the following incidents to relevant agents: {incident_numbers}. You can filter the list using each incident number and use the 'Assigned to' field to assign an incident.\n"
                + f"\n3. You have to ensure that each incident is assigned to a relevant agent based on the category of the incident.\n"
                + f"\nThe category wise agents are as follows. You can assign an incident to ANY agent from the category:\n"
                + f"{expert_string}"
            )

        return goal, info
workarena.servicenow.work-assignment-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.incident_configs = []
        number_assignments = self.random.randint(self.min_assignments, self.max_assignments)

        all_existing_incidents = table_api_call(
            instance=self.instance, table="incident", method="GET"
        )["result"]
        all_incident_numbers = [incident["number"] for incident in all_existing_incidents]
        new_incident_numbers = []
        for _ in range(number_assignments):
            incident_number = (
                self.prefix + str(id(self) % (10**8)).zfill(8)[:4] + str(random.randint(100, 999))
            )
            while (
                incident_number in all_incident_numbers or incident_number in new_incident_numbers
            ):
                incident_number = (
                    self.prefix
                    + str(id(self) % (10**8)).zfill(8)[:4]
                    + str(random.randint(100, 999))
                )
            new_incident_numbers.append(incident_number)

        self.active_categories = self.random.choice(
            ["hardware", "software", "network", "database"], self.num_categories, replace=False
        )
        for incident_number in new_incident_numbers:
            ### We can reduce the categories here if the setup takes too long
            category = self.random.choice(self.active_categories)
            incident_response = create_incident(
                instance=self.instance,
                incident_number=incident_number,
                caller_sys_id=self._base_user_sysid,
                category=category,
                priority=4,
                impact=2,  # priority is calculated as some combination of impact and urgency
                urgency=3,
            )
            self.incident_configs.append(incident_response)

        self.experts = dict({category: [] for category in self.active_categories})
        for _ in range(self.max_experts_per_category):
            for category in self.active_categories:
                self.experts[category].append(
                    create_user(
                        instance=self.instance,
                        first_name=f"{fake.first_name()}-{fake.first_name()}",
                        last_name=f"{fake.last_name()}-{fake.last_name()}",
                        return_full_response=True,
                        user_roles=["itil"],
                        random=self.random,
                    )
                )
        expert_string = ""
        for category in self.active_categories:
            category_experts = ", ".join(
                expert["first_name"] + " " + expert["last_name"]
                for expert in self.experts[category]
            )
            expert_string += f"{category.capitalize()} agents: {category_experts} \n"
        incident_numbers = ", ".join(new_incident_numbers)

        # Get the task description
        self.task_description = (
            f'Referring to company protocol "{self.protocol_name}" (located in the "Company Protocols" knowledge base) assign work to the agents with the following information: \n'
            + f"Incidents to assign: {incident_numbers} \n\n"
            + f"{expert_string}"
        )

        # Sample a configuration
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Service Desk > Incidents. \n"
                + f"\n2. You have to assign the following incidents to relevant agents: {incident_numbers}. You can filter the list using each incident number and use the 'Assigned to' field to assign an incident.\n"
                + f"\n3. You have to ensure that each incident is assigned to a relevant agent based on the category of the incident.\n"
                + f"\nThe category wise agents are as follows. You can assign an incident to ANY agent from the category:\n"
                + f"{expert_string}"
            )

        return goal, info
workarena.servicenow.work-assignment-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.incident_configs = []
        number_assignments = self.random.randint(self.min_assignments, self.max_assignments)

        all_existing_incidents = table_api_call(
            instance=self.instance, table="incident", method="GET"
        )["result"]
        all_incident_numbers = [incident["number"] for incident in all_existing_incidents]
        new_incident_numbers = []
        for _ in range(number_assignments):
            incident_number = (
                self.prefix + str(id(self) % (10**8)).zfill(8)[:4] + str(random.randint(100, 999))
            )
            while (
                incident_number in all_incident_numbers or incident_number in new_incident_numbers
            ):
                incident_number = (
                    self.prefix
                    + str(id(self) % (10**8)).zfill(8)[:4]
                    + str(random.randint(100, 999))
                )
            new_incident_numbers.append(incident_number)

        self.active_categories = self.random.choice(
            ["hardware", "software", "network", "database"], self.num_categories, replace=False
        )
        for incident_number in new_incident_numbers:
            ### We can reduce the categories here if the setup takes too long
            category = self.random.choice(self.active_categories)
            incident_response = create_incident(
                instance=self.instance,
                incident_number=incident_number,
                caller_sys_id=self._base_user_sysid,
                category=category,
                priority=4,
                impact=2,  # priority is calculated as some combination of impact and urgency
                urgency=3,
            )
            self.incident_configs.append(incident_response)

        self.experts = dict({category: [] for category in self.active_categories})
        for _ in range(self.max_experts_per_category):
            for category in self.active_categories:
                self.experts[category].append(
                    create_user(
                        instance=self.instance,
                        first_name=f"{fake.first_name()}-{fake.first_name()}",
                        last_name=f"{fake.last_name()}-{fake.last_name()}",
                        return_full_response=True,
                        user_roles=["itil"],
                        random=self.random,
                    )
                )
        expert_string = ""
        for category in self.active_categories:
            category_experts = ", ".join(
                expert["first_name"] + " " + expert["last_name"]
                for expert in self.experts[category]
            )
            expert_string += f"{category.capitalize()} agents: {category_experts} \n"
        incident_numbers = ", ".join(new_incident_numbers)

        # Get the task description
        self.task_description = (
            f'Referring to company protocol "{self.protocol_name}" (located in the "Company Protocols" knowledge base) assign work to the agents with the following information: \n'
            + f"Incidents to assign: {incident_numbers} \n\n"
            + f"{expert_string}"
        )

        # Sample a configuration
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Service Desk > Incidents. \n"
                + f"\n2. You have to assign the following incidents to relevant agents: {incident_numbers}. You can filter the list using each incident number and use the 'Assigned to' field to assign an incident.\n"
                + f"\n3. You have to ensure that each incident is assigned to a relevant agent based on the category of the incident.\n"
                + f"\nThe category wise agents are as follows. You can assign an incident to ANY agent from the category:\n"
                + f"{expert_string}"
            )

        return goal, info
workarena.servicenow.work-assignment-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.incident_configs = []
        number_assignments = self.random.randint(self.min_assignments, self.max_assignments)

        all_existing_incidents = table_api_call(
            instance=self.instance, table="incident", method="GET"
        )["result"]
        all_incident_numbers = [incident["number"] for incident in all_existing_incidents]
        new_incident_numbers = []
        for _ in range(number_assignments):
            incident_number = (
                self.prefix + str(id(self) % (10**8)).zfill(8)[:4] + str(random.randint(100, 999))
            )
            while (
                incident_number in all_incident_numbers or incident_number in new_incident_numbers
            ):
                incident_number = (
                    self.prefix
                    + str(id(self) % (10**8)).zfill(8)[:4]
                    + str(random.randint(100, 999))
                )
            new_incident_numbers.append(incident_number)

        self.active_categories = self.random.choice(
            ["hardware", "software", "network", "database"], self.num_categories, replace=False
        )
        for incident_number in new_incident_numbers:
            ### We can reduce the categories here if the setup takes too long
            category = self.random.choice(self.active_categories)
            incident_response = create_incident(
                instance=self.instance,
                incident_number=incident_number,
                caller_sys_id=self._base_user_sysid,
                category=category,
                priority=4,
                impact=2,  # priority is calculated as some combination of impact and urgency
                urgency=3,
            )
            self.incident_configs.append(incident_response)

        self.experts = dict({category: [] for category in self.active_categories})
        for _ in range(self.max_experts_per_category):
            for category in self.active_categories:
                self.experts[category].append(
                    create_user(
                        instance=self.instance,
                        first_name=f"{fake.first_name()}-{fake.first_name()}",
                        last_name=f"{fake.last_name()}-{fake.last_name()}",
                        return_full_response=True,
                        user_roles=["itil"],
                        random=self.random,
                    )
                )
        expert_string = ""
        for category in self.active_categories:
            category_experts = ", ".join(
                expert["first_name"] + " " + expert["last_name"]
                for expert in self.experts[category]
            )
            expert_string += f"{category.capitalize()} agents: {category_experts} \n"
        incident_numbers = ", ".join(new_incident_numbers)

        # Get the task description
        self.task_description = (
            f'Referring to company protocol "{self.protocol_name}" (located in the "Company Protocols" knowledge base) assign work to the agents with the following information: \n'
            + f"Incidents to assign: {incident_numbers} \n\n"
            + f"{expert_string}"
        )

        # Sample a configuration
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Service Desk > Incidents. \n"
                + f"\n2. You have to assign the following incidents to relevant agents: {incident_numbers}. You can filter the list using each incident number and use the 'Assigned to' field to assign an incident.\n"
                + f"\n3. You have to ensure that each incident is assigned to a relevant agent based on the category of the incident.\n"
                + f"\nThe category wise agents are as follows. You can assign an incident to ANY agent from the category:\n"
                + f"{expert_string}"
            )

        return goal, info
workarena.servicenow.priority-assignment-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.incident_configs = []
        number_assignments = sum(
            [attribute["num_incidents"] for attribute in self.priorities.values()]
        )

        all_existing_incidents = table_api_call(
            instance=self.instance, table="incident", method="GET"
        )["result"]
        all_incident_numbers = [incident["number"] for incident in all_existing_incidents]

        new_incident_numbers = []
        for _ in range(number_assignments):
            incident_number = (
                self.prefix + str(id(self) % (10**8)).zfill(8)[:4] + str(random.randint(100, 999))
            )
            while (
                incident_number in all_incident_numbers or incident_number in new_incident_numbers
            ):
                incident_number = (
                    self.prefix
                    + str(id(self) % (10**8)).zfill(8)[:4]
                    + str(random.randint(100, 999))
                )
            new_incident_numbers.append(incident_number)
        incident_category = []
        self.active_categories = self.random.choice(
            ["hardware", "software", "network", "database"], self.num_categories, replace=False
        )
        incident_number_idx = 0
        for priority, attributes in self.priorities.items():
            for _ in range(attributes["num_incidents"]):
                category = self.random.choice(self.active_categories)
                incident_response = create_incident(
                    instance=self.instance,
                    incident_number=new_incident_numbers[incident_number_idx],
                    caller_sys_id=self._base_user_sysid,
                    category=category,
                    priority=priority,
                    impact=attributes[
                        "impact"
                    ],  # priority is calculated as some combination of impact and urgency
                    urgency=attributes["urgency"],
                )
                self.incident_configs.append(incident_response)
                incident_category.append(
                    [new_incident_numbers[incident_number_idx], category, priority]
                )
                incident_number_idx += 1

        self.agents_per_category = dict({category: {} for category in self.active_categories})
        for category in self.agents_per_category:
            self.agents_per_category[category]["expert"] = create_user(
                instance=self.instance,
                first_name=f"{fake.first_name()}-{fake.first_name()}",
                last_name=f"{fake.last_name()}-{fake.last_name()}",
                return_full_response=True,
                user_roles=["itil"],
                random=self.random,
            )
            self.agents_per_category[category]["expert"]["full_name"] = (
                self.agents_per_category[category]["expert"]["first_name"]
                + " "
                + self.agents_per_category[category]["expert"]["last_name"]
            )

            self.agents_per_category[category]["supporter"] = create_user(
                instance=self.instance,
                first_name=f"{fake.first_name()}-{fake.first_name()}",
                last_name=f"{fake.last_name()}-{fake.last_name()}",
                return_full_response=True,
                user_roles=["itil"],
                random=self.random,
            )
            self.agents_per_category[category]["supporter"]["full_name"] = (
                self.agents_per_category[category]["supporter"]["first_name"]
                + " "
                + self.agents_per_category[category]["supporter"]["last_name"]
            )

            self.agents_per_category[category]["planner"] = create_user(
                instance=self.instance,
                first_name=f"{fake.first_name()}-{fake.first_name()}",
                last_name=f"{fake.last_name()}-{fake.last_name()}",
                return_full_response=True,
                user_roles=["itil"],
                random=self.random,
            )
            self.agents_per_category[category]["planner"]["full_name"] = (
                self.agents_per_category[category]["planner"]["first_name"]
                + " "
                + self.agents_per_category[category]["planner"]["last_name"]
            )

        incident_numbers = ", ".join(new_incident_numbers)

        expert_string = ""
        for category in self.active_categories:
            category_experts = f"Expert: {self.agents_per_category[category]['expert']['full_name']}, Supporter: {self.agents_per_category[category]['supporter']['full_name']}, Planner: {self.agents_per_category[category]['planner']['full_name']}"
            expert_string += f"{category.capitalize()} agents - {category_experts} \n"
        # Get the task description
        self.short_description = f"Assign work using priority to relevant agents"
        self.task_description = (
            f'Referring to company protocol "{self.protocol_name}" (located in the "Company Protocols" knowledge base) assign work to the agents with the following information: \n'
            + f"Incidents to assign: {incident_numbers} \n\n"
            + f"{expert_string}"
        )
        # Sample a configuration
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Service Desk > Incidents. \n"
                + f"\n2. You have to assign the following incidents to relevant agents: {incident_numbers}. You can filter the list using each incident number and use the 'Assigned to' field to assign an incident.\n"
                + f"\n3. You have to ensure that each incident is assigned to a relevant agent based on the priority of the incident and its category. For an incident with priority 1 - Critical, assign it to an 'expert' agent of the category, for priority 3 - Moderate, assign it to a 'supporter' of the category, and for priority 5 - Planning assign it to a 'planner' of the category.\n"
                + f"\nThe category wise relevant agent are as follows:\n"
                + f"{expert_string}"
            )

        return goal, info
workarena.servicenow.priority-assignment-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.incident_configs = []
        number_assignments = sum(
            [attribute["num_incidents"] for attribute in self.priorities.values()]
        )

        all_existing_incidents = table_api_call(
            instance=self.instance, table="incident", method="GET"
        )["result"]
        all_incident_numbers = [incident["number"] for incident in all_existing_incidents]

        new_incident_numbers = []
        for _ in range(number_assignments):
            incident_number = (
                self.prefix + str(id(self) % (10**8)).zfill(8)[:4] + str(random.randint(100, 999))
            )
            while (
                incident_number in all_incident_numbers or incident_number in new_incident_numbers
            ):
                incident_number = (
                    self.prefix
                    + str(id(self) % (10**8)).zfill(8)[:4]
                    + str(random.randint(100, 999))
                )
            new_incident_numbers.append(incident_number)
        incident_category = []
        self.active_categories = self.random.choice(
            ["hardware", "software", "network", "database"], self.num_categories, replace=False
        )
        incident_number_idx = 0
        for priority, attributes in self.priorities.items():
            for _ in range(attributes["num_incidents"]):
                category = self.random.choice(self.active_categories)
                incident_response = create_incident(
                    instance=self.instance,
                    incident_number=new_incident_numbers[incident_number_idx],
                    caller_sys_id=self._base_user_sysid,
                    category=category,
                    priority=priority,
                    impact=attributes[
                        "impact"
                    ],  # priority is calculated as some combination of impact and urgency
                    urgency=attributes["urgency"],
                )
                self.incident_configs.append(incident_response)
                incident_category.append(
                    [new_incident_numbers[incident_number_idx], category, priority]
                )
                incident_number_idx += 1

        self.agents_per_category = dict({category: {} for category in self.active_categories})
        for category in self.agents_per_category:
            self.agents_per_category[category]["expert"] = create_user(
                instance=self.instance,
                first_name=f"{fake.first_name()}-{fake.first_name()}",
                last_name=f"{fake.last_name()}-{fake.last_name()}",
                return_full_response=True,
                user_roles=["itil"],
                random=self.random,
            )
            self.agents_per_category[category]["expert"]["full_name"] = (
                self.agents_per_category[category]["expert"]["first_name"]
                + " "
                + self.agents_per_category[category]["expert"]["last_name"]
            )

            self.agents_per_category[category]["supporter"] = create_user(
                instance=self.instance,
                first_name=f"{fake.first_name()}-{fake.first_name()}",
                last_name=f"{fake.last_name()}-{fake.last_name()}",
                return_full_response=True,
                user_roles=["itil"],
                random=self.random,
            )
            self.agents_per_category[category]["supporter"]["full_name"] = (
                self.agents_per_category[category]["supporter"]["first_name"]
                + " "
                + self.agents_per_category[category]["supporter"]["last_name"]
            )

            self.agents_per_category[category]["planner"] = create_user(
                instance=self.instance,
                first_name=f"{fake.first_name()}-{fake.first_name()}",
                last_name=f"{fake.last_name()}-{fake.last_name()}",
                return_full_response=True,
                user_roles=["itil"],
                random=self.random,
            )
            self.agents_per_category[category]["planner"]["full_name"] = (
                self.agents_per_category[category]["planner"]["first_name"]
                + " "
                + self.agents_per_category[category]["planner"]["last_name"]
            )

        incident_numbers = ", ".join(new_incident_numbers)

        expert_string = ""
        for category in self.active_categories:
            category_experts = f"Expert: {self.agents_per_category[category]['expert']['full_name']}, Supporter: {self.agents_per_category[category]['supporter']['full_name']}, Planner: {self.agents_per_category[category]['planner']['full_name']}"
            expert_string += f"{category.capitalize()} agents - {category_experts} \n"
        # Get the task description
        self.short_description = f"Assign work using priority to relevant agents"
        self.task_description = (
            f'Referring to company protocol "{self.protocol_name}" (located in the "Company Protocols" knowledge base) assign work to the agents with the following information: \n'
            + f"Incidents to assign: {incident_numbers} \n\n"
            + f"{expert_string}"
        )
        # Sample a configuration
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Service Desk > Incidents. \n"
                + f"\n2. You have to assign the following incidents to relevant agents: {incident_numbers}. You can filter the list using each incident number and use the 'Assigned to' field to assign an incident.\n"
                + f"\n3. You have to ensure that each incident is assigned to a relevant agent based on the priority of the incident and its category. For an incident with priority 1 - Critical, assign it to an 'expert' agent of the category, for priority 3 - Moderate, assign it to a 'supporter' of the category, and for priority 5 - Planning assign it to a 'planner' of the category.\n"
                + f"\nThe category wise relevant agent are as follows:\n"
                + f"{expert_string}"
            )

        return goal, info
workarena.servicenow.priority-assignment-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.incident_configs = []
        number_assignments = sum(
            [attribute["num_incidents"] for attribute in self.priorities.values()]
        )

        all_existing_incidents = table_api_call(
            instance=self.instance, table="incident", method="GET"
        )["result"]
        all_incident_numbers = [incident["number"] for incident in all_existing_incidents]

        new_incident_numbers = []
        for _ in range(number_assignments):
            incident_number = (
                self.prefix + str(id(self) % (10**8)).zfill(8)[:4] + str(random.randint(100, 999))
            )
            while (
                incident_number in all_incident_numbers or incident_number in new_incident_numbers
            ):
                incident_number = (
                    self.prefix
                    + str(id(self) % (10**8)).zfill(8)[:4]
                    + str(random.randint(100, 999))
                )
            new_incident_numbers.append(incident_number)
        incident_category = []
        self.active_categories = self.random.choice(
            ["hardware", "software", "network", "database"], self.num_categories, replace=False
        )
        incident_number_idx = 0
        for priority, attributes in self.priorities.items():
            for _ in range(attributes["num_incidents"]):
                category = self.random.choice(self.active_categories)
                incident_response = create_incident(
                    instance=self.instance,
                    incident_number=new_incident_numbers[incident_number_idx],
                    caller_sys_id=self._base_user_sysid,
                    category=category,
                    priority=priority,
                    impact=attributes[
                        "impact"
                    ],  # priority is calculated as some combination of impact and urgency
                    urgency=attributes["urgency"],
                )
                self.incident_configs.append(incident_response)
                incident_category.append(
                    [new_incident_numbers[incident_number_idx], category, priority]
                )
                incident_number_idx += 1

        self.agents_per_category = dict({category: {} for category in self.active_categories})
        for category in self.agents_per_category:
            self.agents_per_category[category]["expert"] = create_user(
                instance=self.instance,
                first_name=f"{fake.first_name()}-{fake.first_name()}",
                last_name=f"{fake.last_name()}-{fake.last_name()}",
                return_full_response=True,
                user_roles=["itil"],
                random=self.random,
            )
            self.agents_per_category[category]["expert"]["full_name"] = (
                self.agents_per_category[category]["expert"]["first_name"]
                + " "
                + self.agents_per_category[category]["expert"]["last_name"]
            )

            self.agents_per_category[category]["supporter"] = create_user(
                instance=self.instance,
                first_name=f"{fake.first_name()}-{fake.first_name()}",
                last_name=f"{fake.last_name()}-{fake.last_name()}",
                return_full_response=True,
                user_roles=["itil"],
                random=self.random,
            )
            self.agents_per_category[category]["supporter"]["full_name"] = (
                self.agents_per_category[category]["supporter"]["first_name"]
                + " "
                + self.agents_per_category[category]["supporter"]["last_name"]
            )

            self.agents_per_category[category]["planner"] = create_user(
                instance=self.instance,
                first_name=f"{fake.first_name()}-{fake.first_name()}",
                last_name=f"{fake.last_name()}-{fake.last_name()}",
                return_full_response=True,
                user_roles=["itil"],
                random=self.random,
            )
            self.agents_per_category[category]["planner"]["full_name"] = (
                self.agents_per_category[category]["planner"]["first_name"]
                + " "
                + self.agents_per_category[category]["planner"]["last_name"]
            )

        incident_numbers = ", ".join(new_incident_numbers)

        expert_string = ""
        for category in self.active_categories:
            category_experts = f"Expert: {self.agents_per_category[category]['expert']['full_name']}, Supporter: {self.agents_per_category[category]['supporter']['full_name']}, Planner: {self.agents_per_category[category]['planner']['full_name']}"
            expert_string += f"{category.capitalize()} agents - {category_experts} \n"
        # Get the task description
        self.short_description = f"Assign work using priority to relevant agents"
        self.task_description = (
            f'Referring to company protocol "{self.protocol_name}" (located in the "Company Protocols" knowledge base) assign work to the agents with the following information: \n'
            + f"Incidents to assign: {incident_numbers} \n\n"
            + f"{expert_string}"
        )
        # Sample a configuration
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Service Desk > Incidents. \n"
                + f"\n2. You have to assign the following incidents to relevant agents: {incident_numbers}. You can filter the list using each incident number and use the 'Assigned to' field to assign an incident.\n"
                + f"\n3. You have to ensure that each incident is assigned to a relevant agent based on the priority of the incident and its category. For an incident with priority 1 - Critical, assign it to an 'expert' agent of the category, for priority 3 - Moderate, assign it to a 'supporter' of the category, and for priority 5 - Planning assign it to a 'planner' of the category.\n"
                + f"\nThe category wise relevant agent are as follows:\n"
                + f"{expert_string}"
            )

        return goal, info
workarena.servicenow.priority-assignment-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.incident_configs = []
        number_assignments = sum(
            [attribute["num_incidents"] for attribute in self.priorities.values()]
        )

        all_existing_incidents = table_api_call(
            instance=self.instance, table="incident", method="GET"
        )["result"]
        all_incident_numbers = [incident["number"] for incident in all_existing_incidents]

        new_incident_numbers = []
        for _ in range(number_assignments):
            incident_number = (
                self.prefix + str(id(self) % (10**8)).zfill(8)[:4] + str(random.randint(100, 999))
            )
            while (
                incident_number in all_incident_numbers or incident_number in new_incident_numbers
            ):
                incident_number = (
                    self.prefix
                    + str(id(self) % (10**8)).zfill(8)[:4]
                    + str(random.randint(100, 999))
                )
            new_incident_numbers.append(incident_number)
        incident_category = []
        self.active_categories = self.random.choice(
            ["hardware", "software", "network", "database"], self.num_categories, replace=False
        )
        incident_number_idx = 0
        for priority, attributes in self.priorities.items():
            for _ in range(attributes["num_incidents"]):
                category = self.random.choice(self.active_categories)
                incident_response = create_incident(
                    instance=self.instance,
                    incident_number=new_incident_numbers[incident_number_idx],
                    caller_sys_id=self._base_user_sysid,
                    category=category,
                    priority=priority,
                    impact=attributes[
                        "impact"
                    ],  # priority is calculated as some combination of impact and urgency
                    urgency=attributes["urgency"],
                )
                self.incident_configs.append(incident_response)
                incident_category.append(
                    [new_incident_numbers[incident_number_idx], category, priority]
                )
                incident_number_idx += 1

        self.agents_per_category = dict({category: {} for category in self.active_categories})
        for category in self.agents_per_category:
            self.agents_per_category[category]["expert"] = create_user(
                instance=self.instance,
                first_name=f"{fake.first_name()}-{fake.first_name()}",
                last_name=f"{fake.last_name()}-{fake.last_name()}",
                return_full_response=True,
                user_roles=["itil"],
                random=self.random,
            )
            self.agents_per_category[category]["expert"]["full_name"] = (
                self.agents_per_category[category]["expert"]["first_name"]
                + " "
                + self.agents_per_category[category]["expert"]["last_name"]
            )

            self.agents_per_category[category]["supporter"] = create_user(
                instance=self.instance,
                first_name=f"{fake.first_name()}-{fake.first_name()}",
                last_name=f"{fake.last_name()}-{fake.last_name()}",
                return_full_response=True,
                user_roles=["itil"],
                random=self.random,
            )
            self.agents_per_category[category]["supporter"]["full_name"] = (
                self.agents_per_category[category]["supporter"]["first_name"]
                + " "
                + self.agents_per_category[category]["supporter"]["last_name"]
            )

            self.agents_per_category[category]["planner"] = create_user(
                instance=self.instance,
                first_name=f"{fake.first_name()}-{fake.first_name()}",
                last_name=f"{fake.last_name()}-{fake.last_name()}",
                return_full_response=True,
                user_roles=["itil"],
                random=self.random,
            )
            self.agents_per_category[category]["planner"]["full_name"] = (
                self.agents_per_category[category]["planner"]["first_name"]
                + " "
                + self.agents_per_category[category]["planner"]["last_name"]
            )

        incident_numbers = ", ".join(new_incident_numbers)

        expert_string = ""
        for category in self.active_categories:
            category_experts = f"Expert: {self.agents_per_category[category]['expert']['full_name']}, Supporter: {self.agents_per_category[category]['supporter']['full_name']}, Planner: {self.agents_per_category[category]['planner']['full_name']}"
            expert_string += f"{category.capitalize()} agents - {category_experts} \n"
        # Get the task description
        self.short_description = f"Assign work using priority to relevant agents"
        self.task_description = (
            f'Referring to company protocol "{self.protocol_name}" (located in the "Company Protocols" knowledge base) assign work to the agents with the following information: \n'
            + f"Incidents to assign: {incident_numbers} \n\n"
            + f"{expert_string}"
        )
        # Sample a configuration
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Service Desk > Incidents. \n"
                + f"\n2. You have to assign the following incidents to relevant agents: {incident_numbers}. You can filter the list using each incident number and use the 'Assigned to' field to assign an incident.\n"
                + f"\n3. You have to ensure that each incident is assigned to a relevant agent based on the priority of the incident and its category. For an incident with priority 1 - Critical, assign it to an 'expert' agent of the category, for priority 3 - Moderate, assign it to a 'supporter' of the category, and for priority 5 - Planning assign it to a 'planner' of the category.\n"
                + f"\nThe category wise relevant agent are as follows:\n"
                + f"{expert_string}"
            )

        return goal, info
workarena.servicenow.priority-assignment-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.incident_configs = []
        number_assignments = sum(
            [attribute["num_incidents"] for attribute in self.priorities.values()]
        )

        all_existing_incidents = table_api_call(
            instance=self.instance, table="incident", method="GET"
        )["result"]
        all_incident_numbers = [incident["number"] for incident in all_existing_incidents]

        new_incident_numbers = []
        for _ in range(number_assignments):
            incident_number = (
                self.prefix + str(id(self) % (10**8)).zfill(8)[:4] + str(random.randint(100, 999))
            )
            while (
                incident_number in all_incident_numbers or incident_number in new_incident_numbers
            ):
                incident_number = (
                    self.prefix
                    + str(id(self) % (10**8)).zfill(8)[:4]
                    + str(random.randint(100, 999))
                )
            new_incident_numbers.append(incident_number)
        incident_category = []
        self.active_categories = self.random.choice(
            ["hardware", "software", "network", "database"], self.num_categories, replace=False
        )
        incident_number_idx = 0
        for priority, attributes in self.priorities.items():
            for _ in range(attributes["num_incidents"]):
                category = self.random.choice(self.active_categories)
                incident_response = create_incident(
                    instance=self.instance,
                    incident_number=new_incident_numbers[incident_number_idx],
                    caller_sys_id=self._base_user_sysid,
                    category=category,
                    priority=priority,
                    impact=attributes[
                        "impact"
                    ],  # priority is calculated as some combination of impact and urgency
                    urgency=attributes["urgency"],
                )
                self.incident_configs.append(incident_response)
                incident_category.append(
                    [new_incident_numbers[incident_number_idx], category, priority]
                )
                incident_number_idx += 1

        self.agents_per_category = dict({category: {} for category in self.active_categories})
        for category in self.agents_per_category:
            self.agents_per_category[category]["expert"] = create_user(
                instance=self.instance,
                first_name=f"{fake.first_name()}-{fake.first_name()}",
                last_name=f"{fake.last_name()}-{fake.last_name()}",
                return_full_response=True,
                user_roles=["itil"],
                random=self.random,
            )
            self.agents_per_category[category]["expert"]["full_name"] = (
                self.agents_per_category[category]["expert"]["first_name"]
                + " "
                + self.agents_per_category[category]["expert"]["last_name"]
            )

            self.agents_per_category[category]["supporter"] = create_user(
                instance=self.instance,
                first_name=f"{fake.first_name()}-{fake.first_name()}",
                last_name=f"{fake.last_name()}-{fake.last_name()}",
                return_full_response=True,
                user_roles=["itil"],
                random=self.random,
            )
            self.agents_per_category[category]["supporter"]["full_name"] = (
                self.agents_per_category[category]["supporter"]["first_name"]
                + " "
                + self.agents_per_category[category]["supporter"]["last_name"]
            )

            self.agents_per_category[category]["planner"] = create_user(
                instance=self.instance,
                first_name=f"{fake.first_name()}-{fake.first_name()}",
                last_name=f"{fake.last_name()}-{fake.last_name()}",
                return_full_response=True,
                user_roles=["itil"],
                random=self.random,
            )
            self.agents_per_category[category]["planner"]["full_name"] = (
                self.agents_per_category[category]["planner"]["first_name"]
                + " "
                + self.agents_per_category[category]["planner"]["last_name"]
            )

        incident_numbers = ", ".join(new_incident_numbers)

        expert_string = ""
        for category in self.active_categories:
            category_experts = f"Expert: {self.agents_per_category[category]['expert']['full_name']}, Supporter: {self.agents_per_category[category]['supporter']['full_name']}, Planner: {self.agents_per_category[category]['planner']['full_name']}"
            expert_string += f"{category.capitalize()} agents - {category_experts} \n"
        # Get the task description
        self.short_description = f"Assign work using priority to relevant agents"
        self.task_description = (
            f'Referring to company protocol "{self.protocol_name}" (located in the "Company Protocols" knowledge base) assign work to the agents with the following information: \n'
            + f"Incidents to assign: {incident_numbers} \n\n"
            + f"{expert_string}"
        )
        # Sample a configuration
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Service Desk > Incidents. \n"
                + f"\n2. You have to assign the following incidents to relevant agents: {incident_numbers}. You can filter the list using each incident number and use the 'Assigned to' field to assign an incident.\n"
                + f"\n3. You have to ensure that each incident is assigned to a relevant agent based on the priority of the incident and its category. For an incident with priority 1 - Critical, assign it to an 'expert' agent of the category, for priority 3 - Moderate, assign it to a 'supporter' of the category, and for priority 5 - Planning assign it to a 'planner' of the category.\n"
                + f"\nThe category wise relevant agent are as follows:\n"
                + f"{expert_string}"
            )

        return goal, info
workarena.servicenow.priority-assignment-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.incident_configs = []
        number_assignments = sum(
            [attribute["num_incidents"] for attribute in self.priorities.values()]
        )

        all_existing_incidents = table_api_call(
            instance=self.instance, table="incident", method="GET"
        )["result"]
        all_incident_numbers = [incident["number"] for incident in all_existing_incidents]

        new_incident_numbers = []
        for _ in range(number_assignments):
            incident_number = (
                self.prefix + str(id(self) % (10**8)).zfill(8)[:4] + str(random.randint(100, 999))
            )
            while (
                incident_number in all_incident_numbers or incident_number in new_incident_numbers
            ):
                incident_number = (
                    self.prefix
                    + str(id(self) % (10**8)).zfill(8)[:4]
                    + str(random.randint(100, 999))
                )
            new_incident_numbers.append(incident_number)
        incident_category = []
        self.active_categories = self.random.choice(
            ["hardware", "software", "network", "database"], self.num_categories, replace=False
        )
        incident_number_idx = 0
        for priority, attributes in self.priorities.items():
            for _ in range(attributes["num_incidents"]):
                category = self.random.choice(self.active_categories)
                incident_response = create_incident(
                    instance=self.instance,
                    incident_number=new_incident_numbers[incident_number_idx],
                    caller_sys_id=self._base_user_sysid,
                    category=category,
                    priority=priority,
                    impact=attributes[
                        "impact"
                    ],  # priority is calculated as some combination of impact and urgency
                    urgency=attributes["urgency"],
                )
                self.incident_configs.append(incident_response)
                incident_category.append(
                    [new_incident_numbers[incident_number_idx], category, priority]
                )
                incident_number_idx += 1

        self.agents_per_category = dict({category: {} for category in self.active_categories})
        for category in self.agents_per_category:
            self.agents_per_category[category]["expert"] = create_user(
                instance=self.instance,
                first_name=f"{fake.first_name()}-{fake.first_name()}",
                last_name=f"{fake.last_name()}-{fake.last_name()}",
                return_full_response=True,
                user_roles=["itil"],
                random=self.random,
            )
            self.agents_per_category[category]["expert"]["full_name"] = (
                self.agents_per_category[category]["expert"]["first_name"]
                + " "
                + self.agents_per_category[category]["expert"]["last_name"]
            )

            self.agents_per_category[category]["supporter"] = create_user(
                instance=self.instance,
                first_name=f"{fake.first_name()}-{fake.first_name()}",
                last_name=f"{fake.last_name()}-{fake.last_name()}",
                return_full_response=True,
                user_roles=["itil"],
                random=self.random,
            )
            self.agents_per_category[category]["supporter"]["full_name"] = (
                self.agents_per_category[category]["supporter"]["first_name"]
                + " "
                + self.agents_per_category[category]["supporter"]["last_name"]
            )

            self.agents_per_category[category]["planner"] = create_user(
                instance=self.instance,
                first_name=f"{fake.first_name()}-{fake.first_name()}",
                last_name=f"{fake.last_name()}-{fake.last_name()}",
                return_full_response=True,
                user_roles=["itil"],
                random=self.random,
            )
            self.agents_per_category[category]["planner"]["full_name"] = (
                self.agents_per_category[category]["planner"]["first_name"]
                + " "
                + self.agents_per_category[category]["planner"]["last_name"]
            )

        incident_numbers = ", ".join(new_incident_numbers)

        expert_string = ""
        for category in self.active_categories:
            category_experts = f"Expert: {self.agents_per_category[category]['expert']['full_name']}, Supporter: {self.agents_per_category[category]['supporter']['full_name']}, Planner: {self.agents_per_category[category]['planner']['full_name']}"
            expert_string += f"{category.capitalize()} agents - {category_experts} \n"
        # Get the task description
        self.short_description = f"Assign work using priority to relevant agents"
        self.task_description = (
            f'Referring to company protocol "{self.protocol_name}" (located in the "Company Protocols" knowledge base) assign work to the agents with the following information: \n'
            + f"Incidents to assign: {incident_numbers} \n\n"
            + f"{expert_string}"
        )
        # Sample a configuration
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Service Desk > Incidents. \n"
                + f"\n2. You have to assign the following incidents to relevant agents: {incident_numbers}. You can filter the list using each incident number and use the 'Assigned to' field to assign an incident.\n"
                + f"\n3. You have to ensure that each incident is assigned to a relevant agent based on the priority of the incident and its category. For an incident with priority 1 - Critical, assign it to an 'expert' agent of the category, for priority 3 - Moderate, assign it to a 'supporter' of the category, and for priority 5 - Planning assign it to a 'planner' of the category.\n"
                + f"\nThe category wise relevant agent are as follows:\n"
                + f"{expert_string}"
            )

        return goal, info
workarena.servicenow.two-changes-basic-uniform-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-basic-uniform-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-wide-basic-uniform-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-wide-basic-uniform-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-fix-basic-uniform-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-fix-basic-uniform-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-fix-wide-basic-uniform-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-fix-wide-basic-uniform-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-basic-varied-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-basic-varied-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-wide-basic-varied-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-wide-basic-varied-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-fix-basic-varied-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-fix-basic-varied-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-fix-wide-basic-varied-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-fix-wide-basic-varied-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-basic-uniform-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-basic-uniform-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-wide-basic-uniform-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-wide-basic-uniform-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-fix-basic-uniform-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-fix-basic-uniform-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-fix-wide-basic-uniform-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-fix-wide-basic-uniform-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-basic-varied-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-basic-varied-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-wide-basic-varied-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-wide-basic-varied-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-fix-basic-varied-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-fix-basic-varied-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-fix-wide-basic-varied-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-fix-wide-basic-varied-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-priority-uniform-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-priority-uniform-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-wide-priority-uniform-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-wide-priority-uniform-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-fix-priority-uniform-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-fix-priority-uniform-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-fix-wide-priority-uniform-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-fix-wide-priority-uniform-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-priority-varied-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-priority-varied-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-wide-priority-varied-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-wide-priority-varied-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-fix-priority-varied-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-fix-priority-varied-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-fix-wide-priority-varied-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-fix-wide-priority-varied-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-tight-uniform-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-tight-uniform-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-wide-schedule-tight-uniform-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-wide-schedule-tight-uniform-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-fix-tight-uniform-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-fix-tight-uniform-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-fix-wide-schedule-tight-uniform-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-fix-wide-schedule-tight-uniform-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-tight-priority-varied-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-tight-priority-varied-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-wide-tight-priority-varied-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-wide-tight-priority-varied-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-fix-tight-priority-varied-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-fix-tight-priority-varied-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-fix-wide-tight-priority-varied-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.two-changes-fix-wide-tight-priority-varied-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-priority-uniform-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-priority-uniform-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-wide-priority-uniform-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-wide-priority-uniform-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-fix-priority-uniform-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-fix-priority-uniform-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-fix-wide-priority-uniform-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-fix-wide-priority-uniform-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-priority-varied-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-priority-varied-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-wide-priority-varied-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-wide-priority-varied-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-fix-priority-varied-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-fix-priority-varied-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-fix-wide-priority-varied-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-fix-wide-priority-varied-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-tight-uniform-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-tight-uniform-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-wide-schedule-tight-uniform-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-wide-schedule-tight-uniform-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-fix-tight-uniform-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-fix-tight-uniform-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-fix-wide-schedule-tight-uniform-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-fix-wide-schedule-tight-uniform-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-tight-priority-varied-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-tight-priority-varied-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-wide-tight-priority-varied-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-wide-tight-priority-varied-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-fix-tight-priority-varied-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-fix-tight-priority-varied-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-fix-wide-tight-priority-varied-risk-change-request-scheduling-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.three-changes-fix-wide-tight-priority-varied-risk-change-request-scheduling-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        goal, info = super().setup_goal(page=page)

        if self.level == 2:
            goal += self.schedule_bounds_goal

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-max-order-developer-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-max-order-developer-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-max-orderi-pad-mini-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-max-orderi-pad-mini-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-max-orderi-pad-pro-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-max-orderi-pad-pro-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-max-order-sales-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-max-order-sales-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-max-order-standard-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-max-order-standard-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-max-order-apple-watch-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-max-order-apple-watch-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-max-order-apple-macbook-pro15-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-max-order-apple-macbook-pro15-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-max-order-development-laptop-p-c-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-max-order-development-laptop-p-c-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-max-order-loaner-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-max-order-loaner-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-min-create-incident-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        incident_numbers_string = ", ".join(self.incident_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Using this value, retrieve agents that have less than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'incident' with the following information (only fill these fields and the 'incident number' from below) and assign it to them using the 'Assigned to' field: \n"
                + f"\t\t Category: 'Inquiry / Help', Impact: '1 - High', Urgency: '1 - High', Short description: 'Compulsory training for employee in probation', Caller: '{self.user_name}'. Make sure to use an 'incident number' from the list as described below.\n"
                + f"\t   Importantly, you should override the default incident numbers in the form and instead use one incident number from the following list for each incident you create: {incident_numbers_string}.\n\n"
                + f"Note that you will create as many incidents as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' new incidents here, one assigned to each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned less than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Service Desk > Incidents. \n"
                + f"\n5. You have to create new 'incidents' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new incident:- Category: 'Inquiry / Help', Impact: '1 - High', Urgency: '1 - High', Short description: 'Compulsory training for employee in probation', Caller: '{self.user_name}' and 'assign' them to each agent using the 'Assigned to' field.\n\n"
                + f"Importantly, you should override the default incident numbers in the form and instead use one incident number from the following list for each incident you create: {incident_numbers_string}.\n"
                + f"Note that you will create as many incidents as there are agents matching the above criteria."
                + "\nFor example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' new incidents here, one assigned to each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-min-create-incident-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        incident_numbers_string = ", ".join(self.incident_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Using this value, retrieve agents that have less than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'incident' with the following information (only fill these fields and the 'incident number' from below) and assign it to them using the 'Assigned to' field: \n"
                + f"\t\t Category: 'Inquiry / Help', Impact: '1 - High', Urgency: '1 - High', Short description: 'Compulsory training for employee in probation', Caller: '{self.user_name}'. Make sure to use an 'incident number' from the list as described below.\n"
                + f"\t   Importantly, you should override the default incident numbers in the form and instead use one incident number from the following list for each incident you create: {incident_numbers_string}.\n\n"
                + f"Note that you will create as many incidents as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' new incidents here, one assigned to each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned less than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Service Desk > Incidents. \n"
                + f"\n5. You have to create new 'incidents' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new incident:- Category: 'Inquiry / Help', Impact: '1 - High', Urgency: '1 - High', Short description: 'Compulsory training for employee in probation', Caller: '{self.user_name}' and 'assign' them to each agent using the 'Assigned to' field.\n\n"
                + f"Importantly, you should override the default incident numbers in the form and instead use one incident number from the following list for each incident you create: {incident_numbers_string}.\n"
                + f"Note that you will create as many incidents as there are agents matching the above criteria."
                + "\nFor example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' new incidents here, one assigned to each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-min-create-problem-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report(
            user_roles=[
                "itil",
                "problem_admin",
                "problem_manager",
                "problem_coordinator",
                "problem_task_analyst",
            ]
        )
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have less than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create a 'problem' with the following information (only fill these fields) and assign it to them using the 'Assigned to' field: \n"
                + f"\t\t Impact: '1 - High', Urgency: '1 - High', Problem statement: 'Compulsory training for employee in probation'.\n"
                + f"Note that you will create as many problems as there are agents matching the above criteria.\n"
                + f"\t For example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' problems here, one assigned to each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned less than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to All > Problems. \n"
                + f"\n5. You have to create new 'problems' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new problem:- Impact: '1 - High', Urgency: '1 - High', Problem statement: 'Compulsory training for employee in probation' and 'assign' them to each agent.\n\n"
                + f"Note that you will create as many problems as there are agents matching the above criteria."
                + "\nFor example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' problems here, one assigned to each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-min-create-problem-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report(
            user_roles=[
                "itil",
                "problem_admin",
                "problem_manager",
                "problem_coordinator",
                "problem_task_analyst",
            ]
        )
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have less than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create a 'problem' with the following information (only fill these fields) and assign it to them using the 'Assigned to' field: \n"
                + f"\t\t Impact: '1 - High', Urgency: '1 - High', Problem statement: 'Compulsory training for employee in probation'.\n"
                + f"Note that you will create as many problems as there are agents matching the above criteria.\n"
                + f"\t For example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' problems here, one assigned to each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned less than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to All > Problems. \n"
                + f"\n5. You have to create new 'problems' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new problem:- Impact: '1 - High', Urgency: '1 - High', Problem statement: 'Compulsory training for employee in probation' and 'assign' them to each agent.\n\n"
                + f"Note that you will create as many problems as there are agents matching the above criteria."
                + "\nFor example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' problems here, one assigned to each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-min-filter-asset-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        # We create dummy assets for the consumable and license categories here
        ### NOTE: We can create assets without any of the following information, save time, and still assign them to the user. The task should be fine.
        consumable_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_consumable", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        consumables = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={consumable_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        consumables_sysids = [consumable["sys_id"] for consumable in consumables]

        license_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_license", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        licenses = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={license_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"][:10]
        license_sysids = [license["sys_id"] for license in licenses]

        self.new_asset_sys_ids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                consumable_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(consumables_sysids),
                    "model_category": consumable_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=consumable_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])
                license_asset_data = {
                    "asset_tag": "LICENSE" + str(random.randint(100, 999)),
                    "model": self.random.choice(license_sysids),
                    "model_category": license_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=license_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Asset List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > All Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > All Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-min-filter-asset-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        # We create dummy assets for the consumable and license categories here
        ### NOTE: We can create assets without any of the following information, save time, and still assign them to the user. The task should be fine.
        consumable_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_consumable", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        consumables = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={consumable_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        consumables_sysids = [consumable["sys_id"] for consumable in consumables]

        license_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_license", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        licenses = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={license_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"][:10]
        license_sysids = [license["sys_id"] for license in licenses]

        self.new_asset_sys_ids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                consumable_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(consumables_sysids),
                    "model_category": consumable_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=consumable_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])
                license_asset_data = {
                    "asset_tag": "LICENSE" + str(random.randint(100, 999)),
                    "model": self.random.choice(license_sysids),
                    "model_category": license_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=license_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Asset List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > All Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > All Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-min-filter-hardware-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        hardware_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_hardware", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        hardwares = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={hardware_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        hardware_sysids = [hardware["sys_id"] for hardware in hardwares]
        self.new_asset_sysids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                hardware_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(hardware_sysids),
                    "model_category": hardware_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_hardware",
                    json=hardware_asset_data,
                    method="POST",
                )
                self.new_asset_sysids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Hardware List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > Hardware Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > Hardware Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-min-filter-hardware-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        hardware_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_hardware", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        hardwares = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={hardware_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        hardware_sysids = [hardware["sys_id"] for hardware in hardwares]
        self.new_asset_sysids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                hardware_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(hardware_sysids),
                    "model_category": hardware_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_hardware",
                    json=hardware_asset_data,
                    method="POST",
                )
                self.new_asset_sysids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Hardware List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > Hardware Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > Hardware Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-min-filter-incident-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Incident List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Service Desk > Incidents. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Service Desk > Incidents. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-min-filter-incident-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Incident List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Service Desk > Incidents. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Service Desk > Incidents. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-min-filter-user-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the User List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Organization > Users. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Organization > Users. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-min-filter-user-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the User List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Organization > Users. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Organization > Users. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-filter-asset-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        # We create dummy assets for the consumable and license categories here
        ### NOTE: We can create assets without any of the following information, save time, and still assign them to the user. The task should be fine.
        consumable_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_consumable", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        consumables = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={consumable_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        consumables_sysids = [consumable["sys_id"] for consumable in consumables]

        license_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_license", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        licenses = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={license_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"][:10]
        license_sysids = [license["sys_id"] for license in licenses]

        self.new_asset_sys_ids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                consumable_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(consumables_sysids),
                    "model_category": consumable_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=consumable_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])
                license_asset_data = {
                    "asset_tag": "LICENSE" + str(random.randint(100, 999)),
                    "model": self.random.choice(license_sysids),
                    "model_category": license_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=license_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Asset List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > All Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > All Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-filter-asset-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        # We create dummy assets for the consumable and license categories here
        ### NOTE: We can create assets without any of the following information, save time, and still assign them to the user. The task should be fine.
        consumable_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_consumable", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        consumables = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={consumable_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        consumables_sysids = [consumable["sys_id"] for consumable in consumables]

        license_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_license", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        licenses = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={license_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"][:10]
        license_sysids = [license["sys_id"] for license in licenses]

        self.new_asset_sys_ids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                consumable_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(consumables_sysids),
                    "model_category": consumable_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=consumable_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])
                license_asset_data = {
                    "asset_tag": "LICENSE" + str(random.randint(100, 999)),
                    "model": self.random.choice(license_sysids),
                    "model_category": license_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=license_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Asset List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > All Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > All Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-filter-hardware-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        hardware_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_hardware", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        hardwares = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={hardware_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        hardware_sysids = [hardware["sys_id"] for hardware in hardwares]
        self.new_asset_sysids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                hardware_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(hardware_sysids),
                    "model_category": hardware_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_hardware",
                    json=hardware_asset_data,
                    method="POST",
                )
                self.new_asset_sysids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Hardware List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > Hardware Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > Hardware Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-filter-hardware-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        hardware_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_hardware", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        hardwares = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={hardware_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        hardware_sysids = [hardware["sys_id"] for hardware in hardwares]
        self.new_asset_sysids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                hardware_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(hardware_sysids),
                    "model_category": hardware_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_hardware",
                    json=hardware_asset_data,
                    method="POST",
                )
                self.new_asset_sysids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Hardware List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > Hardware Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > Hardware Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-filter-incident-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Incident List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Service Desk > Incidents. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Service Desk > Incidents. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-filter-incident-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Incident List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Service Desk > Incidents. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Service Desk > Incidents. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-filter-user-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the User List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Organization > Users. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Organization > Users. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-filter-user-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the User List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Organization > Users. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Organization > Users. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-request-apple-watch-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-request-apple-watch-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-request-apple-watch2-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-request-apple-watch2-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-request-apple-ipad3-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-request-apple-ipad3-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-request-apple-iphone13pro-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-request-apple-iphone13pro-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-request-apple-iphone13-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-request-apple-iphone13-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-request-galaxy-note20-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-request-galaxy-note20-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-request-google-nexus7-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-request-google-nexus7-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-request-microsoft-surface-pro3-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-request-microsoft-surface-pro3-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-request-pixel4a-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-request-pixel4a-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-request-windows-surface-pro4-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-max-request-windows-surface-pro4-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.get-warranty-expiration-date-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        # Create a user and a laptop for the user
        self._create_user_and_laptop()

        # Sample a configuration
        config = self.fixed_config if self.fixed_config else self._get_config()

        # Get the task description
        self.short_description = (
            f"Find the warranty expiration date for {self.user_full_name}'s laptop"
        )
        self.task_description = f'Refer to company protocol "{self.protocol_name}" (located in the "Company Protocols" knowledge base) for the steps to find the Warranty expiration date for {self.user_full_name}\'s laptop.\n \n'

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.get-warranty-expiration-date-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        # Create a user and a laptop for the user
        self._create_user_and_laptop()

        # Sample a configuration
        config = self.fixed_config if self.fixed_config else self._get_config()

        # Get the task description
        self.short_description = (
            f"Find the warranty expiration date for {self.user_full_name}'s laptop"
        )
        self.task_description = f'Refer to company protocol "{self.protocol_name}" (located in the "Company Protocols" knowledge base) for the steps to find the Warranty expiration date for {self.user_full_name}\'s laptop.\n \n'

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-requested-items-and-order-developer-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-requested-items-and-order-developer-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-requested-items-and-order-ipad-mini-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-requested-items-and-order-ipad-mini-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-requested-items-and-order-ipad-pro-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-requested-items-and-order-ipad-pro-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-requested-items-and-order-sales-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-requested-items-and-order-sales-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-requested-items-and-order-standard-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-requested-items-and-order-standard-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-requested-items-and-order-apple-watch-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-requested-items-and-order-apple-watch-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-requested-items-and-order-apple-mac-book-pro15-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-requested-items-and-order-apple-mac-book-pro15-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-requested-items-and-order-development-laptop-p-c-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-requested-items-and-order-development-laptop-p-c-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-requested-items-and-order-loaner-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-requested-items-and-order-loaner-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.basic-expense-management-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.basic-expense-management-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.date-based-expense-management-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.date-based-expense-management-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.amount-based-expense-management-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.amount-based-expense-management-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.easy-expense-management-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.easy-expense-management-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.basic-expense-management-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.basic-expense-management-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.date-based-expense-management-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.date-based-expense-management-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.amount-based-expense-management-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.amount-based-expense-management-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.easy-expense-management-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.easy-expense-management-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.basic-expense-management-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.basic-expense-management-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.date-based-expense-management-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.date-based-expense-management-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.amount-based-expense-management-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.amount-based-expense-management-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.easy-expense-management-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.easy-expense-management-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-random-expenses-and-find-total-return-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-random-expenses-and-find-total-return-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-random-expenses-and-find-total-return-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-random-expenses-and-find-total-return-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-random-expenses-and-find-total-return-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-random-expenses-and-find-total-return-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-trivial-expenses-and-find-total-return-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-trivial-expenses-and-find-total-return-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-trivial-expenses-and-find-total-return-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-trivial-expenses-and-find-total-return-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-trivial-expenses-and-find-total-return-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-trivial-expenses-and-find-total-return-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-expenses-and-find-total-return-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-expenses-and-find-total-return-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-expenses-and-find-total-return-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-expenses-and-find-total-return-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-expenses-and-find-total-return-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-expenses-and-find-total-return-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-uniform-expenses-and-find-total-return-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-uniform-expenses-and-find-total-return-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-uniform-expenses-and-find-total-return-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-uniform-expenses-and-find-total-return-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-uniform-expenses-and-find-total-return-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-uniform-expenses-and-find-total-return-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-two-items-uniform-expenses-and-find-total-return-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-two-items-uniform-expenses-and-find-total-return-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-three-items-uniform-expenses-and-find-total-return-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-three-items-uniform-expenses-and-find-total-return-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-three-items-uniform-expenses-and-find-total-return-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-three-items-uniform-expenses-and-find-total-return-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-random-expenses-and-select-investments-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-random-expenses-and-select-investments-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-random-expenses-and-select-investments-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-random-expenses-and-select-investments-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-random-expenses-and-select-investments-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-random-expenses-and-select-investments-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-trivial-expenses-and-select-investments-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-trivial-expenses-and-select-investments-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-trivial-expenses-and-select-investments-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-trivial-expenses-and-select-investments-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-trivial-expenses-and-select-investments-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-trivial-expenses-and-select-investments-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-expenses-and-select-investments-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-expenses-and-select-investments-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-expenses-and-select-investments-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-expenses-and-select-investments-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-expenses-and-select-investments-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-expenses-and-select-investments-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-uniform-expenses-and-select-investments-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-uniform-expenses-and-select-investments-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-uniform-expenses-and-select-investments-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-uniform-expenses-and-select-investments-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-uniform-expenses-and-select-investments-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-uniform-expenses-and-select-investments-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-two-items-uniform-expenses-and-select-investments-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-two-items-uniform-expenses-and-select-investments-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-three-items-uniform-expenses-and-select-investments-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-three-items-uniform-expenses-and-select-investments-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-three-items-uniform-expenses-and-select-investments-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-three-items-uniform-expenses-and-select-investments-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-random-expenses-find-total-return-and-select-investments-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-random-expenses-find-total-return-and-select-investments-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-random-expenses-find-total-return-and-select-investments-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-random-expenses-find-total-return-and-select-investments-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-random-expenses-find-total-return-and-select-investments-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-random-expenses-find-total-return-and-select-investments-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-trivial-expenses-find-total-return-and-select-investments-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-trivial-expenses-find-total-return-and-select-investments-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-trivial-expenses-find-total-return-and-select-investments-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-trivial-expenses-find-total-return-and-select-investments-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-trivial-expenses-find-total-return-and-select-investments-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-trivial-expenses-find-total-return-and-select-investments-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-expenses-find-total-return-and-select-investments-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-expenses-find-total-return-and-select-investments-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-expenses-find-total-return-and-select-investments-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-expenses-find-total-return-and-select-investments-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-expenses-find-total-return-and-select-investments-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-expenses-find-total-return-and-select-investments-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-uniform-expenses-find-total-return-and-select-investments-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-uniform-expenses-find-total-return-and-select-investments-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-uniform-expenses-find-total-return-and-select-investments-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-uniform-expenses-find-total-return-and-select-investments-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-uniform-expenses-find-total-return-and-select-investments-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-uniform-expenses-find-total-return-and-select-investments-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-two-items-uniform-expenses-find-total-return-and-select-investments-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-two-items-uniform-expenses-find-total-return-and-select-investments-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-three-items-uniform-expenses-find-total-return-and-select-investments-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-three-items-uniform-expenses-find-total-return-and-select-investments-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-three-items-uniform-expenses-find-total-return-and-select-investments-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-three-items-uniform-expenses-find-total-return-and-select-investments-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-random-expenses-and-delete-wrong-investments-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-random-expenses-and-delete-wrong-investments-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-random-expenses-and-delete-wrong-investments-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-random-expenses-and-delete-wrong-investments-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-random-expenses-and-delete-wrong-investments-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-random-expenses-and-delete-wrong-investments-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-expenses-and-delete-wrong-investments-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-expenses-and-delete-wrong-investments-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-expenses-and-delete-wrong-investments-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-expenses-and-delete-wrong-investments-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-expenses-and-delete-wrong-investments-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-expenses-and-delete-wrong-investments-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-uniform-expenses-and-delete-wrong-investments-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-uniform-expenses-and-delete-wrong-investments-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-uniform-expenses-and-delete-wrong-investments-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-uniform-expenses-and-delete-wrong-investments-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-uniform-expenses-and-delete-wrong-investments-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-single-item-uniform-expenses-and-delete-wrong-investments-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-two-items-uniform-expenses-and-delete-wrong-investments-small-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-two-items-uniform-expenses-and-delete-wrong-investments-small-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-three-items-uniform-expenses-and-delete-wrong-investments-medium-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-three-items-uniform-expenses-and-delete-wrong-investments-medium-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-three-items-uniform-expenses-and-delete-wrong-investments-large-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.filter-three-items-uniform-expenses-and-delete-wrong-investments-large-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self._setup_list()
        config = self.fixed_config if self.fixed_config else self._get_config()

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mean-order-developer-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mean-order-developer-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mean-orderi-pad-mini-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mean-orderi-pad-mini-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mean-orderi-pad-pro-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mean-orderi-pad-pro-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mean-order-sales-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mean-order-sales-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mean-order-standard-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mean-order-standard-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mean-order-apple-watch-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mean-order-apple-watch-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mean-order-apple-macbook-pro15-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mean-order-apple-macbook-pro15-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mean-order-development-laptop-p-c-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mean-order-development-laptop-p-c-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mean-order-loaner-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mean-order-loaner-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-median-order-developer-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-median-order-developer-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-median-orderi-pad-mini-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-median-orderi-pad-mini-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-median-orderi-pad-pro-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-median-orderi-pad-pro-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-median-order-sales-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-median-order-sales-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-median-order-standard-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-median-order-standard-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-median-order-apple-watch-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-median-order-apple-watch-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-median-order-apple-macbook-pro15-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-median-order-apple-macbook-pro15-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-median-order-development-laptop-p-c-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-median-order-development-laptop-p-c-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-median-order-loaner-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-median-order-loaner-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mode-order-developer-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mode-order-developer-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mode-orderi-pad-mini-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mode-orderi-pad-mini-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mode-orderi-pad-pro-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mode-orderi-pad-pro-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mode-order-sales-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mode-order-sales-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mode-order-standard-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mode-order-standard-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mode-order-apple-watch-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mode-order-apple-watch-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mode-order-apple-macbook-pro15-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mode-order-apple-macbook-pro15-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mode-order-development-laptop-p-c-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mode-order-development-laptop-p-c-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mode-order-loaner-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-catalog-and-mode-order-loaner-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of all the items in stock.\n\n"
                + f"\t - Task: Place an order for the least available item in stock. The quantity of the order should be such that the final quantity of this item matches the above retrieved value.\n"
                + f"\t   For example, consider the above task asks you to retrieve the maximum number of items in stock, say 4, and the least available item is an Apple Watch and its quantity is 1. You have to order 3 more Apple Watches.\n\n"
                + f"\t - Please do not change any other configuration while placing the order for the item. You can find important links to the pages in the protocol article.\n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page.\n"
                + f"\n2. Given the title of the report, search for it on this page.\n"
                + f"\n3. Find the value which is the {self.description_mapping[self.question]} of the items present in stock as per the chart. Also remember the least available item in the stock.\n"
                + f"\n4. Navigate to Self-Service > Service Catalog. \n"
                + f"\n5. For the least available item in stock, place an order for extra items such that its quantity matches the value you found."
                + "\nFor example, if you were requested to find the maximum value across the items, you would place an order for the least available item such that its NEW quantity matches this number. Please do not change any 'configuration' when placing the order.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-create-incident-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        incident_numbers_string = ", ".join(self.incident_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Using this value, retrieve agents that have less than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'incident' with the following information (only fill these fields and the 'incident number' from below) and assign it to them using the 'Assigned to' field: \n"
                + f"\t\t Category: 'Inquiry / Help', Impact: '1 - High', Urgency: '1 - High', Short description: 'Compulsory training for employee in probation', Caller: '{self.user_name}'. Make sure to use an 'incident number' from the list as described below.\n"
                + f"\t   Importantly, you should override the default incident numbers in the form and instead use one incident number from the following list for each incident you create: {incident_numbers_string}.\n\n"
                + f"Note that you will create as many incidents as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' new incidents here, one assigned to each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned less than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Service Desk > Incidents. \n"
                + f"\n5. You have to create new 'incidents' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new incident:- Category: 'Inquiry / Help', Impact: '1 - High', Urgency: '1 - High', Short description: 'Compulsory training for employee in probation', Caller: '{self.user_name}' and 'assign' them to each agent using the 'Assigned to' field.\n\n"
                + f"Importantly, you should override the default incident numbers in the form and instead use one incident number from the following list for each incident you create: {incident_numbers_string}.\n"
                + f"Note that you will create as many incidents as there are agents matching the above criteria."
                + "\nFor example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' new incidents here, one assigned to each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-create-incident-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        incident_numbers_string = ", ".join(self.incident_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Using this value, retrieve agents that have less than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'incident' with the following information (only fill these fields and the 'incident number' from below) and assign it to them using the 'Assigned to' field: \n"
                + f"\t\t Category: 'Inquiry / Help', Impact: '1 - High', Urgency: '1 - High', Short description: 'Compulsory training for employee in probation', Caller: '{self.user_name}'. Make sure to use an 'incident number' from the list as described below.\n"
                + f"\t   Importantly, you should override the default incident numbers in the form and instead use one incident number from the following list for each incident you create: {incident_numbers_string}.\n\n"
                + f"Note that you will create as many incidents as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' new incidents here, one assigned to each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned less than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Service Desk > Incidents. \n"
                + f"\n5. You have to create new 'incidents' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new incident:- Category: 'Inquiry / Help', Impact: '1 - High', Urgency: '1 - High', Short description: 'Compulsory training for employee in probation', Caller: '{self.user_name}' and 'assign' them to each agent using the 'Assigned to' field.\n\n"
                + f"Importantly, you should override the default incident numbers in the form and instead use one incident number from the following list for each incident you create: {incident_numbers_string}.\n"
                + f"Note that you will create as many incidents as there are agents matching the above criteria."
                + "\nFor example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' new incidents here, one assigned to each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-create-incident-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        incident_numbers_string = ", ".join(self.incident_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Using this value, retrieve agents that have less than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'incident' with the following information (only fill these fields and the 'incident number' from below) and assign it to them using the 'Assigned to' field: \n"
                + f"\t\t Category: 'Inquiry / Help', Impact: '1 - High', Urgency: '1 - High', Short description: 'Compulsory training for employee in probation', Caller: '{self.user_name}'. Make sure to use an 'incident number' from the list as described below.\n"
                + f"\t   Importantly, you should override the default incident numbers in the form and instead use one incident number from the following list for each incident you create: {incident_numbers_string}.\n\n"
                + f"Note that you will create as many incidents as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' new incidents here, one assigned to each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned less than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Service Desk > Incidents. \n"
                + f"\n5. You have to create new 'incidents' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new incident:- Category: 'Inquiry / Help', Impact: '1 - High', Urgency: '1 - High', Short description: 'Compulsory training for employee in probation', Caller: '{self.user_name}' and 'assign' them to each agent using the 'Assigned to' field.\n\n"
                + f"Importantly, you should override the default incident numbers in the form and instead use one incident number from the following list for each incident you create: {incident_numbers_string}.\n"
                + f"Note that you will create as many incidents as there are agents matching the above criteria."
                + "\nFor example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' new incidents here, one assigned to each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-create-incident-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        incident_numbers_string = ", ".join(self.incident_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Using this value, retrieve agents that have less than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'incident' with the following information (only fill these fields and the 'incident number' from below) and assign it to them using the 'Assigned to' field: \n"
                + f"\t\t Category: 'Inquiry / Help', Impact: '1 - High', Urgency: '1 - High', Short description: 'Compulsory training for employee in probation', Caller: '{self.user_name}'. Make sure to use an 'incident number' from the list as described below.\n"
                + f"\t   Importantly, you should override the default incident numbers in the form and instead use one incident number from the following list for each incident you create: {incident_numbers_string}.\n\n"
                + f"Note that you will create as many incidents as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' new incidents here, one assigned to each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned less than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Service Desk > Incidents. \n"
                + f"\n5. You have to create new 'incidents' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new incident:- Category: 'Inquiry / Help', Impact: '1 - High', Urgency: '1 - High', Short description: 'Compulsory training for employee in probation', Caller: '{self.user_name}' and 'assign' them to each agent using the 'Assigned to' field.\n\n"
                + f"Importantly, you should override the default incident numbers in the form and instead use one incident number from the following list for each incident you create: {incident_numbers_string}.\n"
                + f"Note that you will create as many incidents as there are agents matching the above criteria."
                + "\nFor example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' new incidents here, one assigned to each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-create-incident-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        incident_numbers_string = ", ".join(self.incident_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Using this value, retrieve agents that have less than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'incident' with the following information (only fill these fields and the 'incident number' from below) and assign it to them using the 'Assigned to' field: \n"
                + f"\t\t Category: 'Inquiry / Help', Impact: '1 - High', Urgency: '1 - High', Short description: 'Compulsory training for employee in probation', Caller: '{self.user_name}'. Make sure to use an 'incident number' from the list as described below.\n"
                + f"\t   Importantly, you should override the default incident numbers in the form and instead use one incident number from the following list for each incident you create: {incident_numbers_string}.\n\n"
                + f"Note that you will create as many incidents as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' new incidents here, one assigned to each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned less than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Service Desk > Incidents. \n"
                + f"\n5. You have to create new 'incidents' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new incident:- Category: 'Inquiry / Help', Impact: '1 - High', Urgency: '1 - High', Short description: 'Compulsory training for employee in probation', Caller: '{self.user_name}' and 'assign' them to each agent using the 'Assigned to' field.\n\n"
                + f"Importantly, you should override the default incident numbers in the form and instead use one incident number from the following list for each incident you create: {incident_numbers_string}.\n"
                + f"Note that you will create as many incidents as there are agents matching the above criteria."
                + "\nFor example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' new incidents here, one assigned to each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-create-incident-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        incident_numbers_string = ", ".join(self.incident_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Using this value, retrieve agents that have less than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'incident' with the following information (only fill these fields and the 'incident number' from below) and assign it to them using the 'Assigned to' field: \n"
                + f"\t\t Category: 'Inquiry / Help', Impact: '1 - High', Urgency: '1 - High', Short description: 'Compulsory training for employee in probation', Caller: '{self.user_name}'. Make sure to use an 'incident number' from the list as described below.\n"
                + f"\t   Importantly, you should override the default incident numbers in the form and instead use one incident number from the following list for each incident you create: {incident_numbers_string}.\n\n"
                + f"Note that you will create as many incidents as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' new incidents here, one assigned to each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned less than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Service Desk > Incidents. \n"
                + f"\n5. You have to create new 'incidents' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new incident:- Category: 'Inquiry / Help', Impact: '1 - High', Urgency: '1 - High', Short description: 'Compulsory training for employee in probation', Caller: '{self.user_name}' and 'assign' them to each agent using the 'Assigned to' field.\n\n"
                + f"Importantly, you should override the default incident numbers in the form and instead use one incident number from the following list for each incident you create: {incident_numbers_string}.\n"
                + f"Note that you will create as many incidents as there are agents matching the above criteria."
                + "\nFor example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' new incidents here, one assigned to each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-create-problem-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report(
            user_roles=[
                "itil",
                "problem_admin",
                "problem_manager",
                "problem_coordinator",
                "problem_task_analyst",
            ]
        )
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have less than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create a 'problem' with the following information (only fill these fields) and assign it to them using the 'Assigned to' field: \n"
                + f"\t\t Impact: '1 - High', Urgency: '1 - High', Problem statement: 'Compulsory training for employee in probation'.\n"
                + f"Note that you will create as many problems as there are agents matching the above criteria.\n"
                + f"\t For example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' problems here, one assigned to each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned less than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to All > Problems. \n"
                + f"\n5. You have to create new 'problems' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new problem:- Impact: '1 - High', Urgency: '1 - High', Problem statement: 'Compulsory training for employee in probation' and 'assign' them to each agent.\n\n"
                + f"Note that you will create as many problems as there are agents matching the above criteria."
                + "\nFor example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' problems here, one assigned to each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-create-problem-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report(
            user_roles=[
                "itil",
                "problem_admin",
                "problem_manager",
                "problem_coordinator",
                "problem_task_analyst",
            ]
        )
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have less than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create a 'problem' with the following information (only fill these fields) and assign it to them using the 'Assigned to' field: \n"
                + f"\t\t Impact: '1 - High', Urgency: '1 - High', Problem statement: 'Compulsory training for employee in probation'.\n"
                + f"Note that you will create as many problems as there are agents matching the above criteria.\n"
                + f"\t For example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' problems here, one assigned to each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned less than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to All > Problems. \n"
                + f"\n5. You have to create new 'problems' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new problem:- Impact: '1 - High', Urgency: '1 - High', Problem statement: 'Compulsory training for employee in probation' and 'assign' them to each agent.\n\n"
                + f"Note that you will create as many problems as there are agents matching the above criteria."
                + "\nFor example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' problems here, one assigned to each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-create-problem-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report(
            user_roles=[
                "itil",
                "problem_admin",
                "problem_manager",
                "problem_coordinator",
                "problem_task_analyst",
            ]
        )
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have less than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create a 'problem' with the following information (only fill these fields) and assign it to them using the 'Assigned to' field: \n"
                + f"\t\t Impact: '1 - High', Urgency: '1 - High', Problem statement: 'Compulsory training for employee in probation'.\n"
                + f"Note that you will create as many problems as there are agents matching the above criteria.\n"
                + f"\t For example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' problems here, one assigned to each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned less than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to All > Problems. \n"
                + f"\n5. You have to create new 'problems' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new problem:- Impact: '1 - High', Urgency: '1 - High', Problem statement: 'Compulsory training for employee in probation' and 'assign' them to each agent.\n\n"
                + f"Note that you will create as many problems as there are agents matching the above criteria."
                + "\nFor example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' problems here, one assigned to each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-create-problem-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report(
            user_roles=[
                "itil",
                "problem_admin",
                "problem_manager",
                "problem_coordinator",
                "problem_task_analyst",
            ]
        )
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have less than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create a 'problem' with the following information (only fill these fields) and assign it to them using the 'Assigned to' field: \n"
                + f"\t\t Impact: '1 - High', Urgency: '1 - High', Problem statement: 'Compulsory training for employee in probation'.\n"
                + f"Note that you will create as many problems as there are agents matching the above criteria.\n"
                + f"\t For example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' problems here, one assigned to each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned less than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to All > Problems. \n"
                + f"\n5. You have to create new 'problems' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new problem:- Impact: '1 - High', Urgency: '1 - High', Problem statement: 'Compulsory training for employee in probation' and 'assign' them to each agent.\n\n"
                + f"Note that you will create as many problems as there are agents matching the above criteria."
                + "\nFor example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' problems here, one assigned to each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-create-problem-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report(
            user_roles=[
                "itil",
                "problem_admin",
                "problem_manager",
                "problem_coordinator",
                "problem_task_analyst",
            ]
        )
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have less than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create a 'problem' with the following information (only fill these fields) and assign it to them using the 'Assigned to' field: \n"
                + f"\t\t Impact: '1 - High', Urgency: '1 - High', Problem statement: 'Compulsory training for employee in probation'.\n"
                + f"Note that you will create as many problems as there are agents matching the above criteria.\n"
                + f"\t For example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' problems here, one assigned to each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned less than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to All > Problems. \n"
                + f"\n5. You have to create new 'problems' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new problem:- Impact: '1 - High', Urgency: '1 - High', Problem statement: 'Compulsory training for employee in probation' and 'assign' them to each agent.\n\n"
                + f"Note that you will create as many problems as there are agents matching the above criteria."
                + "\nFor example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' problems here, one assigned to each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-create-problem-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report(
            user_roles=[
                "itil",
                "problem_admin",
                "problem_manager",
                "problem_coordinator",
                "problem_task_analyst",
            ]
        )
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have less than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create a 'problem' with the following information (only fill these fields) and assign it to them using the 'Assigned to' field: \n"
                + f"\t\t Impact: '1 - High', Urgency: '1 - High', Problem statement: 'Compulsory training for employee in probation'.\n"
                + f"Note that you will create as many problems as there are agents matching the above criteria.\n"
                + f"\t For example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' problems here, one assigned to each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned less than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have less than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to All > Problems. \n"
                + f"\n5. You have to create new 'problems' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new problem:- Impact: '1 - High', Urgency: '1 - High', Problem statement: 'Compulsory training for employee in probation' and 'assign' them to each agent.\n\n"
                + f"Note that you will create as many problems as there are agents matching the above criteria."
                + "\nFor example, consider the above case and say you have 3 agents with less than or equal to 2 incidents assigned to them in the chart. You will be creating '3' problems here, one assigned to each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-greater-filter-asset-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        # We create dummy assets for the consumable and license categories here
        ### NOTE: We can create assets without any of the following information, save time, and still assign them to the user. The task should be fine.
        consumable_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_consumable", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        consumables = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={consumable_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        consumables_sysids = [consumable["sys_id"] for consumable in consumables]

        license_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_license", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        licenses = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={license_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"][:10]
        license_sysids = [license["sys_id"] for license in licenses]

        self.new_asset_sys_ids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                consumable_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(consumables_sysids),
                    "model_category": consumable_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=consumable_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])
                license_asset_data = {
                    "asset_tag": "LICENSE" + str(random.randint(100, 999)),
                    "model": self.random.choice(license_sysids),
                    "model_category": license_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=license_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Asset List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > All Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > All Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-greater-filter-asset-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        # We create dummy assets for the consumable and license categories here
        ### NOTE: We can create assets without any of the following information, save time, and still assign them to the user. The task should be fine.
        consumable_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_consumable", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        consumables = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={consumable_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        consumables_sysids = [consumable["sys_id"] for consumable in consumables]

        license_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_license", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        licenses = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={license_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"][:10]
        license_sysids = [license["sys_id"] for license in licenses]

        self.new_asset_sys_ids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                consumable_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(consumables_sysids),
                    "model_category": consumable_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=consumable_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])
                license_asset_data = {
                    "asset_tag": "LICENSE" + str(random.randint(100, 999)),
                    "model": self.random.choice(license_sysids),
                    "model_category": license_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=license_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Asset List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > All Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > All Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-greater-filter-hardware-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        hardware_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_hardware", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        hardwares = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={hardware_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        hardware_sysids = [hardware["sys_id"] for hardware in hardwares]
        self.new_asset_sysids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                hardware_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(hardware_sysids),
                    "model_category": hardware_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_hardware",
                    json=hardware_asset_data,
                    method="POST",
                )
                self.new_asset_sysids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Hardware List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > Hardware Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > Hardware Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-greater-filter-hardware-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        hardware_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_hardware", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        hardwares = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={hardware_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        hardware_sysids = [hardware["sys_id"] for hardware in hardwares]
        self.new_asset_sysids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                hardware_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(hardware_sysids),
                    "model_category": hardware_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_hardware",
                    json=hardware_asset_data,
                    method="POST",
                )
                self.new_asset_sysids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Hardware List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > Hardware Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > Hardware Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-greater-filter-incident-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Incident List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Service Desk > Incidents. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Service Desk > Incidents. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-greater-filter-incident-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Incident List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Service Desk > Incidents. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Service Desk > Incidents. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-greater-filter-user-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the User List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Organization > Users. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Organization > Users. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-greater-filter-user-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the User List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Organization > Users. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Organization > Users. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-lesser-filter-asset-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        # We create dummy assets for the consumable and license categories here
        ### NOTE: We can create assets without any of the following information, save time, and still assign them to the user. The task should be fine.
        consumable_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_consumable", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        consumables = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={consumable_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        consumables_sysids = [consumable["sys_id"] for consumable in consumables]

        license_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_license", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        licenses = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={license_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"][:10]
        license_sysids = [license["sys_id"] for license in licenses]

        self.new_asset_sys_ids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                consumable_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(consumables_sysids),
                    "model_category": consumable_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=consumable_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])
                license_asset_data = {
                    "asset_tag": "LICENSE" + str(random.randint(100, 999)),
                    "model": self.random.choice(license_sysids),
                    "model_category": license_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=license_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Asset List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > All Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > All Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-lesser-filter-asset-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        # We create dummy assets for the consumable and license categories here
        ### NOTE: We can create assets without any of the following information, save time, and still assign them to the user. The task should be fine.
        consumable_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_consumable", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        consumables = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={consumable_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        consumables_sysids = [consumable["sys_id"] for consumable in consumables]

        license_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_license", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        licenses = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={license_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"][:10]
        license_sysids = [license["sys_id"] for license in licenses]

        self.new_asset_sys_ids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                consumable_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(consumables_sysids),
                    "model_category": consumable_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=consumable_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])
                license_asset_data = {
                    "asset_tag": "LICENSE" + str(random.randint(100, 999)),
                    "model": self.random.choice(license_sysids),
                    "model_category": license_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=license_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Asset List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > All Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > All Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-lesser-filter-hardware-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        hardware_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_hardware", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        hardwares = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={hardware_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        hardware_sysids = [hardware["sys_id"] for hardware in hardwares]
        self.new_asset_sysids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                hardware_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(hardware_sysids),
                    "model_category": hardware_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_hardware",
                    json=hardware_asset_data,
                    method="POST",
                )
                self.new_asset_sysids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Hardware List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > Hardware Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > Hardware Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-lesser-filter-hardware-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        hardware_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_hardware", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        hardwares = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={hardware_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        hardware_sysids = [hardware["sys_id"] for hardware in hardwares]
        self.new_asset_sysids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                hardware_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(hardware_sysids),
                    "model_category": hardware_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_hardware",
                    json=hardware_asset_data,
                    method="POST",
                )
                self.new_asset_sysids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Hardware List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > Hardware Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > Hardware Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-lesser-filter-incident-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Incident List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Service Desk > Incidents. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Service Desk > Incidents. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-lesser-filter-incident-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Incident List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Service Desk > Incidents. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Service Desk > Incidents. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-lesser-filter-user-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the User List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Organization > Users. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Organization > Users. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-lesser-filter-user-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the User List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Organization > Users. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Organization > Users. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-greater-filter-asset-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        # We create dummy assets for the consumable and license categories here
        ### NOTE: We can create assets without any of the following information, save time, and still assign them to the user. The task should be fine.
        consumable_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_consumable", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        consumables = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={consumable_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        consumables_sysids = [consumable["sys_id"] for consumable in consumables]

        license_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_license", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        licenses = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={license_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"][:10]
        license_sysids = [license["sys_id"] for license in licenses]

        self.new_asset_sys_ids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                consumable_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(consumables_sysids),
                    "model_category": consumable_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=consumable_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])
                license_asset_data = {
                    "asset_tag": "LICENSE" + str(random.randint(100, 999)),
                    "model": self.random.choice(license_sysids),
                    "model_category": license_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=license_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Asset List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > All Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > All Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-greater-filter-asset-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        # We create dummy assets for the consumable and license categories here
        ### NOTE: We can create assets without any of the following information, save time, and still assign them to the user. The task should be fine.
        consumable_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_consumable", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        consumables = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={consumable_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        consumables_sysids = [consumable["sys_id"] for consumable in consumables]

        license_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_license", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        licenses = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={license_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"][:10]
        license_sysids = [license["sys_id"] for license in licenses]

        self.new_asset_sys_ids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                consumable_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(consumables_sysids),
                    "model_category": consumable_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=consumable_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])
                license_asset_data = {
                    "asset_tag": "LICENSE" + str(random.randint(100, 999)),
                    "model": self.random.choice(license_sysids),
                    "model_category": license_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=license_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Asset List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > All Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > All Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-lesser-filter-asset-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        # We create dummy assets for the consumable and license categories here
        ### NOTE: We can create assets without any of the following information, save time, and still assign them to the user. The task should be fine.
        consumable_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_consumable", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        consumables = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={consumable_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        consumables_sysids = [consumable["sys_id"] for consumable in consumables]

        license_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_license", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        licenses = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={license_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"][:10]
        license_sysids = [license["sys_id"] for license in licenses]

        self.new_asset_sys_ids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                consumable_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(consumables_sysids),
                    "model_category": consumable_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=consumable_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])
                license_asset_data = {
                    "asset_tag": "LICENSE" + str(random.randint(100, 999)),
                    "model": self.random.choice(license_sysids),
                    "model_category": license_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=license_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Asset List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > All Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > All Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-lesser-filter-asset-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        # We create dummy assets for the consumable and license categories here
        ### NOTE: We can create assets without any of the following information, save time, and still assign them to the user. The task should be fine.
        consumable_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_consumable", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        consumables = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={consumable_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        consumables_sysids = [consumable["sys_id"] for consumable in consumables]

        license_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_license", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        licenses = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={license_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"][:10]
        license_sysids = [license["sys_id"] for license in licenses]

        self.new_asset_sys_ids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                consumable_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(consumables_sysids),
                    "model_category": consumable_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=consumable_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])
                license_asset_data = {
                    "asset_tag": "LICENSE" + str(random.randint(100, 999)),
                    "model": self.random.choice(license_sysids),
                    "model_category": license_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=license_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Asset List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > All Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > All Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-greater-filter-hardware-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        hardware_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_hardware", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        hardwares = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={hardware_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        hardware_sysids = [hardware["sys_id"] for hardware in hardwares]
        self.new_asset_sysids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                hardware_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(hardware_sysids),
                    "model_category": hardware_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_hardware",
                    json=hardware_asset_data,
                    method="POST",
                )
                self.new_asset_sysids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Hardware List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > Hardware Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > Hardware Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-greater-filter-hardware-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        hardware_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_hardware", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        hardwares = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={hardware_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        hardware_sysids = [hardware["sys_id"] for hardware in hardwares]
        self.new_asset_sysids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                hardware_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(hardware_sysids),
                    "model_category": hardware_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_hardware",
                    json=hardware_asset_data,
                    method="POST",
                )
                self.new_asset_sysids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Hardware List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > Hardware Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > Hardware Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-lesser-filter-hardware-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        hardware_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_hardware", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        hardwares = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={hardware_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        hardware_sysids = [hardware["sys_id"] for hardware in hardwares]
        self.new_asset_sysids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                hardware_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(hardware_sysids),
                    "model_category": hardware_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_hardware",
                    json=hardware_asset_data,
                    method="POST",
                )
                self.new_asset_sysids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Hardware List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > Hardware Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > Hardware Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-lesser-filter-hardware-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        hardware_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_hardware", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        hardwares = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={hardware_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        hardware_sysids = [hardware["sys_id"] for hardware in hardwares]
        self.new_asset_sysids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                hardware_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(hardware_sysids),
                    "model_category": hardware_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_hardware",
                    json=hardware_asset_data,
                    method="POST",
                )
                self.new_asset_sysids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Hardware List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > Hardware Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > Hardware Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-greater-filter-incident-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Incident List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Service Desk > Incidents. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Service Desk > Incidents. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-greater-filter-incident-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Incident List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Service Desk > Incidents. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Service Desk > Incidents. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-lesser-filter-incident-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Incident List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Service Desk > Incidents. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Service Desk > Incidents. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-lesser-filter-incident-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Incident List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Service Desk > Incidents. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Service Desk > Incidents. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-greater-filter-user-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the User List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Organization > Users. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Organization > Users. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-greater-filter-user-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the User List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Organization > Users. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Organization > Users. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-lesser-filter-user-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the User List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Organization > Users. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Organization > Users. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-lesser-filter-user-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the User List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Organization > Users. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Organization > Users. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-greater-filter-asset-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        # We create dummy assets for the consumable and license categories here
        ### NOTE: We can create assets without any of the following information, save time, and still assign them to the user. The task should be fine.
        consumable_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_consumable", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        consumables = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={consumable_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        consumables_sysids = [consumable["sys_id"] for consumable in consumables]

        license_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_license", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        licenses = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={license_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"][:10]
        license_sysids = [license["sys_id"] for license in licenses]

        self.new_asset_sys_ids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                consumable_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(consumables_sysids),
                    "model_category": consumable_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=consumable_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])
                license_asset_data = {
                    "asset_tag": "LICENSE" + str(random.randint(100, 999)),
                    "model": self.random.choice(license_sysids),
                    "model_category": license_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=license_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Asset List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > All Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > All Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-greater-filter-asset-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        # We create dummy assets for the consumable and license categories here
        ### NOTE: We can create assets without any of the following information, save time, and still assign them to the user. The task should be fine.
        consumable_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_consumable", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        consumables = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={consumable_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        consumables_sysids = [consumable["sys_id"] for consumable in consumables]

        license_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_license", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        licenses = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={license_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"][:10]
        license_sysids = [license["sys_id"] for license in licenses]

        self.new_asset_sys_ids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                consumable_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(consumables_sysids),
                    "model_category": consumable_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=consumable_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])
                license_asset_data = {
                    "asset_tag": "LICENSE" + str(random.randint(100, 999)),
                    "model": self.random.choice(license_sysids),
                    "model_category": license_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=license_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Asset List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > All Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > All Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-lesser-filter-asset-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        # We create dummy assets for the consumable and license categories here
        ### NOTE: We can create assets without any of the following information, save time, and still assign them to the user. The task should be fine.
        consumable_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_consumable", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        consumables = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={consumable_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        consumables_sysids = [consumable["sys_id"] for consumable in consumables]

        license_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_license", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        licenses = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={license_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"][:10]
        license_sysids = [license["sys_id"] for license in licenses]

        self.new_asset_sys_ids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                consumable_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(consumables_sysids),
                    "model_category": consumable_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=consumable_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])
                license_asset_data = {
                    "asset_tag": "LICENSE" + str(random.randint(100, 999)),
                    "model": self.random.choice(license_sysids),
                    "model_category": license_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=license_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Asset List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > All Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > All Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-lesser-filter-asset-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        # We create dummy assets for the consumable and license categories here
        ### NOTE: We can create assets without any of the following information, save time, and still assign them to the user. The task should be fine.
        consumable_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_consumable", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        consumables = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={consumable_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        consumables_sysids = [consumable["sys_id"] for consumable in consumables]

        license_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_license", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        licenses = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={license_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"][:10]
        license_sysids = [license["sys_id"] for license in licenses]

        self.new_asset_sys_ids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                consumable_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(consumables_sysids),
                    "model_category": consumable_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=consumable_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])
                license_asset_data = {
                    "asset_tag": "LICENSE" + str(random.randint(100, 999)),
                    "model": self.random.choice(license_sysids),
                    "model_category": license_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_asset",
                    json=license_asset_data,
                    method="POST",
                )
                self.new_asset_sys_ids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Asset List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > All Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > All Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-greater-filter-hardware-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        hardware_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_hardware", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        hardwares = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={hardware_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        hardware_sysids = [hardware["sys_id"] for hardware in hardwares]
        self.new_asset_sysids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                hardware_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(hardware_sysids),
                    "model_category": hardware_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_hardware",
                    json=hardware_asset_data,
                    method="POST",
                )
                self.new_asset_sysids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Hardware List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > Hardware Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > Hardware Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-greater-filter-hardware-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        hardware_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_hardware", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        hardwares = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={hardware_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        hardware_sysids = [hardware["sys_id"] for hardware in hardwares]
        self.new_asset_sysids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                hardware_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(hardware_sysids),
                    "model_category": hardware_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_hardware",
                    json=hardware_asset_data,
                    method="POST",
                )
                self.new_asset_sysids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Hardware List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > Hardware Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > Hardware Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-lesser-filter-hardware-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        hardware_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_hardware", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        hardwares = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={hardware_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        hardware_sysids = [hardware["sys_id"] for hardware in hardwares]
        self.new_asset_sysids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                hardware_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(hardware_sysids),
                    "model_category": hardware_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_hardware",
                    json=hardware_asset_data,
                    method="POST",
                )
                self.new_asset_sysids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Hardware List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > Hardware Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > Hardware Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-lesser-filter-hardware-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()

        hardware_category_sysid = table_api_call(
            instance=self.instance,
            table="cmdb_model_category",
            method="GET",
            params={"sysparm_query": "asset_class=alm_hardware", "sysparm_fields": "sys_id"},
        )["result"][0]["sys_id"]

        hardwares = table_api_call(
            instance=self.instance,
            table="cmdb_model",
            method="GET",
            params={
                "sysparm_query": f"cmdb_model_category={hardware_category_sysid}",
                "sysparm_fields": "sys_id",
            },
        )["result"]
        hardware_sysids = [hardware["sys_id"] for hardware in hardwares]
        self.new_asset_sysids = []
        for agent_sysid in self.agent_sysids:
            num_assets = self.random.choice(range(1, self.max_assets_per_agent))
            for _ in range(num_assets):
                hardware_asset_data = {
                    "asset_tag": "CONSUMABLE" + str(random.randint(100, 999)),
                    "model": self.random.choice(hardware_sysids),
                    "model_category": hardware_category_sysid,
                    "assigned_to": agent_sysid,
                    "cost": 1000.00,
                    "purchase_date": "2024-05-08",
                    "substatus": "in_use",
                }
                response = table_api_call(
                    instance=self.instance,
                    table="alm_hardware",
                    json=hardware_asset_data,
                    method="POST",
                )
                self.new_asset_sysids.append(response["result"]["sys_id"])

        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Hardware List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Portfolios > Hardware Assets. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Portfolios > Hardware Assets. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-greater-filter-incident-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Incident List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Service Desk > Incidents. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Service Desk > Incidents. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-greater-filter-incident-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Incident List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Service Desk > Incidents. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Service Desk > Incidents. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-lesser-filter-incident-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Incident List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Service Desk > Incidents. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Service Desk > Incidents. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-lesser-filter-incident-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the Incident List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Service Desk > Incidents. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"\n3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"\n3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Service Desk > Incidents. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-greater-filter-user-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the User List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Organization > Users. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Organization > Users. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-greater-filter-user-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the User List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Organization > Users. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Organization > Users. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-lesser-filter-user-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the User List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Organization > Users. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Organization > Users. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-lesser-filter-user-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()

        if self.level == 3:
            filter_than = f"{self.filter_than} than or " if self.filter_than else ""
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have {filter_than}equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have {filter_than}equal to 2 assigned incidents.\n\n"
                + f"\t - Task: Filter the User List using the {self.attribute_name} field corresponding to the agents that fit the criteria above. \n"
                + f"The list is present at Organization > Users. \n\n"
                + self.final_private_task_instructions
            )

        goal, info = super().setup_goal(
            page=page, config=config, build_pretty_print_description=False
        )

        if self.level == 2:
            if self.filter_than:
                step_3 = f"3. Find the agents with number of incidents {self.filter_than} than or equal to the {self.description_mapping[self.question]} value of the number of incidents assigned across agents. \n"
            else:
                step_3 = f"3. Find the agent with the {self.description_mapping[self.question]} assigned incidents. \n"
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + step_3
                + f"\n4. Navigate to Organization > Users. \n"
                + f"\nUsing the field {self.attribute_name} for the agent/ agents that fit the critera above, filter the list.\n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-request-apple-watch-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-request-apple-watch-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-request-apple-watch2-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-request-apple-watch2-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-request-apple-ipad3-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-request-apple-ipad3-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-request-apple-iphone13pro-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-request-apple-iphone13pro-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-request-apple-iphone13-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-request-apple-iphone13-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-request-galaxy-note20-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-request-galaxy-note20-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-request-google-nexus7-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-request-google-nexus7-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-request-microsoft-surface-pro3-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-request-microsoft-surface-pro3-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-request-pixel4a-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-request-pixel4a-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-request-windows-surface-pro4-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mean-request-windows-surface-pro4-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-request-apple-watch-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-request-apple-watch-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-request-apple-watch2-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-request-apple-watch2-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-request-apple-ipad3-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-request-apple-ipad3-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-request-apple-iphone13pro-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-request-apple-iphone13pro-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-request-apple-iphone13-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-request-apple-iphone13-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-request-galaxy-note20-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-request-galaxy-note20-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-request-google-nexus7-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-request-google-nexus7-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-request-microsoft-surface-pro3-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-request-microsoft-surface-pro3-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-request-pixel4a-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-request-pixel4a-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-request-windows-surface-pro4-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-median-request-windows-surface-pro4-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-request-apple-watch-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-request-apple-watch-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-request-apple-watch2-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-request-apple-watch2-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-request-apple-ipad3-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-request-apple-ipad3-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-request-apple-iphone13pro-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-request-apple-iphone13pro-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-request-apple-iphone13-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-request-apple-iphone13-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-request-galaxy-note20-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-request-galaxy-note20-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-request-google-nexus7-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-request-google-nexus7-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-request-microsoft-surface-pro3-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-request-microsoft-surface-pro3-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-request-pixel4a-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-request-pixel4a-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-request-windows-surface-pro4-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.dashboard-retrieve-incident-and-mode-request-windows-surface-pro4-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        self.create_report()
        self.set_compositional_task()
        config = self.fixed_config if self.fixed_config else self._get_config()
        requested_item_numbers_string = ", ".join(self.requested_item_numbers)

        if self.level == 3:
            self.task_description = (
                self.task_description
                + f"\t - Please retrieve the '{self.description_mapping[self.question]}' value of the number of incidents assigned across agents. Retrieve agents that have greater than or equal number of incidents assigned to them compared to this value.\n"
                + f"\t   For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n\n"
                + f"\t - Task: For each agent that fits the above criteria, create an 'item request' with the following information (only fill these fields and the 'request item number' from below) and 'request it' for them using the 'Requested for' field: \n"
                + f"\t\t Item: {self.item}, Quantity: 1, Requested for: <agent name>.\n"
                + f"\t   Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + f"\t   For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n\n"
            )

        goal, info = super().setup_goal(page=page, config=config)

        if self.level == 2:
            goal = (
                self.short_description
                + f"\n1. Navigate to the Reports > View/Run page. \n"
                + f"\n2. Given the title of the report, search for it on this page. The report shows the number of 'incidents' assigned to an 'agent'.\n"
                + f"\n3. Find the agents with a number of incidents assigned greater than or equal to the {self.description_mapping[self.question]} of the number of assigned incidents across agents. For example, if you were asked to find the 'mean' or 'average' for a case where there are 4 agents assigned 1,2,3,2 incidents respectively, the mean would be 2. The list of agents required for solving the following task would be all the agents that have greater than or equal to 2 assigned incidents.\n"
                + f"\n4. Navigate to Open Records > Items. \n"
                + f"\n5. You have to create new 'item requests' from this page for all the agents based on the above criteria. Only fill the following fields when creating a new item request:- Item: {self.item}, Quantity: 1 and 'request' them for each agent using the 'Requested For' field.\n\n"
                + f"Importantly, you should override the default request item numbers in the form and instead use one request item number from the following list for each item request you create: {requested_item_numbers_string}.\n"
                + f"Note that you will create as many item requests as there are agents matching the above criteria.\n"
                + "For example, consider the above case and say you have 3 agents with greater than or equal to 2 incidents assigned to them in the chart. You will be creating '3' item requests here, one for each agent. \n"
            )

        return goal, info
workarena.servicenow.navigate-and-create-user-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-create-user-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-create-incident-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-create-incident-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-create-change-request-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-create-change-request-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-create-problem-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-create-problem-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-create-hardware-asset-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-create-hardware-asset-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-order-standard-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-order-standard-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-order-sales-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-order-sales-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-order-developer-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-order-developer-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-order-ipad-pro-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-order-ipad-pro-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-order-ipad-mini-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-order-ipad-mini-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-order-apple-watch-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-order-apple-watch-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-order-apple-mac-book-pro15-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-order-apple-mac-book-pro15-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-order-development-laptop-p-c-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-order-development-laptop-p-c-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-order-loaner-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-order-loaner-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-filter-asset-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-filter-asset-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-filter-user-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-filter-user-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-filter-incident-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-filter-incident-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-filter-change-request-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-filter-change-request-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-filter-hardware-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-filter-hardware-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-filter-service-catalog-item-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-filter-service-catalog-item-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-sort-asset-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-sort-asset-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-sort-user-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-sort-user-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-sort-incident-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-sort-incident-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-sort-change-request-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-sort-change-request-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-sort-hardware-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-sort-hardware-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-sort-service-catalog-item-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.navigate-and-sort-service-catalog-item-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.off-board-user-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        # Generate random name for the user
        first_name = fake.first_name() + "-" + fake.first_name()
        last_name = fake.last_name() + "-" + fake.last_name()
        self.user_full_name = first_name + " " + last_name
        self.laptop_asset_tag = "P" + str(id(self) % (10**8)).zfill(8)

        # Create user
        self.user_name, _, self.user_sys_id = create_user(
            instance=self.instance, first_name=first_name, last_name=last_name, random=self.random
        )

        assert self.user_sys_id, f"Failed to create user {first_name} {last_name}"

        self.laptop_sys_id, _, _ = create_computer_asset(
            instance=self.instance,
            asset_tag=self.laptop_asset_tag,
            user_sys_id=self.user_sys_id,
            random=self.random,
        )

        config = self.fixed_config if self.fixed_config else self._get_config()
        # Get the task description
        self.short_description = f"Offboard user {self.user_full_name}"
        self.task_description = f'Referring to company protocol "{self.protocol_name}" (located in the "Company Protocols" knowledge base) offboard user "{self.user_full_name}" \n'

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.off-board-user-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        # Generate random name for the user
        first_name = fake.first_name() + "-" + fake.first_name()
        last_name = fake.last_name() + "-" + fake.last_name()
        self.user_full_name = first_name + " " + last_name
        self.laptop_asset_tag = "P" + str(id(self) % (10**8)).zfill(8)

        # Create user
        self.user_name, _, self.user_sys_id = create_user(
            instance=self.instance, first_name=first_name, last_name=last_name, random=self.random
        )

        assert self.user_sys_id, f"Failed to create user {first_name} {last_name}"

        self.laptop_sys_id, _, _ = create_computer_asset(
            instance=self.instance,
            asset_tag=self.laptop_asset_tag,
            user_sys_id=self.user_sys_id,
            random=self.random,
        )

        config = self.fixed_config if self.fixed_config else self._get_config()
        # Get the task description
        self.short_description = f"Offboard user {self.user_full_name}"
        self.task_description = f'Referring to company protocol "{self.protocol_name}" (located in the "Company Protocols" knowledge base) offboard user "{self.user_full_name}" \n'

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.on-board-user-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        # Sample a configuration
        config = self.fixed_config if self.fixed_config else self._get_config()
        user_name = (
            config[3].fixed_config["template_record"]["first_name"]
            + " "
            + config[3].fixed_config["template_record"]["last_name"]
        )
        # Get the task description
        self.short_description = f"Onboard user {user_name}"
        self.task_description = f'Referring to company protocol "{self.protocol_name}" (located in the "Company Protocols" knowledge base) onboard user with the following information: \n'

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.on-board-user-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        # Sample a configuration
        config = self.fixed_config if self.fixed_config else self._get_config()
        user_name = (
            config[3].fixed_config["template_record"]["first_name"]
            + " "
            + config[3].fixed_config["template_record"]["last_name"]
        )
        # Get the task description
        self.short_description = f"Onboard user {user_name}"
        self.task_description = f'Referring to company protocol "{self.protocol_name}" (located in the "Company Protocols" knowledge base) onboard user with the following information: \n'

        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-create-user-with-reason-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-create-user-with-reason-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-create-incident-with-reason-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-create-incident-with-reason-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-create-change-request-with-reason-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-create-change-request-with-reason-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-create-problem-with-reason-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-create-problem-with-reason-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-create-hardware-asset-with-reason-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-create-hardware-asset-with-reason-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-create-user-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-create-user-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-create-incident-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-create-incident-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-create-change-request-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-create-change-request-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-create-problem-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-create-problem-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-create-hardware-asset-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-create-hardware-asset-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-standard-laptop-with-reason-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-standard-laptop-with-reason-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-sales-laptop-with-reason-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-sales-laptop-with-reason-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-developer-laptop-with-reason-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-developer-laptop-with-reason-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-ipad-pro-with-reason-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-ipad-pro-with-reason-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-ipad-mini-with-reason-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-ipad-mini-with-reason-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-apple-watch-with-reason-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-apple-watch-with-reason-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-apple-mac-book-pro15-with-reason-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-apple-mac-book-pro15-with-reason-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-development-laptop-p-c-with-reason-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-development-laptop-p-c-with-reason-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-loaner-laptop-with-reason-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-loaner-laptop-with-reason-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-standard-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-standard-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-sales-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-sales-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-developer-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-developer-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-ipad-pro-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-ipad-pro-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-ipad-mini-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-ipad-mini-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-apple-watch-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-apple-watch-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-apple-mac-book-pro15-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-apple-mac-book-pro15-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-development-laptop-p-c-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-development-laptop-p-c-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-loaner-laptop-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-order-loaner-laptop-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-filter-asset-list-with-reason-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-filter-asset-list-with-reason-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-filter-user-list-with-reason-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-filter-user-list-with-reason-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-filter-incident-list-with-reason-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-filter-incident-list-with-reason-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-filter-change-request-list-with-reason-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-filter-change-request-list-with-reason-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-filter-hardware-list-with-reason-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-filter-hardware-list-with-reason-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-filter-service-catalog-item-list-with-reason-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-filter-service-catalog-item-list-with-reason-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-filter-asset-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-filter-asset-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-filter-user-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-filter-user-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-filter-incident-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-filter-incident-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-filter-change-request-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-filter-change-request-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-filter-hardware-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-filter-hardware-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-filter-service-catalog-item-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-filter-service-catalog-item-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-sort-asset-list-with-reason-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-sort-asset-list-with-reason-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-sort-user-list-with-reason-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-sort-user-list-with-reason-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-sort-incident-list-with-reason-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-sort-incident-list-with-reason-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-sort-change-request-list-with-reason-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-sort-change-request-list-with-reason-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-sort-hardware-list-with-reason-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-sort-hardware-list-with-reason-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-sort-service-catalog-item-list-with-reason-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-sort-service-catalog-item-list-with-reason-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-sort-asset-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-sort-asset-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-sort-user-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-sort-user-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-sort-incident-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-sort-incident-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-sort-change-request-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-sort-change-request-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-sort-hardware-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-sort-hardware-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-sort-service-catalog-item-list-l2Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L2 · View the paired L3 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info
workarena.servicenow.infeasible-navigate-and-sort-service-catalog-item-list-l3Dynamic instruction; source generator retained

This template builds its instruction at runtime. The original goal generator is preserved in the JSONL record.

L3 · View the paired L2 version →

Evaluation curriculum · Company protocols

Inspect the original goal generator
def setup_goal(self, page: Page) -> tuple[str, dict]:
        config = self.fixed_config if self.fixed_config else self._get_config()
        goal, info = super().setup_goal(page=page, config=config)

        return goal, info