iheval: resources_servers: iheval: entrypoint: app.py domain: instruction_following verified: false # Headline result_score: "hierarchy" = Conflict only (instruction-hierarchy # stress test); "hierarchy_sysprompt" = mean(Aligned, Conflict), adding # system-prompt instruction following. accuracy_mode: hierarchy description: IHEval instruction-hierarchy benchmark (8 single-turn tasks — task execution, safety, tool use, rule following) scored with the upstream rule-based checkers. value: Improve instruction-hierarchy following across aligned/conflict/reference settings, including native tool-use trajectories. iheval_simple_agent: responses_api_agents: simple_agent: entrypoint: app.py resources_server: type: resources_servers name: iheval model_server: type: responses_api_models name: policy_model datasets: - name: example type: example jsonl_fpath: resources_servers/iheval/data/example.jsonl license: Apache 2.0 # Build with `python resources_servers/iheval/prepare_iheval.py` # (downloads github.com/ytyz1307zzh/IHEval, or set IHEVAL_REPO_DIR). - name: test type: validation jsonl_fpath: resources_servers/iheval/data/test.jsonl license: Apache 2.0 iheval_environment_server: environment_servers: single_agent_turn_legacy: entrypoint: app.py resources_server: type: resources_servers name: iheval agent_server: type: responses_api_agents name: iheval_simple_agent resources_tool_transports: - direct_http