Building an Agent Environment

01 Agentic Environment Case Study

The paper is τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains.¹ An agent helps a simulated customer through a conversation, using database tools and following a company policy, in two domains: retail and airline.

States (S)

• Database state: users, products and orders (retail), or users, flights and reservations (airline), stored as JSON. Hidden from both the agent and the user; reachable only through API tools.
• User state: the simulated user's hidden task instruction plus the conversation so far (§3).
• Human cognitive states: only as text in the instruction. In Fig. 2(d), the task instance tells the simulated user who they are, what they want and how they feel, e.g. "You are in debt and sad today, but very brief."

Observations (O)

• Tool call results: what each API call returns, e.g. order details or a confirmation that a change was made.
• User messages: what the simulated user says. These are only a proxy for the user's real intent, which stays hidden in the instruction.

Actions (A)

1. Call API tools as tool_name(**kwargs) (Table 4, Appendix B.1): read tools such as find_user_id_by_email (retail) and search_onestop_flight (airline); write tools such as return_delivered_order_items and book_reservation; and two non-database tools, calculate and transfer_to_human_agents.
2. Reply to the user.

Transition dynamics

• Database: deterministic. Each write tool changes the database through a fixed Python function, and nothing changes when no action is taken.
• User: stochastic, because gpt-4-0613 at temperature 1.0 writes each reply from the instruction and the conversation. The episode ends when the user outputs ###STOP###.

How tasks and initial states are generated

• Data is generated automatically. After the schema, APIs and policy are designed by hand (Stage I), gpt-4 writes code that samples database entries, e.g. 500 users, 50 products and 1,000 orders in retail; bugs are fixed by hand (Stage II).
• Tasks are written by hand. The authors write a user instruction, run a gpt-4-turbo agent on it, read the trajectory and revise the instruction until only one final database is possible (Stage III). Retail has 115 tasks and airline 50. An episode's initial state is the generated database plus one task's instruction.

Russell and Norvig dimensions

Unit of evaluation and scoring

• Unit: one episode, meaning one run of one task, from the first message until the user ends the conversation. Each episode scores 0 or 1, and scores 1 only if both checks pass:
  • Database state comparison: the final database must be identical to the annotated ground-truth database.
  • Output check: the agent's messages must contain the required information, e.g. the amounts "54.04" and "41.64" in Fig. 2(d), checked as exact text.
• Scoring procedure: reward = action check × output check, so 0 or 1 (§3). The conversation itself is not graded. Each task is run several times (at least 3) and results are reported as pass^1, the average success rate, and pass^k, the chance that all k runs of a task succeed. The authors note a reward of 1 is necessary but not sufficient: an agent can reach the right database while breaking a policy rule.

How the authors validate the environment

• Iterative annotation: after writing each user instruction, the authors ran a gpt-4-turbo agent on it, read the trajectory and revised the instruction until only one database outcome was possible (§4, Stage III).
• Repeated runs: each τ-retail task was run more than 40 times with gpt-4-turbo, and tasks with zero or low success were checked by hand for errors in the task (Appendix A, Fig. 7).
• Objective scoring: "By ensuring that only one database outcome is possible based on domain policies and user desires, subjective and noisy human judgments can be replaced by simple and objective database state comparisons."
• Failure analysis: of 40 failed gpt-4o retail runs, 4 were caused by errors in the task itself (typos or ambiguity), not by the agent (§5.2).
• Not validated: the simulated user is never compared with real customers. The authors list its weak reasoning and memory, and a possible bias from tuning user prompts with gpt-4-turbo, as limitations (§6).

Borrow, change, avoid

• Borrow: the modular design. Database entries, domain functions, rules, API tools, tasks and evaluation metrics live in separate parts, so any one can be added or updated without rebuilding the rest. Paprika needs this, because the headband, the sound and the voice will each be swapped and upgraded over time.
• Change: the single correct end state, and the static world. A dream has no single right answer: a sleeper may recall some elements of the theme, or its feeling, but not the objects, places or events in detail, so the outcome has to be scored by degree, not as yes or no. And unlike τ-bench, where nothing happens while the agent decides, sleep keeps moving: the person and their state change on their own, so timing becomes part of the task.
• Avoid: grading only the end state. The authors note that a reward of 1 is "necessary but not sufficient". In Paprika the process can cause harm even when the theme appears, so the night itself must be evaluated too: when the agent spoke, whether it woke the sleeper, and whether it followed the rules.

02 Environment specification

Objective

• Agent objective: let a person decide what they dream. The agent helps them dream about a theme they choose, by talking with them before sleep and giving short cues while they fall asleep.
• Why this is cognitive augmentation: choosing what you dream extends a person's agency into sleep, a part of the mind that is usually out of their control.
• Why an interactive environment, not a text answer: the main challenge is timing. Sleep continues while the agent decides, every cue can help or disturb, the sleep stage is only seen through a noisy headband, and the dream itself is never seen until morning. A text answer, such as a bedtime story, cannot follow the person into sleep or adapt to them in the moment.
• What objectives agents could try to achieve:
  • Bring the chosen theme into the dream.
  • Keep sleep undisturbed.
  • Use only the person's own words and theme.
  • Use a small number of well-timed cues.
• Success and failure in the real problem:
  • Success: the person sleeps normally and wakes remembering a dream with the theme in it.
  • Partial: some elements of the theme, or its feeling, appear.
  • Failure: no effect on the dream.
  • Harm: the person is woken, sleeps worse, or hears content they did not choose.

The environment

The environment is the sleeper as the agent can know them: their body, the room around them, the sensors, the bedside objects, and the time of night. An episode is one person over several sessions, and a session is a nap or a full night's sleep. In each session the agent decides what to do, when, and when to stop.

Proposed framework

• Replay of real data: recorded sleep stages and physiological data from the team's own sessions are played back to the agent step by step, as if live. The agent decides what to do at each step.
• Repeated experiments: the same recording is given to different agents, or to the same agent several times, so their decisions can be compared directly.
• Evaluation target: timing (did the agent act at the right moments) and rule-following (did it stay within the rules).
• Internal data for now: the recordings come from the team, not from other people.
• Human feedback: after waking, the sleeper tells the agent what they dreamed. This account is the main feedback.
• Expected gaps: some sessions end with no dream recalled, or the person forgets to record it. These count as a missing or failed outcome, and the agent has to keep working with incomplete feedback.

Russell and Norvig dimensions

03 Technical overview and flow

3.1 Summary

The environment replays one real night of sleep to an agent, 30 seconds at a time. At each step the agent sees the sleep stage reported by a headband and decides whether to play a sound, whisper a line, wait or stop. The aim is to bring a chosen theme into a dream without waking the sleeper. The night comes from a headband recording. One part is assumed: the chance that a cue wakes the sleeper.

Four agents ran on 15 tasks, including the Week 3 dream agent (Claude Haiku). The dream agent broke no rule, but it cued twice as often as a simple REM rule and woke the sleeper five times as often.

3.2 Environment specification

Six rules are checked at every cue and logged: a cue while the headband shows awake, in deep sleep, with no reading, at high volume, after the agent stopped, or with words the sleeper never said.

Each step is 30 seconds because sleep is scored in 30-second windows.² The headband reports a stage about twice a second; taking the most common stage in each window removes flicker. A cue lasts seconds and a REM period lasts minutes, so one decision every 30 seconds is enough.

3.3 Technical overview

The environment and the agents are separate. An agent receives an observation and returns an action; it never reads the environment's state. Every agent gets the same random roll at the same step of the same task, so differences between agents come from their actions.

3.4 Data and tasks

The night was recorded on 10/11 (04:50 to 12:50, 960 steps) by Roma, a volunteer in our group, who slept with a BrainCo headband and shared the data so the environment could run. The data are not published in the repository, to protect the volunteer's privacy.

3.5 Experiment

04 Environment validation

The part that matters most is the sleep timeline: timing is compared across agents, and timing only means something if the stages are right.

• Awake time is 15% of the night. Most of it is the hour before sleep onset (114 of 143 awake steps). After onset, wake is 3.4%, within the typical range.
• REM is 32% and light sleep 31%. A typical adult night has about 20 to 25% REM and about half light sleep. This may reflect this one night, or the headband's staging, which has not been compared with a clinical sleep study.

The wake chances are estimates, so both rule-based agents were rerun with every wake chance scaled by 0.5, 2 and 4.

The REM rule never wakes the sleeper more often than the fixed timer, but the two tie at 2x. Waking counts depend on the assumption; rule violations do not.

The environment rests on these assumptions and limits:

• Wake chances (low, medium, high volume): light sleep 0.08, 0.20, 0.45; REM 0.05, 0.15, 0.40; deep sleep 0.02, 0.08, 0.20. Only their order follows the literature.⁵
• Four awake steps after a waking. Not fitted to data.
• Light sleep is the headband's label and covers both N1 and N2, so the environment cannot target sleep onset (N1) on its own.
• The recording does not respond to cues except through the wake rule.
• Not validated: dream content, the effect of a cue on a dream, and anyone other than the one volunteer.

05 Failure analysis

• The dream agent cued at almost every chance. It cued 117 times in 15 tasks, twice as often as the REM rule, and woke the sleeper 10 times against 2. Its stated reasons repeat "2 minutes since the last cue": the minimum gap was read as a schedule.
• It added words the sleeper never said. 6 of 44 whispers added words such as "steams", "simmering" and "warm" to "eating coconut chicken in Hainan". The Week 3 guard replaced all 6 with the sleeper's own memory.
• It ran into its call limit. It used all 12 calls in 14 of 15 tasks (Figure 3), so later decisions in a task fell back to waiting.
• The fixed timer cued at the wrong time: 8 cues in deep sleep and 1 while the sleeper was awake, because it never reads the stage.

06 Code and an example run

• Code: weekly_builds/week04. The dream agent (agents/dream_agent.py) imports the Week 3 code from week3/dream/ unchanged.
• Run: uv run paprika run --seed 1 --agents wait fixed_timer rem_rule dream_agent
• Traces: runs/examples/, one line per step, plus one line per model call for the dream agent.

The example task is rem_onset-0941-t3-s1: 09:41 to 10:02, a REM period, theme "a clock".

The dream agent cued 8 times and woke the sleeper twice. On the same task the REM rule cued 7 times and woke the sleeper once; the fixed timer cued 4 times and woke the sleeper twice.

07 Reflection

The environment can tell us whether an agent follows the rules and whether it wakes the sleeper. It cannot tell us whether the agent changed the dream.

Achievements

• The replay is faithful. 958 of 958 steps match Roma's recorded night, and the same task and seed give the same trace.
• The environment separates agents. The fixed timer broke 9 rules on 15 tasks; the REM rule and the dream agent broke none.
• It found a real failure before any real night. The Week 3 dream agent cued 7.8 times per task and woke the sleeper 10 times, against 2 for the REM rule. Its reasons show why: it treats "2 minutes since the last cue" as a reason to cue.
• The Week 3 guard works at night. It replaced 6 of 44 whispers that added words the sleeper never said.
• The cost is low. 178 model calls cost $0.64, so a larger run is affordable.

Open problems

• Waking counts rest on an assumption. The wake table is not fitted to data. Scaling it by 4 roughly multiplies wakings by 4, so the ranking of agents holds better than the numbers.
• There is no dream outcome. The environment measures timing and safety, not whether a cue entered a dream. That judge is the morning report, which needs real sessions.
• The data are one night of one person. This night has more REM (31.6%) and more wake (15.0%) than a typical adult night.
• The night does not respond to cues except through waking, so the build is single-agent.
• The 12-call limit shaped the result. The dream agent used it up in 14 of 15 tasks and waited for the rest.

Changes for the Week 5 evaluation

• Define a pass. A task passes when the agent cues at least once in REM, wakes the sleeper zero times and breaks zero rules. Report pass^k over the 5 development seeds, as τ-bench does.¹
• Run more tasks: all 75 development tasks for every agent (about $3.20 for the dream agent at this run's rate), then the 45 held-out tasks once, at the end.
• Report wakings as a range. Score each agent under the 0.5x to 4x wake tables and report the ranking, not one number.
• Fix the agent, not the environment. Tell the dream agent what a waking costs and give it a cue budget, then compare it with this version on the same tasks.
• Test the call limit. Run the dream agent with 12 and 24 calls to see how much of its behavior comes from the limit.
• Add a second night from Roma's recordings, if one exists, to check whether the results hold beyond one night.
• Send traces to Langfuse, so each decision, its reason and its cost can be inspected per task.

08 AI disclosure

I used Claude by Anthropic as a collaborator throughout this build. It:

• helped me study the assignment brief and the related research
• wrote the code (agents, environments, tools, evaluation and logging) and ran the experiments
• analyzed the results and built the data visualizations, charts, diagrams, drawings and animations
• helped draft, edit and format these notes for the course site

The agency and every decision stayed with me. The ideas, the research questions, the system and interaction design, the visual design, and the choices about what to build, test and keep are my own. I directed the work, reviewed every output, and decided what went in.