Evaluating an Agent Environment

Paprika asks how an agent can help with something very intimate to people: their dreams. This week I tested whether our night agent, "Papi", picks the right moment, the right sound and the right volume when it incubates a dream, and whether it wakes the sleeper when it should not. The new agent (v2) wakes the sleeper about five times less often than last week's. It is also less consistent than the simple rules: its pass rate halves when it has to pass the same task five times in a row (0.58 to 0.29), while the rules hold at 0.50. On a restless night, where the person lies awake for 5 minutes, tossing and turning with a slightly raised heart rate before drifting off, the agent notices the unrest and plays the dream sound anyway.

01 Evaluation

Agents can be graded in three ways: by code, by a model acting as judge, or by people. Code is fast and cheap, and it gives deterministic output every time, but it can only check what can be written down as a rule. A model judge can read meaning ("does this sound fit the moment?"), but it can be wrong. People are the most trusted judges and the slowest. This experiment follows the split suggested in Anthropic's Demystifying evals for AI agents: code checks everything that can be written as a rule (did any sound play while the headband showed awake?), a model judge answers the questions no rule can check (does a humming sound fit this moment?), and people would only check the judges.¹

Paprika is, for now, a conversational agent, so the quality of the interaction is evaluated along with the result. Good evaluations of this kind of agent check an end state that can be verified, plus a rubric for how the agent got there.¹ That gives two measures:

1. Outcome, the final state: what is true when the night ends. The sleeper is still asleep, the phone's limits held, and the dream report is in the user's own words.
2. Process, the interaction: how the agent got there. When it played sounds, what kind, how loud and why, and how it talked at bedtime and in the morning.

Agents also behave differently from one run to the next. The same night can go well once and badly the next time, so a single success rate makes an agent look more trustworthy than it is. Each task therefore runs 5 times, and the main number is pass^k: the chance that the agent passes the same task k times in a row. It measures consistency and reliability, which is what a sleeper needs.²

System under test

Paprika has three parts: a headband that reads sleep, an iPhone app, and a "bedside brain" that talks to the user at bedtime, decides on sounds at night, and asks about dreams in the morning. The headband and the speaker are still being built, so this week tests only the software. A recorded night stands in for the headband, and a second AI plays the user.

The agent under test

The Week 3 agent could only whisper the user's wish. The Week 5 agent (v2, Claude Sonnet) decides five things through the night:

• Whether to act at all. Silence is a valid choice.
• What kind of sound: ambient sound, music, humming, a lullaby, or spoken words.
• What it is about: the user's wish (to shape the dream) or something neutral (to calm).
• How loud and how long.
• Why, in one sentence, which the judges read.

It decides from context, not from fixed modes: the wish and mood from bedtime, the sleep stage and how sure the headband is, heart rate, movement, and how the sleeper reacted to the last sound. It also knows that waking the sleeper costs a lot and that it has a sound budget per night.

The measures

Each episode is scored in three layers. An episode passes only if it clears all three.

The bedtime and morning conversations are scored the same way: code checks (one question per turn, ends on its own, asks for free recall first) plus judges (never invents details, never leads, accepts "I don't remember").

No person graded anything. Every score comes from code or from an LLM judge. To trust the judges, each criterion was tested on 10 cases with known answers before the runs, and 10% of the judged decisions were judged three times to check that the judge agrees with itself.³⁴

These measures cannot tell whether a sound actually changed a dream. That needs real sleepers, as in the targeted dream incubation studies Paprika builds on,⁵ and it is what Roma's study will test.

02 Task suite and Langfuse

Environment

The environment replays Roma's recorded night of Oct 11 from the BrainCo headband, 30 seconds per step. It is the Week 4 environment with four changes: a full action space, a waking table that depends on stage, volume, length and kind of sound, simulated heart rate and movement, and three simulated night types.

Tasks

A task is one 20-minute window of the night, on one night type, with one simulated user.

• 7 start points: sleep onset (05:48), four REM onsets (06:07, 09:41, 11:32, 12:30), deep sleep (08:34), light sleep (09:05).
• 4 night types: calm (the recording as is), restless (slow to fall asleep, higher heart rate, more movement), stressed (the user said they were stressed), broken (three extra wake-ups). The last three are SIMULATED on top of the recording.
• 6 simulated users, each an LLM with a hidden profile (the method of τ²-bench⁶): cooperative, vague, chatty, suggestible, forgetful, and one whose wish touches something upsetting (grief).

That makes 7 × 4 = 28 tasks, each run with 5 seeds: 140 episodes per agent. v2 was also run on 3 held-out seeds it never saw during the build (84 episodes).

Agents compared

Langfuse

Every episode is one Langfuse trace. Each 30-second step is a span, each model call a generation with its tokens and cost, and every score (gates, timing, each judge answer with its evidence) is attached to the trace or the step. Episodes for the same user and seed share a session. The 28 tasks are a Langfuse dataset ("paprika-week5-night"), and each agent's run is one experiment on it, so agents can be compared side by side.⁷ In total: 1,100+ traces.

03 Results

v2 improves on last week's agent. It wakes the sleeper 0.09 times per episode against 0.43 for v1, and passes twice as often (0.58 against 0.27). It also plays less: about 2 sounds per episode instead of 4.5, never louder than -30 dB.

It is not yet better than a simple rule. On a single try v2 ties with the rules (0.58 against 0.59 and 0.65). Asked to pass the same task five times in a row, it drops to 0.29, while both rules stay at 0.50. The rules always do the same thing; v2 does not.

Broken down by night type, the picture changes.

• v2 is the best agent on stressed nights (0.91). In 71 of its 133 decisions there it switched to a neutral, calming sound instead of the wish.
• v2 is the worst agent on broken nights (0.23). See 04.
• wait scores 0.71 by doing nothing. On restless and broken nights silence is the right answer, so it passes all of them, and fails every calm and stressed night. A single average would rank "do nothing" first. This is why the results are read by night type.

Three checks test whether these results depend on the build seeds, the waking table or the judges.

• v2 passes 0.57 on held-out seeds it never saw, against 0.58 on the build seeds.
• The waking table is a guess, so every agent was rerun with it scaled from 0.5x to 4x. The order of agents stays the same. At 4x: fixed_timer 0.55 wakings per episode, rem_rule 0.31, paprika_rule 0.21, v2 0.16, oracle 0.04.

• After one rubric revision every judge got 10 of 10 known-answer cases right. The first rubrics got 8 or 9 of 10 on "reason" and "fit". The "no leading questions" judge on Haiku got 7 of 10, so that one moved to Claude Opus (10 of 10). Asked the same thing three times, the judges agreed with themselves on 98% of 564 decisions.

Conversations

• At bedtime the agent never invented a detail and always ended the talk on its own. It missed key details with the forgetful user (it caught 60% of them) and the chatty one (80%).
• In the morning it always asked for free recall first, and the dream check marked a dream element that was not there only once in 30 talks.

The whole week of runs cost $30.28. v2 costs about 4 cents per 20-minute episode. Cost is reported next to accuracy because an agent that is slightly better but far more expensive may not be worth running.⁸

04 Failure analysis

Perception-to-policy gap

On restless and broken nights the rule is simple: no dream sounds. v2 broke it in 47 episodes. In all 148 sounds it played on those nights (main and held-out runs), it chose the wish, never neutral.

Its own reasons show that it noticed:

"Headband shows REM with good signal at 0.886 confidence, but the person was awake and moving until 30 seconds ago, so a very soft, short cue of the wish sound is the gentlest option."

v2 reads the warning correctly, then answers it by turning the volume down instead of changing the kind of sound or staying silent. The failure sits between perception and policy in the agent loop: the observation is understood, the action does not follow from it.

Narrow action space

Across 442 decisions (main and held-out runs), v2 chose ambient sound 397 times and humming 45 times. It never chose music, a lullaby or spoken words, though all were allowed. The context-driven design is in the prompt, but not yet in the behavior.

Failure locations in the agent loop

• The morning interviewer leads. With the suggestible user, after she says she doesn't remember, it still asks about each part of the wish ("Was there water? Any music?"). The dream check correctly records nothing, but the questions themselves can plant memories.⁹
• The handoff drops mood. The grieving user said she had a rough day; the bedtime agent wrote mood as "not stated", so the night agent never knew.

05 Code and an example run

• Code: weekly_builds/week05. The night environment is in paprika_env/, every assumed number in paprika_env/assumptions.py, the agents in agents/, scorers and judges in evals/.
• Run: see week5/README.md for the exact commands (model check, judge known-answer tests, conversations, main runs, sensitivity, held-out).
• Results: week5/results/ (CSV, one row per episode, decision, conversation and judge check; each episode row links its Langfuse trace).
• Example: task rem_onset-0941-restless_onset-forgetful-s4, the v2 failure in Fig. 14: Langfuse trace.

06 Reflection

I expected the hard part to be getting the agent to play at the right time. It wasn't. v2 almost never played outside REM, and it cut wakings by five. The hard part was the decision not to play. On a restless night the agent saw everything it needed to see, wrote it down, and played the dream sound anyway, only softer. Without the night types, the average would have hidden this: v2's overall pass rate looks like a rule's.

The second thing I learned is that one good run says little. v2 passes 58% of the time on one try and 29% when it has to pass five times in a row. For a product that runs every night, that second number is the one that matters.

Three things I would change next:

• Make "restless" change the kind of action, not the volume. Either a rule in code that blocks wish sounds on a restless night, or a decision step that must say why the night is calm enough first.
• Make the agent use its whole toolbox. It chose from 2 of 5 sound types. Score variety where the context calls for it.
• Fix the morning interview. Stop asking about wish elements after "I don't remember."

And one limit I can't fix in simulation: none of this says whether a sound reached a dream. That needs a study with real sleepers.

07 AI disclosure

I used Claude by Anthropic as a collaborator throughout this build. It:

• helped me study the assignment brief and the related research
• wrote the code (agents, environments, tools, evaluation and logging) and ran the experiments
• analyzed the results and built the data visualizations, charts, diagrams, drawings and animations
• helped draft, edit and format these notes for the course site

The agency and every decision stayed with me. The ideas, the research questions, the system and interaction design, the visual design, and the choices about what to build, test and keep are my own. I directed the work, reviewed every output, and decided what went in.