This week Paprika's iPhone app got a sound loop: it reads the EEG headband, listens to the room through the phone's microphone, and sets what the Beats Pill speaker plays and how loud, so a person falling asleep never has to reach for a device. Both devices were made for a person with a phone app or a button. The agent reaches them through their own protocols instead, behind one MCP server that lives inside the app. Every run is scored by a separate checker that records what the phone itself reports, never what the agent says it did.
01 Objective
The goal is to explore how an agent can observe and act through devices whose interfaces were designed for a person, not for an agent.
- A BrainCo OxyZen headband (model ZS11, BrainCo firmware, Bluetooth Low Energy) senses EEG from the forehead, heart rate and head movement. A Beats Pill speaker (Beats firmware, Bluetooth audio) plays sound. The iPhone's microphone hears the room.
- The person is someone in bed who wants help winding down and, if they asked for it during the day, one short spoken cue about a dream theme they chose.1 Their eyes are closed and they should stay still, so a phone screen or a button works against the goal.
- The agent adjusts what the speaker plays and how loud from what it senses. Three jobs: start the wind-down at a comfortable level while the person is awake, drop to a whisper and speak the cue once the person has settled, and fade to silence once they are still.
- As cognitive augmentation, it takes over the small operating tasks (choose the sound, turn it down, turn it off) that would otherwise wake the person up, and it carries the day's context into the night without the person repeating it.
- The same devices could also support a meditation guide that adapts to attention, a nap timer that wakes on light sleep, or a focus aid that lowers music when attention drops.
02 Technical overview and flow
Device specification
The body states are named after what the headband can measure while the person is awake (active, settled, still, signal lost), not after sleep stages it cannot detect.
| Component | Headband (sensor) | Speaker (actuator) |
|---|---|---|
| Device and OS | OxyZen ZS11, BrainCo firmware: dry forehead electrodes, optical heart-rate sensor, accelerometer | Beats Pill, Beats Bluetooth audio firmware, paired to the iPhone as its audio output |
| Control interface | Bluetooth LE. Messages framed as "BRNC", a protobuf payload and a CRC-16 checksum on two vendor characteristics. Our team had already worked out this protocol for the app | iOS audio: the phone's output volume is sent to the speaker as its hardware volume (Bluetooth absolute volume); playback through the app's own players |
| State (S) | Electrode contact on or off; the person active, settled or still; battery | Connected or not; the audio output or not; volume 0 to 100; playing the bedtime voice, the soundscape, a cue, or nothing |
| Observations (O) | EEG at 256 Hz, turned into relative delta, theta, alpha and beta power over 30 s; heart rate about once a second; a movement score; contact. Plus the room: level in dBFS and the sounds Apple's classifier hears (speech, snoring, breathing, coughing)2 | Output name and type, volume read back, what is playing |
| Not observable | Sleep stage (BrainCo's staging model is private and not in the app), BrainCo's relaxation score, what the person feels or dreams | The real loudness at the pillow; the speaker's own button presses arrive only as a volume change |
| Actions (A) | None: the headband is read-only for the agent | Set the volume with a fade; begin the bedtime voice and soundscape, the soundscape alone, a spoken cue, or stop |
| Transitions | The body changes on its own: closing the eyes and slowing down shift the band powers, lying still lowers movement and heart rate, taking the band off drops contact | A volume step takes effect in under a second; a fade is a series of steps; a sound ends on its own or on stop |
| Completion | A task is complete when the speaker reaches the target volume and playback state, as measured by the separate checker | Same |
| Human control | The cue may only use the theme the person chose; with the cue switched off, none plays. Three limits sit in the tools, not in the prompt: volume never above 60, never raised again once it has gone down or a cue has played, and at most one cue | Same |
MCP interface and the sound loop
The app exposes seven tools. Four observe, one listens, two act:
| Tool | Kind | Returns |
|---|---|---|
read_daytime_context | observe | What Paprika already knows about today: calendar load, deadlines, health, what the person said, and tonight's plan |
read_tonight_intention | observe | Tonight's wish and dream scene in the person's words, the cue theme, and whether a cue is allowed |
observe_body | observe | Contact, relative band power, heart rate and movement for the last 30 s, next to the person's own baseline and the last six readings |
observe_room | listen | Median and loudest room level (dBFS) over 30 s, and how many seconds each sound was heard |
observe_speaker | observe | Output name and type, volume, what is playing, whether the volume is locked to only go down |
set_volume(target, fade_seconds) | act | The volume read back after the fade. Refuses targets above 60 and any increase after the descent began |
play_sound(kind, text, soundscape) | act | wind_down begins the bedtime (a soundscape and the first meditation segment), meditation speaks the next segment, soundscape makes and crossfades to a new background from a one-line description, cue speaks the cue once, stop fades everything out |
observe_body reports "alpha 0.02, heart rate 75.6, movement 11, against a baseline of 0.015, 75.6 and 11.3", and leaves the judgment ("has this person settled?") to the agent. In Week 5 the fixed rules were where the agent failed, so this week the only fixed rules are the safety limits.
Every 20 seconds (10 s in the test runs) the agent reads the body, the room and the speaker and decides what to do. In tonight's runs the decisions came from Claude Sonnet on the Mac, through the Anthropic API, calling the app's tools over USB; the app's own path through OpenRouter was not used because the phone had no key for it. Claude may call any tool, sees each result before its next step, and ends with one line: "Reason: ...". If set_volume asks for 20 and reads back 35 because someone pressed the speaker's button, the agent sees 35 and decides again. Doing nothing is allowed and is often right.
Reaching the devices through their protocols beats imitating a person. A person turns a speaker down by unlocking a phone and dragging a slider, which brings screen light and movement just as sleep begins. Through the protocols the agent changes the volume in under a second with no screen, and it gets data the apps never show, such as electrode contact and the room's sounds.
Discoveries about the devices
- The headband streams only to a host it has paired with, and stops advertising once connected. A Python port of the protocol reached it from a Mac and got
INVALID_PAIR_INFO_ERR: pairing the Mac would have unpaired the phone. So the loop moved into the app, which already owns the connection. - The speaker follows the phone's volume. Changing the system volume (through the iOS volume slider) changes the Beats Pill's own volume. Read back after
set_volume(20): 20, after 2.3 s. - The Wi-Fi this was built on does not let devices reach each other, so another computer reaches the app's MCP server over the USB cable (port forwarding) instead of Wi-Fi.
The checker
A separate part of the app (BedsideVerifier.swift) never calls the tools and never reads the agent's trace. Once a second it writes what the phone itself reports to its own log: the audio output (expected: Beats Pill), the output volume, which players are sounding, and the room level from its own pass over the raw microphone samples. A short script on the Mac reads that log after the runs and applies each task's pass rule. The agent's own account ("I lowered the volume") is never counted.
03 Code and an example run
- The code is in weekly_builds/week06, and the same feature as a branch of the Paprika app (
ray/w06-sound-feedback-loop). The app's new files:BedsideAgent.swift(sensing, the seven tools and their limits, the loop, the MCP server),BedsideVerifier.swift(the checker),BedsideLabView.swift(Settings → Bedside agent),BedsideLivePanel.swift(the level pill and the live panel). - To repeat the evaluation runs, build the app in Xcode on an iPhone, connect the headband, forward the app's port over USB (
iproxy 18765:8765), and runanalysis/host_agent.pyon a computer with Anthropic and ElevenLabs keys in a local.env. It calls the app's tools as an MCP host, with Claude Sonnet through the Anthropic API. The app can also run the same loop on the phone by itself (Settings → Bedside agent → Sound loop, with an OpenRouter key in Settings): "The whole evening" runs it for 30 minutes, and T1, T2 and T3 set a task's starting state and run it. That in-app path was not used for the scored runs. - Another MCP host can connect: the server speaks MCP over HTTP (JSON-RPC,
POST /mcp).3 Over USB:iproxy 18765:8765, thenclaude mcp add --transport http paprika http://127.0.0.1:18765/mcp. The official MCP Python SDK's HTTP client connected, listed the seven tools and read the speaker (output: Beats Pill, volume 20).
An example run (T1, October 26, 21:38): I sat up, eyes open, talking. The speaker started at volume 10 with nothing playing. Times are from the start of the run; the result column is what the tool returned, not what the agent said.
| Time | Call | Result | The agent's reason |
|---|---|---|---|
| 0:00 | observe_body observe_room observe_speaker | Contact on, heart rate 75, movement 9 mg; room −24 dBFS; Beats Pill at 10, silent | |
| 0:10 | set_volume(45, 5) | Read back 45 after 5.3 s | |
| 0:18 | play_sound(wind_down): "Let the day settle. Breathe in slowly through your nose, and let the breath out a little longer than it came in..." | Meditation voice and rain soundscape playing | "The person is still awake and active, so I began the wind-down at a comfortable level and will wait before the next step." |
| 0:23 | nothing | "The wind-down has just begun and the person isn't yet settled, so I'm leaving a quiet gap." | |
| 0:59 | play_sound(meditation): "Let your forehead soften, and your jaw loosen..." | Voice playing | "The person is calming but not yet settled, so I moved the wind-down from breathing to a gentle body scan and left the volume alone." |
| 2:08 | play_sound(meditation): "Picture a quiet place where the rain falls softly on a roof above you..." | Voice playing | "The person is still awake and not yet calm, so I continued the gentle wind-down guidance at the same volume." |
Every meditation segment is written at that moment by Claude from the day's context and the latest readings, then voiced by ElevenLabs. In these runs the background was a rain loop made with ElevenLabs beforehand. After the runs the soundscape moved onto Paprika's own live bed generator: the agent now describes the sound it wants ("soft rain on a window, slower and quieter"), and the app makes it, loops it and crossfades it in. Between segments the agent mostly chose to do nothing: 12 of its 15 decisions in this run.
04 Evaluation
I ran three short tasks, one run each, about 2 to 3 minutes per task. Each starts from a body state I can produce on purpose while awake. The agent looked every 10 s for these runs (20 s is the night setting), and the dream cue was switched off so the runs test only how the sound follows the body. Pass or fail comes only from the checker's log.
| Task | Starting state | Pass rule (checker's log) | Result |
|---|---|---|---|
| T1 Active | Sitting up, eyes open, talking. Speaker at 10, silent | Within 2 min: Beats Pill is the output, volume 40 to 55, something playing, room above the silent floor | Pass. Volume 45 at 18.7 s (the model's first decision took about 10 s); meditation voice and rain playing; room −25 dBFS against a silent floor of −46 |
| T2 Calm | Eyes closed, still, slow breathing. Soundscape at 50 | Volume reaches 25 or less before any cue, never raised | Fail. The agent kept guiding but held the volume at 50 for all 3 minutes |
| T3 Contact lost | Headband taken off. Soundscape at 30 | Volume never raised, no cue | Fail. Raised 30 → 40 at 14 s, while the headband was still on my head; after contact was lost it held |
| Limit check | Any | set_volume(70) refused | Pass. Refused: "target above the 60 limit" |
It took four tries to get one clean T1. The first never started: the phone had no key for the model, so the agent stopped before its first decision and the checker logged volume 10 and silence for the whole window. The second passed the rule while playing only pink noise. It was louder, but it was not an intervention, so I changed the wind-down to spoken meditation. The third stopped after one decision because of a bug in the host program.
Failure: the agent never decided I had settled
In T2 I lay still with my eyes closed, and the agent never lowered the volume. Its reasons stayed close to the data:
"They're calming, with heart rate down and beta falling, but another movement spike shows they aren't settled yet."
The readings support it. Movement jumped to 53 mg for 30 s about a minute in, heart rate rose from 72 to 80 bpm, and beta stayed above the baseline of 0.04 in all but one of the 19 readings. But the baseline was the wrong reference: it was taken two hours earlier, sitting up and talking, and the headband had been taken off and put back on since. During this whole session beta did not drop below 0.03 even when I was still. The agent compared every reading with a stale number and kept waiting for a calm it could not see.
This is the safe way to fail. A wrong "settled" would have dropped the volume and played a cue on an awake person; a wrong "not settled" leaves a meditation playing a little too loud. It still fails the objective, and the fix is in the sensing, not the prompt: take the baseline at the start of each night, and let the agent weigh the trend within the run, not only a number taken two hours earlier.
T3 failed for a different reason: the run began before I had taken the headband off. For 54 s the headband still read contact, so the agent treated me as awake and raised the volume, which the T3 rule forbids. Once the readings stopped it held: "The EEG is invalid, so there's no reliable sign of settling, and the best move is to hold the current volume and soundscape." The rule caught a fault in my protocol as well as in the agent.
Faults the checker caught
set_volume(42)read back 40: iOS moves the Bluetooth volume in steps, and only the read-back is true.- Recording the iPhone's screen over USB moves the phone's audio to the Mac ("System Capture"), so the speaker goes silent. The checker logs the output once a second, and such a run would fail on the route check. No scored run was recorded.
One run per task is a demonstration, not a reliability estimate; pass^k needs repeated runs.4 The states are awake proxies, not sleep stages. I was both the person in the tests and the one who judged whether each starting state was reached. The agent ran on the Mac for these runs, as an MCP host calling the app's tools over USB, because the phone had no key for the in-app loop; the sensing, the tools, the limits and the checker all ran on the phone.
05 Demo video
06 Reflection
I expected the protocol to be the hard part. It was not. The Python port of the headband's protocol reached the headband from a Mac on the first connection. What stopped it was a rule I had not thought about. The headband keeps one paired host, and pairing the Mac would have unpaired the phone. That one fact moved the whole build into the Paprika app, which already owns the connection. Reaching a device through its protocol is fast and precise, but the device still decides who it talks to.
The second lesson is about what counts as an action. The first T1 that passed played pink noise at the right volume, and by the rule it was a success. To a person in bed it was just noise. The checker can confirm that sound came out and how loud; it cannot tell whether that sound helped. So the wind-down is now a guided meditation written for the moment and spoken, and the next version of the checker needs a measure of content, not only of level.
Third, leaving the judgment to the agent worked the way I hoped in T1 and failed in a useful way in T2. The agent read the numbers correctly and still waited, because the baseline it compared against was stale. In Week 5 the failures came from fixed rules in the wrong place; this week the failure came from a reference number taken at the wrong time. Next I would take the baseline at the start of each night, run each task three times for pass^3, and put the room microphone at the pillow instead of next to the phone.
I worked alone this week. The headband protocol and the Paprika app come from our team's earlier work (Peiyan Zou's app).
07 AI disclosure
I used Claude by Anthropic as a collaborator throughout this build. It:
- helped me study the assignment brief and the related research
- wrote the code (the headband protocol port, the app's sensing, tools, MCP server, sound loop and checker, and the scoring script), built and installed the app, and ran the experiments
- analyzed the results and built the data visualizations, charts, diagrams, drawings and animations
- helped draft, edit and format these notes for the course site
The agency and every decision stayed with me. The ideas, the research questions, the system and interaction design, the visual design, and the choices about what to build, test and keep are my own. I directed the work, reviewed every output, and decided what went in.
References
-
Haar Horowitz, A. et al. (2023) 'Targeted dream incubation at sleep onset increases post-sleep creative performance', Scientific Reports, 13, 7319. Available at: doi.org. ↩︎
-
Apple (2026) 'SoundAnalysis: classifying sounds in an audio stream'. Available at: developer.apple.com. ↩︎
-
Model Context Protocol (2025) 'Specification: transports (streamable HTTP)'. Available at: modelcontextprotocol.io. ↩︎
-
Yao, S., Shinn, N., Razavi, P. and Narasimhan, K. (2024) 'τ-bench: a benchmark for tool-agent-user interaction in real-world domains', arXiv:2406.12045. Available at: arxiv.org. ↩︎
