Designing Human-Agent Interfaces

01 Objective

The domain I want to focus on for the final project is memory, specifically dream memory. I dream a lot and have tried to record my dreams many times. I'm also fascinated by how films like Paprika (Satoshi Kon, 2006) and Until the End of the World (Wim Wenders, 1991) imagine devices that can record or enter our dreams. Pop culture, film and literature are full of these imaginary devices.

Within that domain, the three cognitive outcomes I want to focus on are:

• Dream recall: how much of a dream you remember after waking, and in how much detail.
• Pattern awareness: noticing what keeps coming back across dreams, like people, places or feelings.
• Dream agency: the ability to influence what you dream about.

For this week, my primary objective is dream recall, because it lays the foundation for pattern awareness and dream agency. The most basic step is recording your dreams, and the agent helps with that: you wake up and tell it what you dreamed.

The agent is for people who want to remember their dreams, like people who keep dream journals, and anyone curious about what they dream.

I read the Dormio paper by Haar Horowitz, Cunningham, Maes and Stickgold (2020).¹ Their device detects when you are falling asleep, plays a voice cue, then wakes you and records what you say, and the cue showed up in people's dreams. In that paper the dream is captured by voice, seconds after waking, which is why I made my agent a voice agent with ElevenLabs.

I also read Konkoly et al. (2021),² where they showed you can have a real-time conversation with someone while they are dreaming. What surprised me is that some dreamers remembered a question wrong after waking, even though they had answered it correctly while asleep. This led me to think the agent should capture fast and add nothing, since dreams are already so fragile on their own.

My agent supports recall by being voice only: you keep your eyes closed and talk for about 2 minutes, and it asks neutral questions only about things you already said. Then it reads the dream back in your own words. In the morning, a review screen shows your words and what repeats across nights. The bedtime intention is a first small step toward agency, and mental wellness, like having fewer bad dreams, is a future direction.

The main things that could go wrong are the agent asking a leading question and planting a detail that wasn't in the dream (Chan et al. 2024)³, privacy, over-interpretation, and bad intentions, like incubating ads in people's dreams (Coors tried this in 2021) or selling dream data.

To know if it works, I would look at the words and details I capture, how fast I start talking, and how often I have a white dream (knowing I dreamed but not what). Then I would compare nights with the agent against nights with just a plain voice memo.

02 Interface design

Form factor, modality and relation

I don't want this to be another chatbot, where you wake up and type into a chat box on a bright screen. That would make me less likely to record my dreams. Instead, I imagine the agent living inside a bedside object, like a lamp or an alarm clock, something closer to a mirror or a portal than an app. It is still another actor (alterity)⁴: you talk to it and it talks back. But it has no chat window, only a voice and a soft glow. Unlike Modem's Dream Recorder⁵, which only records and displays, this object listens and asks.

The idea is also inspired by 周公解梦 (Duke of Zhou Interprets Dreams), a traditional Chinese text and popular system for interpreting the hidden meanings of dreams. People have always brought their dreams to someone, and the agent is a kind of Zhou Gong living by the bed.

There is a tension here, however: Zhou Gong interprets dreams, but in Section 1 I said the agent should add nothing. So the agent changes role across the night:

• Night (ambient): the object stays dark and quiet. There is nothing to look at.
• Waking (alterity, as a listener): a calm voice asks short, neutral questions while you keep your eyes closed.
• Morning (hermeneutic): a screen you read, showing your own words, the details pulled from them, and what repeats across nights. Here Zhou Gong can speak, but only in questions ("Trains came up in 3 of 5 dreams. What do trains mean to you?"). The meaning stays with you.

For this week, the phone stands in for that object: a black screen with a gradient circle that breathes while it listens, plus voice in and out. In the full version (Paprika), the object would be a bedside clock and a headband with EEG, so the agent could also sense which sleep stage you are in.

Interaction specification

03 Building an agent with a novel interface

This prototype is a working version of the bedside object from Section 2. The agent is real and runs as a phone app at dream-agent-chi.vercel.app (password protected). The hardware, a bedside lamp or clock and an EEG headband, is simulated with the phone.

3.1 The prototype

The phone moves through four states during the night:

1. Bedtime: I say tonight's intention out loud, and it is saved in my own voice.
2. Asleep (ambient): the screen goes dark with a dim clock and stays awake. Optionally, my recorded intention plays back quietly after 15 minutes, a simulated sleep-onset cue like Dormio.
3. Waking (alterity): I tap the screen and keep my eyes closed. The agent knows from the clock whether it is night or morning. At night it says "Hi, Ray. You're awake, and that's okay" and helps me get back to sleep. In the morning it asks how I slept, whether I dreamed, and brings back words I said during the night. It interviews me one gentle question at a time, reads the dream back in my own words, and saves it.
4. Journal (hermeneutic): each night has a timeline of every wake and a dream card: a title, the story in my words, the mood and a question for the day, plus a soft, low-definition postcard painted from the colors I mentioned (inspired by Modem's Dream Recorder).

3.2 Technical overview and flow

The app is a web page installed on the phone's Home Screen. It talks to a small Python server on Vercel, so my Mac does not need to be on overnight. Dreams and voice recordings are kept in a private Vercel Blob store. Every time I speak, one turn happens:

1. The phone records me until I have been quiet for about 3 seconds.
2. ElevenLabs Scribe turns my voice into text.
3. Simple rules in code handle the easy cases first: silence gets a soft nudge, "I'm done" goes to the read-back, "I can't remember" logs a white dream and ends.
4. Code decides which interview moves are allowed right now, based on the Cognitive Interview (Fisher and Geiselman 1992)⁶: ask how I slept, ask if I dreamed, free recall, follow one of my own words, what happened before or after, senses, feelings, or "are you done?". "What else?" is allowed once at most.
5. The agent, a smolagents CodeAgent running Claude Haiku 4.5, picks one of those moves, saves the new details, and writes the question, all in a single model step.
6. A guard checks that question for any word I never said. If it finds one, it swaps in a neutral question like "Where were you?"
7. ElevenLabs speaks the reply in a slow, calm voice, and the phone plays it.
8. After the read-back, the agent writes a dream card, and code removes anything on it that I did not say.

The agent has four tools:

I chose a CodeAgent, like in Week 2, because one block of code can save several details and pick the next move in a single model call. That takes about 3 to 4.5 seconds per turn, and about 4.5 to 7 seconds in total once speech in and speech out are added. The most important rules (no new details, my words only, past dreams locked, which moves are allowed) live in plain code, not just in the prompt, so the model cannot break them.

3.3 Code and example run

• App: dream-agent-chi.vercel.app (password protected)
• Code: agents2026-ray/weekly_builds/week03

An example run in morning mode (typed in testing mode on my laptop, Oct 5). Each answer changes the move the agent picks next:

The guard checked every question in this run and found no word I had not said, so nothing was replaced. The automated tests (14 passing) cover the case where it does replace a leading question.

3.4 Reflection

Building it made me realize the ritual matters as much as the conversation. The app asks you to open your phone before sleep, put it by the bed and tap Goodnight, which in effect keeps you off your phone: once it is in sleep mode, the phone stops being a screen and becomes a quiet object. Talking with my eyes closed and only hearing short questions also felt much closer to remembering than typing. When I mentioned "yellow lanterns", it asked about the lanterns instead of bringing up something new, so following my own words worked.

At first the agent was honestly quite stupid. It could only say one or two simple things, like "Mm. What else?" and "And then?", again and again, because my orchestration was not designed well. I had to rethink the conversation as a set of interview moves, and it still needs more time to design the structure of the conversation and its different reactions. Each turn also takes about 4.5 to 7 seconds, which feels long when you are half asleep.

The voice input and the agent's feedback need the most work. Recording stops after 3 seconds of quiet, which can cut me off mid-thought, and the wait between turns breaks the flow. I also want to redesign the modality and the rhythm of the whole interaction. The phone may not be the right object at all, and a bedside lamp or clock could fit the ritual better.

So far I have only tried it in test runs, not over many real nights, so I can't say yet whether it improves recall. What I can say is that the saved record stayed in my own words in every run. A real test would compare a week with the agent against a week with a plain voice memo, counting words, details and white dreams.

04 Final project group

Mateo and I talked and decided to work together on a direction around memory and dreams. For now this is a preliminary plan for our group of two.

The common thread is long-term memory: how a personal agent builds context and memory about a person over time, by combining personal devices such as wearables with designed rituals. Dreams are one way into that, and Mateo's work on interview-based digital twins and memory maintenance is another. The shared cognitive objective is helping people hold on to, and reflect on, their own memories.

Link: Final project

05 Course agent issue

I can't scroll back through my conversation with the course agent. If I want to see something it told me 20 messages ago, I can't, so I end up asking the same question again. A visible chat history would help me learn, because I could go back to earlier explanations and build on them instead of starting over.

Issue: Can't access past conversation with the course agent (#14)

06 AI disclosure

I used Claude by Anthropic as a collaborator throughout this build. It:

• helped me study the assignment brief and the related research
• wrote the code (agents, environments, tools, evaluation and logging) and ran the experiments
• analyzed the results and built the data visualizations, charts, diagrams, drawings and animations
• helped draft, edit and format these notes for the course site

The agency and every decision stayed with me. The ideas, the research questions, the system and interaction design, the visual design, and the choices about what to build, test and keep are my own. I directed the work, reviewed every output, and decided what went in.