Building My First Agent

These notes document my first attempts to build an agent from scratch.

01 Trying to make an agent

First I wanted to understand what an agent is by definition:

An artificial intelligence (AI) agent is a system that autonomously performs tasks by designing workflows with available tools.¹

The important word here is "autonomously." A non-agentic chatbot reads what the user types and answers it, then stops and waits for the next instruction. An agentic AI keeps working toward the goal on its own: it picks tools, plans the steps around them, and adapts to the user's preferences over time.

From the vending machine example in class I learned that an agent has four key parts: the environment, the observations, the brain (an LLM), and the actions with their feedback. After following the tutorial through, I set out to build my own minimal agent loop.

1.1 Minimum agent

Here is the link to my Google Colab document: Minimal agent loop

I recently started to rebuild my personal website, www.rayzlz.com, and found it tedious to move content from where I write it, my local Obsidian vault, to the live site. A content management agent could help. It would 1. detect the changes I make locally, or any difference between my vault and the live site; 2. decide what to do about each difference: update a page, create a new one, or delete one; 3. cope with harder cases, such as a source file I moved to another folder, renamed or deleted by accident; and 4. keep feedback and memory across many runs, because I update my content often.

Building a minimal agent loop requires five main code concepts:

1. The environment: a Python object standing in for the physical or digital world. It remembers its state and responds to actions.
2. The decision model: an LLM call that reads the state and decides the next step.
3. The action parser: plain Python that pulls the chosen action out of the model's text.
4. Action execution: running the chosen function against the environment.
5. A bounded agent loop: the core while or for loop running Prompt → Parse → Act → Observe → Repeat.

In my case the environment is two dictionaries, the local vault and the remote site:

LOCAL, REMOTE = {}, {}

I also added a set of pages that the simulated server fails to publish:

BROKEN = set()

The agent's goal is to make every page on the remote site match the local vault:

p += "GOAL: make every page on the remote site match the local vault.\n\n"

The observations that the environment hands back are:

lines.append(f"{name}: missing on remote")
lines.append(f"{name}: out of date")
lines.append(f"{name}: in sync")

The LLM brain is one function:

def ask(prompt, max_new_tokens=40):

The possible actions that this LLM agent can do are:

ACTIONS_HELP = '''list_pages()
publish("<filename>")
done()'''

In short, the loop works like this:

• The agent compares the local vault and the remote site.
• Based on what it observes, it decides which files to publish.
• It observes again, and the next round of actions depends on what it sees.

1.2 The Zette Agent

I have used Obsidian to organize my writing and knowledge for a few years. But I am lazy and often forget to label or tag my notes, let alone file them into folders. Over those years I adopted the Zettelkasten system for managing knowledge.

Zettelkasten was developed by Niklas Luhmann. It is a note-taking system meant to support thinking as well as writing. Luhmann discovered that his note-taking approach helped him remember information and also sparked new ideas and innovations. The word Zettelkasten in German translates to “slip box,” referring to the boxes in which Luhmann stored his notes. The box's primary purpose was simply to hold the notes, but Luhmann called it a Zettelkasten, and the name has since become well known.²

A Zettelkasten has three types of notes:

• Fleeting notes: quick, temporary notes I write myself
• Literature notes: notes on knowledge from outside sources
• Permanent notes: notes deliberately linked to other notes

The system started well, because I was excited about building a second brain. Then I got busy and stopped writing permanent notes. They take a lot of time, mostly because I had to link hundreds of fleeting and literature notes by hand.

What if an agent could do this for me? Here is the blueprint I proposed:

I gave this blueprint to Gemini to write code I could run in Google Colab and test whether the idea works: Zettelkasten Agent.

1.3 The Zette Agent V2

Strictly speaking, this "loop" is not agentic. It is a deterministic pipeline (a retrieval algorithm): it always follows the same fixed path, Input note → Compute math → Output top 5. The embedding model (TaylorAI/bge-micro-v2) is only its eyes. It turns notes into vectors, but it does not make decisions, choose between tools, or change course based on what it sees along the way. Version 2 adds an LLM that judges each candidate link.

02 Agents for cognitive augmentation (project ideas)

These exercises made me curious about how knowledge works in the mind.⁴ They took me back to when I first used Obsidian. The app is made for people who want to build a "second brain" or "memory palace", because it lets them set up a dense web of backlinks across everything they know.

This reminds me of Simon Critchley's book Memory Theatre⁵. In it, Critchley receives boxes of unpublished papers from a French philosopher who died during a savage summer. One box holds a model of Giulio Camillo's sixteenth-century Venetian memory theatre, a space meant to contain the sum of all knowledge.

Camillo's Theatre of Memory⁶ arranges knowledge in space, much as I was trying to build relationships between stand-alone notes in my vault.

Idea 1: An agent for knowledge

This agent is for people who want to grow, manage and produce knowledge. Many people, myself included, write and collect a lot of information but never organize it. A piece of knowledge only becomes useful when it is connected to others, and making those links is slow, hard work. An agent could read the whole knowledge base, suggest links and explain why each one makes sense. That takes the friction out of indexing and lets people focus on producing new knowledge.

Idea 2: An agent for reliving memory

This agent is for anyone with old photos, recordings and letters from a person or a time they can no longer remember clearly. Objects from the past are triggers that can take us back to specific moments. As in the "Eulogy" episode of Black Mirror season 7⁷, an agent could build a virtual scene by connecting those fragments. You walk through it, correct what is wrong as you go, and the agent rebuilds the scene from your corrections, so remembering becomes active reliving.

Idea 3: An agent for recording dreams

This agent is for people who keep dream journals, and for anyone curious about the thinking that happens while they sleep. Dreams fade within minutes of waking and are hard to write down in that state, so most of the material is lost, and whatever survives sits as separate fragments with no pattern you can see. A pair of agents, one generative and one reasoning, could record your dream by voice as soon as you wake, turn it into a visual, interactive record, and track recurring people, places and moods across months. That shows patterns no single dream reveals.⁸

03 AI disclosure

I used Claude by Anthropic as a collaborator throughout this build. It:

• helped me study the assignment brief and the related research
• wrote the code (agents, environments, tools, evaluation and logging) and ran the experiments
• analyzed the results and built the data visualizations, charts, diagrams, drawings and animations
• helped draft, edit and format these notes for the course site

The agency and every decision stayed with me. The ideas, the research questions, the system and interaction design, the visual design, and the choices about what to build, test and keep are my own. I directed the work, reviewed every output, and decided what went in.