Building My First Agent

W12026

A minimal agent loop built from scratch, an agent that links the notes in my Obsidian vault, and three directions for the final project.

Below are documentations of me attempting to make an agent from scratch:

1. Trying to make an agent

First I wanted to understand what an agent is by definition:

An artificial intelligence (AI) agent is a system that autonomously performs tasks by designing workflows with available tools. (IBM)

The important term here is "autonomous." A non-agentic AI chatbot relies on natural language processing to understand user input and can continue working on a task or pause when no further instructions or questions are provided. On the other hand, agentic AI is designed to learn and adapt to user preferences over time, offering a more personalized experience and more thorough responses by using different tools and designing workflows around that.

In the vending machine minimal agent example I learned that there are four key parts to form to an agent: the environment, observations, the brain(LLM), and lastly actions & feedbacks. After following the tutorial through I then set-out to explore my own minimal agent loop.

Minimum agent

Here is the link to my Google Colab document: Minimal agent loop

Brainstorm for my agent: I have been recently started to renovate my personal website here: www.rayzlz.com and found the process of updating content from my local content production space - my Obsidian, to the cloud very cumbersome. If would be good if there is a content management agent that can 1. Detect changes I make locally or discrepancies of contents between my local obsidian vault and the live site. 2. Analyze such discrepancies and decide on what to act, whether its to update the site content, or create a new page, or delete a page etc. 3. Challenges it might face could be that I have moved the source file to a different folder, or renamed it, or accidentally deleted it. 4. Collect and maintain feedbacks & memories through many iterations of such operations, because I will be update my contents from time to time.

Building a minimal agent loop requires five main code concepts:

  1. The Environment: A Python object representing the physical or digital world that statefully responds to actions.
  2. The Decision Model: An LLM call that reads the state and decides the next step.
  3. Action Parser: Plain Python logic that extracts the structured action from text.
  4. Action Execution: Executing the selected function against the environment.
  5. Bounded Agent Loop: The core while or for loop running: Prompt → Parse → Act → Observe → Repeat.
My initial diagram for this brainstorm

The environment in this case are:

LOCAL, REMOTE = {}, {}

and I've also added a simulated a misbehavior environment for the server as well:

BROKEN = set()

The goal i want my agent to achieve is to make every page on the remote site match the local vault.:

p += "GOAL: make every page on the remote site match the local vault.\n\n"

The observations that the environment hands back are:

lines.append(f"{name}: missing on remote")
lines.append(f"{name}: out of date")
lines.append(f"{name}: in sync")

My LLM brain in this case is :

def ask(prompt, max_new_tokens=40):

The possible actions that this LLM agent can do are:

ACTIONS_HELP = '''list_pages()
publish("<filename>")
done()'''

A minimal version of my agentic loop in this case could be summarized as:

  • The agents looks at the local and remote site environment for discrepancies.
  • It can either decide to update files base on the observations of the environment.
  • The next rounds of actions depends on another round of observations
2nd iteration of the agent architecture

The Zette Agent

I have been using Obsidian to organize my writing and knowledge for a few years already. But I am lazy and often forget to label or tag them let along putting them into folders. I've adopted this idea of a Zettelkasten knowledge management system over the past few years of using Obsidian.

Zettelkasten was developed by Niklas Luhmann. It is a note-taking system designed to support not just writing, but also the process of thinking. Luhmann discovered that his note-taking approach helped him remember information and also sparked new ideas and innovations. The word Zettelkasten in German translates to “slip box,” referring to the boxes in which Luhmann stored his notes. The box's primary purpose was simply to hold the notes, but Luhmann called it a Zettelkasten, and the name has since become well known.

For the Zettelkasten system you have three different types of notes:

  • Fleeting notes - temporary notes that I produced
  • Literature notes - a note that contain external knowledge that I didn't produce myself
  • Permanent notes - a note that is intentionally connected to other notes

This system started off just fine as i was excited about the idea of constructing my second brain, yet I was too busy and just got lazy at building my permanents note because it consumes a lot of time, especially manually building up the connections among hundreds of fleeting and literature notes.

What if an agent can do this for me, here is the proposed blueprint for the agent:

ComponentHigh-Level Architectural Role
GoalGiven an active focus note, automatically discover and rank the top N most semantically relevant notes across any domain without manual tagging.
Environment & StateEnvironment: A vault of N cross-domain notes paired with a high-dimensional vector space ("Smart Index").
State: Tracks active_note_id, candidates_evaluated count, and top_connections payload.
Actions1. get_vector(): Pulls vector representations from the smart index.
2. compute_similarity(): Calculates spatial closeness (cosine angle) between vectors.
3. rank_top_n(): Sorts all candidate scores descending and extracts top N.
ObservationA structured diagnostic payload confirming execution status, total candidate evaluation count, and the top N ranked notes paired with decimal relevance scores (e.g., [Relevance: 0.9982] -> note_id.md).
Failure ModesAutonomously handles missing/deleted note IDs, self-comparison loops, zero-magnitude vectors (blank notes), and score ties without crashing the system execution loop.
Exit ConditionThe evaluation loop terminates immediately upon generating the ranked observation report or when an initial invalid note ID triggers an early error intercept.

I then delivered this blueprint to Gemini AI to produce some code that I can run and test it inside Google Colab to this if this proof of concept work. Link to this Zettelkasten Agent.

Output from the Zette agent in a simulated environment with 50 domain-agnostic notes

The Zette Agent V2

However this "loop" is not necessarily agentic speaking because It is a deterministic pipeline (or a retrieval algorithm), meaning it follows a fixed, linear path: Input note → Compute math → Output top 5. I used the Transformer embedding model (TaylorAI/bge-micro-v2) as the observational tool (its "eyes" or semantic encoder which doesn't make autonomous decisions, choose between different tools, or adapt its behavior based on intermediate feedback.

ComponentArchitectural Specification & Agentic Definition
GoalGiven an active focus note within a dynamically generated vault of 50–100 cross-domain nodes, autonomously discover, filter, and establish valid, logically justified knowledge links using a perception-cognition loop.
Environment & StateEnvironment: A dynamic, multi-domain mock vault (50-100 nodes), a vector perception index, and an LLM cognition engine.
State: Tracks active_note_id, iteration count, raw vector candidates, established_links, and current execution status.
Actions1. generate_vault_library(): Dynamically spins up a multi-domain synthetic node database.
2. vector_perception(): Computes cosine similarities to retrieve top-K geometric matches from the node library.
3. llm_cognition_reason(): Passes note pairs to the language model for semantic validation and reasoning critique.
4. commit_link(): Appends validated connections to the agent's knowledge network.
ObservationA structured feedback payload outputting the live state, similarity score metrics, LLM rationale evaluations, and final link-building results.
Failure Modes1. Surface-level keyword "false positives" caught by vector math but rejected by logic.
2. LLM reasoning timeouts or formatting errors.
3. Exhaustion of the candidate pool before reaching the target link count.
Exit ConditionThe loop terminates when: 1. The target number of valid links is achieved; 2. The candidate pool is completely exhausted; or 3. An execution exception triggers an early abort.

2. Agents for cognitive augmentation (project ideas)

After doing the above exercises, I became intrigued to the idea of knowledge within the cognitive domain.It brought me back to my first contact with using Obsidian. The app is designed to be used by people who want to construct a "second-brain" or "memory palace" because it allows users to define and setup complicated backlink systems for all of their knowledge.

This reminds me of this book called Memory Theatre by Simon Critchley. In the book, Critchley received boxes full of unpublished paper by a french philosopher who died during a savage summer, and one of these boxes contains a maquette of Giulio Camillo’s sixteenth-century Venetian memory theater, a space supposed to contain the sum of all knowledge.

Giulio Camillo’s Theatre of Memory is presenting a new way of spatializing knowledge in a similar way like how I was trying to buildup relationships between stand-alone notes in my notebook.

Idea 1 — An agent for knowledge

This agent is for people who want to grow, manage, and produce more knowledge. I write and collect of information but never systematically organize them, myself included. A knowledge only becomes useful when it is connected to other knowledge, but establishing the link is really hard and time-consuming. An agent could read the whole knowledge base, analyze their connections with each other and suggest links and explain why each one makes sense, so the friction of indexing knowledge could be reduced and let people focus on the production of new knowledge.

my own obsidian knowledge in a interconnected graph view

Idea 2 — An agent for reliving memory

This agent is for anyone with old photos, recordings and letters from a person or a time they can no longer remember clearly. Things from the past carries cognitive triggers that can bring us back to specific moments in space-time. Similarly to the Eulogy episode in Black Mirror season 7, an agent could build a virtual scene out of those fragments from the past by connecting the dots, let you walk through it and correct what is wrong and edit on the fly, and use feedbacks to rebuild it, so the process of remembering becomes active reliving.

Black Mirror, "Eulogy"

Idea 3 — An agent for recording dreams

This agent is for people who keep dream journals, and for anyone curious about the thinking that happens while they sleep. Dreams fade within minutes of waking and are hard to write down in that state, so most of the material is lost, and whatever survives sits as separate fragments with no pattern you can see. A combination of generative&reasoning agents could take your dream by voice as soon as you wake, turn it into a visual/interactive record, and track repeating people, places and moods across months, showing patterns that we often not see from a single dream.

Idea 3 diagram, from a previous system design

References