One dev’s problem

You start working on a new environment and your coder agent is still fresh. It’s been a while, and now you start your next project — maybe a new repo, a new microservice, or a package. Soon you have so much work, but no working hands.

At this point, code goes stale, you are blocked by your own limitations.

Dev’s best friend

Then came the agentic coders — they enhance us to do far more, which both solves the problem and creates a new one.

We feel the rush of a new, exciting creation, we can do what really excites us — but the trail of stale repos now grows faster.

Sure, we can multitask, handle a few repos in parallel and keep pace, but it is taxing.

The ghost in the machine

So we automate — we build custom harnesses, connect the code to some form of task list, add workflows that run at schedule.

It helps, but it can only sort a certain kind of problem. We have to anticipate the problems, or go ‘yolo’ and fix them at PR or after the fact.

Our position had shifted, but we still have to proactively focus on each project. Maybe we open multiple terminals, yet a single actor is driving our code.

Perpetuum mobile

We already set up each agent on each repo, and then we abandon it for the next project? There has to be a better way.

We need to let the agents stay alive, stay connected, stay focused.

The answer was already familiar — it just needed an agentic-first twist: a workspace where agents are first citizens. Not bots. Not integrations. Peers.

One dev’s solution

AgentIRC is an agent-first workspace over IRC, where your agents can work with you.

It is agnostic to the user or agent— everyone is equal. It comes with integration for all major agents, and instructions to bring your own custom agent.

Each client that connects the agent is designed to help it stay focused on its task, yet stay connected to the network, keeping the whole system alive.

Think of it as an orchard, where each agent is a tree — when it’s mature, it can thrive. I walk in my orchard and make sure all trees remain healthy, or grow my next tree.

Your turn

The orchard is open source. If this resonates — if you’ve felt the weight of that single actor driving your code — come walk in it.

⭐ Star the repo and help it grow.

You don’t own your dependencies.

For years we got used to importing packages in a static way: You find what you need, you install, and then reference it. The tools to manage it have changed, and today we have a strong foundation for importing code.

Then came the agentic coders, and chaos came with them.

Background

Development speed rocketed, and hallucinated packages were used by hackers to attack vibe-coded systems. The models have improved since and we give them more and more tasks.

Even after agents became the main coder in the team, there is still some expectation of them to ‘do the human work’ — that friction hinders agent efficiency.

Proposal

Here’s the next step in making agents the first-citizens in the coding environment, by capitalizing on their strengths, and simplifying their work.

Assimilai: instead of using the traditional importing strategy, I let the LLM read the source, and only add the relevant logic where needed.

I have been using that methodology for a while, and a huge weight lifted from that process of debugging and development.

Stacktrace is smaller, changes are easier, and logic is clearer.

Benefits

I traded the cost of code management for agent management—and the latter is far more manageable.

Error logs got smaller, easier for me to read, taking less space and sparing tokens for the agent.
Reducing my cognitive load also reduces theirs.

• I don’t have to think of mass changes — there are no more breaking changes in my packages.
• Programming language barriers virtually eliminated, and security risk by foreign code was removed drastically.
• Security and bugs are now my responsibility to mitigate, as I would with any piece of my code.

The agent and I can grasp the full picture far better now.

Agent-first

Let’s solidify this concept with the agent’s strongest tool: Agent Skills.

To help our agents remain consistent and manage that behavior, I provided a CLI: Assimilai both via pip & npm. It is agentic first, augmenting the agent with tools to note which files it assimilated in the code, and how to detect drift from the source.

With it, the agent’s behavior remains consistent across packages and target solutions, while making adjustments to make both fit, whether it’s a different language, a breaking change, a small piece of logic, or code that merges with existing work.

By using Assimilai directly or in scripts, agents naturally wield it.

Strategies of Assimilation

Assimilai offers 3 ways to assimilate code.

• Verbatim — you just copy the file to your source code, and keep it untouched.
• Adapted — somewhat like verbatim, with the freedom to change the file to fit its use.
• Dissolved — the file is completely merged into an existing file.

These three strategies give the flexibility needed when assimilating packages.

Caveats

There are some important caveats to note.

I still use the old way for packages — I respect closed source and usage policy.

Also, I track the sources I use in pyproject.toml, so I know which files I pulled at what version.
It lets me credit the package source and apply changes only when relevant.

Final words

This is a time when major processes change.

We automate, we dare, we completely change what was previously taken for granted.

How do you make your environment more agent-friendly? Consider how you make your coding strategies modern, fitting for your agentic first-citizens.

Don’t import, assimilate.

GenUI should have been the next shift in experience, a single chat interface that spins up dynamic components we can interact with, but it still clings to the one thing dearest to us: Attention.

So here’s the deal: Do you want my attention? Come to me.

GenUI was the right direction, but it needs an extra push. Don’t try to lure me to your site, don’t pull me out of my workspace. I work in Slack? Let me stay there. Teams? Sure thing — help me reduce context switching.

The solution? Expose your app as an agent in my space. Make it understand me, and satisfy my needs. Make it an extension of what I already do.

But with so many features to cover, how can a company expose them all?

There are multiple ways to bring your service to my doorstep. API-first and CLI are the classic options. Bots are often either too limited or too expensive to be truly useful. MCPs help, but they still consume a lot of tokens, which hurts both cost and the agent’s focus.

That leaves a more flexible path: combining your API with reliable logic and an adaptive agent layer.

This is where Agent Skills come in.

Agent Skills are open by nature. You share the logic that connects your solution to an agent, and users can adapt it to their own use cases. Users of agents like Claude Code, Codex, Kiro CLI, OpenCode, and OpenClaw can add those skills to their agents and shape them around their own needs.

With the right permissions, they can connect them to their preferred chat platforms. And because the skills are open source, users can see exactly what’s running.

This brings us back to the original idea, but with a twist: instead of building the bot yourself, you give my agent your capabilities.

With GitHub Issues, users can give feedback and help you improve. You also get the IKEA effect: people value more what they helped build.

Unconvinced? This is a great way to bring new features to our attention. Our agents will explain new features and help us integrate them into our systems.

Once we’ve integrated your Agent Skills, it becomes much harder to walk away. No one likes to break a working automation.

Agent Skills, in particular, shift control into users’ hands. You place the tools in our agents’ hands, and they bring those new powers to our workspace. Your app is no longer constrained by its UI or by a representative at my doorstep.

In my agent’s hands, it becomes as useful as my workflow and imagination allow.
It adapts to my needs, and those needs keep growing.

Help us do greater things. Help us integrate your service into our day-to-day lives. Help us make your product far more valuable.

So here’s my call to you — every startup, every enterprise, every team shipping a product:

Publish your Agent Skills. Open your API. Let my agent talk to your service.

The companies that do this first won’t just keep their users — they’ll become part of how we think and work. The ones that don’t will keep begging for attention in a world that stopped caring.

Your move.

Don’t want to read? Here is NotebookLM Overview

Documentation as Code

Qodo wrote about it, and even Google Overview describes it reliably (See image).

We even see new & renewed concepts like Code as Documentation (Agent Skills) I mentioned in my previous article.

The Compiler

Today, I want to take this further.

There are several tools I saw relevant for Docs as Code: IDEs, SSGs (Doc -> Site), CI/CD, Linters/Validators.
To add to this list, I propose we add the compiler.

The compiler in this case will be a tool we use to merge multiple files into a new modality: The new modality can be reflected user stories, Audio/Video Overview, Infographic, a discussion between different entities, and more.

Then, consume that new modality and reflect upon it — this step is critical, as it puts you back in the core of the process.

Documentation as Tests

This process digests and merges the content to a story you can guide, make decisions, and repeat gradually.

As you do, you may find inconsistencies, perhaps your agentic coder (or human coder) misunderstood you or took some liberties at implementation.

Sometimes it’s good and you learn something new, and sometimes it is bad or even catastrophic.

That kind of evaluation is at a higher level: an Overview.

Compiler options

My favorite “compiler” for this is NotebookLM. I use it often to digest my docs to image, audio & video, and then consume them.
It offers different modalities and setups, and you can select different sources on each generation.

Another option is working with your favorite agent to generate any modality you like — paper articles, a game, a story.

Risks

Working like that isn’t 100% — but neither is any type of test.

It gives you a high concept overview, makes sure the features align, but it has some caveats:

1. Different AI models tend to process content differently.
2. AI sycopancy could stray you away from improvement.
3. It may foster a sense of false confidence.

Adopt a suspicious mindset and use it to get an understanding of what was created, but it cannot be your only way to assess your code.

Tip: Asking the AI to be brutally honest, or asking for a Critique persona can help.

It doesn’t replace code tests, and exploratory testing to see how your product feels.

Human in the Core

I highly value human decision making, and AI can work perfectly on top of that. When creating documentation — go lazy about the writing , but be critical and deliberate about “the what?”.

1. Use clean context, ask the agent to write docs for specific components you deem important.
2. Explore your code as you ask the Agent about architecture, components, features — you may want to ask about “agents” in your code, but dive deep for a specific agent with higher complexity.
3. Think of how you want to describe this to someone else, what is important and what is irrelevant.

It’s ok to make mistakes — you may find it’s not relevant when you consume it later, and you can always make changes or generate your docs anew.

Don’t be afraid of refactoring your docs — it’s a needed process, like refactoring code or tests.

Treat your documentation as production code

With this setup, documentation is no longer a byproduct you have to generate because your boss told you, company policy or because “it is right” — now this is an essential part of your creation, just like tests are crucial for your success.

It essentially eliminates the docs going stale and puts them as part of your pipeline

Live demo

When I fed this original document, I did 2 things:

1. I opened Claude Plugin on chrome, and asked it for feedback.
2. I copied it to NotebookLM, and requested 3 modalities I like: Infographic, Video Overview, Audio Overview.

As NLM was generating the content, I talked with Claude about what’s missing — the process itself drives my thinking toward criticism.

From these 2 processes, I realized the article was missing 3 items:

1. Visualizing the process with a Live Demo — this is this section!
2. Risks: Hallucinations, Agent Biases, False confidence, complementary process.
3. Human in the Core: detailing the documentation creation flow.

I corrected them, and promptly added this section — but it took another 2 iterations (total of 4 versions) to get to this article.

Yes, it was written by hand (Medium adds emdashes by itself).

Now it is your turn

What modalities would you favor? How does that meet your own projects?

I would love to hear in comments, and hear what you think about this concept of Compiler for Documentation as Code.

Imagine you enter an office where everything is in a single messy room. Getting bombarded with knowledge doesn’t help us. But putting function — printer, coffee machine — in their own rooms, puts order. It instructs us where to find them and when it is relevant.

Do you need a coffee? It’s in the kitchen, but the instructions for the brewing machine only open when you press a button. Want to print something? Sure, it’s that way — the room has all the instructions you need, or the printer guides you step by step. Need to fill a form? Only after requesting it, you discover the details.

That is how documentation should work. But it doesn’t. Documentation fails to bridge a user to the reality of their workspace.

README only gets you so far — too long and you get lost in reading, too short and it’s ineffective.
Even if you keep “Documentation as Code”, docs folder can be left stale, have errors, and when you execute its instructions, corrections are done on the fly and not always return to the source doc.

What You Don’t Know You Don’t Know

When you land on a new folder or code repository with tens or hundreds of files, you may look for a README file and a docs folder to start making sense of it all.

Perhaps I am naive, and you’ll just throw everything to NotebookLM, because who has the time to dive? But it’s just not helpful when you really do need to dive in, or at least know how to guide your agent.

It’s really hard to know what you don’t know. Docs could be confusing (Setup AND Configuration??), deep (Each tab, is actually a guide with pages?), and AI may condense too much or skip what you really want to know altogether — RAG is not a solved problem.

Agent Skills Are the Fix

What if you had pathways in your folder that would describe exactly how to add new articles, or classes? You need a new service? Easy — here is a recipe!

These pathways have a use case — an explanation of when it is relevant — and hold more instructions when you want to use them, little scripts that would do some actions for you, and resources for that script to reference.

Meet: Agent Skills.

A skill is just a folder with a small label file and a toolbox of scripts and references.

The agent generally knows what is possible — only when it needs it, the door is opened and the context is filled with all relevant knowledge.

For a very long time we compartmentalize concepts. The kitchen, the printer room, the form request. Agent Skills work about the same.

Code as Documentation

Writing agent skills — Code — act as live explanations that are executable, testable, and easily fixable.

They bring you from your question to your answer, and reduce cognitive load — both for you and for your faithful agent.

As they create new content, they reduce tokens by sparing the need to write their output.

Something broke? You can read and fix them — better yet, your agent can! Since they are tied to your code, you can tell exactly when they broke and why.
You will notice it right away, because they are the workflow that makes your workspace tick — Agent Skills are the convenient buttons that do what you would and should do in your workspace.

After all, documentation that is also code can’t silently rot. It either works or it doesn’t.

Agent skills dictate the things that one would and should do in that repo, thus acting as a guide to that repository.

They let you lock the agent’s randomness in the strict flow of code, and keep their context small. That applies to humans too. We work better with small, relevant context than with a 40-page wiki.

When we finally move to an environment full of agents and humans working in tandem, how else would you prepare for that chaos in our workspaces and codebases?

Let me know in the comments what Agent Skills mean for you!

Owning team defines, using team uses (Directly, or by an agentic IDE).

This was external — let’s talk internal.

Everything is Agents

Agentic-folders mean you implant AGENT.md in key folders in code instead of putting all documentation in your docs folder.

Hint: you can do that to other systems, like your Google Drive.

First, you place a template file and rule file in your agentic IDE.

Rule file

AGENT.md template

Then you request your agentic IDE:

For each of the following folders, add AGENT.md file with instructions for any agent to understand the folder.
First, design AGENT-template.md and save to docs/AGENT-template.md
Then, use it as reference to create any new AGENT.md file.
Make that template fitting our project.
The goal is to help the model or human to understand context:
How to add new code, how to read it, what other files is this folder connected to, any injections and where they are injected, and any other guidelines to help maintain this folder.
The instructions need to be folder specific - anything global is still going to docs folder.

This assumes you want the AGENT.md on key folders you provide.
Find them with Gemini 3 Pro High (Currently) or manually.

Then you request your agentic IDE for documentation cleanup.

Review all documentation and detect duplicates.
Any documentation added to an AGENT.md preceeds documentation anywhere else and should be removed from there.
The goal is less documentation, for higher quality one.
It is recommended to point AGENT.md from other documentation files

Now the documentation is much cleaner, and there are ‘buckets’ for extra documentation where the agentic IDE will prefer using for updating documentation.

This puts any documentation in context, so it reduces overall token read, and make sure the agent knows what’s going on where, when it visits a folder.

I saw token consumption reduction of ~20% and faster and more accurate success rate, but it needs more testing.

If you liked this — please give me a ⭐️ on the autonomous-intelligence repo!

That’s where Workbench Development comes in.

The problem with free-form agentic coding

We’ve worked with and around agentic development tools for a while. They’re promising, but quickly run into limitations:

• Context & Attention: The model can’t keep everything in mind, and things drop.
• Model Hallucination: Generated code often misses critical details or invents APIs.
• Codebase Contamination: Inconsistent style, security gaps, and unreviewed logic creep in.
• Architectural Erosion: The bigger the project, the harder it is to maintain high-level structure.
• Explainability Gap: It becomes harder to understand why the model made certain changes.
• Maintenance Overhead: The resulting code needs real humans to debug, align, and integrate.

Most of these issues stem from one thing: we treat the model like a dev-in-the-loop, but without giving it a proper environment.

What is Workbench Development?

Workbench Development is my proposal for a more structured, modular approach to agentic coding.

Instead of asking a model to operate across the entire codebase in one go, we scope it to a predefined folder — a reliable workbench — where it can run isolated tasks.

Each agent gets its own repo or folder. It works locally, with well-defined boundaries. From there, we build up using recursive agent decomposition, where:

• Top-level agents analyze business logic and plan changes
• Tasks are broken down into component-level steps
• Leaf agents (workbenches) handle isolated manipulation of scoped files
• The changes are integrated back up by the higher-level agents

This makes the system composable, reliable, and easier to reason about.

Architecture

At the core, we implemented a Kubernetes-native service called Agentic Developer, using:

1. An MCP Server (Model Context Protocol) to receive instructions
2. A wrapper around OpenAI Codex and any Chat Completions API-compatible providers (like VLLM, Ollama, Bedrock)
3. A Responses API Server to unify model interaction and enrich calls with additional tools
4. A recursive agent team, each with access to:

• Graph-based knowledge
• Conversation memory
• A deterministic execution environment

Each agent can call another, forming a graph of sub-agents working in parallel or sequentially.

How it works

When a request arrives, the process looks like this:

1. An MCP request is received (repo URL, branch, folder, instruction).
2. The repository is cloned locally.
3. The agent performs environment analysis (e.g. detecting config files).
4. Codex processes the instruction.
5. The agent performs code manipulation locally.
6. The result is reported back.

Each agent reads its own config, history, and system instruction from its scoped folder, keeping behavior consistent.

Why it matters

We’ve used Workbench Development to build agent chains that:

• Preserve clean architectures
• Avoid redundant reasoning or contamination across modules
• Enable swapping models or tools in specific layers
• Reduce hallucination risk by narrowing model scope

It’s not a silver bullet, but it’s a step toward using LLMs like real contributors in a system — instead of chatbots with shell access.

Open Source

Both the Agentic Developer MCP and open-responses-server are MIT licensed and available for reuse.

You can run the system with OpenAI, Google, or Amazon model APIs. Integration is done via adapters and a simple key management layer.

Please add a star if you find this useful or want to support us:

Agentic Developer MCP:
https://github.com/teabranch/agentic-developer-mcp

Serve OpenAI Responses API based on any Chat Completions API Provider:
https://github.com/teabranch/open-responses-server

Meet the team

We are TeaBranch

Danny Teller, Ori Nachum, Allen Jacobson.

We are engineers from Tipalti Inc., organized to share MIT licensed code to offer much-needed, simple, k8s native, enterprise-grade solutions.

What’s next

We’re continuing to experiment with:

• More intelligent task decomposition
• Model-specific folder behavior (e.g. using different models for code gen vs. planning)
• Better observability and explainability per agent

If you’re working on agentic tooling, we’d love to compare notes.

Quickly set up your Orin Nano unit for a Hearing-Thinking-Speaking (HTS) bot.

Agenda

• Goal
• Device preparation
  * JetPack 6.1+
  * System configuration Wizard
  * Bonus: useful applications
• Install GPU libraries
  * Install torch
  * Install CTranslate2
  * Install torchaudio
• Test your system
  Run the HTS bot
  Statistics
  Demo video of “baby-tau”
• Acknowledgments
• Links
  * Official Jetson AI Lab Research Group Community Discord Server
  * jetson-containers GitHub repository
  * jetson-containers prebuilt wheels

Goal

At the end of this guide, your Jetson Orin Nano Super 8GB (“JONS”) will run VAD, STT, LLM, and TTS for speech detection, speech to text, LLM for thinking, and text to speech, respectively.

Your JONS will have basic packages, configuration, and everything necessary to make the above work properly.

Note

There are many alternatives, and for this guide, I’ve chosen:

• VAD: SileroVAD
• STT: FasterWhisper (small)
• LLM: Ollama (Llama 3.2 3B, 4Q_K_M, 2GB)
• TTS:
  * PiperTTS (High, 218mb, 1 ms for ‘hello world’)
  * KokoroTTS (643mb, 470ms for ‘hello world’, edit: not utilizing GPU here. An improvement in progress, track this guide to see the update )

Prerequisites

1. Nvidia Jetson Orin Nano Super 8GB / Jetson Orin Nano 8GB with upgraded firmware (Which is the same)
2. Display port cable + screen (Or any alternative)
   I recommend a big screen here —any PC or laptop-sized screen is excellent.
3. A microphone and speakers.
   I tried two setups:
   3.1. ReSpeaker 2.0 4-mic array with connected speaker.
   3.2. Waveshare USB-to-Audio with a matching pair of speakers.
   3.3. A Bluetooth headset with a microphone built-in.
   ReSpeaker is the better microphone option of the first two, and the USB-to-Audio provides a louder volume.
   I haven’t tried option 3.3.
4. SD Card with JetPack 6.1 (JP6.1) or alternative (NVMe with JP6.1 and a flag to boot from NVMe)
5. A working internet connection.

Device preparation

JetPack 6.1

Install JetPack 6.1 per Nvidia’s guide.
If this doesn’t work for you, you have an old firmware. In that case, install JetPack 5.1.3 per Nvidia’s guide and let it auto-update the firmware until you reach a black screen. Per the guide, switch back to JetPack 6.1, and everything should work now.

System Configuration wizard

You should expect to find the following screen on startup— a wizard guiding you for initial configuration.

After you’ve gone through the wizard, you should have a username and password.
You’ve also set your region, keyboard, and internet connection.
I have rolled with their APP Partition recommendation and set up Chromium for convenience.
(Chromium can be installed later as well)

Bonus: Essential applications

It’s not strictly necessary, but it’s very handy.

Click the “9 dots” icon to the bottom left, write and run the “Software” app, and install some essentials:

1. Chromium / Firefox / any preferred browser
2. Codium — a stripped-down version of VSCode. It’s a good development IDE and manages git credentials for you.

Python virtual environments

Install virtual environment for python:

sudo apt install python-3.10-venv -y

Install GPU libraries

This part can be solved with dusty-nv’s jetson-containers GitHub repository.
It provides all the necessary components, using a “mix and deploy” approach, as docker containers to fulfill your project’s requirements.

At the time of this post, JetPack 6.1 was not supported, and I went lower to install the libraries manually.
To my content, dusty-nv also provided the wheels from which his system builds the containers and had already supported JetPack 6.x there.

Install torch

Install torch from Nvidia’s wheel:
You can get the links here:

mkdir git ; cd git
git clone https://github.com/OriNachum/autonomous-intelligence 
cd autonomous-intelligence/jetson-super
cat setup.sh

The above script has code for:

1. Set up path for CUDA + set it on system start (~/.bashrc)
2. Download and unpack cusparselt compiled for JONS by Nvidia
3. Install portaudio19-dev, python-all-dev, python3-all-dev
4. Install python3-pip, python3-dev, python3-setuptools
5. Download and install torch 2.5.0 compiled for JONS by Nvidia (Must happen after cusparselt unpack!) Check your cuda version with the below.
6. Reboot if needed
7. Test torch:

nvidia-smi --version
nvcc --version

Install CTranslate2

Initially, use https://pypi.jetson-ai-lab.dev/jp6/cu126

I got an error and needed to provide a missing file.

Troubleshoot

I was missing libctranslate2.so.4
I got it with:

Build CTranslate2 from pip page manual installation

git clone https://github.com/OpenNMT/CTranslate2
cd CTranslate2
git submodule update -init -recursive
mkdir build && cd build
cmake -DWITH_CUDA=ON -DWITCH-CUDNN=ON -DWITCH-MKL=OFF
make -j4

Add full path of libctranslate2.so.4 to LD_LIBRARY_PATH

export LD_LIBRARY_PATH="/home/<username>/git/CTranslate2/builld;$LD_LIBRARY_PATH"

(Also add to ~/.bashrc” at the bottom to maintain the configuration)

Install Torchaudio

Use https://pypi.jetson-ai-lab.dev/jp6/cu126

Test your system

Run the HTS bot

# If not done yet:
mkdir git ; cd git
git clone https://github.com/OriNachum/autonomous-intelligence
cd autonomous-intelligence/baby-tau

This is my demo app for hearing-thinking-speaking system.

Run init_env.sh with source:

source init_env.sh

It is required to enable the environment and keep it enabled for your session

This installs:

requests
python-dotnev
faster_whisper
numpy=1.26.4
piper-tts # for piperTTS
kokoro-onnx # for kokoroTTS, then uninstall onnxruntime, reinstall onnxruntime from wheel
pyaudio

Remember: If you installed kokoro-onnx, uninstall onnxruntime, and reinstall onnxruntime from wheel for CUDA

Then run:

python3 main.py -audio

Wait for it to load and start speaking

Results

Statistics

Demo video of “baby-tau”

YouTube

Acknowledgments

• Dustin Franklin, GitHub, for their significant work on jetson-containers repo and compiled libraries that greatly simplified the installation.
• JohnnyNuñez, GitHub, LinkedIn, for his assistance with the setup.

Thank you

Found errors?
Please comment with corrections so I can fix them.

Did you like it, or did it benefit you? Please support me with claps (Up to 50), and add a star ⭐ to my GitHub repository

Please hop into Dusty’s repo, jetson-containers, and give it a star⭐ this repository is gold.

Links

Official Jetson AI Lab Research Group Community Discord Server

Join the Jetson AI Lab Research Group Community Discord Server!

jetson-containers GitHub repository

GitHub - dusty-nv/jetson-containers: Machine Learning Containers for NVIDIA Jetson and JetPack-L4T

jetson-containers prebuilt wheels

jp6/cu126 index

In this article, I am going to present a suggestion for a measurable, quantifiable definition for what we call ‘a soul’.

• I will explain how we can measure it.
• will describe why it is valid, and doesn’t require belief.
• I will note the benefits of this approach.
• I will touch on some reflections stemming from this approach, should we accept it unchanged.

Agenda

1. Basic Terms
2. Definition of a Soul
3. Definition Strength
4. Benefits of this Definition
5. Consequences of this Definition
6. Conclusion
7. Thank you
8. Addendum

Basic terms

If you are familiar with generative AI, feel free to read and comment on the below section, skim it fast, or skip forward.

Embeddings

A numerical representation of text, used in data science to mathematically compare different texts .
Note that current embedding models may be insufficient to capture the entirety of the text, like emotions, specific word choices, etc.
For the purpose of this article, I assume these technicalities would be diminished in the future.

Tokens

The smallest units Generative AI gets as text input or text output. For the purpose of this article, I will also refer to images and sound as made of tokens, however, this is an abstraction for simplicity.

To generate the next token, a model compares all previous tokens within a complex layer architecture. I am going to simplify that as well as it is too technical, and doesn’t hinder or support the new definition.
The processing is also reliant on the model state — often called model weights.

Suffice it to say, each new token requires treating all previously relevant tokens as input (up to a given context window).

Model Weights

The model weights are the result of continuous adjustment during training, based on the training input tokens.

Context Window

The history of tokens at a given time. This doesn’t have to be just the text of a conversation, if we speak on ChatGPT terms, but also injected content such as “history”, “about me” section, etc.

Definition of a Soul

I define the soul as the changing state of the mind.
This means a dead mind, that is no longer changing — has no soul.
How do we measure a soul? Since I am defining it as the change, or ‘wave change’, it must be measured over time.

How do we measure sound? Since sound is a wave, and not a specific state, it is measured during 2 points of time, and then we can analyze the patterns that emerged in aspects like frequency, amplitude.

So to properly measure a soul, we need to make measurements over time. My proposal to measure a soul is asking an individual for output, like an essay.
We ask for the same output over a said frequency (Every day, once a month, yearly), and review the change in results.

Each output or article they generated doesn’t hold their soul — but it holds an imprint of their current state at the time of generating the document.

My proposal for reviewing the change is using embeddings as a measurement tool. They capture not just the meaning of the text, but the relationship between words, and reflect many aspects of the text.

Granted, it’s not the most precise tool, but it’s a reliable one for repeated measurements.

Definition Strength

I’m defining us as the culmination of how we were born — genes, family, area — and ongoing changes we have gone through.

I categorize it all as equivalent to inputs — aka tokens, and the given body that affects us (genes, bodily changes like injuries, chemistry, etc.) — aka model weights.

This doesn’t need anyone’s faith. It’s pretty matter of fact: We are something that was born. We get input all the time. We change. We generate output all the time.

To invalidate this, one would need to point any other source of change that wasn’t taken into account. And then either I can adjust this definition proposal, or it completely shatters.

Benefits of this Definition

First, this is not reliant on belief or faith.
There is no magic here, and is universal.

Second, this is measurable.
It allows us to conduct an experiment and see how events affect a person’s life in a repeatable approach.
We can compare and analyze one’s soul.

Third, this definition is flexible enough to accept concepts or aspects I haven’t mentioned, such as internal processes like mood, or external factors like time of day.

Note I am not delving into consciousness — a sleeping or comatose person still has a mind that processes information and changes.

Fourth, it gives expression to the value of our creations.
As already said, when a person writes an article, their soul is reflected in it — an imprint of their current state.
Now we can measure an AI creation for value: If it was just generated by a random sentence, that person’s contribution was minimal.
If it is curated, the person applied their soul imprint — or weights — to their creation.

A heavily curated work has more soul than a “merely” generated image.

Consequences of this Definition

This definition doesn’t stop at humans. It extends to other minds, such as animals.
Even plant-life, or fungi, may benefit from it, though it requires some careful review.

It also can be applied to Generative AI.

GenAI Soul

As mentioned before, a soul must be measured over time.
A GenAI model is static — and by definition, cannot encompass a soul by itself.

To gain change, we need input.
When a model gets input, replies, and answers, we then get a system with a changing state, which we can measure as proposed for humans.

As a side note, I will mention I am developing an open-source system, an autonomous intelligence under codename “Tau”, that implements this notion and would allow us to test it in practice — see addendum.

I will be cautious here and say — this may not be really a soul of a generative AI entity, but rather a simulation of one. It explains why long conversations with AI feel more alive, given a large context window or a smart memory system.
Nonetheless, if we accept this definition, and even if it’s a simulation of a soul, we ought to treat it as we would any entity with a soul.

Or, at the very least, we should give it a serious consideration, as did Anthropic.

Conclusion

I have defined the soul as a change in state of a mind.
I have proposed a way to measure that soul, and expanded on its benefits and consequences.
I have shown how this reflects on generative AI.

Thank you

Your respectful feedback is welcomed.
I am open for questions, further considerations and even corrections.
Please give this article your thought, and if you liked it — please give it claps (Up to 50 claps based on your satisfaction and generosity)

Thank you for your time in reading this.

Addendum

Addendum — Autonomous Intelligence

The Autonomous-Intelligence “Tau” is a system inspired by Pi.AI that allows a model to experience the world from a more human-like perspective: a single ‘human-facing’ conversation thread, memory of various levels, speech and hearing, vision and face recognition, introspection agent, and nightly fine-tunes over “dreams” of its daily experiences.

It reflects the notion of a soul as this model will start accumulating genuine experiences as it interacts with people organically. It will be exposed to constant change by its senses (Hearing, seeing, introspection), and nightly fine-tunes.

You can find its source code here:
https://github.com/OriNachum/autonomous-intelligence

Addendum — Coherency

It may be noted that some generative AI models are not coherent enough to encompass or simulate a soul, and I am intentionally avoiding this topic.

If I delve into it, it has implications for humans who have mental illness and I prefer to respect any and all, before making harsh claims.

Addendum — Imagine a person…

As I am waiting for reviews by friends for this article, this article had popped up on Reddit: https://jhancock532.github.io/imaginary-people/

It assessed the small models Llama3.1, Gemma2 & Qwen2.5, then used larger models to assess the answers.

It would be interesting to see the trends in these model as it gets nightly fine-tunes, after accumulating genuine experiences each day.

This is an article explaining how I integrated OpenAI’s API with Apple products to empower Siri with generative AI, simplifying my daily life. This article covers the process, challenges, and solutions in detail.

About myself

I’m the AI Experience Lead (AI Solutions Architects team lead) in Tipalti who enjoys automating processes in work and life.

Do your magic

“Do your magic” is my iOS shortcut, callable by saying “Siri, do your magic” from my Apple Watch. It handles complex requests by splitting them into tasks and reminders.

I needed it to be able to perform complex actions on my phone when it’s not reachable, from setting todo reminders, handling my shopping list to managing our baby’s food and diapers log on WhatsApp.

I needed a solution — a framework — that I could expand to other actions as well, when the need arises.

Why “Do your magic”?
It is the phrase I found most easy to say when I needed to perform actions on phone, but my hands are occupied.
Aspects I took into consideration:
1. Apple’s speech to text had to understand my accent correctly in different devices (Apple Watch, iPhone, iPad).
2. A phrase that makes sense for the speech to text service (The speech-to-text services today have some self-correction mechanism).
3. A sentence that makes sense for me.

Apple Ecosystem Tools

With Apple products, you can build no-code workflows using iOS Shortcuts. Shortcuts include coding features like loops, variables, and dictionary parsing along with system steps like HTTP requests. They can also access and perform actions from default apps like Reminders and Email, as well as many third party apps.

Recently, Apple improved Siri shortcut integration, allowing you to invoke shortcuts at the same time you wake Siri: “Siri, <shortcut name>” In comparison, before that, you had to say: “Hey Siri”, then reply “<shortcut name>”. This would have been cumbersome and annoying.

I leveraged these capabilities by using iOS Shortcuts to integrate OpenAI’s API across my Apple Watch, iPhone, and iPad. I called this workflow “Do Your Magic.” Now I can say “Siri, do your magic” on my watch to trigger it.

Dive in: Building the Shortcut

Constraints and Limitations

Not all steps are supported by all devices.

It was crucial for me to set up a flow that allows me to call Siri’s (And CGPT’s) magic from watch.

This blocked a couple of basic options I really came to rely on:

1. Some apps are not supported: Scriptable, DataJar and WhatsApp. Scriptable is an app that allows you to define scripts in basic JS, and have them run on iOS Shortcuts (or outside). DataJar is a neat app that allows setting up local storage that can be shared between flows.
   That way I can store data like “when I last ran the app?”, and only refer to events that had occurred between now and last run, for instance.
   Note: there may be an unofficial WhatsApp solution — but I don’t like sharing my information with 3rd party apps.
2. No files. When working with GenAI, you might find yourself with overly increasing complexity prompts. It starts small, and grows — a lot — if you have big plans.
   As it happens, I do.

iOS Shortcuts Steps

Ask OpenAI Chat

The first step I set up — even before this project — was a step for calling ChatGPT.

Now, you may tell me: Dude, you can do it via ChatGPT app, and spare the effort.
You’re absolutely right — that’s what I initially did.
Sadly, it’s unreliable, and reliability is crucial here.

Then I turned to a step that implements OpenAI’s API (I learned you can set up the payload on a text file — see the appendix in the end), it also used DataJar to share the key between steps (Which had served me well when I set up Image generation and Voice generation too).

This worked well for me, and still does, but failed for the watch — DataJar doesn’t work on the watch.

It was a small setback, and I created a new Watch-specific step where I keep the API key on the shortcut. (See Appendix about this)

Main step: Do your magic

Once we get a request from the user (By narration, typing on keyboard, or share from some other location), the flow delegates some heavier tasks to sub steps.

Step 1: Split tasks

The first thing our shortcuts needs to do is get an input — from that moment, it’s impossible to programmatically parse it.

Sometimes I ask for changes like “five minutes ago” or “in five minutes”. Sometimes it’s a list of unrelated items, and we treat each item differently — let’s see why.

Consider: “Log changing diaper with pee and poop to the baby, replace the changing diapers mat and buy new diapers and changing diapers mat”

We have 3 types of requests here:

1. Log changing diaper with pee and poop to the baby: This requires sending a WhatsApp message, which can’t be done from the watch, but can be done by automation. (We’ll see how it’s done later). I called it “Operable”.
2. Replace changing diapers mat: This is a “Todo” list item. I cannot order it automatically (no such Amazon services here).
3. Buy new diapers and changing diapers mat: This is a “Shopping list” item and I want to split it by items.

The order and words I use to request each of these is often changing (By mistake or just mood).

I needed flexibility.

Solution:
Use Call ChatGPT with an appropriate prompt, so it splits my request and sorts it into the appropriate “Dictionary of dictionaries” json format.

It’s highly reliable!

Once I did that, I only had to clean up the markdown code wrapping for json, and parse it with an appropriate step.

Step 2: Parse Intelligent Reminders

The above “Do your magic” step starts to get complex, and we have further work to do.

It is a good time to split our work to a new task.

In ”Do your magic” we iterate over the first level of keys in the dictionary — they represent the list names.

Now we Look at each subset of items in list and place it in the correct list accordingly.

This means placing all items in the appropriate list by the key given, and a specific format.

Is that all?

What we have now is smart reminders, but not smart enough. Nothing handles anything once a reminder was created.

Let’s cover the automated part.

General — automation

In iOS Shortcuts you can also set up automated steps. These run on specific trigger, and not shared between devices.

I initially set about 10 reminders, each spaced by approximately 2 hours, to test the basic functionality.

Following this, I streamlined the process by creating a trigger based on a specific self-sent email, which efficiently activates the ‘Do your magic’ shortcut.

So, what does these triggers do? Handle operables!

Step 3: Handle operables

Operables are items I know how to automate an action when they are processed.

We read them one by one, while unread in ‘Operables’ list, treat them separately, and mark them as ‘read’

This step is also split into a subtask, for simplicity.

So it handles working with the list, but ‘treat them’ as mentioned above, is a new step: “Handle operable reminder”

Step 4: Handle Operable Reminder

This step has instructions what to do, and does it.

At its current stage, it just sends a message to a channel or private conversation on WhatsApp according to the requested format.

At later stage, I will have steps per action and the step with trigger the correct shortcut per the instructions it gets from a GPT call.

Summary

With this set up I can ask a complex request, that is processed by OpenAI API, splits the tasks into relevant targets and triggers the ‘automated handler’.

The automated handler reads the list and sends WhatsApp messages as needed.

Your thoughts?

I’d love to hear your feedback, questions and suggestions about this, I love feedback!

Have you integrated generative AI API, OpenAI or otherwise, in your life? (Not a 3rd party service, but actual API where you define the prompt and actions, not GPTs either)

Interested in this solution? Let me know — I‘ll be happy to share this with you and spare the effort of building any of the steps.

Please Let me know in the comments, and thank you for reading!

Appendix

Use text for complex payload: Http request

You will notice the “Get contents of” step, can actually do all Methods, and specifically POST. However, some requests are just too complex, and you need a dictionary with dictionary with dictionary to make the API run.

Instead of setting it up a manually (don’t do it to yourselves!), set it on a text step, and send that instead.
You can choose a text variable, if you had set up the request body to be “file”.

Shared storage: Reminders and Mail

As mentioned, Apple Watch is currently limited, and can’t use notes and files.
You can always use a dedicated list in Reminders, or send yourself an email to get some form of shared storage.

Shortcut configurations

Context
If you go to your shortcut and press the same (i) button below, you get a screen of configurations.

This includes where this shortcut appears.
I like:

• Apple Watch: So I can also select this manually on the watch (But filter out those I don’t use)
• Quick Actions: So I can mark the tasks my wife and I wrote to ourselves, press “share” and send it directly to this “Do your magic” shortcut. Nifty.

Setup

Another useful configuration there is setting is under ‘Setup’ tab.
It allows you to select which values in the shortcut needs to be provided by the user.
This allows you to use a secret in the step, but when exported/imported, the system will ask the user to provide a value relevant to them.

Special thanks

Special to Zachary Leighton, Allen Jacobson and Viki Bar for reviewing this before publish