<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ben Rebernik on Medium]]></title>
        <description><![CDATA[Stories by Ben Rebernik on Medium]]></description>
        <link>https://medium.com/@ben.rebernik?source=rss-899e34ea15a3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*GBIkIDQLTJlmW0k7</url>
            <title>Stories by Ben Rebernik on Medium</title>
            <link>https://medium.com/@ben.rebernik?source=rss-899e34ea15a3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 19:55:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ben.rebernik/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building an Autonomous AI and why AI chatbots aren’t the AI of Today anymore]]></title>
            <link>https://medium.com/@ben.rebernik/building-an-autonomous-ai-and-why-ai-chatbots-arent-the-ai-of-today-anymore-ad4c1ea91890?source=rss-899e34ea15a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad4c1ea91890</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[cost-saving-tips]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[automation]]></category>
            <dc:creator><![CDATA[Ben Rebernik]]></dc:creator>
            <pubDate>Mon, 26 Oct 2026 14:16:25 GMT</pubDate>
            <atom:updated>2026-10-26T14:16:25.420Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>TL;DR</strong></p><ul><li><strong>Chatbots wait.</strong> They predict the next token, answer once, and go quiet until you prompt it again.</li><li><strong>Agents act</strong>, but only when you tell them to. They loop through Perceive, Plan, Execute and Verify, using tools to work in your files and apps.</li><li><strong>Autonomous AI</strong> doesn’t wait for you. It runs on events, like a new email or a note you said out loud, and decides on its own what matters. A morning briefing script is just a timer.</li><li>Watching your screen with screenshots all day is too expensive. My rough math puts it at about $250–600 per month per user, depending on the model and image size. Vision models can now complete desktop tasks well, so cost and privacy are the real problems, not accuracy.</li><li><strong>The fix.</strong> Cheap code on your own machine (voice detection, accessibility APIs) does the watching and only wakes a big model when something real happens. My own estimate for a actually well-built version is about $30–80 a month, I haven’t measured that so that’s an assumption.</li><li><strong>Keep it safe. </strong>Mask private data before it leaves your machine, and make the agent ask a human before anything irreversible.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cftMI_Ft9sOJzhqd.jpeg" /><figcaption><a href="https://www.macwelt.de/article/2947871/napster-meldet-sich-mit-ki-assistenten-fur-den-mac-zuruck.html">https://www.macwelt.de/article/2947871/napster-meldet-sich-mit-ki-assistenten-fur-den-mac-zuruck.html</a></figcaption></figure><p>We all spend hours every day with, scheduling meetings, triaging emails, filling out spreadsheets, and context-switching between app, going in and out of focus and are just generally wasting our time.</p><p>Now just imagine a morning where you sit down normally at your desk and instead of going through inboxes and doing all the boring stuff we all do, a proactive AI assistant has already read your incoming Slack notifications, Gmail, fixed anything cross-referenced your calendar, and drafted your morning updates, tells you what you actually need to react on and what might be important everything before you made your morning coffee.</p><p>So if an AI could continuously listen, see monitor state changes and do something like check your chats the keyboard would stop being the bottleneck of human productivity, instead it could be your mind .</p><p>Though saying that an agent is always listening and collecting data is easy but building one that won’t cost you a fortune and also won’t leak your entire life to the internet, that’s the actual challenge.</p><p>So here is how an AI or LLM (Large Language Model) work, why using chatbots like Claude Chat and ChatGPT as just a text chat is outdated, what is different about this compared to a simple every morning briefing loop, and how to actually build something like this.</p><h3>What is an AI and How Does it Work?</h3><p>Most people today interact with reactive AI, just simple chatbots: you type a prompt into a text box, then it predicts the next sequence of tokens, and it stops until your next command.</p><p>At an AI’s most basic level, its just chatbot with a Large Language Model (LLM) behind it, so on its own it cant do things like tell you what time it is, unless it gets that from a separate tool.</p><p>It also does not “know” stuff like a human could, instead a AI is a extremely advanced <strong>statistical prediction engine</strong>. Think of it as it would be the predictive text on your smartphone but on “crack” like the one where you type and it like recommends you the word you might type for a shortcut, that one.</p><p>So it uses billions of learned numbers (called parameters), trained on trillions of chunks of text, to figure it out.</p><h4>For example:</h4><p>If you asked an ai “What is 1+1?”, the AI wont be using a calculator instead it has already seen millions of math examples during its training, so it has learned that after the prompt “What is 1+1?”, the word with the highest probability to follow is “2”.</p><p>Though this does get slightly different when an AI needs to research on a real new website, then it basically stores the information in its context window and then gives you the answer.</p><h4>Though this begs the question what is a context window?</h4><p><em>Quickly skip this if you already know. It is an ai</em>’s short-term working memory. It is the maximum amount of text, all your prompts, the conversation history, web research and any other kind of information all of this the can quicly read, process, and keep in mind at one single time. Tough it has a limit once it is hit you have to compact it or start a new chat, if that happens the older parts get summarised, so you lose detail, unless sometimes it has the tools to go back and look them up.</p><p>Soo understand why a standard chatbot answers your prompt once and then stops, we have to look at the phases that make this amazing prediction possible the <strong>Training</strong> and <strong>Inference phase</strong>.</p><h3>The Training Phase (Building a Frozen Brain)</h3><p>Long before an AI can exist and do something like reply to you, it must be built.</p><p>And this happens during the training phase of creating an ai, which requires vast amounts of data, months of computing time, and millions of dollars in infrastructure which is why you cant really train a big model easely at home (small ones are possible though).</p><ul><li><strong>For the pre-training:</strong> The model is fed petabytes of raw text data <em>(Yes you heard right petabytes or 1000TB)</em> from books, articles, and our good old internet. It then analyzes this data to learn the foundational patterns, grammar, and structures of our human language.</li><li><strong>Fine-Tuning &amp; Alignment:</strong> The almost done model is then still refined using humans (and sometimes also other AIs) to grade the model’s outputs and to also set rules for safety and teach it how to be a helpful conversational assistant.</li></ul><blockquote>if you want to see how a small GPT-style model gets built from complete scratch, , i recommend watching this video by Andrej Karpathy <a href="https://www.youtube.com/watch?v=kCc8FmEb1nY">https://www.youtube.com/watch?v=kCc8FmEb1nY</a></blockquote><p>Once this training phase is complete, the model’s internal weights and mathematical parameters are completely frozen. That version of the AI cant get smarter or dumber, the “brain” stays frozen until the ai company releases a newer version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*GSAy3Hty-Kdol4ZQx82Esg.png" /></figure><p><strong>Quick side note, how is this different from a morning briefing loop?</strong> A briefing loop is basically a script that runs at 8am, collects your stuff and summarizes it, and then it does nothing until tomorrow. A real autonomous AI doesn’t run on a clock, it runs on events. It notices when something happens (a new email, a note you said out loud) and decides on its own if it matters and what to do about it, even at 3pm.</p><h3>The Inference Phase</h3><p>In extremely short, this is how the model answers you so when you type a prompt, you are triggering the “inference”. This is where the actaul frozen brain does its work and gives you a response.</p><p>First, your text goes through something called tokenization this is where human words are split intos smaller parts called tokens, and then each of the tokens is turned into a number.</p><p>Then the model’s “transformer architecture” which for those who dont know basically a neural network structure of an AI, It then uses its attention mechanism to work out how these tokens relate to each other.</p><p>Like for example take the sentence “The young man went mountain climbing because it should be fun.” Like for example take the sentence “The young man went mountain climbing because it should be fun.”</p><p>The Attention mechanism allows the model to link words based on their actual “contextual relationship” like connecting the “it” back to “mountain climbing” rather than just looking at raw bigram probability across the sentence.</p><p>Now that these two steps are done, the model will run these tokens through its billions of learned parameters (our current frontier models like astra and opus are even bigger but sadly their exact sizes aren’t public).</p><p>The result is then the probability of every next token. The model then picks one of the tokens its usually by sampling from the likely ones and its not always the single top choice. Then it repeats this loop token by token until the full response is done.</p><h4>Here are actual speeds:</h4><ul><li>Per-token Latency is about 10–50 ms</li><li>Generation Speed of the tokens is about 20–100 tokens/sec</li><li>On Ultra-fast Hardware its about 300–1,000+ tokens/sec</li><li>and a typical Response for a simple question is about 2–5 seconds (100 tokens)</li></ul><p>Keep in mind that overall response times vary significantly for complex, multi-step tasks like deep research or code generation.</p><p>Today our core bottleneck of a chatbot is that its simply reactive. It runs this loop once per prompt and then reaches the end of its answer after that it sits idle until your next prompt.</p><h3>The Agentic shift, giving ai models hands</h3><p>Now that we finally get to an interesting part, a AI agent flips this fully passive, only text setup. Instead of stopping after printing text, an agent works as a closed loop made of four continuous steps:</p><h4>Perceive → Plan → Execute → Verify.</h4><p>[Image: a closed-loop circle called “Agent Feedback Loop”. Top: “Perceive” (reading system state and files) and “Plan” (breaking down the task). Bottom: “Execute” (producing a structured tool call) and “Verify” (checking the tool’s output). Arrows cycle between them.]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EZ69D_i3qfQj5mh6asyBaQ.png" /></figure><p>Now take a agent like Claude Code as an example. When you run it in your local project, it doesn’t just spit out code and maybe tell you where to paste it instead it works directly inside your machine. What that means is that it can do something like install an app for you, write code in an app, lounch apps and basically control your laptop.</p><p>An LLM cannot reach directly into your operating system or run code by itself. To bridge that gap, the ai gets a list of tools it may use, such as read_file, edit_file or “run a shell command”.</p><h4>Now lets look at the 4 steps of the loop in detail.</h4><p><strong>Perceive &amp; Plan</strong>, The model will analyzes your local environment, your files, and your recent history for contex it will then create a brief plan for how to handle the task you gave it.</p><p><strong>Execute</strong> is when the model generates structured data like a JSON request specifying a tool name and arguments it will use, instead of just plain human-facing text then a separate program intercepts that JSON, and runs the command on your machine, and captures the result.</p><p><strong>Verify &amp; Loop</strong> is<strong> </strong>the result and whether a successful build was created and if a compiler error or a failed test happened if so then its added back into the model’s conversation as an observation.</p><p>really cool for us is that if such a test fails the agent wont just quit, Instead it’ll read the error from its updated history, then correct its approach to fix it then edit the file and finally run the loop again until the everything passes.</p><p>This is already a huge upgrade from chatbot, but it still only barely scratches the surface of what a agent can actually do. I really highly recommend exploring this topic further, but we still haven’t reached true autonomy, so let’s explore that together right now.</p><h3><em>Understanding Autonomous AI and Its Visual Bottleneck</em></h3><p>A truly autonomous AI moves from simply answering to a prompt to continuously watching in either real time or at key points during the day and doing your task before you ask.</p><p>So instead of waiting for you, an autonomous ai agent will listen for any active changes, such as file changes, system logs, incoming notifications or the state of your desktop, switching apps typing speaking and maybe even camera and location and all of that data lets the ai predict what you actouly need and how to get them done.</p><p>The tech industry’s default approach to building desktop agents like claude code, relies heavily on something like visual screen capture taking constant screenshots to look at a interface like a human would. Every few seconds the system takes a screenshot, then passes the image to a vision-language model, guesses where to click, and clicks. (You can even ask your model for those screenshots) And even though this sounds like complete nonsense it works quite well.</p><p>On a OSWorld benchmark, the original 2024 paper reports that humans actouly complete over 72% of their tasks (72.36%), while the best AI model at the time managed only 12.24%. Since then, scores have climbed a lot and ai has improved, but i cant say the exact numbers.</p><p>though even on current OSWorld-Verified benchmarks, AI models still struggle to reliably outperform human accuracy. While newer evaluations like OSWorld 2.0 continue to raise the bar, benchmark scores rarely translate directly to real-world reliability, where sometimes unexpected popups or UI glitches break those screenshot taking visual loops.</p><p>But our models are still really good so accuracy is no longer the main problem. The two real problems are cost and privacy. Continuously sending screenshots to a cloud model gets very expensive, and it sends everything on your screen to someone else, which most of us wont like but lets do the real math.</p><h3>What an Always-On AI Really Costs and how to lower the cost</h3><p>Imagine an unoptimized visual desktop agent that takes a high-resolution screenshot every 5 seconds to stay aware of your work, and thats just the screenshots not actouly doing any task we’ll get to that in a second.</p><p>Now if we habve 5 screenshots per minute thats 12 a minute and if we have 8 hours in a workday thats 12 × 60 × 8 which equals 5,760.</p><p>Assuming that one screenshot costs roughly 1,000 input tokens (Anthropic’s docs put a image at up to about 1,568 tokens on the standard tier, so this is only on the low side)</p><p>So our daily input tokens would be 5,760 × 1,000 which is a total of 5,760,000 Tokens a day just from making and proccesing screenshots.</p><p>If it cost $3.00 per million input tokens (which is the price of Anthropic’s Sonnet 4.x models. (the current Sonnet 5.5 is listed at $2.00))</p><p>So our daily cost would be 5.76 × $3.00 which is $17.28. Now that at a monthly cost so 22 workdays is $17.28 × 22 which is about $380, and that is quite steep for not even getting anything done.</p><p>Those two are though assumption at the current of $2, the same math gives about $253 a month and you can take lower models like Haiku, or GLM 5.3 Flash so real costs depend on the model, the image size and tricks like downscaling and caching, so you can lower then but they’ll still add up fast.</p><h4><em>The real cost of running an Ai a lot</em></h4><p>We’ve talked about screenshots, becouse its the main part but screenshots aren’t the only thing on the bill. Every time your agent reads an email, a chat message or a voice note and thinks about it, you pay for the tokens going in and the tokens coming out, and then with all that info it still needs to actouly do the task.</p><p>So here’s a made-up example (these are my assumptions, not measurements), an assistant that handles 200 small events a day, with about 2,000 input tokens and 300 output tokens per event, running that for 30 days a month.</p><p>If we are using the prices from Anthropic then per million tokens, (input / output)</p><p><strong>Haiku 5.5 is $0.10 / $0.50. </strong>| <strong>Sonnet 5.5 is $2 / $10 </strong>| <strong>Opus 5.5 is $4 / $20.</strong></p><p>Multiply that by what were going to use it for and you get,</p><p><strong>Haiku 5.5 $0.07, $2.10 | Sonnet 5.5 $1.40, $42.00 | Opus 5.5 $2.80, $84.00</strong></p><p>This setup assumes that 1 in 4 events actually needs work, and that most of those are medium-hard.</p><p>This is quite very expensive just from all this stuff, and these are assumptions it could be a lot more or less depending on your work.</p><p>(There might be a lot of hidden cost like compacting the context and storing informations)</p><h3>How to bring it down</h3><p>How do we bring these cost down so people that cant spend $600 a month are also able use this one simple asnwer is, don’t use one model for everything. Send each task to the cheapest model that can do it well.</p><p>Lets start though with which fetures we can entirely remove from an ai’s work load.</p><p>The rule I willuse is simple, if a task follows a fixed rule, it should be hard coded, not done by AI. Code costs nothing to run and behaves the same way every time. So lets look at everything we can cut.</p><ul><li><strong>Logging what happens,</strong> a small script inside the software will capture every small thing happening such as which app is in front, which window is open, if and when a file gets saved, how long you’ve been idle and many many more, the AI can then read this short log every few minutes instead of watching screenshots, the thing is if we also want to capture what you type it could also capture passwords but we might have a solution for that.</li><li><strong>Filtering, </strong>as soon as any certain apps (like a password manager), or a log in page gets detected the scipt kills itself to not capture anything, and then a new one comes up detecting if you leave the page/app again.</li><li><strong>Urgency rules.</strong> A message from a specific person, a word like “urgent”, or other words or even a calendar event starting in 15 minutes can trigger an alert straight from code. The AI only gets involved when it’s unclear and if so let a cheap model like GLM 5.3 flash handle it thatll might cost less then a cent.</li><li><strong>Reading data.</strong> Calendars and email headers (sender, subject, time) come as text straight to the ai, so there’s no need to read a screenshot and certain emails like newsletters or spam mail can just not be read at all.</li><li><strong>Simple commands.</strong> “Set a timer” or “open this app” can match a pattern and run directly without needing to touch an ai.</li><li><strong>Limits and schedules.</strong> Timers for briefings, a cap on how many times the AI can be called per hour, and a daily spending limit.</li><li><strong>Caching.</strong> If the same question comes in again, reuse the old answer instead of paying for a new one.</li></ul><p>These are just some things im sure theres more but these are the most important ones.</p><p>Now back to using diffrent models, what we can do is asmall model decides whether an event needs any work at all like haiku or GLM 5.3 Flash. Becouse most events dont need an action and just need to be logged so an ai can know more about you generating better results.</p><p>If every request starts with the same instructions, cache them what that means is that the ai wont need to reread your instructions and already has them proccesed on Sonnet 5.5 and Opus 5.5, reading from the cache costs 0.05× the normal input price, which is huge! Writing the cache though costs a bit more (1.25× for the 5-minute kind), but it pays off after a single reuse instantly.</p><p>Collect all task during the day that arent urgent and send them all at once. See Anthropic’s Batch API (or OpenAi’s) gives a 50% discount if you send a lot of tasks at once or send them at night or non realistic work hours that saves us huge chuncks of money on stuff like morning and evening briefings or reacuring tasks.</p><p>Send a lot less, skip or shrink screenshots, trim the memory you attach to each request, and cap how long the answers can be. Also we can run super small jobs just on a quick local model it cost nothing it will use some battery but its private and will handle them without a cost.</p><p>Put everything here together and realisticly you can drop cost <strong>90% to 98%</strong>, absoluley insane right? From a $380 a month it goes down to $7–9, Now keep in mind the $380 to $7–9 is without tasks so it’ll still go up. But the 90% is entirley real from running an completley unoptimized autonomous ai to a fully optimized can lower cost immensly.</p><p><em>Important sources for key numbers, </em><a href="https://platform.claude.com/docs/en/about-claude/pricing"><em>Anthropic pricing</em></a><em>, </em><a href="https://platform.claude.com/docs/en/build-with-claude/vision"><em>Anthropic vision docs</em></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IhDx39hXBVzZM-7FFe67HQ.png" /></figure><h3>How to engineer a autonomous ai</h3><p>Now we just heard incredible news, we have to get back to reality and know how to build an autonomous desktop agent that is going to be reliable and affordable.</p><p>So lets take a look at this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JxIvm1F00vZwKVXIG2aDMw.png" /></figure><p>These are the 4 steps that keep an always-on AI cheap, cheap code on your machine watches for changes, a router picks the cheapest model that can do the job, the model only runs when something real happens, and a tool runner asks you before anything that can’t be undone.</p><p><strong>We will only discuss how to get the ai to work, not where it lives or how to build any software</strong> so here is the blueprint for an efficient ambient pipeline</p><h4>Step 1: Cheap local detectors</h4><p>Instead of running a big model 24/7, the agent should instead use lightweight scripts and local models to do tiny tasks.</p><p>Voice activity detection, a small script could check for louder noises similar to speech and then call a tiny local model that decides if its actouly speach or just background noise, then saves the info and the actual good ai can later check if it needs action. (A live camera feed goes too far just now, so we stay with audio)</p><p>Accessibility APIs, instead of analyzing screenshots with vision models, a lightweight script can read the actual text and buttons inside your active window directly from macOS or Windows. It can detects window switches or text edits instantly. However, some apps (like games or custom web interfaces) don’t support this, so it can’t completely replace visual screen capture.</p><h4>Step 2: Routing task and Skills</h4><p>When a trigger fires, the pipeline builds a compact text payload and hands it to the model together with instructions for how to handle this kind of event. In Claude Code these instructions live in files like</p><p><strong>First let’s learn what a SKILL.md is and a INSTRUCTIONS.md (also calledCLAUDE.md)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tSmg0hW63voIwq8s.png" /></figure><ul><li>A SKILL.md is a md file that contains targeted instructions triggered manually via a slash command or automatically when they are relevant. They focus on specific tasks like designing, writing, or brainstorming and they enforce strict “quality” standards so outputs look polished rather than generic AI slop.</li><li>A INSTRUCTIONS.md is instead the core system prompt for your project. It defines global rules, coding style, tech stack details, and general guidelines the AI must follow whenever it works in that codebase.</li></ul><p>An example would be something like <a href="https://github.com/nextlevelbuilder/ui-ux-pro-max-skill">“ui-ux-pro-max”</a> its a skill that sets specific rules for UI-UX like spacing, where to place text, what to avoid and which things make designs look like ai and how to avoid them. Once you run /ui-ux-pro-max or the model detects it would be usefull it will follow these rules making the design a lot better.</p><p>Feeding the model a short text description of what changed, instead of millions of screenshot pixels, using skills and instructions for better results and a md file on how to not waste tokens should cut down cost a lot and improve the outcome. My own rough estimate is that a well-built version of this could land somewhere around $30 to $80 per user per month.</p><h4>Step 3 &amp; 4 The real work</h4><p>This is simple, we have two main things:</p><ul><li><strong>A loop, </strong>the actual ai (like GLM 5.3 Flash) will check every 5m, if new information came in and if an action needs to be taken.</li><li><strong>The real work, i</strong>f work is detected, the system evaluates the difficoulty of the task and hands it to a frontier model capable of actualy handling it. Then the frontier model executes the work, and pings you if human approval is required after that itll deliver the completed work to you.</li></ul><h3>Under the hood, what Building Autonomy Really Takes</h3><p>Beyond just monthly token costs, a system like this needs real effort in three areas,</p><blockquote><strong>model choice, privacy and safety.</strong></blockquote><p>Lets go through it and answer a few questions that might be unanswered</p><h4>Do you need to train a model?</h4><p>Problably not. Training a frontier model costs a very large amount of money and computing time, like millions which is why almost nobody trains their own big models. Most agents are built on top of existing models using good prompts, tools and skill files. Fine-tuning on your own workflows is sometimes useful, but it is optional.</p><h4>The privacy boundary</h4><p>An always on agent has to watch your incoming chats, documents and file changes, that creates a privacy problem. If all of it goes to a cloud provider without limits, you will leak company secrets or personal credentials without really being able to do something about it, but running something like this locally requires at least $25.000 worth of hardware im assuming.</p><p>A safer option would clean the data on your machine first so passwords, personal data and private code should be removed locally before any text is allowed to leave your device, and a cheap local model should check what will be revealed.</p><h3>The normal sci-fi question, could this take over the world?</h3><p>We have all seen the “Ai will take over the world” bla bla, and sci-fi stories about autonomous AI becoming conscious, breaking free and taking over the world but today’s AI agents don’t work that way.</p><p>They dont got any “i want this” or “i need to be free” feelings instead an agent loop runs because a person or a trigger started it, and it stops when the loop ends but still some researchers do discuss longer-term risks as AI systems get more capable, so I won’t say that thats out of the question but for people who are building agents today, the more real risk are from humans using it for wrong things, like trying to make a virus.</p><p>The real risk and best documented one is prompt injection, someone hides instructions inside content and the agent reads it, like an email or a web page, and then the agent follows them as if you had written them. For example, an email could contain hidden text telling your agent to forward your file, but like i said nothing of this is ai’s foult instead its all humans trying to find new ways of harming other humans.</p><h4>Trusting an Ai with your Data.</h4><p>To deploy an always on agent safely, builders need clear limits.</p><p>An approval gate for permanent actions, the agent must never have unlimited system access. Anything that can’t be undone, like deleting data, sending money or changing production code, has to be approved by a human and cant be automatic, you could even add a feture that it can call or text you for these desicions.</p><p>Theres a lot more that can be down but this is a quick main thing, this article already is increadably long but thanks for reading and feedback and comments on these limits are very welcome!</p><p>If you’d like to connect with me or have more questions be sure to ask me on <a href="https://www.linkedin.com/in/ben-rebernik-b56538431/">linkedin</a>.</p><h3>Sources</h3><ul><li><a href="https://arxiv.org/abs/1706.03762">Attention Is All You Need (Vaswani et al., 2017)</a></li><li><a href="https://huggingface.co/blog/how-to-generate">How to generate text (Hugging Face)</a></li><li><a href="https://huggingface.co/docs/transformers/en/tokenizer_summary">Tokenization algorithms (Hugging Face docs)</a></li><li><a href="https://platform.claude.com/docs/en/build-with-claude/context-windows">Context windows (Anthropic docs)</a></li><li><a href="https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview">Tool use overview (Anthropic docs)</a></li><li><a href="https://code.claude.com/docs/en/how-claude-code-works">How Claude Code works</a></li><li><a href="https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview">Agent Skills overview (Anthropic docs)</a></li><li><a href="https://code.claude.com/docs/en/memory">How Claude remembers your project (CLAUDE.md)</a>, <a href="http://claude.md/">CLAUDE.md</a></li><li><a href="https://code.claude.com/docs/en/permissions">Claude Code permissions</a></li><li><a href="https://arxiv.org/abs/2404.07972">OSWorld paper (arXiv:2404.07972)</a></li><li><a href="http://osworld-v1.xlang.ai/">OSWorld project site</a></li><li><a href="https://presenc.ai/research/osworld-computer-use-leaderboard-2026">OSWorld-Verified leaderboard 2026 (Presenc AI, third-party)</a></li><li><a href="https://platform.claude.com/docs/en/build-with-claude/vision">Vision (Anthropic docs)</a></li><li><a href="https://platform.claude.com/docs/en/about-claude/pricing">Pricing (Anthropic docs)</a></li><li><a href="https://epoch.ai/blog/how-much-does-it-cost-to-train-frontier-ai-models">How much does it cost to train frontier AI models? (Epoch AI)</a></li><li><a href="https://github.com/snakers4/silero-vad">Silero VAD</a></li><li><a href="https://github.com/wiseman/py-webrtcvad">py-webrtcvad</a></li><li><a href="https://developer.apple.com/documentation/applicationservices/axuielement">AXUIElement (Apple Developer Documentation)</a></li><li><a href="https://learn.microsoft.com/en-us/windows/win32/winauto/uiauto-uiautomationoverview">UI Automation overview (Microsoft Learn)</a></li><li><a href="https://genai.owasp.org/llmrisk/llm01-prompt-injection/">LLM01:2025 Prompt Injection (OWASP)</a></li><li><a href="https://simonwillison.net/2023/Apr/14/worst-that-can-happen/">Prompt injection: what’s the worst that can happen? (Simon Willison)</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad4c1ea91890" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an Autonomous AI and why AI chatbots aren’t the AI of Today anymore.]]></title>
            <link>https://medium.com/@ben.rebernik/building-an-autonomous-ai-and-why-ai-chatbots-arent-the-ai-of-today-anymore-d49deab8395c?source=rss-899e34ea15a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/d49deab8395c</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[future-technology]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[autonomy]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Ben Rebernik]]></dc:creator>
            <pubDate>Wed, 30 Sep 2026 09:28:24 GMT</pubDate>
            <atom:updated>2026-09-30T09:28:24.283Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MzuVxUrJ8kPMCVoTciEeyQ.jpeg" /><figcaption><a href="https://www.macwelt.de/article/2947871/napster-meldet-sich-mit-ki-assistenten-fur-den-mac-zuruck.html">https://www.macwelt.de/article/2947871/napster-meldet-sich-mit-ki-assistenten-fur-den-mac-zuruck.html</a></figcaption></figure><p>We all spend hours every day doing small, meaningless tasks and scheduling, doing small parts of work that could just be done easily, right?</p><p>Typing, prompting, it all takes time, so why not a real solution? But not just a new Wispr Flow, but instead a fully autonomous assistant that predicts what you need and handles it automatically. That would just be amazing, right?</p><blockquote>Thats percisly what this article is about, telling you how Ai works and how we could make it even better.</blockquote><p>While Modern assistants still wait you to press a button or an active wake up prompt true autonomy isn’t reactive, it’s damn ambient. When an AI can continuously listen, observe your screen state in real time, and ground intent directly into OS actions, the keyboard stops being the bottleneck of productivity.</p><p>So here’s what it actually takes to build an always-listening agent from the ground up.</p><h3>What Autonomous AI Actually Means, and why chatbots are not the AI of today anymore.</h3><p>Most people today interact with reactive AI, just simple chatbots: you type a prompt into a text box, then it predicts the next sequence of tokens, and it stops until your next command.</p><blockquote>That isn’t an AI agent, that’s just a very fast search engine.<br>Not trying to undermine AI, it’s just that agentic AI is on a completely different level.</blockquote><h4>For those who don’t know how an AI chatbot, or more precisely an LLM, works:</h4><p>An AI chatbot operates as a conversational interface, and it’s driven by a Large Language Model (LLM) that transforms human language into numbers, processes it through its own neural networks, and then statistically predicts simple replies word by word.</p><p>When you type a message, the system does not look up something pre-written. Instead, it coordinates a very complex mathematical sequence to answer your questions.</p><p>The process begins with tokenisation, where input text is split into chunks, allowing a transformer architecture to weigh how different words relate to one another. “Armed” with its context from billions of pre-trained parameters, the LLM acts as a probabilistic engine evaluating the next most likely word. In the end, it fine-tunes the model to act as a safe, helpful assistant, streaming the text back to your screen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*TYt9yBCDDbyp87MygG5acQ.png" /><figcaption><a href="https://blog.promptlayer.com/llm-architecture-diagrams-a-practical-guide-to-building-powerful-ai-applications/">https://blog.promptlayer.com/llm-architecture-diagrams-a-practical-guide-to-building-powerful-ai-applications/</a></figcaption></figure><p>Quite complicated, right? Well, it’s about to get even more so :), but first the simple stuff… Lucky.</p><h3>How an Ai agent works</h3><p>Now that we know how a chatbot and an LLM work, we can dive deep into how agents do.</p><p>Actually, most people that I know mistake chatbots for AI agents. But a chatbot is purely reactive, as we know it can’t see your context, it can’t touch your apps, and it has zero follow-through or messages you back.</p><p>But an AI agent flips this model on its head. Instead of a passive text box, an agent is a closed-loop system it Perceives, then Plans, then Executes, and even Verifies.</p><p>Take coding agents like Claude Code or Codex as a perfect example.<br>They don’t just dump code into a chat window and hope you figure out how to run it. They operate inside a runtime loop right on your machine and through cloud hardware.</p><p>When you give the agent a goal or task, it inspects your real environment. It looks at possibly related files and at what you’ve been doing during your last “chat.” So instead of guessing everything blindly, the model decides which specific tool it needs to call next and then works on your task structurally (e.g., `read_file`, `bash`, or `edit_file`).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/927/1*ztixZZedmzgS1koAoh7C0w.png" /><figcaption><a href="https://www.promptingguide.ai/research/llm-agents">https://www.promptingguide.ai/research/llm-agents</a></figcaption></figure><p>But important is aswell that the agent doesn’t touch your disk directly instead a local wrapper intercepts the tool call and runs the actual command in your terminal or edits the exact lines of code. It reads the command output or test results back into its context. If a test fails or a syntax error pops up, it doesn’t just tell you well, no, but instead reads the error message, course-corrects, and edits the file again until everything passes.</p><p>It can install things for you, which a chatbot can’t. It basically lives in your laptop while being supported by a huge network of cloud computing power.</p><h4>Lets quickly dive in in a more technical level</h4><p>Basicly in very short an agent is just an LLM hooked up to a feedback loop and a few external tools. On its own, an LLM cannot execute code or touch files.</p><p>To make a simple LLM an agent, we supply it with a list of specific tools it can call, like `read_file` or `run_command`. Instead of answering with words for a human, the model outputs structured data (like a JSON file) specifying the exact function name and arguments it wants to run.</p><p>The exact AI behind Claude Code cannot access your operating system directly, though. An external runner application, like the Claude desktop app, intercepts that structured output, runs the command locally in your terminal or files, and captures the result. The output from that command (whether it is file contents, a successful build, or a crash log) is pasted directly back into the LLM’s conversation history as an observation.<br>Now that the model has seen what its action did, it decides what to do next: call another tool, fix a bug from the error message, or return the final answer.</p><p>After the full loop is finished, it gives you the real answer and your output from whatever task you gave.</p><p>For anyone who needs a deeper, more technical dive, I recommend watching this:<br><a href="https://youtu.be/Mvcxa35_PMA?si=iXoxQwJwi2yRVTM7">https://youtu.be/Mvcxa35_PMA?si=iXoxQwJwi2yRVTM7</a></p><h3>What’s Autonomous AI, and what’s holding it back.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jfofgoIYNVnM-EJ8_pE3Wg.png" /><figcaption><a href="https://www.linkedin.com/pulse/what-autonomous-ai-why-does-matter-rajaram-j-thoyc">https://www.linkedin.com/pulse/what-autonomous-ai-why-does-matter-rajaram-j-thoyc</a></figcaption></figure><p>True autonomy is where we cross the line from an agent that follows commands to a system that predicts what needs to be done and handles new tasks by knowing you need them, not by being told.</p><p>An ordinary agent is like an extremely fast, highly talented junior engineer waiting at your desk: it can execute a task from start to finish, but only after you prompt it, outline the goal, and define the boundaries.<br>Autonomous AI flips that dynamic. Instead of waking up only when you hit “Enter,” an autonomous system has:</p><p>Continuous Awareness: it doesn’t rely on a single prompt. It maintains an active perception stream, monitoring system events, your screen, inbound notifications, or codebase changes. It could even listen in and watch your camera, but I think we all know that the camera goes a little too far.</p><p>Given high-level intent (like “keep this server healthy” or “organize incoming client tasks”), it determines its own sub-goals and sub-agents without needing step-by-step guidance. When you give it four recurring tasks, it “hires” four subagents that do them for it, and it handles the main task that needs to be done.</p><p>It can run multi-hour loops completely unsupervised, planning dozens of steps ahead, handling branching decisions, and deciding for itself when a task is actually finished.</p><p>But we also all know some tasks need to be done by a human, so we can set simple guidelines for that or let it do the structuring.</p><h3>In summary:</h3><p>Regular agents give models hands, but autonomous AI gives them agency, shifting the human role from micromanaging every loop to simply setting the objective and auditing the results.</p><h3>The bad news…</h3><p>The bottleneck today isn’t raw model intelligence, it’s our current compute power, inference cost, and execution reliability.</p><p>On real-world desktop benchmarks like OSWorld, human workflows hit over 70% completion rates, while visual multimodal agents struggle around 12%. When an agent relies on parsing raw screen pixels, every interaction requires heavy vision-language model calls, burning dozens of high-end GPU seconds and running up costs of several dollars per complex task just to navigate UI coordinate shifts and unexpected popups.</p><p>The breakthrough is moving away from brute-force vision. Instead of spending massive compute to guess GUI coordinates, modern workflows use structured markdown files (like CLAUDE.md or SKILL.md). Feeding the agent clean text definitions and API boundaries slashes token usage and latency, letting lightweight models execute deterministic system tasks at a fraction of the cost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*48kpQLN8D0V3eQTmsZGHLQ.png" /></figure><blockquote>These text-based skills bridge the gap between messy pixels and clean code execution, which is pretty good.</blockquote><p>Running an autonomous desktop agent is far more expensive than a $20/month Claude Code subscription. If an agent sees raw screen pixels continuously, constant vision-language model calls burn through millions of tokens daily. Unoptimized visual agents easily run $300 to $1,000+ per user each month in raw inference.</p><p>The fix is architecture over brute force: using local change-detection heuristics and structured text definitions (`SKILL.md`) to invoke frontier models only when action is needed. That brings realistic costs down to roughly $30 to $80 per user monthly.</p><p>Basically, a low-tier model or a simple script for voice detection checks whether action needs to be taken, and if so, hands it to the differently powered models depending on the size of that task.</p><h3>Executive Summary</h3><p>Modern AI is moving fast from basic chatbots to truly autonomous agents.<br>While traditional LLMs simply predict the next word in a chat box, agentic systems use local runtime loops, external tools, and automated error recovery to execute real computer workflows.</p><p>The next frontier is ambient autonomy: systems that continuously observe screen context and audio to anticipate needs rather than always waiting on prompts.</p><p>However, brute-force visual parsing makes current agents both unreliable and prohibitively expensive. The practical solution is an event-driven architecture pairing local heuristics with structured text skills, delivering dependable multi-step execution at a fraction of the token cost.<br>Thanks for reading. I hope everything was great, and feedback is very welcome!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d49deab8395c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>