<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Matan Weksler on Medium]]></title>
        <description><![CDATA[Stories by Matan Weksler on Medium]]></description>
        <link>https://medium.com/@matanweks?source=rss-7793a4e59292------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*sUENwVocnA6wF2bCPNveVA.png</url>
            <title>Stories by Matan Weksler on Medium</title>
            <link>https://medium.com/@matanweks?source=rss-7793a4e59292------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 21:32:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@matanweks/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Art and Science of Context Engineering]]></title>
            <link>https://blog.pelles.ai/the-art-and-science-of-context-engineering-4669d56f6d48?source=rss-7793a4e59292------2</link>
            <guid isPermaLink="false">https://medium.com/p/4669d56f6d48</guid>
            <category><![CDATA[construction]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[context-engineering]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Matan Weksler]]></dc:creator>
            <pubDate>Tue, 08 Jul 2025 05:20:53 GMT</pubDate>
            <atom:updated>2025-07-08T05:20:53.508Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why the future of AI isn’t just about shiny new models, clever prompts, but how we are architecting the intelligent systems that power them.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jFbqs26hhuN0PAtjAR4E2Q.png" /></figure><p>In our latest <a href="https://blog.pelles.ai/the-construction-prompt-coach-get-what-you-need-from-ai-fast-f7aa4b961e4a">blog post</a>, we explored the concept of “prompt engineering” — the most well-known method for interacting with large language models (LLMs): craft a clever prompt, get a clever response. While this approach works well for general-purpose chatbots, it falls short in production environments where reliability, context-awareness, and workflow integration are essential. As AI evolves from novelty to mission-critical infrastructure, we need systems that go beyond prompt engineering, systems that understand your domain, adapt to your processes, and anticipate your needs.</p><p>Enter <strong>context engineering</strong>, a holistic discipline that’s quickly becoming the backbone of next-generation AI-based systems. Rather than focusing on a single instruction, context engineering emphasizes the entire informational ecosystem built around the models, dramatically improving the reliability, precision, and power of AI agents.</p><p>In this article, we’ll explore what context engineering truly entails, why it’s the necessary evolution from prompt engineering, and how platforms like Pelles.ai are building the <strong>operating systems</strong> to master it.</p><h4>From Prompting to Context Programming: A Necessary Evolution</h4><p>Prompt engineering often feels like a series of linguistic tricks — adding incentives (“I’ll give you a huge tip for a perfect answer”) or pleading with the model to “think step-by-step.” This method is akin to persuading a human assistant who may or may not fully grasp your intent. It’s brittle and unpredictable.</p><p>But as LLMs become the computational core of complex <strong>workflows and AI agents</strong>, these surface-level interactions are no longer sufficient. The real challenge today isn’t just writing better prompts, but architecting comprehensive contexts that enable models to reason, decide, and act with consistency.</p><p>As thought leaders like Andrej Karpathy have highlighted, we are moving from prompt engineering to “context programming.” LLMs operate not just on a single prompt, but on the entire <strong>context window</strong> — the total information presented to them at any given moment. The quality, structure, and relevance of this context dictate the outcome.</p><p>Imagine prompt engineering as replacing a light bulb, you need the right bulb, maybe a ladder, and if you don’t have one, you improvise with a chair or a stack of books. It’s about quick, non-repetitive, local adjustments that hopefully do the job right<br>Context engineering, however, is installing the entire electrical system in the project.</p><ul><li>Designing the shop drawing (<strong>System Prompt</strong>) ensures a clear structure and purpose.</li><li>Selecting quality materials (<strong>Data Retrieval</strong>) guarantees durability and relevance.</li><li>Employing the proper tools and machinery (<strong>Tool &amp; API Calls</strong>) streamlines the building process and enhances capabilities.</li><li>Maintaining accurate records and project logs (<strong>State Management</strong>) ensures continuity, coherence, and prevents costly mistakes.</li></ul><p>Just as sloppy drawings or the wrong tools can lead to costly rework, a poorly crafted context can undermine AI performance. Too little context, and the model struggles. Too much — or irrelevant — context, and it risks confusion, hallucination, or inflated costs. Striking the right balance is the new frontier in effective AI deployment.</p><h4>The Science of Context</h4><p>Context engineering is a discipline grounded in rigorous research that goes beyond elaborate prompt engineering.</p><p>A <a href="https://cs.stanford.edu/~nfliu/papers/lost-in-the-middle.arxiv2023.pdf?utm_source=chatgpt.com">Stanford</a> study found that structured contexts incorporating curated examples and external knowledge can boost LLM accuracy by as much as 30%. Similarly, <a href="https://news.mit.edu/2025/unpacking-large-language-model-bias-0617?utm_source=chatgpt.com">MIT</a> research emphasizes that thoughtful context window design is critical for enterprise-level AI performance.</p><p>Recent research in <a href="https://www.nature.com/articles/s44271-024-00091-8"><em>Nature</em></a> has also highlighted that LLMs exhibit human-like cognitive biases; something as simple as the order of information can dramatically alter the output.</p><p>This is demonstrated in the “Needle-in-a-Haystack” benchmark, which assesses long-context understanding. A leading model improved its accuracy by over 10% simply by rearranging context data to mitigate positional bias , without any model retraining or fine-tuning. This highlights the impact of smarter, structured context.</p><h4>Context Engineering in Practice: The AI Agent Operating System</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V8FlrvPG1q6imjMUW8yyWw.png" /></figure><p>This is where theory meets reality. Context engineering isn’t an isolated task; it’s the central function of what we at <strong>Pelles.ai</strong> see as an <strong>Operating System for AI Agents</strong>. Building a truly effective AI agent requires orchestrating multiple components in real-time.</p><p>A modern AI application is a sophisticated, real-time context engine — far more than a simple “ChatGPT wrapper.” It must expertly manage:</p><ol><li><strong>State and Memory:</strong> Seamlessly recalling previous parts of a conversation or workflow to maintain coherence.</li><li><strong>Knowledge Integration (RAG):</strong> Retrieving the right document chunks from a vector database at the right time, without overloading the context window.</li><li><strong>Tool Orchestration:</strong> Intelligently deciding when to search the web, query a database, or call a third-party API, then feeding the result back into the context.</li><li><strong>Cost and Latency Optimization:</strong> Caching results, prefetching data, and dynamically routing tasks to the most efficient model</li><li><strong>Safety and Security:</strong> Implementing guardrails, protecting sensitive inputs, and ensuring compliance — tasks that are fundamental to enterprise-grade systems.</li></ol><p>Manually coding this orchestration is the reason so many internal AI projects become mired in complexity and hidden costs. This is precisely the challenge that we at <strong>Pelles.ai</strong> are designed to solve — by providing the structured workflows and infrastructure to manage context at scale.</p><h3>Why This Matters for You</h3><p>For anyone using AI applications, the difference between a cool gadget and a true AI partner comes down to one thing: context.</p><p>Without it, AI tools are amnesiac. They give generic answers because they can’t access your company’s real-time data. You ask for a VAV box submittal and get a web result instead of the approved document from Procore. You ask it to draft an RFI and it spits out a blank template, forcing you to manually hunt down the drawing numbers, spec sections, and gridlines. This isn’t an assistant; it’s another tool that makes <em>you</em> do all the work.</p><p>A context-engineered AI agent, however, changes the game completely. It’s connected to your project files, your systems, and your workflow.</p><p>The conversation shifts from this:<br><em>“Draft an RFI.”</em> → <strong>Result: A blank template and more manual work for you.</strong></p><p>To this:<br><em>“Draft an RFI for the ductwork clash on drawing M-501.”</em></p><p>A context-aware partner doesn’t just respond — it executes. It instantly:</p><ul><li>Finds the specific clash and references the conflicting sprinkler drawing (F-102).</li><li>Pulls the relevant spec section and drawing details.</li><li>Applies your company’s standard RFI format and professional tone.</li></ul><p>This isn’t just about getting a “smarter” answer. It’s about creating AI systems that are fundamentally more <strong>reliable, transparent, and capable</strong>. The end result is a system that you can actually do meaningful work with, transforming it from a novelty you experiment with into an indispensable assistant you can rely on.</p><h4>The Future is Context-Aware</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kmqJYkCxkDGlMqJbKQAZ-Q.png" /></figure><p>The next time you engage with a powerful AI, remember: Are you spending your time coaching it on every single detail, or is it already equipped with the project blueprints?</p><p>While the tech world is buzzing with new models, the real breakthrough for our industry will come from mastering the context they operate in. The future of AI in construction isn’t about writing a more clever prompt; it’s about building AI agents that already know your job, understand the drawings, and help you protect your bottom line.</p><p>The challenge ahead is making this powerful capability accessible. At Pelles.ai, we are building the infrastructure to do just that — turning abstract AI potential into a practical tool that prevents rework, speeds up communication, and makes projects more profitable.</p><p>In this new landscape, the context you build is the most valuable asset you have.</p><p>Thank you for reading. If you found this article helpful, please share it with your network and give it a clap! :)</p><p>Find out more:</p><p><a href="https://www.pelles.ai/">Pelles | AI for Preconstruction, ‍‍Empowering‍ every worker. Automating every workflow.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4669d56f6d48" width="1" height="1" alt=""><hr><p><a href="https://blog.pelles.ai/the-art-and-science-of-context-engineering-4669d56f6d48">The Art and Science of Context Engineering</a> was originally published in <a href="https://blog.pelles.ai/">Pelles.ai</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond the Prompt — From On-Demand to Always-On]]></title>
            <link>https://blog.pelles.ai/beyond-the-prompt-from-on-demand-to-always-on-47350bc352d3?source=rss-7793a4e59292------2</link>
            <guid isPermaLink="false">https://medium.com/p/47350bc352d3</guid>
            <category><![CDATA[construction]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ambient-agents]]></category>
            <category><![CDATA[mep]]></category>
            <dc:creator><![CDATA[Matan Weksler]]></dc:creator>
            <pubDate>Mon, 30 Jun 2025 13:58:41 GMT</pubDate>
            <atom:updated>2025-06-30T14:02:45.377Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Beyond the Prompt — From On-Demand to Always-On</strong></h3><h4>How AI is evolving from a tool you direct to a colleague that anticipates your needs.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FsBUc6k6PDXDaUXFijGj-Q.png" /></figure><p>In our previous article, we introduced the “Iron Man Suit” principle: successful AI tools don’t replace us, they augment us. Tony Stark wasn’t replaced by his suit; he became exponentially more capable with it. The suit provided superhuman strength, flight, and data processing, but Stark’s ingenuity, intuition, and strategic mind remained firmly in the pilot’s seat. This model is built on two pillars: a tight feedback cycle where AI generates and the human verifies, and an autonomy slider that lets the user control how much the AI contributes. This is the foundation for building practical, human-centric AI.</p><p>But this reactive, command-driven approach only scratches the surface of AI’s true potential. The greatest advantage of modern AI is its capacity to work tirelessly in the background, processing vast amounts of information while we focus elsewhere. This capability allows the suit to evolve from a reactive tool into something far more integrated: an <strong>Ambient AI Agent</strong>. This is not just a tool you command, but a digital teammate that manages complex workflows and collaborates with you as a persistent partner.</p><p>The question, then, is how do we build this proactive, ever-learning agent? How do we achieve this vision without falling into the trap of creating unreliable, “black box” systems that ultimately fail us?</p><p>The answer is surprisingly straightforward: You build the Ambient AI Agent by rigorously applying the Iron Man Suit principle. The path to a trusted digital teammate isn’t a reckless leap into full autonomy. It is a deliberate, step-by-step process of building a smarter, more perceptive suit. In this article, we’ll explore how.</p><h3>From Reactive Tool to Proactive Partner</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ns6UcFN9Cb5MezXAw_L09g.png" /></figure><p>The primary limitation of the classic “Iron Man Suit” is its reactive nature. Like any tool, it’s incredibly powerful when you pick it up, but it sits idle until you do. The user must recognize a need, open the application, feed it data, and initiate a command. The human is always the starting pistol for any AI-assisted task. This creates an invisible ceiling on productivity; the AI’s potential is shackled to the user’s moment-to-moment awareness and initiative.</p><p>The Ambient AI Agent shatters this ceiling by fundamentally changing the interaction model. Can be described as an AI that “never sleeps,” this agent is not a tool you open, but a persistent presence that works alongside you. It is “always on” and, crucially, event-driven. Instead of waiting for a command, it is triggered by events in your digital environment — a new email, a calendar invite, a change in a project management board, or an update in a CRM.</p><p>To illustrate this evolution, let’s consider a common task for a project manager: handling a Request for Information (RFI).</p><ul><li><strong>The “Digital Toolkit” (Reactive) Approach:</strong> A project manager, Jane, sees a new email in her inbox. It’s an RFI from the plumbing subcontractor about a pipe specification. She opens her project management software and decides to use its AI chat assistant to speed up the process. She copies the question from the email, pastes it into the chat, and attaches the PDF. The AI helps populate the new RFI form, but Jane still has to review it, look up the correct architect, and manually assign it. Although the AI assistant helped with data entry, the entire process of identifying, logging, and routing the RFI was initiated and directed by Jane.</li><li><strong>The Ambient AI Agent (Proactive) Approach:</strong> Jane’s digital teammate is always monitoring her inbox. It detects the new email from the subcontractor, recognizes it as an RFI based on the sender and keywords, and autonomously begins processing it. It parses the email content and the attached PDF, logs it in the project management system with the correct RFI number, and identifies from the document’s context that the question must be routed to the lead mechanical engineer. It then sends Jane a notification: “New RFI from AquaPlumb logged regarding pipe specs in Area B. I’ve assigned it to the mechanical engineer with a 3-day response deadline. Ready for your review.”</li></ul><p>In the second scenario, the AI has transformed from a passive calculator into a proactive junior partner. It didn’t just augment Jane’s ability to do the task; it augmented her capacity by initiating the task for her, freeing her to focus immediately on the high-level strategic decision: Is this bid worth pursuing?</p><h4>Redefining the “Autonomy Slider” and “Verification Loop”</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M_88grBx83qG_3tS-RFLXw.png" /></figure><p>This shift from a reactive tool to a proactive teammate requires us to rethink the core mechanics of the “Iron Man Suit.” The “Autonomy Slider” and “Generation-Verification Loop” don’t disappear; they evolve to govern a more complex, asynchronous relationship.</p><p>The “Autonomy Slider” transforms from a task-level control to a set of strategic “rules of engagement.” In the classic model, a user might slide autonomy up or down for a single report. In the ambient model, the user sets broader permissions and goals. For her super-estimator agent, Jane might configure the slider with rules like:</p><ul><li><strong>Low Autonomy:</strong> “You may monitor all incoming emails and flag potential bids, but do not start an estimate without my explicit command.”</li><li><strong>Medium Autonomy:</strong> “You may automatically generate preliminary estimates for any bid under $500,000 from our existing client list. Present the draft to me for verification before any further action.”</li><li><strong>High Autonomy:</strong> “For any standard renewal bid from a top-tier client, you may generate the estimate, draft the proposal using our standard template, and schedule a 15-minute review meeting in my calendar.”</li></ul><p>The slider is no longer about how to do a single task, but what the agent is permitted to do within the broader workflow. It’s a shift from micromanagement to strategic delegation.</p><p>Similarly, the “Generation-Verification Loop” becomes more asynchronous and milestone-driven. The tight, second-by-second loop of “generate-verify-generate” is ill-suited for an agent that works in the background. Instead, the loop stretches over hours or even days. The agent generates, refines, and cross-references data autonomously. The “verification” step then occurs at critical junctures when human judgment is essential.</p><p>The “morning briefing” is the perfect example of this new loop. The agent works overnight, sifting through communications, project updates, and data dashboards. In the morning, it presents a synthesized summary: “Here are the three most urgent emails, here is the status of Project X which is falling behind schedule, and here is the preliminary bid I drafted for you.” The human’s verification is not on every small calculation, but on the prioritized output and the strategic next steps. The loop is completed when the human provides feedback or direction, which in turn launches the agent’s next cycle of autonomous work.</p><h4>Building Trust with Your New Digital Teammate</h4><p>Granting an AI this level of proactive autonomy naturally raises critical challenges, primarily centered on trust. How can we rely on an agent working in the background without constant supervision? This is where concerns about data quality and the “black box” problem — not knowing why an AI made a certain decision — become paramount.</p><p>The solution lies in framing the agent not as an infallible oracle, but as an “ever-learning apprentice.” This approach is fundamental to building trust over time. You wouldn’t hire a new human employee and immediately give them the keys to your most sensitive accounts. You start them with low-risk tasks, observe their performance, provide feedback, and gradually grant more responsibility as they demonstrate competence.</p><p>The same principle applies to the digital teammate:</p><ul><li><strong>Start Small:</strong> Initially, the ambient agent is tasked with low-risk, high-value activities. It might only be allowed to categorize incoming requests, summarize long documents, or flag action items from meeting transcripts.</li><li><strong>Learn from Feedback:</strong> Every time the human corrects the agent — re-categorizing a request, editing a summary, or ignoring a flagged item — it provides a crucial data point. This is the asynchronous “Verification Loop” in action, serving as a continuous training mechanism.</li><li><strong>Earn More Autonomy:</strong> As the agent’s accuracy and reliability improve, the user gains confidence. They might then adjust the “Autonomy Slider” to allow the agent to draft responses, perform preliminary analyses, or even manage calendar scheduling. The agent literally <em>earns</em> its autonomy by proving its value.</li></ul><p>This model keeps the human firmly in the loop, reinforcing the core philosophy of the “Iron Man Suit”. The goal is not to create an autonomous entity that replaces human judgment, but to build a reliable partner that can be trusted to handle an increasing scope of work. Human oversight remains the ultimate backstop, ensuring that the AI’s powerful capabilities are always aligned with human goals and values.</p><h4>Conclusion: The Future of Work is a Collaboration</h4><p>The journey of workplace AI is a clear evolutionary path. We began by building an “Iron Man Suit” — a powerful, reactive tool that amplified our abilities on command. We mastered the principles of the “Generation-Verification Loop” and the “Autonomy Slider” to make these tools effective and safe. Then, that suit fuses with our digital environment, transforming into a proactive, persistent digital teammate.</p><p>This Ambient AI Agent represents the next frontier of productivity. It retains the foundational principles of human-machine collaboration — human oversight and adjustable autonomy — but applies them in a more sophisticated, asynchronous manner. By handling the constant churn of monitoring, filtering, and preliminary processing, these agents will liberate human workers from the tyranny of the urgent and the tedium of the repetitive.</p><p>The future of work is not a contest between humans and machines. It is a collaboration. It’s a future where every knowledge worker is supported by a dedicated digital teammate, one that prepares their day, anticipates their needs, and executes routine tasks flawlessly. This frees us, the human pilots, to focus on what we do best: strategic thinking, creative problem-solving, building relationships, and making the critical judgments that will always require a human mind at the helm. The suit is no longer just something we wear; it’s the partner we work with.</p><p>Thank you for reading. If you found this article helpful, please share it with your network and give it a clap! :)</p><p>Find out more:</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=47350bc352d3" width="1" height="1" alt=""><hr><p><a href="https://blog.pelles.ai/beyond-the-prompt-from-on-demand-to-always-on-47350bc352d3">Beyond the Prompt — From On-Demand to Always-On</a> was originally published in <a href="https://blog.pelles.ai/">Pelles.ai</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Construction Prompt Coach: Get What You Need from AI, Fast.]]></title>
            <link>https://blog.pelles.ai/the-construction-prompt-coach-get-what-you-need-from-ai-fast-f7aa4b961e4a?source=rss-7793a4e59292------2</link>
            <guid isPermaLink="false">https://medium.com/p/f7aa4b961e4a</guid>
            <category><![CDATA[construction]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <dc:creator><![CDATA[Matan Weksler]]></dc:creator>
            <pubDate>Mon, 23 Jun 2025 13:07:11 GMT</pubDate>
            <atom:updated>2025-06-23T13:07:11.657Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Tkjq0RsebYsXVYwDIT1fQ.png" /></figure><p>You’ve heard the hype about AI. You’ve even tried using ChatGPT or a similar tool to help with your work. You ask it to write an email about a delay or summarize a safety issue, and what you get back is… generic. Vague. Unusable.</p><p>It feels like another overhyped tech toy that doesn’t understand the grit and reality of the work.</p><p>Here’s the truth: The problem isn’t the AI. It’s the instructions we give it. In the world of AI, these instructions are called “prompts.” Vague prompts get you vague results. It’s the digital version of <strong>“garbage in, garbage out”</strong></p><p>As a subcontractor, your time is your most valuable asset. You don’t have hours to go back and forth with an AI chatbot, rephrasing your request. You need the right document, the right email, or the right summary <em>now</em>.</p><p>Instead of guessing, let’s use a playbook. By combining a simple framework with core strategies from OpenAI’s own guide, you can turn a generic AI into a world-class project coordinator that truly understands how your business works.</p><h3>Your Blueprint for Perfect Prompts: The CRAFT Framework</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cGzfSyn2viGhuUe8WM4O2A.png" /></figure><p>A great way to structure your requests is with the <strong>CRAFT</strong> framework. It’s a simple model to ensure you cover all the key details. <a href="https://www.geeky-gadgets.com/craft-prompt-framework/?utm_source=chatgpt.com">The CRAFT framework</a> (Context, Role, Audience, Format, Task) was popularized by Lawton Solutions as a memorable and effective method for prompt engineering. We are adapting this structure for the specific needs of construction.</p><h4>C for Context: The “Where, Why, and With What”</h4><p>Never assume AI knows what you’re talking about. Providing reference text is one of the best ways to get better results. Set the scene, provide the facts, and give the AI all the information it needs to do its job.</p><ul><li><strong>Vague Prompt:</strong> “Write an email about the problem with the ducts.”</li><li><strong>Prompt with Context:</strong> “We are the HVAC subcontractor on the ‘Oak Valley Community Hospital’ project. In the west wing, second floor, the recently installed plumbing pipes are clashing with our planned ductwork route shown on drawing M-201. This prevents us from installing the main supply trunk line. Use the foreman’s daily log as reference: ‘End of day, confirmed clash with P-301 piping run. Notified GC foreman on site.’”</li></ul><h4>R for Role: The “Who You Are”</h4><p>We calls this asking the model to adopt a persona. It’s a magic key. By assigning a role, you tell the AI which “database” of experience, vocabulary, and tone to use.</p><ul><li><strong>Vague Prompt:</strong> “Explain the ductwork problem.”</li><li><strong>Prompt with Role:</strong> “Act as a Senior Project Manager for an HVAC subcontractor with 20 years of experience. You are professional, firm, and solutions-oriented.”</li></ul><h4>A for Audience: The “Who It’s For”</h4><p>Who are you writing to? An email to the GC’s Superintendent requires a different tone and level of detail than a toolbox talk for your crew or a formal notice to the architect.</p><ul><li><strong>Vague Prompt:</strong> “Write about the clash.”</li><li><strong>Prompt with Audience:</strong> “The audience for this RFI is the General Contractor’s Project Manager. They are focused on schedule and budget. We need to clearly explain the issue and the potential for a delay if not resolved quickly.”</li></ul><h4>F for Format: The “What It Should Look Like”</h4><p>Don’t leave it up to the AI model to decide. It’s important to clearly state the format you want for the response — whether that’s a formal letter, a quick email, a bullet-point list, or just a short summary. Be specific.</p><ul><li><strong>Vague Prompt:</strong> “Give me the details.”</li><li><strong>Prompt with Format:</strong> “Format the response as a formal Request for Information (RFI). Use a clear subject line. The body must have three sections: 1. The Issue, 2. The Location and Drawing Reference, 3. The Requested Action.”</li></ul><h4>T is for Task: The “What to Do”</h4><p>Finally, give a clear, direct command. Use strong action verbs like “write,” “summarize,” “analyze,” “compare,” or “create a list.”</p><ul><li><strong>Vague Prompt:</strong> “Help with this situation.”</li><li><strong>Prompt with Task:</strong> “Write a draft email to the GC to accompany the formal RFI. The email should briefly summarize the clash, state that the RFI has been submitted, and request a coordination meeting within 48 hours.”</li></ul><h3>Go Deeper: Pro-Level Strategies from OpenAI’s Playbook</h3><p>Once you’ve mastered CRAFT, you can incorporate these advanced strategies directly from OpenAI’s documentation to tackle more complex work.</p><p><strong>1. Split Complex Tasks into Simpler Subtasks<br></strong>Asking AI to do something complex in one shot is a recipe for a bad result. Guide it step-by-step. For instance, when creating a change order justification:</p><ul><li><strong>Step 1:</strong> “Summarize the key differences between the originally specified ‘Model A’ light fixture and the new ‘Model B’ fixture based on these two spec sheets…”</li><li><strong>Step 2:</strong> “Based on that summary, create a bulleted list of the impacts on our electrical scope. Include material cost differences, additional labor hours, and schedule delays.”</li><li><strong>Step 3:</strong> “Now, combine that information into a formal Change Order Request narrative.”</li></ul><p><strong>2. Provide Examples to Steer the Output (AKA Few-Shot Prompting)<br></strong>If you need AI to follow a very specific style or format, give it an example. This is one of the most powerful techniques.</p><ul><li><strong>Prompt with an Example:</strong> “I need to write a daily safety observation summary. Always follow this format: <strong>Issue:</strong> [One-sentence description]. <strong>Location:</strong> [Specific area]. <strong>Action Taken:</strong> [What was done immediately]. Here is today’s observation: ‘A worker on the third floor wasn’t tied off while working near the edge.’ Now, generate the summary in the correct format.”</li></ul><p><strong>3. Give AI “Thinking Time”<br></strong>For complex reasoning tasks, ask the model to work out the solution for itself before giving you the final answer. This forces a more rigorous, logical process and reduces errors.</p><ul><li><strong>Prompt Without Thinking Time:</strong> “Does this contract clause expose us to risk? ‘Subcontractor shall be responsible for all site cleanup.’” <em>(This may give a simple yes/no).</em></li><li><strong>Prompt WITH Thinking Time:</strong> “First, identify any ambiguous terms in this contract clause: ‘Subcontractor shall be responsible for all site cleanup.’ Second, list potential scenarios where this ambiguity could lead to a dispute with a General Contractor. Third, based on your analysis, explain the primary financial risk this clause presents for a subcontractor. Finally, answer whether this clause exposes us to significant risk.”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k6UebiqkrjFQYjAz4U3b8Q.png" /></figure><h3>The Pelles Philosophy: Computers should learn us, Not the Other Way Around</h3><p>For decades, technology has forced us to adapt. We learn its quirks, menus, and language. We believe that’s backward. The future of truly useful AI is one where the system learns <em>you</em>.</p><p>The strategies above are how you <em>manually</em> coach an AI today. At Pelles, our mission is to build this coaching capability directly into our platform through a three-phase evolution, so you can <strong>trigger</strong> <strong>actions</strong>, not write a story about them.</p><h4>Phase 1: The User Interface — Your Command Center</h4><p>First, we give you the tools to make providing instructions effortless. The Pelles interface is designed to systematize the CRAFT framework internally, removing the guesswork and busywork.</p><ul><li><strong>Your Centralized Context Hub:</strong> Instead of copy-pasting, you can connect Pelles to your project documents or simply drag-and-drop a daily report, submittal, or drawing to give the AI the specific facts it needs.</li><li><strong>Structured Inputs/Outputs, Not Guesswork:</strong> The platform gives you the ability to define your <strong>Role</strong>, <strong>Audience</strong>, and create your desired <strong>Formats</strong>, to eliminate the wild-wild west of AI</li></ul><h4>Phase 2: AI-Assisted Collaboration — The System Gets Smarter</h4><p>As you use the interface, Pelles learns. This is where we use <strong>AI-powered suggestions to better understand your needs for a seamless human-computer collaboration</strong>.</p><ul><li><strong>You say:</strong> “Draft an RFI about the pipe clash.”</li><li><strong>Pelles suggests, </strong>“Draft an RFI about the pipe clash, Here are the drawing references […], here’s my RFI template […] and here is an RFI example ‘Pipe Clash Near HVAC Duct — Needs Resolution’ […] from a previous project.</li></ul><p>This collaborative loop is driven by continuous learning: each time you accept or refine a suggestion, you’re actively guiding the system. It adapts to your projects, recognizes your key contacts, memorizes your communication style, and remembers your preferred document formats, gradually building a personalized understanding of how you work.</p><h4>Phase 3: Agentic Intelligence — Your Automated Partner</h4><p>Once the system has learned your recipes, it graduates from collaborator to an autonomous partner. This is where we implement <strong>agentic optimizations for prompt improvements</strong>. The Pelles system becomes an “agent” that can act on your behalf, intelligently using your templates and data to execute complex tasks from simple commands.</p><ul><li><strong>You say:</strong> “Draft a clash RFI for the new plumbing issue on the 3rd floor.”</li><li><strong>Pelles executes:</strong> It retrieves your “Clash RFI” template, applies your “Project Manager” role, addresses the GC, and prefills the project name. It then intelligently incorporates the new details (“plumbing issue on the 3rd floor”) into the pre-built structure, delivering a perfect draft in seconds — all <strong>based on your past user data and the information we have in the system.</strong></li></ul><p>Our goal is to free you from prompt engineering, giving you an intelligent partner that understands your workflows and anticipates your needs.</p><h3>Start Building Your Automated Partner Today</h3><p>These professional strategies are the training methodology for the future of work. By using them, you’re not just getting work done faster today; you are developing the precise communication habits that will power the automated systems of tomorrow.</p><p>This is the logic we’re building at Pelles — a system that listens, learns, and ultimately anticipates your needs. As subcontractors, we work in a world of specifics. It’s time our tools did, too.</p><p><strong>Build Smarter.</strong></p><p>Thank you for reading. If you found this article helpful, please share it with your network and give it a clap! :)</p><p>Find out more:</p><p><a href="https://www.pelles.ai/">Pelles | AI for Preconstruction, ‍‍Empowering‍ every worker. Automating every workflow.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f7aa4b961e4a" width="1" height="1" alt=""><hr><p><a href="https://blog.pelles.ai/the-construction-prompt-coach-get-what-you-need-from-ai-fast-f7aa4b961e4a">The Construction Prompt Coach: Get What You Need from AI, Fast.</a> was originally published in <a href="https://blog.pelles.ai/">Pelles.ai</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Guide to Building AI That Actually Works: The Iron Man Suit Principle]]></title>
            <link>https://medium.com/@matanweks/guide-to-building-ai-that-actually-works-the-iron-man-suit-principle-9f4aa961f398?source=rss-7793a4e59292------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f4aa961f398</guid>
            <category><![CDATA[andrej-karpathy]]></category>
            <category><![CDATA[human-ai-collaboration]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[agents]]></category>
            <dc:creator><![CDATA[Matan Weksler]]></dc:creator>
            <pubDate>Mon, 23 Jun 2025 10:55:51 GMT</pubDate>
            <atom:updated>2025-06-23T10:55:51.686Z</atom:updated>
            <content:encoded><![CDATA[<h4>This post is based on Andrej Karpathy’s talk at Y Combinator’s AI Startup School — <a href="https://youtu.be/LCEmiRjPEtQ?si=sNLWdQU_j1OWbR5M">Link</a></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*m6_p2wXX-g_yE3j0uNG5DQ.jpeg" /></figure><p>The AI hype train is moving at breakneck speed. Every week, we see flashy demos of fully autonomous agents promising to revolutionize everything from coding to customer service. But as anyone who has tried to build a real-world product knows, the gap between a flashy demo and a reliable tool is vast.</p><p>So, how do we bridge that gap? In a great talk at Y Combinator’s AI Startup School, AI expert Andrej Karpathy laid out a pragmatic and powerful framework for building successful AI products today. The core idea? Stop trying to build an all-powerful <strong>Iron Man robot</strong>. Instead, focus on building an <strong>Iron Man suit</strong> — a tool that augments human capabilities through partial autonomy.</p><p>This approach is built on two key principles: the <strong>Generation-Verification Loop</strong> and the <strong>Autonomy Slider</strong>.</p><h4>The Human-AI Dance: The Generation-Verification Loop</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4n5A9UJM78UAZcKtK-mf_Q.png" /><figcaption><em>image source: Andrej Karpathy - Software Is Changing (Again)</em></figcaption></figure><p>At its heart, the most effective human-AI collaboration today is a dance. The AI takes the lead on <strong>generation</strong> — creating code, drafting emails, or designing layouts. The human then steps in for <strong>verification</strong> — auditing, refining, and approving the AI’s output.</p><p>The magic happens when this loop is incredibly fast and intuitive. Karpathy emphasizes that the user interface (UI) and user experience (UI/UX) are not just afterthoughts; they are central to making this partnership work.</p><p>Here’s the secret: <strong>Make verification easy and fast.</strong></p><p>Humans are visual creatures. We can process a color-coded code diff or a visual layout change in a fraction of a second, whereas reading and parsing a dense block of text is slow and cognitively demanding. The best AI applications leverage this by building custom GUIs that present the AI’s suggestions in a way that our brains can instantly process. Instead of forcing us to work in a terminal-like chat, they give us visual tools to quickly say “yes” or “no.”</p><h4>Keeping Agents on a Leash: The Autonomy Slider</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XAAtbcQk4SDzpDqLIJe9xw.png" /></figure><p>We’ve all seen AI get a little too “creative.” Given too much freedom, agents can become overreactive, hallucinate, or get “lost in the woods,” producing overwhelming and often incorrect results. This is why Karpathy advocates for keeping AI “on a tight leash.”</p><p>This “leash” is best imagined as an <strong>autonomy slider</strong>. A great partial autonomy product allows the user to dynamically adjust how much control they cede to the AI.</p><p>Let’s look at some examples:</p><ol><li><strong>AI-Assisted Coding (e.g., Cursor):</strong> This is a perfect illustration. The autonomy slider ranges from low to high:</li></ol><ul><li><strong>Low Autonomy:</strong> Simple tab-completion for a single line.</li><li><strong>Medium Autonomy:</strong> Highlighting a function and asking the AI to refactor just that block.</li><li><strong>High Autonomy:</strong> Giving the AI control over an entire file.</li><li><strong>Full Autonomy (Agent Mode):</strong> Letting the AI attempt changes across the entire repository.<br>The user is always in the pilot’s seat, choosing the right level of assistance for the task at hand. This builds trust and makes the tool genuinely useful.</li></ul><p>2.<strong> AI in Education:</strong> Simply prompting an LLM to “teach me physics” is a recipe for disaster. The AI has no constraints. A better, “leashed” approach would be a two-part system:</p><ul><li><strong>Course Creation App:</strong> A human expert (the teacher) uses AI to create a structured, auditable curriculum — the “leash.”</li><li><strong>Course Serving App:</strong> The AI then serves this pre-approved content to the student, providing a consistent and accurate learning experience.</li></ul><p>3.<strong> Tesla Autopilot:</strong> This is a classic real-world example. Over the years, Tesla has been gradually moving the autonomy slider. It started with simple lane-keeping and has since added more complex tasks like stopping at traffic lights and navigating intersections. The human driver remains the ultimate verifier, ready to take control at any moment.</p><h4>The Takeaway: Build a Suit, Not a Robot</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7DsPJqCVwG00Xs-6v5K_ug.png" /></figure><p>The message for builders is clear: the path to building indispensable AI products today isn’t through building fully autonomous agents that try (and often fail) to do everything.</p><p>It’s about building elegant <strong>Iron Man suits</strong>.</p><p>Focus on creating <strong>partial autonomy products</strong> with custom UI/UX that speeds up the generation-verification loop. Give your users an autonomy slider, empowering them to choose the level of help they need. By augmenting human intelligence instead of trying to replace it, you’ll build tools that are not just impressive in a demo, but truly invaluable in the real world.</p><p>Thank you for reading. If you found this article helpful, please share it with your network and give it a clap! :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f4aa961f398" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Next Hire Might Be an AI That Never Sleeps — What is an Ambient AI agent?]]></title>
            <link>https://blog.pelles.ai/your-next-hire-might-be-an-ai-that-never-sleeps-what-is-an-ambient-ai-agent-a4496337c98a?source=rss-7793a4e59292------2</link>
            <guid isPermaLink="false">https://medium.com/p/a4496337c98a</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[construction]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ambient-agents]]></category>
            <dc:creator><![CDATA[Matan Weksler]]></dc:creator>
            <pubDate>Tue, 17 Jun 2025 09:49:33 GMT</pubDate>
            <atom:updated>2025-06-17T09:49:33.087Z</atom:updated>
            <content:encoded><![CDATA[<h3>Your Next Hire Might Be an AI That Never Sleeps — What is an <strong>Ambient AI agent?</strong></h3><p>If you’re in the construction industry, you know the drill. The preconstruction phase is a high-wire act of juggling tight deadlines, razor-thin margins, and mountains of paperwork. For mechanical, electrical, and plumbing (MEP) contractors and other trades, it’s a frantic race to prepare bids, perform quantity takeoffs, and coordinate with a dozen other teams, all before a single shovel hits the ground. It’s stressful, labor-intensive, and a single mistake can send ripples of cost overruns and delays through the entire project.</p><p>What if you had a team member who could read every page of a 1,000-page spec book in minutes, never miss a design change, and work 24/7 to find risks before they become problems? This isn’t science fiction. It’s the promise of <strong>Long-running Ambient</strong> <strong>AI agents</strong>, a new class of artificial intelligence poised to become the most valuable player on your preconstruction team.</p><h3>From Simple AI Tools to Digital Teammates</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0saCm8AUXp4tuShKZqDvdg.png" /></figure><p>We’ve all used AI tools like ChatGPT. You ask a question, and it gives you an answer. It’s powerful, but it’s like a power drill: useful, but only when you pull the trigger.</p><p>Long-running AI agents are something else entirely. They’re not just tools, they’re collaborators. Think of them as the craftsman, not the drill.</p><p>Instead of issuing one-off commands, you give an agent a goal, say, “Prepare a competitive bid for Project Titan.” From there, it operates like a capable project engineer, breaking the task into manageable steps:</p><ul><li><strong>Plan</strong>: “First, I’ll review the bid documents. Then I’ll do a quantity takeoff, price materials and labor, and finally draft the proposal.”</li><li><strong>Act</strong>: It taps into your systems — accessing documents, integrating with your BIM models, and pulling data from cost databases.</li><li><strong>Refine</strong>: It doesn’t stop at a first draft. If it notices missing information, it might ask, “The plans don’t specify site conditions. Can you clarify?” before proceeding.</li></ul><p>This agent keeps context. It remembers your company’s past projects, preferences, and the feedback you’ve given it. It’s a persistent, evolving partner.</p><p>But we can go even further, with ambient, reactive agents that add a new layer of intelligence:</p><ul><li><strong>Always On, Always Aware</strong>: These agents monitor ongoing activity — document uploads, model changes, emails, schedule shifts, and act when it matters.</li><li><strong>Parallel Collaboration</strong>: Multiple agents can run simultaneously, each focused on a different task — bidding, clash detection, supply tracking. No more single-threaded chatbot sessions.</li><li><strong>No Rush, Just Results</strong>: These agents don’t need to respond instantly. They can plan, reflect, query multiple APIs, and even coordinate among themselves to get it right.</li></ul><p>This is more than automation. It’s a shift from tools to teammates — persistent, adaptive, and always learning from your projects and your feedback.</p><h3>Giving Your Preconstruction Team Superpowers</h3><p>So, how does this digital teammate solve the real-world headaches of preconstruction?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BuBWsNU7-k-f2-Ca7lxlkw.png" /></figure><h4>1. The Super-Estimator: From Weeks to Hours</h4><p>Preparing a bid is a high-stakes grind. Teams spend days, even weeks, manually poring over drawings to count every fixture and pipe run (quantity takeoffs) and pricing it all in unwieldy spreadsheets. A single missed detail can wipe out a project’s profit.</p><p>An AI agent acts as a super-estimator. It can ingest an entire bid package and, in minutes, highlight every requirement relevant to your trade — no more manual searching through hundreds of pages for that one buried clause about a specific fire code. It then analyzes the 2D or 3D plans to perform an automated, highly accurate quantity takeoff. One plumbing sub using a similar AI-assisted tool cut a three-day takeoff task down to just eight hours, avoiding costly errors.</p><p>Finally, the agent can suggest pricing by analyzing your company’s historical cost data and real-time market prices, giving you a data-driven starting point for your bid. The human expert is still in charge, but they’re now focused on strategy and final review, not tedious grunt work.</p><h4>2. The Ultimate Coordinator: No More Collisions</h4><p>One of the costliest problems in construction is poor coordination. An HVAC duct clashing with a structural beam, or electrical conduits vying for the same space as plumbing lines, can lead to expensive rework on site.</p><p>An AI agent is the ultimate coordinator, acting as a vigilant air traffic controller for your project. Instead of running clash reports periodically, the agent can <strong>continuously monitor the BIM model</strong>. When a conflict arises, it doesn’t just flag it; it can notify the relevant parties and even suggest potential solutions. Imagine an agent for the HVAC trade and an agent for the fire sprinkler trade autonomously negotiating a rerouting solution and presenting it to the human designers for approval.</p><p>Furthermore, it ensures everyone is working from the latest plans. The moment an architect uploads a revised drawing, the agent detects it, compares it to the old version, and sends a summary of the changes to your team: “Heads up, the wall on Level 2 moved six inches.” This simple act prevents the catastrophic mistake of building from an outdated blueprint.</p><h4>3. The Ever-Learning Apprentice: Capturing Experience</h4><p>One of the biggest challenges in construction is losing institutional knowledge when a senior expert retires. All those years of wisdom on which clients are difficult, which materials have long lead times, and what to watch out for on certain project types can simply walk out the door.</p><p>A long-running AI agent solves this by <strong>learning from every project</strong>.</p><p>It tracks the project’s entire lifecycle. If an estimate was off, it analyzes <em>why</em> — was it an unforeseen site condition or an optimistic labor estimate? It then incorporates that lesson into its model. The next time a similar job comes up, its recommendations will be smarter. If you correct the AI (“No, don’t use that supplier, their lead times are too long”), it remembers.</p><p>Over time, the agent becomes a living archive of your company’s collective experience, constantly improving and turning every project — good or bad — into a learning opportunity that makes the next one better.</p><h3>What Makes Those Agents Game-Changing?</h3><p>Long running — ambient agents differ due to several key attributes:</p><ul><li><strong>Event-Driven Core</strong>: Triggered by document uploads, calendar events, site alerts, not by explicit user commands.</li><li><strong>Human-in-the-Loop UX</strong>: They notify you, ask clarifying questions, or request review/approval only as needed.</li><li><strong>Persistence &amp; Time‑Travel</strong>: Agents maintain state across sessions: you can pause their work, inspect step #10 of 100, tweak it, and resume, crucial for trust and transparency .</li><li><strong>Parallel &amp; Complex Workflows</strong>: Since latency isn’t critical, agents can carry out multi-step processes — planning, calling tools, reflection without user waiting.</li></ul><h3>The Road Ahead: From Assistant to Digital Worker</h3><p>Of course, this transformation won’t happen overnight. There are challenges to overcome.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JKrnrt4pI6Be5P6Dwn8jpA.png" /></figure><ul><li><strong>Data Quality:</strong> AI is only as smart as the data it’s fed. Construction data is often messy and siloed in different systems. Companies will need to get their digital house in order to get the most out of AI.</li><li><strong>Trust and People:</strong> Construction is a relationship-based industry, and many are skeptical of “black box” technology. Building trust will require transparency, starting with small, low-risk tasks and demonstrating clear value before handing over more critical responsibilities.</li><li><strong>Human Oversight:</strong> An AI agent is an assistant, not the boss. The final decisions and responsibility will always rest with human experts. The goal is to augment human intelligence, not replace it.</li></ul><p>But the vision is clear and compelling. We are moving from AI as a passive tool to AI as an active <strong>worker</strong>.</p><p>Imagine a future where your AI agent gives you a morning briefing:</p><ul><li><em>“Good morning. I’ve flagged a potential clash between the new plumbing layout and the steel on Level 3. I’ve scheduled a coordination meeting with the structural team.”</em></li><li><em>“The supplier for the primary switchgear just announced a 2-week delay. I’ve analyzed the schedule and suggest resequencing the drywall installation to avoid crew downtime.”</em></li><li><em>“Based on the last five projects of this type, we are at high risk of a budget overrun in Division 26. I’ve highlighted the specific line items to review.”</em></li></ul><p>This isn’t about replacing project managers; it’s about augmenting them with a level of insight and proactivity that was never before possible. It’s about freeing up your best people from the drudgery of data-wrangling so they can focus on what they do best: building relationships, solving complex problems, and delivering exceptional projects.</p><p>The blueprints for this new era are drawn. It’s time to let the digital workforce help us build.</p><p>Thank you for reading. If you found this article helpful, please share it with your network and give it a clap! :)</p><p>Find out more:</p><p><a href="https://www.pelles.ai/">Pelles | AI for Preconstruction, ‍‍Empowering‍ every worker. Automating every workflow.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a4496337c98a" width="1" height="1" alt=""><hr><p><a href="https://blog.pelles.ai/your-next-hire-might-be-an-ai-that-never-sleeps-what-is-an-ambient-ai-agent-a4496337c98a">Your Next Hire Might Be an AI That Never Sleeps — What is an Ambient AI agent?</a> was originally published in <a href="https://blog.pelles.ai/">Pelles.ai</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Construction AI Agents — from tools to workers]]></title>
            <link>https://blog.pelles.ai/construction-ai-agents-from-tools-to-workers-6ef56d80d666?source=rss-7793a4e59292------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ef56d80d666</guid>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[mep]]></category>
            <category><![CDATA[construction]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Matan Weksler]]></dc:creator>
            <pubDate>Tue, 07 May 2024 15:05:59 GMT</pubDate>
            <atom:updated>2024-05-07T19:49:54.567Z</atom:updated>
            <content:encoded><![CDATA[<h3>Construction AI Agents — From Tools to Workers</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QaPNF15fam6sIwtR" /><figcaption>Created with Dall-E (prompt - a man-in-black meme on AI construction Agents)</figcaption></figure><p>A new buzzword is circling around the AI block these days — Agents. While these hypes tend to come and go, Agents might be here to stay and create some game-changing capabilities for AI in general, and for preconstruction workflows specifically. But what are they? And what&#39;s really the big deal?</p><p>Let&#39;s start with an analogy, think of common AI algorithms as a tool, say a drill or an angle grinder, they are useful and for the right tasks, ones they were designed for, they get the job done. But when do you use which? And what if you need to first cut and only then drill? Would you just try both and see what happens? (please don’t). And what if you grind the edges, but they remain sharp and need another go?</p><p>While being trivial to even young tradesmen, for a robot (or a computer for that matter), it’s extremely hard.</p><p>This is where AI Agents come in, they are an ensemble of AI modules, able to infer what they are up against (and if proven wrong — reiterate), choose an action to perform (or multiple actions), utilize external modules (e.g. a search engine or a database) and iteratively refine their output if needed, providing more accurate, comprehensive results to a wide variety of tasks.</p><p>So If an AI algorithm is a tool — an AI Agent is (growing to be) the tradesman.</p><p>We’ll stop with the analogy at this point, and unfortunately ground things a bit, as AI is yet to autonomously decide when to drill holes and cut metal (though it is definitely getting there). What it is capable of though, is to autonomously decide when to look online to get you the spec of that piece of equipment it found on the schedule, or reiterate on that RFI it wrote to make it more professional and domain-specific. It could even get the relevant code book for your project based on the specs and project details to ensure compliance, not because you asked it to, but because it reasoned you’re working on a takeoff, choosing equipment ought to be compliant, or that you need that max load figure from the (latest) spec sheet.</p><h4>The Power of Agentic Reasoning in Construction</h4><p>AI Agents have the potential to revolutionize various aspects of the construction process, especially in preconstruction planning. By employing agentic reasoning, AI systems can outperform advanced models on complex tasks by iteratively improving their outputs and recovering from failures along the way, dynamically creating pipelines and workflows all towards flexible, general purpose AI.</p><p>For the brave of heart, let&#39;s get a bit more technical on key design patterns for construction AI Agents:</p><p><strong>Reflection</strong>: Encouraging AI Agents to evaluate and refine their own outputs, identifying errors, and suggesting improvements, ensuring outputs are reliable and accurate based on the related data and construction knowledge.</p><p><strong>Tools</strong>: Enabling AI Agents to leverage a broad range of external tools and data sources is crucial for enhancing their functionality and making the system domain and project-specific and following the right regulations. These data sources could include drawings and blueprints, BIM APIs, specs and code books, schedules, calculators, and other systems and databases. Those could contain project information such as material properties, vendor lists, pricing catalogs, and many more. By effectively harnessing these diverse data sources, AI Agents can perform a variety of complex tasks, such as generating detailed model analyses, preparing accurate cost estimations, ensuring code compliance, and calculating material quantities based on project schedules.</p><p><strong>Planning</strong>: Combining the above design patterns with additional Agents allows for the autonomous planning and execution of multi-step processes, such as writing proposals, preparing shop drawings, or coordinating MEP clash detection, while recovering from potential setbacks.</p><p><strong>Multi-Agent Collaboration</strong>: Deploying multiple AI Agents that can collaborate and divide tasks among themselves enhances performance in areas like risk management and MEP design.</p><h4>Optimizing Agent Architectures for Construction</h4><p>The ideal AI Agent architecture for construction projects depends on the specific task and project complexity. Successful construction Agent systems often incorporate well-defined prompts tailored to construction domains, dedicated reasoning/planning, execution, and evaluation workflows, dynamic multi-agent collaboration, human feedback and oversight mechanisms, and Intelligent filtering of relevant project information.</p><p>By carefully combining these elements, we can develop optimized Agent architectures that excel across different tasks — navigating complexities through iteration and collaboration.</p><h4>The Future of AI Agents in Construction</h4><p>While Agents and agentic reasoning is still an emerging field, AI Agents can already significantly expand the capabilities of construction experts helping them work through bid sets, create deliverables and enhance their everyday work.</p><p>Furthermore, combining agentic reasoning with ever-improving language models and faster token generation capabilities could lead to breakthroughs in areas previously thought to be out of reach for AI systems in construction.</p><p>As the construction industry continues to embrace digital transformation, agentic reasoning, and multi-agent systems are poised to play a crucial role in pushing the boundaries of what is possible in areas like MEP design, and pre-construction planning. By harnessing the power of AI Agents, construction professionals can unlock new levels of efficiency, accuracy, and productivity, driving the industry toward a more sustainable and streamlined future.</p><p>Thank you for reading. If you found this article helpful, please share it with your network and give it a clap! :)</p><p>Find out more:</p><p><a href="https://www.pelles.ai/">Pelles | AI for Preconstruction, ‍‍Empowering‍ every worker. Automating every workflow.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ef56d80d666" width="1" height="1" alt=""><hr><p><a href="https://blog.pelles.ai/construction-ai-agents-from-tools-to-workers-6ef56d80d666">Construction AI Agents — from tools to workers</a> was originally published in <a href="https://blog.pelles.ai/">Pelles.ai</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlocking the Potential of Data in Construction: Key Challenges]]></title>
            <link>https://blog.pelles.ai/unlocking-the-potential-of-data-in-construction-key-challenges-16a2e54344b0?source=rss-7793a4e59292------2</link>
            <guid isPermaLink="false">https://medium.com/p/16a2e54344b0</guid>
            <category><![CDATA[mep-engineering]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[construction]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Matan Weksler]]></dc:creator>
            <pubDate>Mon, 02 Oct 2023 13:20:22 GMT</pubDate>
            <atom:updated>2023-10-04T09:20:35.275Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UkbYsHdm05SgPwHw1_209w.jpeg" /><figcaption>Photo Generated with DALL-E 3</figcaption></figure><p>The construction industry is experiencing a shift with the integration of innovative technologies. These advancements are helping to streamline processes, improve efficiency, and ultimately reduce costs, which are critical for the success and sustainability of any construction project. However, despite these positive strides, there are still significant data challenges in construction that deserve our attention and demand innovative solutions. These challenges not only affect the efficiency and productivity of construction projects but also have far-reaching implications on the overall performance and profitability of the industry. As we move towards a more digitized and interconnected world, it is crucial that we address these data challenges head-on and work towards creating a more robust and resilient construction ecosystem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mWmqsD6PrgD62hZjCVjDsg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@alinnnaaaa?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Alina Grubnyak</a> on <a href="https://unsplash.com/photos/ZiQkhI7417A?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>1. Underutilized Data</h3><p>Approximately 80% of construction data remains untapped, driving up storage costs and missing out on intelligence insights and automation opportunities. Construction companies collect vast amounts of data daily, from drawings, blueprints, schedules, project progress, and more. Unfortunately, a significant portion of this data remains underutilized or entirely unused. This not only increases storage costs but also deprives the industry of valuable insights and automation opportunities that could enhance operational efficiency and decision-making.</p><h3>2. Complex Data Journey</h3><p>Data in construction goes through approximately 6–7 process stages before becoming useful, highlighting the need for streamlining. From data collection, storage, cleaning, and analysis, to visualization and decision-making, each stage is crucial and requires careful handling. Any inefficiency or error in one stage can have a cascading effect on the subsequent stages, leading to inaccurate insights and suboptimal decisions. Therefore, there is an urgent need for streamlined processes and tools that can facilitate the seamless flow of data through its journey.</p><h3>3. Lack of Data Reusability</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AnNvJPT9q9I8zpBK15SfiA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@john_cameron?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">John Cameron</a> on <a href="https://unsplash.com/photos/7zocFMzvbpc?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Naturally, every construction project is unique, holding a unique design and functionality. However, unsurprisingly, there’s a fundamental uniformity in the nature of the information that all construction projects share. Despite the wealth of data and experiences encapsulated in each project, its substantial information often remains confined within its lifecycle, seldom benefitting future endeavors. This scenario leads to the inability to leverage valuable insights nurtured in each project resulting in the reliance on experience or perpetual reinvention of the wheel, consuming unnecessary resources and time. By intertwining data from diverse projects, we could refine construction processes, improve design efficiency, and elevate overall industry standards. This approach would allow every unique project to contribute to the evolution of construction knowledge, pushing the boundaries of what is achievable.</p><h3>4. Information Overload</h3><p>Each project involves a vast amount of information, with crucial details often buried within extensive and generic documents. The sheer volume of data collected during construction processes can be overwhelming, making it challenging to discern vital information from the non-essential. Professionals face the daunting task of sifting through endless streams of data to find those nuggets of information crucial to their specific needs and roles. This inundation of generic information can lead to vital insights being overlooked, ultimately impacting the quality and success of construction projects.</p><h3>Turning Challenges into Opportunities</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jOBuq0wmjf-T_FuMixumIg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@helloimnik?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Nik</a> on <a href="https://unsplash.com/photos/umFPf301OjQ?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Data can be the blueprint for future success in the construction industry. By addressing these challenges head-on and implementing the necessary tools and processes, we can turn these challenges into stepping stones toward a brighter, data-driven future.</p><h3>Conclusion</h3><p>The construction industry is on the cusp of a digital revolution, and data is at the heart of this transformation. By addressing the challenges of underutilized data, complex data journey, uniformity paradox, and more we can unlock the full potential of data in construction. This will not only lead to more efficient and cost-effective operations but also foster innovation and drive the industry towards a more sustainable and prosperous future.</p><p>Let’s take proactive steps to overcome these data challenges and build a brighter, data-driven future for the construction industry!</p><p>Thank you for reading. If you found this article helpful, please share it with your network and give it a clap! :)</p><p>Find out more:</p><p><a href="https://pelles.ai/">HOME | Pelles.ai</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=16a2e54344b0" width="1" height="1" alt=""><hr><p><a href="https://blog.pelles.ai/unlocking-the-potential-of-data-in-construction-key-challenges-16a2e54344b0">Unlocking the Potential of Data in Construction: Key Challenges</a> was originally published in <a href="https://blog.pelles.ai/">Pelles.ai</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unleashing Efficiency: The Power of Multimodal AI in Pre-Construction and MEP Design]]></title>
            <link>https://blog.pelles.ai/unleashing-efficiency-the-power-of-multimodal-ai-in-pre-construction-and-mep-design-1c838ab45f1a?source=rss-7793a4e59292------2</link>
            <guid isPermaLink="false">https://medium.com/p/1c838ab45f1a</guid>
            <dc:creator><![CDATA[Matan Weksler]]></dc:creator>
            <pubDate>Thu, 10 Aug 2023 12:23:30 GMT</pubDate>
            <atom:updated>2023-08-10T12:23:30.458Z</atom:updated>
            <content:encoded><![CDATA[<p>In the ever-evolving world of AI, innovation knows no bounds. One of the latest groundbreaking advancements that are reshaping the landscape is multimodal artificial intelligence.</p><p>Multimodal AI refers to models that can process and understand information from multiple modalities or sources of data. A modality in this context refers to a specific type of data or sensory input, such as text, images, audio, video, and more. Multimodal AI aims to combine and leverage information from different modalities to gain a more comprehensive and holistic understanding of the data, leading to more accurate and nuanced AI-driven tasks.</p><p>The concept of multimodal AI is particularly relevant in scenarios where information is presented in various formats and where a single modality might not provide a complete picture. For example, understanding a scene in a video might require analyzing both the visual content and the accompanying audio. Similarly, processing social media content might involve considering both the textual information and any associated images.</p><h4><strong>Understanding Multimodal AI in Pre-Construction and MEP Design</strong></h4><p>At its core, Multimodal involves the fusion of various data sources, to gain a comprehensive understanding of complex scenarios.</p><p>In the context of pre-construction and MEP design, multimodal AI transcends traditional silos of information. It blends textual descriptions, 2D and 3D visual representations, structural data, equipment specifications, code requirements, and spatial relationships into a unified framework. When applied to floor plan analysis and MEP design, this innovative approach offers engineers, and construction professionals a powerful toolkit to optimize design processes and outcomes. Laying the foundation for more intelligent decision-making throughout the construction lifecycle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e_FZArE3UkwEZF1y" /><figcaption>Multimodal AI in Pre-Construction (Source autor)</figcaption></figure><h4><strong>Enhancing MEP Design with AI</strong></h4><p>As construction projects grow in complexity, the demand for innovative solutions becomes more pressing. It becomes cliché to say that AI answers this call, yet multimodal AI is nonetheless a key milestone in his ability to do so. The integration of various inputs enables computers to better understand, contextualize, and reason across construction projects, allowing the creation of tools to empower professionals, de-risk projects and increase efficiency.</p><p>Find out more:</p><p><a href="https://www.pelles.ai/">Pelles | AI for Preconstruction, ‍‍Empowering‍ every worker. Automating every workflow.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c838ab45f1a" width="1" height="1" alt=""><hr><p><a href="https://blog.pelles.ai/unleashing-efficiency-the-power-of-multimodal-ai-in-pre-construction-and-mep-design-1c838ab45f1a">Unleashing Efficiency: The Power of Multimodal AI in Pre-Construction and MEP Design</a> was originally published in <a href="https://blog.pelles.ai/">Pelles.ai</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[So retrieval is what we needed?]]></title>
            <link>https://pub.towardsai.net/so-retrieval-is-what-we-needed-9485e4f9e939?source=rss-7793a4e59292------2</link>
            <guid isPermaLink="false">https://medium.com/p/9485e4f9e939</guid>
            <category><![CDATA[deepmind]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[retrieval]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[transformers]]></category>
            <dc:creator><![CDATA[Matan Weksler]]></dc:creator>
            <pubDate>Sun, 09 Jan 2022 06:36:33 GMT</pubDate>
            <atom:updated>2022-01-21T05:57:22.324Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsai.net/p/category/artificial-intelligence">Artificial Intelligence</a></h4><p>Last month DeepMind published their new NLP model called <a href="https://arxiv.org/pdf/2112.04426.pdf">RETRO </a>(Retrieval-Enhanced TRansfOrmer) which according to the paper, is a leap forward in the NLP world in multiple aspects. A notable one is that while this model achieved comparable results to SOTA architecture (e.g., <a href="https://arxiv.org/abs/2005.14165">GPT-3</a>) it’s X25 times smaller with only 7.5B parameters compared to the 178B parameters of <a href="https://uploads-ssl.webflow.com/60fd4503684b466578c0d307/61138924626a6981ee09caf6_jurassic_tech_paper.pdf">AI21 Jurassic-1</a>.</p><p>This breaks the presumption that bigger models mean better accuracy.</p><p>The main advantage of smaller models is that they mean shorter training time thus less computing time and energy and overall a smaller carbon footprint. Just to put things in perspective, It would take 355 years to train GPT-3 on a single NVIDIA Tesla V100 GPU with an estimated cost of around $5M in compute time (to train GPT-3 OpenAI Uses 1,024x A100 GPUs still reaching an astronomical training time of around 34 days). Carbon-wise, it produced the equivalent of 85,000 kg of carbon dioxide during its training! That’s as much carbon dioxide as driving 120 cars for a whole year! Adding to that, due to their high training cost, re-training large language models regularly to incorporate new data, languages, norms, or updating the model in terms of fairness and bias is prohibitively expensive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/0*7oePLzdXEb9LiEsz" /><figcaption><em>Source: </em>S&amp;P global</figcaption></figure><p>An additional advantage of smaller models is that while large language models can perfectly memorize parts of their training data when coupled with large training datasets gathered from the web or other sources, imposing privacy and safety implications, smaller models are less prone to such scenarios, <em>as shown in</em><a href="https://aclanthology.org/2020.findings-emnlp.301/"><em> Gehman et al. (2020)</em></a><em>. </em><a href="https://dl.acm.org/doi/10.1145/3442188.3445922"><em>Bender et al. (2021)</em></a></p><h4>So why have model sizes increased so much in recent years?</h4><p>Up until recently when we spoke about NLU models we analyzed the language logic behind the text and responded accordingly. In recent years, we are trying (and hoping) to solve more knowledge-intensive tasks such as question answering, and to do so we need to integrate into the model more knowledge about the world. An example to question answering tasks might be — <em>When was the movie Matrix released?</em></p><p>With linguistics utilization, we can only understand that we are searching for a date and not a name but if we are interested in getting the right answer the model needs to store this knowledge somewhere. Another reason for the growth is the increasing availability of computing power and the advances in GPU processors explicitly allow the rapid growth of models and the training cycles needed.</p><h4><strong>The Retrieval Philosophy</strong></h4><p>Instead of increasing the size of the model and training on more data, you equip models with the ability to directly access a large database as they perform predictions — a semi-parametric approach.</p><p>The best analogy to the retrieval method is the difference between Open-Book vs. Closed-Book exams.</p><p>While in a closed-book exam one needs to study and remember all the test material. In an open-book exam, you only need to understand the essence of the material, where to look for the right answer, and how to extract from the given material the correct one.</p><p>The concept of retrieval (in NLP models) is not new and has been suggested for NLU models in several papers including “Improving Neural Language Models with a Continuous Cache” by <a href="https://openreview.net/forum?id=B184E5qee">Grave et al.</a> and Retrieval Augmented Generation from <a href="https://arxiv.org/abs/2005.11401">Meta-AI</a> or the in the work of machine translation that retrieves translation pairs based on edit distance between source sentences and guide the translation output using the closest retrieved target sentences such as <a href="https://arxiv.org/abs/1804.02559">Zhang et al. (2018)</a> and <a href="https://www.aaai.org/GuideBook2018/17282-74380-GB.pdf">Gu et al. (2018)</a>.</p><p>Below I focus on the retrieval mechanism implemented by DeepMind which to my knowledge (and theirs), is the first work that shows the benefits of scaling the retrieval database to trillions of tokens for large parametric language models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/0*j3arAUSnJZXe0mps" /><figcaption><em>Source: Improving language models by retrieving from trillions of tokens</em></figcaption></figure><h4><strong>So what’s this retrieval method that makes the magic happen?</strong></h4><p>At a high level, retrieval models take the input and from a given database retrieve information similar to the input to incorporate it into the model to improve the predictions of the network.</p><p>The novelty of the retrieval-enhanced architecture of DeepMind’s RETRO model is this it’s designed to be capable of retrieving data from a database with trillions of items and being able to incorporate it into the model using a chunked cross-attention scheme. That enables a linear correlation to computational time and storage requirements.</p><p>First thing, in order to implement a retrieval mechanism, we need to construct a key-value database, where values are raw chunks of text tokens and keys are frozen embeddings. The key in the RETRO model is generated by a frozen pre-trained BERT model to avoid having to periodically-compute embeddings over the entire database during training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VwAqGrGjulRXlh7V" /><figcaption><em>Source: </em>author</figcaption></figure><p>Depending on the task, the value can be a wide variety of things, for example in the question-answering task the value would be a tuple of neighbors (questions) — continuation (answer).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/0*kb5hvqRDDG957WnA" /><figcaption><em>Source: </em>author</figcaption></figure><p>During the training phase, each training sequence is divided into chunks, which are then embedded using frozen BERTs. From the database, each chunk is retrieved his k-nearest neighbors to be augmented later in the RETRO block.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*tt1pw1969Vr7GJH8" /><figcaption><em>Source: </em>author</figcaption></figure><p>Then the encoder-decoder architecture integrates retrieval chunks into the model’s predictions.</p><p>The retrieved information is passed through a transformer encoding block to be able to be incorporated at the decoder part of the RETRO model. In practice, at the RETRO model, it’s being incorporated every third block starting from the 9th RETRO block (i.e, 9, 12, 15…32)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*b91yCCAfVxAuC7Yu" /><figcaption><em>Source: Improving language models by retrieving from trillions of tokens: Retro architecture — Retro architecture. Left: simplified version where a sequence of length 𝑛 = 12 is split into 𝑙 = 3 chunks of size 𝑚 = 4. For each chunk, we retrieve 𝑘 = 2 neighbors of 𝑟 = 5 tokens each. The retrieval pathway is shown on top. Right: Details of the interactions in the Cca operator. Causality is maintained as neighbors of the first chunk only affect the last token of the first chunk and tokens from the second chunk.</em></figcaption></figure><h4><strong>The advantages of retrieval</strong></h4><p>The obvious advantage of retrieval is that you can use a smaller model without losing accuracy. In practice, this has a beneficial impact on training (or re-training for model update) and inference time. More than that, keeping the retrieval models up-to-date, may be sufficient only to update the retrieval database, which is orders of magnitude cheaper than re-training a model from scratch.</p><p>Additionally, if portions of the training data are found to be biased or toxic outputs after training, retrieval allows for some correction, as the offending retrieval data can be retroactively filtered.</p><h4><strong>Summary</strong></h4><p>I intentionally did not want to go too deeply into the mechanics of the model. The model has already been analyzed in a number of articles, and of course, you can always read the full article by Deepmind (<a href="https://arxiv.org/pdf/2112.04426.pdf">Link</a>).</p><p>In my opinion, the concept of retrieval is revolutionary and worth the extra attention (pun intended). The paradigm shift is that instead of collecting and labeling huge amounts of data to teach our models, let’s see how we can teach a model to analyze information and just give the model the information we think might be relevant to him to extract this data.</p><p>Nowadays, access to information is endless, it is much more intuitive for us as humans to search the web and analyze the given information rather than remember everything, so why shouldn’t models do the same?</p><p>Just as “attention” allowed us to see the contexts within a sentence, methods like retrieval could help us link the model to the information relevant to it.</p><p>This could be the first step towards General-AI models, by embedding into the model the ability to extract task-specific information from a huge database to improve accuracy in a broad spectrum of tasks.</p><p>Furthermore, since we can directly visualize or modify the neighbors that the retrieval mechanism provides, we can gain more insights into the outputs of a model, thereby improving the explainability of the model.</p><p>Looking ahead, the next questions to ask could be; how can we improve our existing language models today? Is it possible to extend this retrieval method to other less intuitive places? What would be our database size limit? Could we improve our database query? and many more.</p><p>Days will tell</p><p>BTW, The Matrix was released on March 24, 1999 :)</p><h4>References</h4><p><a href="https://deepmind.com/research/publications/2021/improving-language-models-by-retrieving-from-trillions-of-tokens">https://deepmind.com/research/publications/2021/improving-language-models-by-retrieving-from-trillions-of-tokens</a></p><p>Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., … &amp; Sifre, L. (2021). Improving language models by retrieving from trillions of tokens. <em>arXiv preprint arXiv:2112.04426</em> - <a href="https://arxiv.org/pdf/2112.04426.pdf">https://arxiv.org/pdf/2112.04426.pdf</a></p><p>Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., … &amp; Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. <em>arXiv preprint arXiv:2005.11401</em>. <a href="https://arxiv.org/abs/2005.11401">https://arxiv.org/abs/2005.11401</a></p><p>Grave, E., Joulin, A., &amp; Usunier, N. (2016). Improving neural language models with a continuous cache. <em>arXiv preprint arXiv:1612.04426</em>. -<a href="https://openreview.net/forum?id=B184E5qee">https://openreview.net/forum?id=B184E5qee</a></p><p><a href="http://jalammar.github.io/illustrated-retrieval-transformer/">http://jalammar.github.io/illustrated-retrieval-transformer/</a></p><p>Gehman, S., Gururangan, S., Sap, M., Choi, Y., &amp; Smith, N. A. (2020). Realtoxicityprompts: Evaluating neural toxic degeneration in language models. <em>arXiv preprint arXiv:2009.11462</em> - <a href="https://aclanthology.org/2020.findings-emnlp.301/">https://aclanthology.org/2020.findings-emnlp.301/</a></p><p>Zhang, J., Utiyama, M., Sumita, E., Neubig, G., &amp; Nakamura, S. (2018). Guiding neural machine translation with retrieved translation pieces. <em>arXiv preprint arXiv:1804.02559</em> -<a href="https://arxiv.org/abs/1804.02559">https://arxiv.org/abs/1804.02559</a></p><p>Gu, J., Wang, Y., Cho, K., &amp; Li, V. O. (2018, April). Search engine guided neural machine translation. In <em>Proceedings of the AAAI Conference on Artificial Intelligence</em> (Vol. 32, №1)-<a href="https://www.aaai.org/GuideBook2018/17282-74380-GB.pdf">https://www.aaai.org/GuideBook2018/17282-74380-GB.pdf</a></p><p>Bender, E. M., Gebru, T., McMillan-Major, A., &amp; Shmitchell, S. (2021, March). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?🦜. In <em>Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency</em> (pp. 610–623) - <a href="https://dl.acm.org/doi/10.1145/3442188.3445922">https://dl.acm.org/doi/10.1145/3442188.3445922</a></p><p>Lieber, O., Sharir, O., Lenz, B., &amp; Shoham, Y. (2021). Jurassic-1: Technical details and evaluation. <em>White Paper. AI21 Labs</em>. — <a href="https://uploads-ssl.webflow.com/60fd4503684b466578c0d307/61138924626a6981ee09caf6_jurassic_tech_paper.pdf">https://uploads-ssl.webflow.com/60fd4503684b466578c0d307/61138924626a6981ee09caf6_jurassic_tech_paper.pdf</a></p><p><a href="https://fortune.com/2021/04/21/ai-carbon-footprint-reduce-environmental-impact-of-tech-google-research-study/">https://fortune.com/2021/04/21/ai-carbon-footprint-reduce-environmental-impact-of-tech-google-research-study/</a></p><p><a href="https://www.spglobal.com/marketintelligence/en/news-insights/trending/HyvwuXMO9YgqHfj7J6tGlA2">https://www.spglobal.com/marketintelligence/en/news-insights/trending/HyvwuXMO9YgqHfj7J6tGlA2</a></p><p><a href="https://www.theregister.com/2020/11/04/gpt3_carbon_footprint_estimate/">https://www.theregister.com/2020/11/04/gpt3_carbon_footprint_estimate/</a></p><p><a href="https://ai.facebook.com/blog/retrieval-augmented-generation-streamlining-the-creation-of-intelligent-natural-language-processing-models/">https://ai.facebook.com/blog/retrieval-augmented-generation-streamlining-the-creation-of-intelligent-natural-language-processing-models/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9485e4f9e939" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/so-retrieval-is-what-we-needed-9485e4f9e939">So retrieval is what we needed?</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Developing an RL agent for cognitive impact]]></title>
            <link>https://medium.com/@matanweks/developing-an-rl-agent-for-cognitive-impact-bd37bfddea11?source=rss-7793a4e59292------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd37bfddea11</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[imitation-learning]]></category>
            <category><![CDATA[human-robot-interaction]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <dc:creator><![CDATA[Matan Weksler]]></dc:creator>
            <pubDate>Thu, 23 Dec 2021 16:31:09 GMT</pubDate>
            <atom:updated>2021-12-23T16:31:09.637Z</atom:updated>
            <content:encoded><![CDATA[<h4>In this post, I will describe the algorithmic side of a joint project with Ido Glanz under the supervision of Prof. Erez Karpas and prof. Tzipi Horowitz-Kraus of a Story-telling robotic teddy-bear with an LTL constrained reinforcement and imitation learning algorithms designed to maximize children’s interaction and attention</h4><p>I’m sure the parents amongst us are familiar with the dilemma of handing your kid an iPad to watch YouTube and get a few minutes of rest, knowing it might not be the best pedagogical experience (even if we won’t admit it publicly).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*asHlz8Mufy0gmr_A.jpg" /></figure><p>To put this in a more serious manner, a study by Prof. Horowitz-Kraus seeks to examine the cognitive impact of this dilemma. An experiment was conducted in a number of kindergartens to determine whether a child’s cognitive development is affected when a person reads him a story as opposed to when he watches the same story on a tablet.</p><p>I would recommend reading the full articles, but in brief, unsurprisingly the child shows more cognitive improvement in his social and language skills when there is a social interaction compared to the interaction with the tablet. even if the same story was played.</p><p>While most of these situations don’t necessarily come from laziness, sometimes there are situations where we are just not able to have one-on-one experiences with the children, for instance in hospitals. Adding to that, the advancements that technology brings to the table open a wide range of new experiences, and the ability to learn new skills that we never could before.</p><h4><strong>From that, we started our journey to explore — Would it be possible to create a computer that is able to interact with a better cognitive experience?</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/0*yAN7OUNo8NUngJz2.jpg" /></figure><p>The technological advancements in artificial intelligence allow computers to accomplish amazing things that we couldn’t even dream of a decade ago like autonomous driving, language translation, and beating the world champion at Go.</p><p>Technology now occupies an increasingly large part of our lives, and most of the day (except maybe while we sleep) we are interacting with some kind of device.</p><p>Human-Robot interaction researchers are looking at this very interaction between humans and technology. In this research area, the focus is to investigate our interaction with machines and enable a more intuitive interaction similar to when we work together with another individual. As a simplistic example, do we need to invest time and learn all the functions of the new microwave, or can the new microwave just figure out on its own what we want to heat and do it for us?</p><h4><strong>Can we create a robot that tells a story to children while also giving them the interaction benefits that are being lost while using technology?</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3HhYRZ3TRimldviT" /></figure><p>For this purpose, we approached the problem by looking at places where the algorithm is more environmentally aware and can react to environmental signals. The research area where it happens is called reinforcement learning.</p><p>Reinforcement learning refers to the study of how intelligent agents ought to take actions in an environment in order to maximize the notion of cumulative reward</p><p>I would like to share our experience with building a real-world RL-based robotic system from designing the algorithmic side through training consideration toward on-device AI agents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*GP1dg31A-i_gDpPUb2uMWg.jpeg" /></figure><p>The idea was to build an agent that tells the story with the goal of maximum attention and active interaction based on the observable environment, for instance when there is a noise he will ask to be quiet or when he feels the concentration level decreases he will ask a question or even if kids being too “concentrated” the agent will perform an action (e.g. moving its limbs) to break this blank-gazing vicious cycle and so on.</p><p>Our state vector was constructed from metrics derived from both sound and video analysis, depicting the attention, engagement, and concentration of the kids using tools such as gaze analysis, movement monitoring, fluctuations in sound, etc.</p><p>We implemented a policy-gradient RL algorithm with an actor-critic scheme together with an added LTL constraints module aimed to allow a way to elaborate constraints in a way the algorithm could interpret and capture in its learning process.</p><p>The general flow can be seen in the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/0*UWDCBPZu5jhOxB2U" /></figure><p>Generally speaking, the algorithm is fed with a state vector constructed from the accumulated metrics and calculates the next action in order to maximize/optimize the engagement of the children w.r.t a rewarding mechanism accounting for both childrens state and the confinement to a given LTL (Linear Temporal Logic) formula pre-defined by the user.</p><p>LTL statements are pre-configured per the user’s best understanding of the task and a general sense of creating “fairness” and “rationality” of the problem and to enable the algorithm to better capture those understandings without the need to learn them entirely from scratch. An example of such a constraint could be <em>“Eventually(play next page)” </em>or in LTL syntax: <em>“F(play_next_page)” </em>to avoid a long loop of repetitive actions such as asking a question or moving hands/head causing to eventually lose the children’s attention. Should be noted that these constraints are not strictly limiting the robot’s actions (i.e. blocking it from asking 3 questions in a row) but are integrated as part of his reward mechanism so it would learn these but in a more explicit manner than just through observing the general state of the children.</p><p>The LTL is evaluated using a DFA (Deterministic Finite Automata) which is fed a state-trace with a time-sequence vector of previous actions which is also a part of the state vector fed to the policy function (thus the state space is extended per the LTL statements needs and changing the LTL would/could consequent in the need to re-train the algorithm), if the state-sequence confines to the LTL statement the robot is rewarded accordingly and penalized otherwise.</p><p>As the nature of this work/algorithm doesn’t allow the easy collection of data and each episode has to be run with actual groups of children, and while fearing the lack of pre-training before actually letting it engage with children, we added an imitation learning option (a manual mode) allowing for a human supervisor to manually choose the next action, thus replace the algorithm inference.</p><p>That said, while playing, the algorithm is still running in the background and is learning the humans’ choices in a supervised manner, i.e. viewing the supervisor’s choice as a ground truth/label. This way, the first few trials of the robot could be manually run while the algorithm is learning in the background and potentially would have some prior knowledge/intuition when moving back to agent mode.</p><p>This is a high-level description of the algorithmic side of our story-telling robot. We wanted to share our experience with developing a real-world AI agent, and I hope you found it useful.</p><p>This post is part of a series of posts (which we will publish in the near future) that will give a brief overview of all major components of the robot, including insights from the cognitive impact study. Stay tuned.</p><p>Feel free to DM Ido or me for further questions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd37bfddea11" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>