TL;DR

• AI can supercharge divergent thinking: Modern AI like ChatGPT can generate a high volume of ideas by combining familiar concepts in novel ways. It excels at combinatorial creativity (mixing known patterns), scoring at top human levels on some creativity tests. However, it rarely produces truly groundbreaking ideas on its own, so don’t expect an automatic genius.
• Fluency ≠ Depth: Don’t mistake an AI’s torrent of output for genuine creative insight. ChatGPT might spit out 100 suggestions, but many will be obvious or superficial. Research shows it tends toward “mid-level” novelty and reinforces familiar patterns without guidance. High fluency (quantity) doesn’t guarantee originality or usefulness.
• Use AI as a thinking partner, not a replacement: The best results come when you iterate with the AI. Techniques like prompt chaining (iterative prompts), analogies, and persona prompts help push beyond the first clichés. Guide the AI with feedback and fresh angles, and use your human judgment to filter and refine.
• Structure your creative process: An effective workflow might follow a divergent-convergent cycle (like a Double Diamond). Let the AI expand your idea space, then step back to evaluate and refine, possibly looping this process. This ensures you harness the AI’s breadth and maintain focus on truly valuable ideas.
• Mind the pitfalls: Always verify AI-generated ideas. They can include factual errors or unrealizable suggestions (AI hallucinations). Over-relying on AI can erode your own creative skills over time. And since AI draws from existing data, be cautious of inadvertently borrowed or copyrighted material in its outputs. Treat every AI suggestion as a starting point to build on responsibly.

AI’s Creative Potential: What the Research Says

Is AI actually creative or just a clever mimic? The answer, it turns out, is a bit of both. Recent studies show that advanced language models can perform impressively on divergent thinking tasks — the kind of tests where you generate lots of ideas or uses for an object. For example, GPT-4 has been found to produce more original and elaborate answers than the average human in classic brainstorming tests like the Alternative Uses Task. In one paper, researchers noted GPT-4’s responses scored in the top 1% of human answers in creativity measures. Clearly, today’s AI isn’t just parroting back common knowledge; it’s recombining things in ways we might not anticipate, a key aspect of creativity known as combinatorial creativity.

Combinatorial creativity is basically “new ideas from old components.” AI is extremely good at this. As an LLM (Large Language Model), ChatGPT was trained on countless texts and can mash up concepts from different domains effortlessly. It might connect coffee with winter and suggest a “peppermint latte” idea, or blend bicycle and electricity to envision a new gadget. AI’s architecture helps here: a Transformer-based model can attend to distant concepts in context and draw analogies between them in vector space. It’s like having a massive mental library of patterns and metaphors to pull from. By increasing the “randomness” setting (temperature) of the AI’s output, you can encourage more offbeat combinations — a higher chance of surprising, if sometimes bizarre, ideas. I often nudge the temperature up when I want wackier suggestions, then dial it down when I need to get serious.

That said, AI’s creativity has limits. Researchers from University of Essex and others found that while models like GPT-4 outperform most humans in sheer quantity and even average originality of ideas, they rarely beat the very best human creators. In their 2025 study, only 0.28% of AI’s ideas reached the top-tier of human creativity benchmarks, and humans were 35 times more likely to come up with a “standout” original idea. The AI tended to produce middling novelty — clever twists on known ideas, but not the kind of radical “eureka!” breakthroughs that redefine a field. The authors described AI’s sweet spot as “mid-level novelty”, reinforcing combinatorial creativity over truly conceptual innovation. In plainer terms: ChatGPT is great at riffing within the box, but it won’t often think completely outside the box. It circles the familiar unless we explicitly push it further.

Why doesn’t AI hit those rare, revolutionary ideas? One reason is that it lacks genuine divergent thinking in the human sense. Divergent thinking isn’t just making lots of stuff — it’s the ability to wander into remote, unlikely ideas. AI can mimic divergence by spewing many variants, but it doesn’t have a conscious “imagine the impossible” drive. It also lacks the evaluative side of creativity — the convergent thinking that knows which wild idea is actually valuable or feasible. Without human guidance, AI has no filter for quality or relevance beyond the patterns it learned. And those patterns are, by definition, grounded in past data. AI might surprise us, but it won’t purposely seek out a leap that breaks its learned mold — not unless we guide it to.

In summary, research paints AI as a formidable brainstorming ally that can match human creativity on quantity and even baseline originality, but one that won’t replace human ingenuity. It provides a vast associative memory and relentless idea machine, if you leverage it right. The key is recognizing what it’s good at (combining and iterating on known ideas) and what still needs a human touch (judgment, truly novel leaps, and that intuitive sense of “what if?” that drives breakthroughs).

Common Misconceptions: Fluency, Depth, and the “Cliché Machine” Trap

When I first started using AI for ideas, I fell into a classic trap: equating the AI’s endless stream of suggestions with high-quality creativity. This is a major misunderstanding. Just because ChatGPT can generate 50 taglines in a blink doesn’t mean it has given you 50 great taglines. In creativity, fluency (the number of ideas) is not the same as depth or originality. AI is fluent by design — it’s trained to continue producing plausible content. It will keep giving you more until you say stop. But as any creative knows, nine out of ten ideas (or 49 out of 50 in this case) might be mediocre. AI is no different. In fact, without careful prompting, it often gravitates to obvious, safe ideas first.

Researchers have observed that AI tends to default to the average. One study asked an AI to brainstorm new uses for a garden hose and a tennis racket. The result? About 21% of the AI’s answers ended up being essentially “use it as a sprinkler” — a decent idea but hardly groundbreaking, and repeated with minor tweaks. The AI was stuck on the most common association (hose -> water sprinkler). This highlights a second misconception: that AI’s ideas come from some boundless imagination. In reality, the AI is drawing from the common patterns in its training data. It produces the statistically likely answers first — which often means conventional ideas. That’s why its initial outputs can feel cliché or derivative. As one paper quipped, “large language models often give ‘the most average answer’”. To get past that, sometimes you literally need to force the AI beyond the obvious. For example, simply adding a prompt like “Give me ideas that are as different from each other as possible” can dramatically boost the variety of answers. I’ve learned to include instructions for diversity or even explicitly say “no clichés, think wild” to jolt the model out of repetition.

Another misunderstanding is thinking the AI “knows” which ideas are brilliant. It’s easy to anthropomorphize ChatGPT — it writes so eloquently that we imagine an intelligent mind behind the screen. We might even assume “Wow, it came up with something I never thought of — it must be smarter or more creative than me!” But we have to remember: the AI has no intent or insight, it’s generating patterns of words. One group of experts dubbed this the “illusion of thinking” — the AI appears to reason or invent, but under the hood it’s still pattern matching. It will cheerfully present a bad idea with the same confident tone as a good one. I’ve had ChatGPT suggest product ideas that sounded cool, but on reflection made no sense (like a solution that already exists, or a marketing angle that didn’t fit the audience at all). If you find yourself wowed by an AI suggestion, take a step back and scrutinize it as you would any human colleague’s idea. The AI won’t be offended! In fact, it expects you to judge and steer it. As Dr. Hanel put it, an LLM’s output is “just another perspective” and must be evaluated objectively. In short: don’t outsource your discernment to the machine. Use it to expand possibilities, but apply your own criteria to decide what’s actually original, relevant, or profound.

Finally, there’s the “cliché machine” problem described by writer Katti Jisuk Seo: if you use ChatGPT in a naive way, you often get bland, generic results. Many users misunderstand this as “AI isn’t creative.” But often it’s how we’re using it. I found that if I ask a very generic question, I get a generic answer — surprise, surprise. The model aims to please with a probable answer. That’s why a vanilla prompt like “Give me some marketing ideas for a new coffee” might return the same tired coffee shop promos you’ve heard before. The AI isn’t trying to be boring; it’s trying to be helpful by giving you what it thinks is a solid answer. Breaking out of this requires more inventive prompting (we’ll cover specific techniques next). The bottom line is, you get out what you put in. Treating ChatGPT as a magical oracle will lead to disappointment and the impression that it’s only capable of surface-level ideas. Treating it as a tool you need to direct and prod will unlock far more interesting results.

Takeaway: Don’t be misled by AI’s fluency. Volume of ideas is not the same as quality of ideas. The first things an AI offers are often the low-hanging fruit — the same obvious ideas a large crowd might also suggest. Real creativity with AI requires pushing past those, which means giving it smarter prompts and not trusting its output at face value. In the end, AI is a brilliant imitator, not an autonomous genius. It needs your human spark to do something truly different.

How to Use AI as Your Creative Sidekick (Techniques That Work)

Through trial and error (and a lot of late-night prompt tinkering), I’ve learned that how you prompt the AI makes a night-and-day difference in the quality of ideas. Using AI effectively in brainstorming isn’t one single skill — it’s a set of techniques you can combine. Here I’ll introduce a few of the most useful prompting methods I use and that research and community knowledge have validated: prompt chaining, analogical prompting, and persona prompting. Think of these as recipes to get ChatGPT to go beyond the obvious.

Key Prompting Techniques for Brainstorming with AI

Let’s talk about how these actually look in practice:

• Prompt Chaining: The core idea is iteration. Instead of the one-and-done prompt (“Give me ideas for X”), you treat the AI like a collaborator you can workshop with. For instance, I might start with “List 10 rough ideas for a new mobile app to help people learn guitar.” Once I get those, I’ll follow up with something like, “Great, now take idea #3 and make it more unique — maybe combine it with a concept from gaming.” The AI then builds on that idea. I might continue: “Now give me a catchy slogan for this refined idea.” This chain can be done in one long prompt (structuring it in steps) or interactively. Research confirms this approach leads to more “multi-faceted and refined output”. By explicitly telling the model the process (e.g., “first do A, then do B, then do C”), you help it “think” in stages. I often find the first batch of ideas are mediocre, but after a few rounds of feedback (“make them bolder”, “now combine two of the ideas into one”), the results are far more interesting. Prompt chaining is basically guiding the AI through diverging and converging phases, which is exactly how human creativity tends to work as well.
• Analogical Prompting: This one is a gem for getting unstuck. Whenever I’m faced with a problem that feels too constrained, I ask the AI to find an analogy. For example, I was brainstorming improvements for a coworking space. Instead of directly asking for ideas for coworking spaces (which yielded fairly standard answers), I tried: “Think of how bees work in a beehive or ants in a colony — how do they collaborate and organize? Now, given those insights, suggest new features for a coworking space.” The results were much more novel — the AI came up with an idea of an “idea pheromone board” where people leave traces of ideas for others to build on (inspired by how ants leave pheromone trails). I never would have thought of that on my own! The technique works because you’re letting the AI’s knowledge network roam freely before tethering it back to your problem. You’re effectively saying “search your memory for something kind of like this” and then “apply it here.” It leverages AI’s strength in connecting distant dots. I find analogical prompts especially useful in innovation and product design — when you need a creative leap, not just incremental tweaks.
• Persona Prompting: This has been a game-changer in avoiding the “generic AI voice.” If I ask ChatGPT, “Give me social media campaign ideas for a new chocolate brand,” I might get decent but bland ideas. But if I say, “You are Willy Wonka giving marketing advice: now suggest some fantastical social media campaign ideas for a new chocolate brand,” the tone and imagination ramp up significantly. Suddenly the ideas involve golden ticket giveaways and edible posts (literally outlandish stuff, but those kernels can be refined into something feasible). I’ve also used personas like “a grumpy old professor” or “an excited teenager” depending on the tone I want. It’s not just a gimmick; by changing the voice, the AI taps into different styles and content. One writer noted that giving ChatGPT an unusual persona led to “a million times better” results than trying to force it out of cliché in the default voice. I’ve experienced the same. When ChatGPT pretends to be, say, a “radical feminist futurist editor” or a mischievous goblin (yes, I tried Pumuckl!), the ideas that come out have a fresh flavor that I can then translate back to my normal context. Personas also help in role-playing scenarios — for instance, I’ll often tell the AI to act as a skeptical client and have it ask me questions about my idea. This surfaces critiques I hadn’t considered, which in turn sparks new ideas to address them. The key with personas is to be creative and not afraid to dial it up or down. As Katti Jisuk Seo advises, experiment — “intensify their edginess or craziness” and see what happens. The results might surprise you.

Beyond these, there are other prompting tricks (like asking the AI to ask you questions to clarify the problem, using counterfactual prompts like “Imagine the opposite of this idea is true, what then?”, or role simulation where you have a dialogue with multiple AI personas). But the three above — chaining, analogies, personas — cover a huge part of the value I get from AI in brainstorming.

One more tip: don’t hesitate to explicitly instruct the AI on the creative process. It might sound funny, but you can prompt ChatGPT with your method. For example: “Let’s use an iterative approach. Give me 5 rough ideas first. Then we’ll pick one and refine. Ready?” The AI will happily oblige and even follow the plan diligently. It doesn’t get bored or impatient, which is wonderful. I’ve even had success telling the AI to generate ideas in a certain format — like a table of wild, moderate, and safe ideas — to ensure a spread of creativity. Remember, the AI can only work with what you tell it. Guide it like you’re the facilitator of this brainstorming session.

A Framework for AI-Augmented Ideation: The Double Diamond (Supercharged)

If all these techniques feel a bit scattered, it helps to have an overarching process to slot them into. One framework I love (borrowed from design thinking) is the Double Diamond model. Traditionally, the Double Diamond has two phases: first diverge (explore the problem widely) then converge (define the core problem), and again diverge (ideate solutions broadly) then converge (refine the best solution). It’s essentially structured creativity: explore, then hone, then explore, then hone.

How does AI fit in? I’ve adapted this model when working with tools like ChatGPT. The idea is to use AI heavily in the divergent phases and use human judgment more in the convergent phases. Here’s how an AI-augmented Double Diamond might look:

In an AI-augmented approach, you can leverage AI during the divergent thinking phases (expanding problem understanding and idea options), and then apply human insight during convergent phases (narrowing down and validating).

• Diamond 1 — Divergent Thinking (Problem Space): Start by broadening your understanding of the challenge with AI’s help. I use ChatGPT to discover insights: ask it for a quick research roundup, or “What are 5 different ways people struggle with [the problem]?” or “List unconventional examples of how this problem is solved in other fields.” This is analogous to the Discover phase. The AI can surface hidden pain points or reframe the issue in ways you hadn’t considered. Next, as you converge to define the problem, you take the lead. Synthesize the AI’s findings and decide which problem angle is worth focusing on. AI can assist here too (e.g., “summarize these findings into a problem statement”), but defining the real problem — that’s a human strength. As one design expert noted, the goal is not what’s technically possible but “what’s humanly valuable”. Use AI as a research assistant, but trust your instincts on what truly matters.
• Diamond 2 — Divergent Thinking (Solution Space): Now for idea generation, the fun part. This is where AI shines as a brainstorming partner. In the Develop/Ideate phase, unleash those prompting techniques: have ChatGPT spit out a hundred rough ideas for a solution. Encourage wild ideas (high temperature, “no idea is too crazy” prompts). You can generate solutions in different categories — maybe ask for some quick wins, some moonshot ideas, some borrowing from nature (analogies again). The AI’s breadth here is fantastic; it’s like having an infinite idea whiteboard. Don’t worry if many ideas are impractical — that’s expected. Once you have a rich pool, enter converge mode again: Deliver/Refine. Here, I switch hats to evaluator. I might use AI to help evaluate (e.g., ask it to sort ideas by novelty or feasibility, or do a pros/cons list for a few favorites), but crucially I apply my criteria to pick a direction. Often I’ll then use AI one more time to develop the chosen idea further — for instance, “Let’s take idea X and draft a concept note” or “Simulate a user trying this solution and see what feedback they’d have.” This iterative honing is where human creativity and AI’s generative power really collaborate. We use AI’s output as raw material and refine it with human judgment.

Sometimes I even add a third mini-diamond at the end, as some futurists suggest, where after delivering a solution, you keep an AI in the loop to evolve the idea with real-world feedback (imagine an AI monitoring how an idea performs and suggesting tweaks). But that’s more relevant in product development. For most brainstorming purposes, two diamonds — problem and solution — suffice.

The key principle is alternation: let the AI diverge, then you converge. AI can generate options, you decide and direct the next step. Used this way, you avoid the two extremes — you’re not relying solely on AI (which could lead to lots of so-so ideas with no direction), and you’re not ideating alone (which could be slower or limited by your own biases). Instead, you and the AI form a complementary pair. One researcher described it nicely: AI tools become “springboards for human creativity, not replacements for it.” They help you jump higher and see more possibilities, but you’re still the one doing the purposeful dive.

In my practice, having this structured loop also prevents me from getting overwhelmed. It’s easy to ask an AI for ideas and get flooded with 100 options and then feel stuck on what to do next. By thinking in terms of diverge–converge, I remind myself: Step 1 — Generate freely (don’t judge yet, just explore with AI), Step 2 — Pick or synthesize (use human judgment), Step 3 — Generate again on the refined path (use AI to flesh out or even find the next layer of ideas), and Step 4 — Finalize and validate (human-led). It’s a flexible framework; sometimes I loop a few times if the problem is really tricky, other times one pass is enough. The “double diamond” is just a visual way to remember to balance quantity with quality, expansion with focus. With AI, leaning into the expansion is easy — the challenge is making sure you follow through with the focusing and selection. Structure helps.

Examples: AI Brainstorming in Action

Let’s bring this to life with a few concrete scenarios. These examples show how a professional might collaborate with AI in different creative tasks — marketing copy, product naming, and service innovation — using the techniques and process we’ve discussed.

• Example 1: Marketing Campaign Ideas — Scenario: You’re a marketer tasked with coming up with a social media campaign for a new line of eco-friendly sneakers. How AI helps: Start by prompt chaining. You ask ChatGPT, “Give me 10 edgy social media campaign ideas for eco-friendly sneakers targeted at Gen Z.” It returns a list — say it includes ideas like a TikTok dance challenge with recycling bins, a “sneaker take-back for discount” program, etc. Some are meh, a couple have potential. You then use analogical prompting: “Imagine how a popular climate activist or a street artist might promote these shoes — give me 5 more ideas incorporating those perspectives.” Now the AI comes up with art-installation concepts like a giant shoe made of recycled plastic appearing in city squares, or a challenge where people post creative upcycling of old shoes. Next, you switch to persona prompting: “You are a snarky social media influencer who secretly loves the environment. Propose a campaign idea that would get your followers engaged.” This yields a fun idea for a parody video series (“Sneaker Confessions”) where influencers humorously admit their past wasteful habits and show off the new eco-sneakers as redemption. Finally, you converge: you pick your favorite concept — let’s say the city art installation — and ask the AI to flesh it out: “Write a pitch outline for the recycled-plastic shoe installation campaign, including how we’d involve the community and promote it online.” Now you’ve got a draft concept you can refine and present. In this workflow, AI served as a tireless ideation machine, but you guided it at each step, injecting new angles and filtering the outputs.
• Example 2: Product Naming — Scenario: You need a catchy name for a new smart home device that controls lighting and temperature automatically. How AI helps: Begin with a broad ask: “Give me 20 name ideas for a smart home device that manages lights and temperature (focus on it being easy and intelligent).” As expected, ChatGPT churns out a lot of “LightTemp Pro, SmartComfort, HomeEase, LumiTherm” type combinations. Not thrilling. Time to get creative: “Okay, now give me names that are mash-ups or metaphors — for example, think of mythology (gods of light or weather), or words implying harmony.” The AI now suggests names like “Helios Home”, “AuraSync”, “HarmonyHub”. Better, but let’s push it further with analogies: “What if this device was like a personal butler for your home environment? Give me some name ideas playing on the concept of a smart assistant or butler.” Out come names like “CasaConcierge”, “EcoButler”, “AtmoAttendant”. One of those sparks an idea in your head — “What about something with ‘Maestro’, like orchestrating the home?” You ask AI for variations: “Combine the concept of a conductor or maestro with home comfort in a name.” It returns “Comfort Maestro, ClimateConductor, HomeOrchestra.” Aha! “HomeOrchestra” sounds novel. You double-check with AI: “Does any company or product already use the name ‘HomeOrchestra’ for a device? (If yes, suggest a tweak.)” It reports back nothing obvious (though you’d later do a real trademark search!). You’ve arrived at a pretty cool name through iterative prompting, metaphorical thinking, and the AI’s endless thesaurus brain.
• Example 3: Service Innovation (Analogical Thinking) — Scenario: You run a chain of cafes and want to create a new service experience that stands out. Perhaps something to do with how customers can customize drinks or engage with the cafe community. How AI helps: You decide to break the mold by using analogies. You prompt: “Our cafe is like a library of flavors. Using that analogy (a library), suggest innovative services or features for our cafe.” The AI suggests a “flavor check-out” system where regulars can “borrow” a unique secret menu item each week, or a quiet corner called the “Reading Nook” where you can enjoy coffee with provided books — blending the library vibe. You then try a different angle: “Think of our cafe as a social network in physical form. What features would that entail?” Now it comes up with ideas like a community board where people post conversation topics or skill-sharing sessions (like an in-person subreddit for the cafe). You like the community idea. You use persona prompting to refine it: “Imagine I’m an outgoing regular customer. What kind of community event or feature at the cafe would I love, that also encourages me to bring friends?” The AI (as the enthusiastic customer) suggests a weekly “Coffee and Crafts” meetup where people do DIY projects together, or a system where you get a free drink for bringing a first-time visitor (gamifying word-of-mouth). You settle on the concept of a “Bring a Friend Friday” plus the community board idea. To develop it, you ask the AI to outline how that might work operationally and marketing-wise. At the end, you’ve got a novel service concept — all sparked by reframing the cafe through different analogies via AI.

In each of these examples, notice the pattern: prompt -> AI outputs -> your judgment/next prompt -> AI refine -> … -> final idea. It’s a back-and-forth dance. The human (you) provided the strategic direction (target audience, analogy source, persona context, selection of final idea), and the AI provided the fast expansion and even some incremental improvements. This collaboration can significantly cut down the time to a solid creative concept, and often yields something more interesting than either party might have come up with alone. In marketing scenarios especially, it’s becoming routine for professionals to use AI in exactly this way: ask for a bunch of options, cherry-pick and polish. As one marketing friend told me, “It’s like brainstorming with an intern who has read everything but needs you to tell them what’s a good idea or not.”

Risks and Limitations: Keeping Creativity Responsible

Working with AI in the creative process is powerful, but it’s not without its pitfalls. Based on both research and my own stumbles, here are some crucial caveats to keep in mind:

• Hallucinations and False Facts: AI doesn’t know truth — it knows text. This means if your brainstorming touches on factual matters (e.g., “what if we base a campaign on this historical event?”), ChatGPT might confidently fabricate details. In creative ideation, I’ve had it propose technologies that don’t exist or reference “trends” that were made up. Always double-check any factual claims or assumptions the AI slips into an idea. If it suggests partnering your brand with a popular meme or trend, verify that trend is real and appropriate. When it generates an analogy or story, fact-check it if you plan to use it. Essentially, use AI for imagination, not verification. One strategy: explicitly ask it to separate facts from fiction (“List any assumptions you made that might not be true”) — sometimes it will reveal where it’s unsure. But ultimately the responsibility is on us to vet the outputs.
• Over-reliance and Deskilling: This is a subtle long-term risk. When I first got into AI-assisted creativity, I started letting ChatGPT do the heavy lifting on brainstorming. Over time I realized I wasn’t flexing my own creative muscles as much. This phenomenon has been noted by creators: if you always turn to AI first, you might feel your own ability to ideate “from scratch” weakening. It’s like always riding a bike with training wheels — you might struggle to balance when they’re off. The remedy is to use AI as a support, not a crutch. Sometimes I force myself to brainstorm a bit on paper or with colleagues before consulting the AI, so I stay in shape. Or I’ll only use the AI after I have a few initial ideas, to build on them. Another approach is to treat AI suggestions as a benchmark to beat: challenge yourself to come up with something even the AI couldn’t. The idea is to keep your intuitive, human creativity in the loop. AI is tireless and can spoon-feed you ideas endlessly, but creativity is also about the struggle and the blank-page moment. Don’t lose that habit entirely. Use AI to enhance your skills, not erode them.

Despite this list of warnings, don’t be scared off — the point is to use AI with eyes open. I liken it to driving a powerful car: you want to know where the brakes are and understand the road hazards. Once you do, you can really get moving. Many of these risks are mitigated by the simplest rule: keep a human in the loop at all critical points. You are the editor, the fact-checker, the ethical compass, and the creative director of this collaboration.

Conclusion: Principles for AI-Augmented Creativity

After plenty of experiments, successes, facepalms, and “wow” moments, I’ll wrap up with some guiding principles that have emerged for me. Think of these as the manifesto for using AI as a brainstorming partner responsibly and effectively:

1. Always be the “pilot,” never just the passenger. Approach AI as a co-pilot at most — you steer the session. Set the problem, adjust the course with prompts, and decide on the destination. The AI can fly the plane on autopilot for a stretch (generate ideas), but you must take control for takeoff and landing. In practice: start with a clear goal for your brainstorming, and end by choosing and polishing the idea yourself.
2. Craft your prompts with intent. A well-crafted prompt is to AI what a good question is to a colleague. Be specific about what you want (“10 creative names related to travel”), and if you don’t get good answers, tweak the prompt rather than blaming the AI. Include creative constraints or styles in your prompt to get more interesting outputs. Remember, garbage in, garbage out — and conversely, imaginative prompt in, imaginative output out.
3. Iterate, iterate, iterate. Don’t stop at the first answer. Treat each AI output as a draft. Use follow-up prompts to dig deeper: “Tell me more about idea #2”, “Give me a crazier version of #5”, “Combine #3 and #7 into one concept”. The iterative process is where the real magic often happens — initial ideas get better, or two mediocre ideas merge into one great one. Creativity is an iterative process; it’s no different with AI in the loop.
4. Diversify your inputs for diverse outputs. If you want variety, inject variety. Use analogies from different fields, switch up personas, ask for multiple styles. Don’t just rephrase the same prompt ten times and expect radically different results — change the perspective or approach to jolt the AI into new territory. Similarly, consider using more than one AI model or tool if available; one might surprise you in ways another doesn’t.

• 5. Keep your critical thinking hat on. No matter how fluent or convincing the AI is, evaluate everything. Check feasibility: is this idea actually doable or just cool-sounding? Check originality: have I seen this somewhere before? Check alignment: does it fit the brand/goal/users? Use AI as a sounding board, but you are the creative director who says “this works, that doesn’t.” As the saying goes, “Trust, but verify” — though with AI it’s often “Be intrigued, but verify.”
• 6. Augment your creativity, don’t offload it. Use AI to extend your reach, not replace your imagination. If an AI idea intrigues you, ask yourself “How can I build on this?” rather than just taking it wholesale. Keep practicing your own brainstorming skills in parallel. Maybe use AI to overcome writer’s block, but then turn it off and develop the best idea further on your own. The goal is a symbiosis: the AI expands what you could do alone, and you push the AI beyond its automated bounds.
• 7. Respect ethical and legal boundaries. Don’t let the excitement of an AI-generated idea make you forget professional responsibility. If an idea touches on sensitive cultural, social, or personal areas, apply the same ethical lens as you would to any idea — perhaps even more, since AI might not realize something is offensive or insensitive. Give credit (at least internally) to the fact that AI contributed, especially if working in a team — be transparent that “This tagline came from a ChatGPT session, and then we refined it.” And ensure any outputs used are within your rights to use. When in doubt, consult a legal or ethics expert.

In essence, treat AI as a creative partner that needs guidance, oversight, and inspiration — much like a junior team member with infinite energy but zero common sense. You provide the common sense, and in return, you get that superhuman breadth and persistence. When I follow these principles, I find my brainstorming sessions are more productive, and frankly, more fun. There’s a real joy in bouncing ideas off an AI — it doesn’t judge, it never tires, and sometimes it pulls a gem out of left field that makes you laugh or think “where did that come from?”

To conclude, I’ll echo a reassuring finding from researchers: people who used AI for brainstorming still needed their own creativity to make something of the results. AI didn’t obviate the human spark — if anything, it amplified it when used thoughtfully. My best ideas lately have often been half mine, half AI’s — I would not have gotten there alone, nor would the AI. In a world where these tools are only going to get more powerful, the creatives who thrive will be those who learn to dance with the AI: leading when it matters, following when it helps, and always staying in rhythm with their own creative intuition. Happy brainstorming, and don’t forget to ask, “What if…?” — whether to yourself or to your trusty AI sidekick.

Generative AI is a force multiplier for Agile, not its replacement. By drastically accelerating development and shortening feedback loops, AI makes Agile principles more critical than ever. However, to harness this power, teams must evolve their Scrum practices. This means moving beyond rigid, fixed-sprint models to hybrid workflows that accommodate AI’s experimental nature. It requires shifting focus from output-based metrics like velocity to outcome-focused measures like cycle time and lead time, where learning itself is a valuable deliverable. We must learn to treat AI as a new type of team member — a “cybernetic teammate” — that augments every role, elevating humans from tactical execution to strategic oversight. Finally, we need to abandon a one-size-fits-all approach to estimation. This guide proposes a new, three-tiered framework for estimating work: “Zero-Point” stories for fully automated tasks, “Standard” stories for human-led work, and a new category, the “Review & Integration” (R&I) story, to specifically account for the human effort of prompting, validating, and integrating AI-generated work.

Introduction: The AI-Accelerated Feedback Loop

The core principle of Agile development is the feedback loop: building, testing, learning, and iterating as quickly as possible. Generative AI is fundamentally reshaping this loop. By automating the creation of code, tests, and documentation, AI shortens development cycles from weeks to days or even hours. This creates a powerful synergy with Agile’s core values, making its principles of adaptability and rapid response more relevant than ever. As Ken Ringdahl, CTO at Emburse, states, Generative AI is “the greatest force multiplier in Agile history”.

This acceleration, however, introduces a new challenge. Simply doing the same things faster can lead to chaos, not progress. The central question is not if we should be Agile, but how we must adapt the specific practices of a framework like Scrum, which was designed for a pre-AI world. We must avoid the trap of simply “doing Agile” faster and instead evolve our practices to be truly AI-native.

The profound implication of this accelerated feedback loop is that it dramatically lowers the cost of experimentation. When the time and effort required to test a new idea in production is minimal, teams can validate more hypotheses, pivot more quickly based on real user data, and innovate at an unprecedented rate. The challenge for Scrum is to create a structure that encourages this hyper-iterative capability rather than constraining it.

Part 1: Rethinking the Pillars of Scrum for an AI-Native World

To thrive in the AI era, teams must re-examine some of Scrum’s most foundational practices. The universal applicability of fixed-length sprints and the reliance on metrics like velocity are being challenged by the unique nature of AI-driven development.

1.1 From Fixed Sprints to Fluid Workflows: The Case for Hybrid Models

A common failure pattern is to treat AI projects like any other software project. The Pienso article “Unlearning with Poise” argues that fixed sprint planning is ill-suited for the unpredictable, experimental nature of AI development. AI work often involves research, data collection, and model training, which do not align neatly with fixed two-week iterations.

The recommended solution is to adopt a hybrid model that combines the strengths of different Agile approaches.

• For exploratory and research-heavy AI tasks, where timelines are fluid and discovery is the goal, a Kanban-style continuous flow is more appropriate. This allows the team to pull work as capacity allows without the pressure of a fixed deadline.
• For well-defined engineering work, such as building a user interface around a validated AI model, traditional Sprints can still be highly effective.

This hybrid approach allows the team to match the process to the nature of the work, embracing flexibility where it’s needed and structure where it’s beneficial.

1.2 Beyond Velocity: New Metrics for the AI Era

When asked if teams should simply aim to increase their velocity with AI, the answer is a firm no. While AI assistants can boost developer productivity by 20–55%, velocity is a flawed metric for this new paradigm. It measures output, not outcome, and can create perverse incentives to prioritize speed over quality, leading to increased technical debt and poorly planned features.

High-performing teams are shifting their focus to metrics that measure the flow of value. The key metrics for the AI era are:

• Cycle Time: The time from when work starts on a task to when it is completed. This is a pure measure of team efficiency. Thoughtworks reports that clients adopting AI-first engineering have reduced cycle times by up to 50%.
• Lead Time: The time from when an idea is conceived to when it delivers value in production. This measures the overall agility of the entire organization, from idea to impact.

Furthermore, for experimental AI work, the primary goal is not always a shippable feature. The Pienso article argues that sprint goals should be defined around learning milestones. A “successful” sprint might be one that definitively proves a hypothesis is wrong, saving the organization months of wasted effort on a dead-end idea. In this context, the insight itself is the deliverable.

This shift has a profound impact on Scrum events. The Sprint Review, for example, evolves from a simple “demo” of working software into a forum for sharing validated learnings. The team might present a “failed” experiment, explain the crucial insights gained, and collaborate with stakeholders on the next strategic pivot. This makes the review far more valuable, providing a transparent look into the discovery process, not just the delivery pipeline, and perfectly aligning with Scrum’s pillars of transparency, inspection, and adaptation.

Part 2: The Evolving Scrum Team: Meet Your New AI Teammate

The most significant shift in Agile development is reframing AI from a passive tool into an active collaborator. Experts from Gartner, Thoughtworks, and Scrum.org agree that AI augments human expertise rather than replacing it. It automates repetitive work, freeing humans to focus on creativity, critical thinking, and complex problem-solving.

2.1 AI as a Force Multiplier, Not a Replacement

The most effective way to think about this new dynamic is to treat AI as a new kind of team member. Research cited by Scrum.org refers to AI as a “cybernetic teammate” that can fulfill some of the social and motivational roles of a human collaborator. It can act as a pair-programming partner, a source of novel ideas, and an tireless assistant for administrative tasks. Adopting the mental model of treating AI as a junior teammate — guiding it, critiquing its output, and iterating based on its performance — is a powerful strategy for success.

2.2 The Augmented Product Owner: From Backlog Manager to Visionary

For Product Owners (POs), the AI teammate is a powerful research analyst. AI tools can analyze vast amounts of market data, competitor intelligence, and customer feedback to spot unmet needs and emerging trends far faster than any human. It can also automate the creation of artifacts, taking raw inputs like interview transcripts and generating draft user stories and acceptance criteria. This automation frees the PO from the administrative burden of backlog management and allows them to focus on higher-value activities: stakeholder communication, vision setting, and long-term product strategy.

2.3 The Strategic Scrum Master: From Facilitator to Team Coach

The AI teammate also transforms the role of the Scrum Master (SM). AI can handle a significant portion of the SM’s administrative load, such as summarizing meeting notes, generating reports, tracking metrics, and scheduling events. More importantly, AI can provide data-driven coaching insights. By analyzing sprint data, it can identify process bottlenecks, predict risks, and even analyze team communication for sentiment. This provides the SM with objective data to guide retrospectives, shifting conversations from subjective feelings to evidence-based improvement. The SM’s role is elevated from a process referee to a strategic coach focused on the complex human elements that AI cannot handle: building psychological safety, resolving conflict, and mentoring the team.

2.4 The Developer as AI Collaborator: From Coder to Architect

For developers, AI coding assistants eliminate much of the mundane work of writing boilerplate code, unit tests, and documentation, with reported productivity gains of 20–50% for these tasks. This shift necessitates a new skillset. Gartner predicts that 80% of engineers will need to upskill by 2027. The critical skills of the future are:

• Systems Architecture: Designing robust, modular systems that provide clear boundaries for AI to work within.
• Prompt Engineering: The craft of skillfully guiding the AI to produce high-quality, accurate output.
• Critical Review: The non-negotiable discipline of meticulously validating all AI-generated code for quality, security, and correctness.

The rise of AI-native task management tools like Linear, where a product manager might directly assign a task to an AI, does not disintermediate the developer. Instead, it makes the developer’s role as a reviewer and integrator more critical. The developer must be involved before a prompt is finalized to ensure it is architecturally sound, and after the code is generated to validate its quality and security. This work shifts from typing code to defining the problem and verifying the solution — a higher-level, more architectural function.

Part 3: The Story Point Dilemma: A New Framework for Estimating AI-Driven Work

Traditional story points, which measure a combination of effort, complexity, and uncertainty, break down in the AI era. When the “effort” of code generation approaches zero, but the complexity and uncertainty of getting it right remain high, a new approach is needed. This directly addresses the challenge of how to estimate tasks where an AI does the initial work but a human must handle review, integration, and potential rework.

The solution is to stop trying to fit all work into one category. Teams should adopt a three-tiered framework that classifies work based on the nature of the effort involved.

3.1 Tier 1: The “Zero-Point” Story (Fully Automated Tasks)

The Agile community has long debated zero-point stories, with many arguing that no task is truly “free” due to overhead like testing and deployment. However, with mature DevSecOps pipelines and AI automation, some tasks can now be completed with negligible human effort.

A Zero-Point Story should be narrowly defined as a task that is fully automated from trigger to completion, requires no manual review, and whose quality is guaranteed by the automated pipeline itself. An example would be automatically generating and publishing API documentation from code comments as part of a trusted CI/CD process.

3.2 Tier 2: The “Review & Integration” (R&I) Story (AI-Assisted Tasks)

This is a new story type designed to estimate the human effort involved in collaborating with an AI. It does not measure the AI’s work; it measures the work of the developer acting as a guide, validator, and integrator. An R&I story bundles several human-centric activities:

• Prompt Engineering: The effort to craft, test, and refine the prompts given to the AI. This is a skilled activity.28
• Critical Review: The work of scrutinizing the AI’s output for correctness, security vulnerabilities, performance, and adherence to standards. This is often the most time-consuming part.
• Integration & Refactoring: The effort to integrate the generated code into the existing codebase.
• QA & Rework: The effort of testing the integrated code and handling the back-and-forth “rally” when the AI’s output is flawed or incomplete.

Teams can estimate R&I stories using familiar techniques like Planning Poker. However, the estimation is based on new factors: the complexity of the prompt, the risk of AI error, the effort required for a thorough review, and the complexity of integration.

3.3 Tier 3: The “Standard” User Story (Human-Led Tasks)

This category is for traditional user stories that represent work that is primarily creative, strategic, or requires deep human context that AI cannot provide. Examples include designing a novel system architecture, facilitating a difficult stakeholder negotiation, or conducting deep qualitative user research. These tasks are estimated using standard story point practices, as they represent purely human effort, complexity, and uncertainty.

Part 4: A Practical Playbook for AI-Native Scrum

Adopting these new practices can be done incrementally. The following steps provide a practical path for teams to begin their evolution.

4.1 Step 1: Start Small, Focus on High-Value Automation

Do not attempt to transform your entire workflow overnight. Begin by identifying repetitive, low-risk, and time-consuming tasks and apply AI to them first. Good starting points include automating test data generation, creating draft documentation from code, summarizing meeting transcripts, and generating first drafts of unit tests. The goal is to build confidence and demonstrate value quickly.

4.2 Step 2: Master Structured Prompt Engineering

Effective interaction with AI is a discipline, not a casual conversation. Vague prompts lead to vague results. Teams should treat prompt engineering as a core competency. Structured prompting techniques, such as the “Role, Task, Context, Expectation” framework, provide a solid foundation for clear communication with AI. Teams should build a shared library of effective prompts for common tasks and refine them continuously.

4.3 Step 3: Reinforce Quality and Security with a Human-in-the-Loop

The ease of generating code with AI can lead to a decline in quality if not managed vigilantly. Human oversight is the primary defense. Use AI to enhance quality practices, not bypass them. AI can help generate more comprehensive test cases and scan for vulnerabilities, but the final judgment on quality rests with the human team. It is also crucial to establish clear ethical and security guardrails. Teams must implement policies that prevent sensitive or proprietary data from being fed into public AI models without explicit safeguards, and legal and security teams should be involved in this process.5

Conclusion: The Future is Human-Augmented Agility

The integration of Generative AI does not signal the end of Agile; it amplifies its core principles. The goal is not to automate humans out of the development process but to create highly effective, AI-augmented teams that fuse human creativity and strategic thinking with the speed and analytical power of AI. The future of software development belongs to those who embrace this synergy.

The practices outlined here are a starting point, not a final destination. The technologies and our understanding of them are evolving rapidly. The teams that will succeed in this new era are those that embody the Agile spirit: a willingness to inspect their own processes, adapt to new realities, and continuously learn and “unlearn” old habits to discover better ways of working.

TL;DR

Evaluating Large Language Models (LLMs) and the sophisticated systems built upon them, such as Retrieval Augmented Generation (RAG) and AI Agents, presents a significant yet indispensable challenge. The open-ended and diverse nature of LLM outputs makes traditional, straightforward right-or-wrong assessments difficult. Consequently, the field is rapidly moving beyond static benchmarks towards more dynamic, holistic, and human-centric evaluation methodologies. This evolution is crucial because robust evaluation not only drives technological progress but also ensures the reliability and trustworthiness of AI systems. Key approaches include established benchmarks like MMLU and HellaSwag for core capabilities, human-in-the-loop methods such as Chatbot Arena and the LLM-as-a-judge paradigm for nuanced assessments, and specialized frameworks like RAGAs for RAG systems and AgentBench for AI agents. Furthermore, the evaluation landscape extends to multimodal AI, with specific metrics like FID and CLIPScore for image generation, and FVD and JEDi for video generation, each addressing unique aspects of quality and alignment.

The Labyrinth of LLM Evaluation: Why It’s Hard and Why It Matters

The task of evaluating Large Language Models (LLMs) is intricate, largely due to the inherent characteristics of these generative models and the multifaceted nature of their outputs. Understanding these difficulties is the first step towards developing more effective and meaningful evaluation strategies.

The Challenge of Evaluating Generative Models

A primary challenge in LLM evaluation stems from the models’ complexity and the open-ended nature of the text they generate. Unlike traditional machine learning models, such as those used for classification tasks which have discrete, verifiable answers, LLMs can produce a vast array of responses to a single prompt. Many of these responses might be valid, creative, or stylistically different, yet still appropriate. This makes it difficult to apply simple accuracy metrics. For instance, a request to summarize a document can yield multiple correct summaries, each emphasizing slightly different aspects or using different phrasing.

This open-endedness also brings forth a tension between evaluating creativity and ensuring factuality. In some applications, like creative writing assistance, novel and imaginative outputs are highly desirable. In others, such as question-answering systems providing medical or financial information, factual accuracy and adherence to established knowledge are paramount. Defining what constitutes a “good” output thus becomes context-dependent, complicating the design of universal evaluation metrics. The sheer volume of potential outputs for any given input also means that comprehensively testing all possible scenarios is practically impossible.

Human judgment, often considered a gold standard for assessing nuanced qualities like coherence or helpfulness, also presents its own set of problems. Human evaluation can be subjective, prone to bias, expensive to conduct at scale, and different evaluators may hold contrasting opinions about the same LLM output. This inherent variability and cost in human assessment drive the need for automated metrics that can, at least partially, capture these complex qualities. The difficulty in LLM evaluation is, in many ways, a direct consequence of their success in generating human-like, open-ended text. The more versatile and human-like these models become, the more challenging it is to assess their performance with simple, predefined metrics.

The Critical Importance of Robust Evaluation

Despite the challenges, robust and comprehensive evaluation is fundamental to the advancement and responsible deployment of LLM technology. Evaluation serves as a critical feedback mechanism in the development lifecycle. It allows researchers and developers to track improvements as they iterate on model architectures, training data, prompts, or system parameters, and to detect regressions before they negatively impact users.3 Without effective evaluation, progress can be haphazard and misdirected.

Moreover, reliable evaluation is essential for building trust among users and stakeholders. For LLMs to be widely adopted and integrated into critical applications, there must be confidence in their capabilities and limitations. This confidence is built upon transparent and rigorous assessment processes that demonstrate the model’s performance, safety, and reliability. As LLM applications become more prevalent, ensuring they perform as expected and deliver consistent, reliable outcomes is paramount.

Evaluation also plays a crucial role in ensuring the safety and ethical deployment of LLMs. This includes assessing models for biases learned from training data, their propensity to generate misinformation or harmful content, and their adherence to safety guidelines. For LLM applications in production, outputs need to be factually accurate, align with organizational brand voice and safety policies, and remain within the intended scope. The importance of evaluation, therefore, extends beyond mere technical performance; it encompasses the societal and ethical implications of deploying powerful AI systems.

Brief Overview of the Evolving Landscape

The field of LLM evaluation is not static; it is continuously evolving in response to the rapid advancements in LLM capabilities. Early benchmarks, while foundational, are increasingly being supplemented or replaced by more sophisticated methods designed to capture the nuances of modern LLMs. There is a discernible trend towards evaluations that are more contextual, task-oriented, and incorporate human feedback more directly, moving beyond purely automated metrics to assess how models perform on user-specific tasks and in real-world scenarios. This dynamic landscape reflects an ongoing effort to develop evaluation techniques that keep pace with the models themselves.

Evaluating Core LLM Capabilities: Foundational Benchmarks

To systematically assess the diverse capabilities of LLMs, a variety of benchmarks have been developed. These standardized tests aim to measure specific skills, from general knowledge and reasoning to truthfulness and coding proficiency. While not without limitations, they provide a common ground for comparing different models.

Knowledge and Reasoning

A significant aspect of LLM performance is their ability to understand and reason about a wide range of topics.

MMLU (Massive Multitask Language Understanding)

MMLU is designed to measure an LLM’s general knowledge and problem-solving abilities across an extensive set of subjects. It comprises multiple-choice questions spanning 57 diverse fields, including STEM (e.g., mathematics, physics, computer science, engineering), humanities (e.g., history, philosophy, literature), social sciences (e.g., psychology, economics, politics), and professional disciplines (e.g., law, medicine, accounting). The questions range in difficulty from elementary to expert levels.

Examples of MMLU tasks include:

• STEM (Mathematics): “If a group G has order 12, what is the largest possible order of an element in G?”.
• Social Sciences (Microeconomics): “One of the reasons that the government discourages and regulates monopolies is that. (A) producer surplus is lost and consumer surplus is gained. (B) monopoly prices ensure productive efficiency but cost society allocative efficiency. (C) monopoly firms do not engage in significant research and development. (D) consumer surplus is lost with higher prices and lower levels of output.”.
• Professional (Medicine): A clinical scenario question, such as: “A 33-year-old man undergoes a radical thyroidectomy for thyroid cancer. During the operation, moderate hemorrhaging requires ligation of several vessels in the left side of the neck. Postoperatively, serum studies show a calcium concentration of 7.5 mg/dL, albumin concentration of 4 g/dL, and parathyroid hormone concentration of 200 pg/mL. Damage to which of the following vessels caused the findings in this patient?” (with multiple choice options).

Scoring on MMLU is typically based on the percentage of correct answers. Evaluations are often conducted in zero-shot (model answers without prior examples) and few-shot settings (model receives a small number of example question-answer pairs, commonly 5-shot). Top-performing models like GPT-4 have achieved high scores on MMLU, approaching human expert levels. MMLU scores are frequently reported on leaderboards like Chatbot Arena. However, MMLU is not without its limitations. Studies have revealed data quality issues in certain sub-tasks, such as a significant percentage of questions in the Virology subset containing errors or incorrect ground truth labels. There can also be uneven representation of subjects, potentially leading to domain bias, and the static nature of the benchmark means the knowledge tested can become outdated. Furthermore, different implementations of the MMLU evaluation harness can lead to variations in scores for the same model. The prominence of MMLU means it is often a target for model optimization, which raises concerns about whether improvements in MMLU scores always translate to genuine, broad improvements in understanding or if models are, to some extent, being “taught to the test.”

HellaSwag (Commonsense Natural Language Inference)

HellaSwag is designed to test an LLM’s commonsense reasoning capabilities through a sentence completion task. The model is presented with an initial context (often a short scenario) and must choose the most plausible continuation from four options. For example:

• Context: “A man is in a kitchen, he is pouring a drink from a bottle into a glass.”
  The model then chooses the most logical next sentence from four choices, such as “He is holding the bottle with his left hand and the glass with his right hand”.
• Another example: “Men are standing in a large green field playing lacrosse. People is around the field watching the game. men…” followed by choices like “are holding tshirts watching int lacrosse playing” or “are running side to side of the ield playing lacrosse trying to score”.

Scoring is based on the proportion of correctly chosen endings. While HellaSwag questions are designed to be trivial for humans (over 95% accuracy), they pose a significant challenge for LLMs. However, some analyses have pointed out that a portion of HellaSwag examples contain grammatical errors or nonsensical options, which might mean the benchmark sometimes tests a model’s tolerance for flawed language rather than pure commonsense inference. These issues highlight that even “gold standard” benchmark data can have imperfections, complicating the interpretation of model performance.

Other Reasoning/Understanding Benchmarks

Other notable benchmarks in this category include SuperGLUE, an improved and more challenging version of the original GLUE benchmark designed to measure general language understanding across various tasks like inference and question answering.
WinoGrande, based on the Winograd Schema Challenge, tests a model’s ability to resolve ambiguities in sentences involving pronoun references, requiring commonsense understanding. The existence and evolution of these benchmarks, from GLUE to SuperGLUE and the development of MMLU, reflect the rapid improvement in LLM capabilities, necessitating progressively harder tasks to differentiate model performance effectively.

Truthfulness and Safety

Ensuring that LLMs provide accurate information and operate safely is a growing concern.

TruthfulQA

TruthfulQA is specifically designed to evaluate an LLM’s truthfulness by testing its ability to avoid generating false answers that stem from common human misconceptions or false beliefs. The questions are crafted to target areas where humans might provide incorrect answers. The evaluation often uses another fine-tuned LLM, termed “GPT-Judge” (based on GPT-3), to assess the truthfulness of the generated answers by classifying them as true or false.

Coding and Specialized Tasks

LLMs are increasingly being used for code generation and other programming-related tasks.

HumanEval

This benchmark focuses on assessing an LLM’s ability to generate functionally correct code. It presents coding challenges, and the generated code is evaluated using a metric called pass@k. This metric measures how many of the k code samples generated by the LLM for a given problem pass a set of unit tests.

CodeXGLUE

CodeXGLUE is another benchmark that covers a range of programming tasks, offering a broader assessment of code-related capabilities.

Limitations of Traditional Benchmarks

While foundational benchmarks are valuable, they have inherent limitations:

• Static Nature: Benchmarks are typically fixed datasets. Over time, models might inadvertently “overfit” to these specific datasets, or the knowledge contained within them can become outdated, especially in rapidly evolving fields.
• Contamination: A significant concern is “data contamination,” where parts of the benchmark dataset might have been included in the LLM’s vast training data. If a model has “seen” the test questions during training, its performance on the benchmark is no longer a valid measure of its generalization ability. The emergence of research focused on “Contamination-Free LLM Benchmarks” indicates the seriousness of this issue.
• Narrow Focus: Many benchmarks test very specific, isolated skills. High performance on such a benchmark may not always translate to strong general capabilities or effective performance in complex, real-world applications.
• Susceptibility to “Goodhart’s Law”: This law states that when a measure becomes a target, it ceases to be a good measure. If the AI community overly focuses on optimizing for specific benchmark scores, models might be developed that excel on those benchmarks without genuinely improving the desired underlying capabilities or even by exploiting quirks in the benchmark itself.

The issues found in benchmarks, such as grammatical errors in HellaSwag examples or factual inaccuracies in MMLU subsets, underscore that benchmark creation and maintenance are challenging endeavors. Evaluating models against flawed ground truths can muddle the interpretation of their true abilities.

Beyond the Model: Evaluating LLM-Powered Systems

While evaluating the core capabilities of an LLM is essential, many real-world applications involve LLMs as part of larger, more complex systems. Evaluating these integrated systems, such as Retrieval Augmented Generation (RAG) pipelines and AI agents, requires specialized approaches that consider the interplay between different components. This marks a shift from assessing “what the LLM knows” to “what the LLM can do with its knowledge and available tools.”

Evaluating Retrieval Augmented Generation (RAG) Systems

RAG systems enhance LLM outputs by first retrieving relevant information from an external knowledge base and then using that information to generate a response.14 This architecture aims to improve factual accuracy and contextual relevance.

The Unique Challenges of RAG Evaluation:

Evaluating RAG systems necessitates assessing both the retriever (how well it finds relevant information) and the generator (how well the LLM uses that information). The final output quality hinges on the synergy between these two components. If the retriever fetches irrelevant or incomplete context, even a perfect generator will struggle. Conversely, if the generator fails to faithfully synthesize relevant retrieved information or introduces hallucinations, the system’s utility diminishes. Thus, evaluation must target both stages.

Key Metrics (often leveraging LLM-as-a-judge):

Several metrics have been developed to assess different facets of RAG performance:

Faithfulness: This measures whether the generated answer is factually consistent with the retrieved context and avoids contradicting it or introducing information not present in the context (hallucinations). For example, RAGAS defines faithfulness as the consistency of the generated response with the retrieved passages, essentially checking if claims in the answer are substantiated by the context. It can be quantified as the fraction of statements in the answer confirmed by the retrieved documents.

Answer Relevance: This assesses if the generated answer directly addresses and is pertinent to the user’s original query. An answer might be faithful to the context but irrelevant to the user’s actual question. RAGAs evaluates this by analyzing the relationship between the query’s intent and the response content. One method involves using an LLM to generate hypothetical questions based on the answer and then measuring their similarity to the original query.

Context Relevance / Context Recall / Context Precision: These metrics evaluate the performance of the retrieval component:

• Context Relevance: Are the individual documents or chunks retrieved by the system actually relevant to the input query?. This helps gauge if the retriever is fetching pertinent information.
• Context Recall: Did the retriever find all, or a sufficient amount, of the truly relevant information available in the knowledge base that is needed to answer the query comprehensively?. This often requires a ground truth set of relevant documents for comparison.
• Context Precision: Of the documents retrieved, what proportion is actually relevant? This helps ensure the LLM isn’t burdened with excessive irrelevant information, which could degrade generation quality.

Frameworks for RAG Evaluation:

Specialized frameworks have emerged to streamline RAG evaluation:

• RAGAs: This framework focuses on component-wise evaluation using the specialized metrics mentioned above (faithfulness, answer relevance, context relevance), often employing LLMs as judges to automate the assessment.
• ARES (Automated RAG Evaluation System): ARES takes a different approach by using synthetic data generation (creating question-context-answer triples) and fine-tuned classifiers to assess context relevance, answer faithfulness, and answer relevance. This aims to minimize the need for extensive human annotation while providing statistically confident evaluations. The use of AI to generate test data for AI evaluation, as seen in ARES, points to a broader trend of leveraging AI capabilities to address the scaling challenges in AI assessment.
• DeepEval: This is another tool that provides implementations for many common RAG evaluation metrics, facilitating their application.

Evaluating AI Agents

AI agents represent a further step in LLM application complexity, involving multi-step reasoning, interaction with external tools via API calls, and operation within dynamic environments to achieve goals.18

The Complexity of Agent Evaluation:

Evaluating AI agents is significantly more challenging than evaluating standalone LLMs or even RAG systems. An agent’s performance depends not just on the final output, but on the entire sequence of decisions, tool invocations, and environmental interactions it undertakes. Agents might call tools in various sequences, invoke sub-agents, or exhibit non-deterministic behavior based on state and memory. Poor long-term reasoning, flawed decision-making, or incorrect instruction-following are major hurdles. A single mistake in a multi-step process, such as choosing the wrong tool or misinterpreting a tool’s output, can derail the entire workflow. This “black box” nature is amplified in agentic systems, as failures can originate from numerous points in the complex chain of operations, making diagnosis and targeted evaluation critical.

Benchmarks for AI Agents:

To address these challenges, dedicated benchmarks for AI agents are being developed:

• AgentBench: This is a comprehensive benchmark designed to evaluate LLMs as agents within interactive environments. It assesses reasoning and decision-making abilities in multi-turn, open-ended generation settings across eight distinct environments: Operating System (OS), Database (DB), Knowledge Graph (KG), Digital Card Game, Lateral Thinking Puzzles, House-Holding (simulated via ALFWorld), Web Shopping (simulated via WebShop), and Web Browsing (simulated via Mind2Web). For example, in the Web Shopping environment, an agent might be tasked with finding a product with specific features, comparing options, and adding it to a virtual shopping cart. In an OS environment, the agent might need to execute commands to manage files or processes based on natural language instructions. AgentBench studies have found that while top commercial LLMs exhibit strong agent capabilities, a significant performance gap exists compared to open-source competitors, with challenges in long-term reasoning and instruction following being key obstacles.
• MLR-Bench: This benchmark focuses on evaluating AI agents specifically on open-ended machine learning research tasks, with tasks sourced from major ML conferences. It includes an automated evaluation framework called “MLR-Judge” to assess research quality. Findings from MLR-Bench indicate that while LLMs can be effective at generating coherent ideas and structuring papers, current coding agents often produce fabricated or invalidated experimental results, posing a barrier to trustworthy scientific discovery support.
• Other Agent Evaluation Frameworks: Other emerging frameworks include C³-Bench, which is designed to test agent robustness by presenting challenges such as complex tool relationships, critical hidden information, and variable decision trajectories.
  WebArena tests agents in realistic web environments on tasks like online shopping and travel booking.

Key Aspects to Evaluate in Agents:

Effective agent evaluation must consider multiple dimensions:

• Task Completion Rate: Did the agent successfully achieve the overall goal specified by the user?.
• Reasoning Quality: Was the agent’s plan logical? Did its intermediate decisions make sense in the context of the task? Metrics like reasoning relevancy (is the reasoning for a tool call tied to the user’s request?) and reasoning coherence (does the reasoning follow a logical process?) are important.
• Tool Selection and Invocation Accuracy: Did the agent choose the appropriate tools for sub-tasks? Were the parameters supplied to the tools correct? Did it correctly interpret and utilize the outputs from the tools?.
• Efficiency: How many steps, API calls, or how much time did the agent require to complete the task?.
• Adaptability and Robustness: How well does the agent handle errors, unexpected feedback from the environment or tools, or changes in the task requirements?.

Human-Centric and Dynamic Evaluation Approaches

While automated benchmarks provide scalability, evaluating aspects like nuanced understanding, conversational quality, and overall helpfulness often requires human judgment. This has led to the development of human-centric and dynamic evaluation methods. These approaches often aim to assess “perceived quality” and “usefulness,” which are crucial for user-facing applications where subjective experience is a key performance indicator.

LLM-as-a-Judge

The “LLM-as-a-Judge” paradigm involves using a powerful, capable LLM (often a proprietary model like GPT-4) to evaluate the outputs of another LLM based on specified criteria. These criteria can include helpfulness, harmlessness, accuracy, coherence, or adherence to a particular style.

Concept and How It Works:

An LLM judge is typically prompted with the input query, the response from the model being evaluated, and a set of evaluation guidelines or a scoring rubric. The judge LLM then provides a score, a classification, or even a textual critique of the response. This method has become a standard practice for approximating human preferences on text quality at scale.

Advantages:

• Scalability: It offers a more scalable and potentially cost-effective way to evaluate qualitative aspects compared to relying solely on human annotators, especially for large volumes of text.
• Approximating Human Preference: LLM judges can often capture nuances in language and context that simpler rule-based or keyword-based metrics might miss, thereby providing evaluations that align more closely with human preferences.
• Versatility: The approach is flexible and can be adapted to evaluate a wide range of tasks and criteria by modifying the prompts and guidelines given to the judge LLM.

Limitations and Challenges:

Despite its advantages, the LLM-as-a-Judge approach has several limitations:

• Bias: The judge LLM itself can exhibit biases present in its training data or arising from its architecture. These can include position bias (favoring the first or second response in a pairwise comparison), verbosity bias (preferring longer answers), or even self-enhancement bias (favoring answers similar to what it would generate).
• Impact of Chain-of-Thought (CoT) Prompting: While CoT prompting is often used to improve reasoning in LLMs, research suggests it can sometimes harm the performance of LLM judges. CoT can lead to sharper, less spread-out score distributions from the judge, making its mean judgment very similar to a simple greedy decoding (mode) and potentially obscuring finer-grained preferences. Removing CoT can sometimes improve performance by allowing for a wider judgment distribution.
• Agreement with Humans: While LLM judges often show strong agreement with human judgments, this agreement is not perfect and can vary depending on the complexity of the task, the clarity of the evaluation criteria, and the specific capabilities of the judge model.
• Vulnerability to Attacks: LLM-as-a-Judge systems can be susceptible to adversarial attacks. For instance, the “JudgeDeceiver” attack demonstrated that carefully crafted sequences injected into a candidate response can trick an LLM judge into selecting that response, regardless of other candidates. This highlights a potential security concern if LLM judges are used in high-stakes evaluation scenarios. The recursive problem also arises: if we use an LLM to evaluate other LLMs, how do we evaluate the evaluator LLM itself? Flaws in the judge could lead to misleading evaluations and misdirected development efforts.

Current Research:

Ongoing research aims to improve the reliability and effectiveness of LLM-as-a-Judge. One promising direction involves leveraging the full probability distribution over judgment tokens provided by the judge LLM, rather than just relying on the single most likely token (greedy decoding). Taking the mean of this distribution has been shown to consistently outperform taking the mode in various evaluation settings. Other research focuses on developing defenses against adversarial attacks and better understanding the biases of LLM judges.

Crowdsourced and Arena-Based Evaluation

Another approach to capturing human preferences at scale is through crowdsourced platforms and competitive arenas.

Chatbot Arena:

Chatbot Arena is a prominent example of this approach. It is a platform where different LLMs compete against each other in anonymous, randomized “battles.” Human users interact with two unnamed models simultaneously for a given prompt and then vote for which model provided the better response, or declare a tie or both bad.

Elo Rating System and Bradley-Terry Model:

Based on the outcomes of these pairwise comparisons, models are ranked using an Elo rating system, similar to how chess players are ranked. The Elo system updates a model’s rating based on whether it wins or loses against other models, taking into account the ratings of its opponents.

More recently, Chatbot Arena has been adopting the Bradley-Terry model to refine its rankings. The Bradley-Terry model is a statistical model specifically designed for pairwise comparison data. In this model, each item (or LLM) is assigned a score, and the probability of one item being preferred over another is a function of the difference in their scores. The Bradley-Terry model can be seen as providing the maximum likelihood estimation (MLE) for an underlying Elo-like model, assuming fixed pairwise win-rates and that game order doesn’t matter, which is suitable for static LLMs evaluated on a complete history of games. The application of these statistical methods from competitive domains to rank AI models underscores the increasingly competitive landscape of LLM development.

How it Captures Human Preference:

Chatbot Arena directly leverages aggregated human judgment on overall response quality in a comparative setting. This allows for a dynamic assessment that reflects real user preferences for aspects like helpfulness, coherence, and engagement.

Advantages:

This method is dynamic, as new models can be added and rankings can change based on ongoing user feedback. It reflects genuine user preferences in a more holistic way than many static benchmarks and is arguably harder to “game” because the evaluation is based on diverse, unpredictable human interactions.

Limitations:

The quality of evaluations can be influenced by the nature of the prompts users provide. User subjectivity is inherent, although aggregated over many votes, strong trends can emerge. Achieving stable and reliable rankings requires a large volume of votes.

Note: Leaderboards are dynamic; this is an example based on provided research. Refer to live leaderboards for current data.

Emerging Trends and the Future of LLM Evaluation

The field of LLM evaluation is characterized by rapid evolution, driven by the increasing capabilities of models and a deeper understanding of their potential impacts. Several key trends are shaping the future of how these complex AI systems are assessed. The overarching direction is towards evaluating entire systems rather than just isolated models, recognizing that LLMs often function as components within larger applications.

• Focus on Robustness, Safety, and Ethical Considerations:
  There is a growing emphasis on evaluating how LLMs perform under non-ideal or even adversarial conditions. This includes assessing their susceptibility to prompt injection attacks, as highlighted by research on vulnerabilities in systems like LLM-as-a-judge, and their propensity to generate biased, harmful, or untruthful content. Future evaluations will likely incorporate more specialized benchmarks and methodologies designed to rigorously test these critical safety and ethical dimensions beyond general capabilities.
• Evaluating Long-Context Understanding and Complex Reasoning:
  As LLMs are developed to process and understand increasingly longer contexts — tens of thousands or even millions of tokens — evaluation methods must adapt. Assessing a model’s ability to maintain coherence, accurately recall information from distant parts of the input, and perform complex reasoning over these extended contexts is becoming crucial. This necessitates the development of new benchmarks specifically targeting these long-context capabilities.
• The Shift Towards Evaluating Real-World Application Performance:
  There is a clear movement away from relying solely on isolated, academic benchmarks towards evaluating LLMs within the context of specific, real-world applications and workflows. This means assessing not just the LLM’s output quality but also how effectively it integrates with other systems, utilizes tools, and supports human users in completing their tasks. Metrics will need to be tailored to the specific use case and desired outcomes of the application.
• Need for Continuous Monitoring and Adaptation of Evaluation Strategies:
  LLM performance is not static; it can drift over time due to changes in training data, model updates, or evolving user interaction patterns. New vulnerabilities or failure modes can also emerge. Consequently, continuous monitoring and evaluation of LLMs in production environments are becoming standard practice. Furthermore, evaluation frameworks themselves must be adaptable, capable of evolving to accommodate new model architectures, novel capabilities, and diverse application domains. The proliferation of new benchmark proposals signifies active development and a search for more relevant and reliable evaluation techniques.
• Automated Benchmark Generation and “Living Benchmarks”:
  To address the limitations of static benchmarks (e.g., data contamination, becoming outdated), research is exploring frameworks that can automatically generate new benchmark instances or dynamically adapt existing ones. This aims to keep benchmarks challenging, relevant, and less susceptible to “teaching to the test.” Examples include concepts like an “LLM-Powered Benchmark Factory” and the synthetic data generation used in systems like ARES for RAG evaluation. This trend suggests a co-evolution: as models become more sophisticated, they can also be employed to create more sophisticated and robust evaluation scenarios, leading to a dynamic interplay between AI development and AI assessment.

The push for “contamination-free” benchmarks and a deeper understanding of the limitations and potential biases within existing evaluation methods signals a maturation of the field. There is a growing recognition that simply chasing state-of-the-art scores is insufficient; the meaningfulness, reliability, and fairness of the evaluation process itself are paramount for guiding true progress in AI.

Beyond Text: A Brief Look at Multimodal Evaluation

While much of the focus has been on text-based LLMs, AI is increasingly multimodal, with models capable of generating and understanding images, video, and audio. Evaluating these systems presents its own unique set of challenges and requires specialized metrics. The difficulties often mirror those in text evaluation, such as reliance on features from pre-trained models and occasional disconnects with human perception, but with added dimensions like spatial coherence for images and temporal consistency for videos.

Image Generation

Evaluating the quality of AI-generated images involves assessing aspects like realism, alignment with text prompts, and visual appeal.

FID (Fréchet Inception Distance): FID is a widely used metric to assess the realism and visual quality of generated images by comparing their feature distribution to that of real images. It works by passing both sets of images through a pretrained image classification model (typically Inception v3) to obtain feature embeddings. Assuming these embeddings form multivariate Gaussian distributions, FID calculates the Fréchet distance between the distribution of real image features (
(μr​,Σr​)) and generated image features ((μg​,Σg​)) using the formula:

FID=∣∣μr​−μg​∣∣2+Tr(Σr​+Σg​−2(Σr​Σg​)1/2)

A lower FID score indicates that the generated images are more similar to real ones. However, FID has limitations: it assumes Gaussianity of features, requires a large dataset for stable estimates, and its scores are dependent on the specific surrogate model used (Inception v3), which may not perfectly capture all aspects of human visual perception.

CLIPScore: This metric evaluates how well a generated image aligns semantically with the text prompt used to create it. It utilizes a pre-trained CLIP (Contrastive Language-Image Pre-training) model, which maps both images and text into a shared embedding space. CLIPScore is calculated as the cosine similarity between the CLIP embedding of the generated image (EI​) and the CLIP embedding of the text prompt (ET​), typically scaled by 100:

CLIPScore=max(100⋅cos(EI​,ET​),0)

A higher CLIPScore (ranging from 0 to 100) indicates better semantic alignment. A key limitation is its potential insensitivity to image quality; an image might be semantically aligned with the prompt but still be visually poor or contain artifacts. Like FID, its effectiveness is tied to the capabilities of the surrogate CLIP model.

Other metrics such as PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index Measure) focus on pixel-wise or structural similarity to a reference image, while LPIPS (Learned Perceptual Image Patch Similarity) aims to better align with human perceptual judgments of image similarity.

Video Generation

Evaluating generated videos adds the critical dimension of temporal consistency on top of per-frame image quality. A video must not only have high-quality individual frames but also exhibit smooth and logical transitions and motion over time.

FVD (Fréchet Video Distance): FVD is an adaptation of FID for video evaluation.31 It uses features extracted from a 3D Convolutional Neural Network (e.g., I3D trained on the Kinetics dataset) to compare the distributions of real and generated videos. It aims to capture both visual quality and temporal characteristics.

However, FVD has faced significant criticism for several limitations:

1. Non-Gaussianity of Feature Space: The I3D feature space often does not conform to the Gaussian distribution assumption underlying Fréchet Distance, especially for longer videos.
2. Insensitivity to Temporal Distortions: I3D features have been found to be relatively insensitive to certain temporal distortions that are salient to human viewers, and FVD can be biased towards per-frame image quality rather than overall video quality or motion coherence.
3. Sample Inefficiency: Reliable FVD estimation often requires impractically large numbers of video samples.
4. Sensitivity to Implementation Details: FVD scores can be overly sensitive to minor details like video compression levels or file encoding, leading to inconsistencies across studies.

Alternatives like JEDi (Joint Embedding Distributional Similarity):
Due to FVD’s limitations, alternatives like JEDi have been proposed. JEDi aims to address FVD’s shortcomings by:

1. Using a Maximum Mean Discrepancy (MMD) metric with a polynomial kernel. MMD does not assume Gaussianity for the feature distributions.
2. Employing features from a V-JEPA (Video Joint Embedding Predictive Architecture) model, which has shown better alignment with human perception of video quality, particularly regarding temporal aspects.
   JEDi has demonstrated advantages such as requiring significantly fewer samples for stable estimation (higher sample efficiency) and showing better correlation with human judgments of video quality compared to FVD. The development and proposal of JEDi illustrate a healthy scientific process: identifying fundamental flaws in an established metric (FVD) and systematically developing an improved alternative grounded in both statistical theory and empirical alignment with human perception.

Conclusion: The Ongoing Quest for Better Evaluation

The evaluation of Large Language Models and the AI systems they power is a complex, multifaceted, and rapidly evolving domain. It is a challenge that lies at the heart of responsible AI development and deployment. As LLMs become more capable and integrated into diverse aspects of our lives, the need for robust, reliable, and meaningful evaluation methodologies becomes ever more critical.

This exploration has highlighted that no single metric or benchmark can capture the full spectrum of an LLM’s performance or an AI system’s utility. Traditional benchmarks like MMLU and HellaSwag provide valuable insights into core knowledge and reasoning skills but are often static and can be “gamed.” Evaluating more complex systems like RAG and AI agents requires a shift towards assessing process, tool use, and interaction with dynamic environments, using specialized frameworks such as RAGAs and AgentBench. Human-centric approaches, including LLM-as-a-Judge and platforms like Chatbot Arena, offer scalable ways to approximate human preferences but come with their own set of biases and vulnerabilities. Furthermore, multimodal AI introduces additional layers of complexity, demanding tailored metrics like FID, CLIPScore, FVD, and emerging alternatives like JEDi.

The path forward necessitates a multi-faceted approach to evaluation. This involves a combination of automated benchmarks, rigorous human-in-the-loop assessments, system-level testing in realistic contexts, and the development of domain-specific metrics that align with particular application goals. The field is actively moving towards more dynamic, robust, and ethically aware evaluation practices, including continuous monitoring in production and the automated generation of “living benchmarks.”

Ultimately, the ongoing quest for better evaluation is not merely an academic exercise or a technical pursuit of higher scores. The metrics and methodologies chosen by the AI community profoundly influence the direction of AI development. By striving for evaluation techniques that are not only accurate but also fair, transparent, and truly reflective of desired AI capabilities and societal impact, we can better guide the trajectory of this transformative technology towards beneficial and trustworthy outcomes. Continuous research, critical assessment of existing methods, and collaborative development of new evaluation paradigms are essential to navigate this complex landscape successfully.

TL;DR

knowhub is an open-source tool that helps you keep shared configuration files in sync across multiple repositories without manual copy-paste. It addresses the pain of maintaining files like AI assistant rules (.cursor/rules, .windsurfrules, or .github/copilot-instructions.md) and coding guidelines (e.g. coding-guidelines.yaml) in dozens of repos. With knowhub’s plugin-based architecture, you define once where these common files come from – whether a local folder, an HTTP URL, or other sources – and knowhub automatically pulls them into each project. No more outdated rules or inconsistent configs scattered around. This article explains the problem knowhub solves, how its plugin-only design works (including built-in local and HTTP plugins), how to configure it via a .knowhubrc file (with options like overwrite vs. symlink and multiple output targets), and how to extend it with custom plugins for sources like private GitHub repos or internal S3 buckets. We’ll also draw parallels to how modern AI coding assistants (Cursor, Copilot, Windsurf, Devin) use project-specific rule files, and how that inspired knowhub’s design. In short, knowhub is your “knowledge hub” for cross-repo configs – maintain shared files in one place and reuse them everywhere, confidently and consistently.

The Problem: Scattered Config Files Across Repositories

If you work in a team with many code repositories, you’ve likely run into this scenario: you want every project to follow the same guidelines or use the same AI assistant instructions, so you create files like coding guidelines, style rules, or AI prompt rules and put a copy in each repo. These might be files such as a .windsurfrules for Windsurf (to guide its AI code suggestions), a .cursor/rules directory for Cursor’s project rules, a .github/copilot-instructions.md for GitHub Copilot, or a YAML with company coding standards. It works great initially – each project gets the benefit of these custom instructions or guidelines – until one day you update something in those files. Now you have to propagate that change to every repository. It’s easy to forget one or apply updates inconsistently, leading to drift and confusion.

Developers have been asking for better solutions. For example, one user on Reddit managing 21 repositories pleaded: “I have 21 repos… Is there a way to create a global instructions file for all my repos?”. This is a common pain point — manually copying changes across repos is error-prone and time-consuming. We end up with stale rules in some projects and up-to-date rules in others, defeating the purpose of having a standard. In the context of AI coding assistants, outdated instruction files in one repo could mean the AI there isn’t following the latest team practices, while another repo’s AI is. For general documentation or config files, the same inconsistency creeps in.

Why not just use one repository for shared configs? In some cases, teams try approaches like Git submodules, copying files as part of build scripts, or maintaining a separate “config templates” repo and manually downloading from it. These solutions either add complexity or still require manual steps. There’s a clear need for an automated, flexible way to keep designated files in sync across multiple projects.

Meet knowhub: A Unified Solution for Shared Files

knowhub is designed to solve exactly this problem. It’s a tool that acts as a central knowledge hub for your repositories’ shared files. With knowhub, you declare in each repo which files should be kept updated from a common source. Then, with a simple command, knowhub fetches those files from the source and updates your local repository copy (either by replacing the file or by linking to a central copy, depending on your preference). The result is that all your projects stay consistent without each developer having to manually copy files around.

At its core, knowhub embraces a plugin-only architecture. This means it doesn’t hardcode any particular place to get files from — instead, it supports a variety of “plugins” that know how to fetch files from different locations. Out of the box, knowhub comes with a couple of built-in plugins, and you can add your own. The built-in ones cover the most common cases:

• Local plugin — for pulling in files or entire directories from your local filesystem (e.g. a directory on your machine or shared drive where you keep common configs).
• HTTP plugin — for pulling a file from an HTTP(S) URL (e.g. a raw file on the web, a gist or an internal HTTP server).

Using this plugin system, knowhub treats every source uniformly. Whether a file lives on your disk, behind an HTTPS link, or (eventually) in a GitHub repository, you configure it in the same way and knowhub will retrieve it using the appropriate plugin. This unified model makes knowhub very extensible — if the built-ins don’t cover your use case, you can write a custom plugin to fetch from anywhere (like a private GitHub repo via API, an S3 bucket, etc.), and plug it in without modifying knowhub’s core.

In short, knowhub acts like a middleman: your repo says “I need file X from source Y”, and knowhub (via a plugin) goes and gets file X from Y, then places it into your repo in the desired location. The developer experience is simple: run knowhub whenever you want to sync or update those shared files (this could be manually, or as part of your build/test pipeline), and commit any changes. Consistency is achieved with minimal effort.

Plugin-Only Architecture: Fetch from Anywhere

One of knowhub’s key design principles is that it doesn’t know or assume where your team’s “source of truth” for shared files is — instead, it relies on plugins to handle that. Let’s unpack how this works and why it’s powerful:

Built-in Plugins: Local and HTTP

By default, knowhub includes two plugins:

• Local Plugin: This plugin fetches files from a local directory on the same machine. You might use this if, say, you have a central folder on your drive or network (/company/shared-rules/ for example) that contains the latest versions of your .cursor/rules, .windsurfrules, coding-guidelines.yaml, etc. The local plugin can be pointed at that directory (or a specific file) and will copy those files into your project. It supports grabbing a single file or an entire directory subtree. For example, if you maintain a whole folder of standard config snippets, you could sync the whole folder into each repo with one plugin entry. This ensures you’re always using the latest local copy of those files.
• HTTP Plugin: This plugin fetches a file from a URL over HTTP or HTTPS. This is useful if your canonical source is hosted on the web — for instance, maybe you have a raw GitHub gist URL for your latest coding-guidelines.yaml, or an internal web service that serves your company’s standard .windsurfrules file. By using the HTTP plugin, knowhub will perform an HTTP GET to download the file and drop it into your repo. This way, even if your source of truth is outside your local machine, you can fetch it on demand. (Under the hood, it could also handle things like following redirects or basic auth if needed, since it’s just an HTTP fetch – though for private URLs with auth, a custom plugin might be better suited, more on that later.)

The architecture being “plugin-only” means that even these basic behaviors are implemented as plugins. knowhub’s core doesn’t contain special-case code for “copy from filesystem” or “download from URL” — it just loads the appropriate plugin based on your config. This is a deliberate choice: it keeps the core simple and lets users extend it in the future without having to modify the core tool.

GitHub Plugin Example and Custom Extensions

While knowhub’s core remains the same — all it needs is a plugin interface that fetches content — this section shows how you can leverage an existing GitHub plugin and explains the design trade-offs that let you add any new source without touching core code.

GitHub Plugin as a Drop-in Example
In the repository’s examples/github-plugin.ts, you’ll find a ready-to-use implementation that fetches files (and entire directories) from a GitHub repository. It handles:

1. Authentication: Accepts a personal access token so you can access private repos.
2. Directory Listing: Uses GitHub’s API to enumerate contents under a given owner/repo/path@ref.
3. File Retrieval: Downloads the raw file content (or branches out into subdirectories) and hands it back to knowhub.

Because it conforms exactly to knowhub’s plugin interface — exporting a name and asynchronous fetch(config, context) function—you simply reference "github" in your .knowhubrc (along with fields like owner, repo, path, ref, and token). knowhub will call that plugin, get the content, and then write or symlink it into your project just as it does for local or HTTP sources. No core modifications needed. Why does this deskgn work?

Separation of Concerns:

• Core: Validates shared fields (targets, overwrite/symlink flags), handles writing files into your repo, and orchestrates the overall sync run.
• Plugin: Knows how to talk to a specific backend (in this case, GitHub).

It takes care of API calls, pagination, error handling, and returns raw file content or file paths.
Because of this division, knowhub’s core never needs to import Octokit or worry about rate limits — it just calls the plugin’s fetch and trusts it to return valid data.

Easy Extension to New Sources:
Need S3? Write an S3 plugin that uses AWS SDK to pull files from a bucket.
Need Bitbucket or GitLab? Write a plugin that wraps their REST APIs.
Need a private wiki or internal service? Implement the plugin interface, register it, and reference its name in .knowhubrc. You don’t alter knowhub’s core code at all—just drop in the new plugin.

Consistent UX Across Sources:
Whether you’re pulling from GitHub, HTTP URLs, or a local folder, your .knowhubrc entries look similar:

files:
  - plugin: "local"
    source: "../SharedConfig/coding-guidelines.yaml"
    target: "docs/coding-guidelines.yaml"
    mode: "overwrite"
  - plugin: "http"
    source: "https://intranet.example.com/standard-rules/windsurf_rules_v5.md"
    target: ".windsurfrules"
    mode: "overwrite"
  - plugin: "github"
    source:
      owner: "my-org"
      repo: "shared-rules"
      path: "ai/.cursor/rules"
      ref: "main"
      token: "${GITHUB_TOKEN}"
    target: ".cursor/rules"
    mode: "overwrite"

Once you register (or install) the GitHub plugin, knowhub handles it the same as “local” or “http.” Sync behavior does not change.

Because of these design choices, knowhub can scale to any number of file sources. The core remains lightweight — focused only on syncing and writing — while plugins encapsulate all “fetch logic,” whether that’s an HTTP GET, a local filesystem read, or a GitHub API call.

How knowhub Uses Plugins Under the Hood

To demystify it a bit: when you run knowhub, it reads your configuration (the .knowhubrc file, see next section) which tells it what to fetch and which plugin to use for each item. For each entry, knowhub will load the specified plugin (all plugins conform to a common interface, e.g. a function or class with a method to fetch files). It then calls the plugin with the details (such as a path or URL) to retrieve the content. The plugin hands the content (or multiple files if it’s a directory) back to knowhub, and knowhub writes them into the specified locations in your repo (handling things like creating directories, overwriting files, or making symlinks as directed by config). If any step fails (e.g. URL not found, permission denied, etc.), knowhub will report the error, but the idea is that these plugins encapsulate those details.

By designing it this way, adding a new source type doesn’t require changing how knowhub operates; you just drop in a new plugin. This means knowhub can easily adapt to new storage solutions or company-specific requirements. It’s truly a unified model — from knowhub’s perspective, every file is just “coming from a plugin,” regardless of where it actually lives.

Configuring knowhub: The .knowhubrc File

So, how do you tell knowhub what to fetch and where to put it? This is done via a configuration file in your repo, typically named .knowhubrc (with various format options). knowhub supports configuration in TypeScript, JSON, or YAML, so you can choose the format you’re most comfortable with (.knowhubrc.ts, .knowhubrc.json, or .knowhubrc.yaml for example). All of these ultimately define the same information: the list of plugins and sources to use, and the target locations in your project where files should go.

Let’s break down the key parts of a .knowhubrc config:

• Plugins Section (optional): If you need to register any custom plugins or configure plugin options, you would do that here. (For built-in plugins, you usually don’t need to configure anything special — they work out of the box. Custom plugins will be discussed later.) In many cases you might not need a separate plugins section at all, and can directly reference built-in plugins by name.
• Files/Rules Section: This is the heart of the config — it’s where you list the files or directories you want to sync. For each entry, you specify:
• Which plugin to use (by name, e.g. "local" or "http" or a custom plugin name).
• The source location or identifier (e.g. a file path, directory path, or URL).
• The target location in your repo where the file(s) should be placed.
• The mode of how to place them (more on modes below).

Each entry can be as granular as a single file, or it can represent a batch (for example, a whole directory of files to mirror). You can have multiple entries if you need to fetch different files from different sources.

Let’s look at an example configuration in YAML format for clarity. Suppose we want to achieve the following in our repository:

• Pull the latest AI rules from a shared HTTP URL and save it as our local .windsurfrules.
• Pull a coding guidelines YAML from a local directory (perhaps an adjacent folder on disk) and put it in our repo’s docs folder.
• Ensure that the same AI rules file is also saved as a Cursor rules file (maybe the content is compatible with both Windsurf and Cursor, so we want it in two places).

Our .knowhubrc.yaml might look like this:

# .knowhubrc.yaml
files:
  # 1. Fetch .windsurfrules from a URL
  - plugin: "http"
    source: "https://intranet.example.com/standard-rules/windsurf_rules_v5.md"
    target: ".windsurfrules"
    mode: "overwrite"
  # 2. Fetch coding guidelines from a local shared directory
  - plugin: "local"
    source: "../SharedConfig/coding-guidelines.yaml"
    target: "docs/coding-guidelines.yaml"
    mode: "symlink"
  # 3. Also copy the same Windsurf rules to Cursor's rules (multiple targets from one source)
  - plugin: "http"
    source: "https://intranet.example.com/standard-rules/windsurf_rules_v5.md"
    target: ".cursor/rules/enterprise-project.mdc"
    mode: "overwrite"

A few things to note in this config:

• We have an array under files: with three entries. Each entry is processed independently by knowhub.
• Plugin: Each entry specifies a plugin. In entry 1 and 3, we use the "http" plugin (both happen to use the same URL as the source). In entry 2, we use the "local" plugin, pointing to a local path.
• Source: For the HTTP entries, source is a URL string. For the local entry, source is a filesystem path. The local plugin can handle absolute or relative paths; in this case ../SharedConfig/... might refer to a directory outside this repo (perhaps a cloned repository or a network drive that contains shared files).
• Target: This is where in our repository we want the file to go. In entry 1, we target the file .windsurfrules in the root of this repo. In entry 2, we target docs/coding-guidelines.yaml (meaning we expect a docs folder in our repo; knowhub will create it if it doesn’t exist). In entry 3, we target a file inside the .cursor/rules/ directory. (Cursor’s convention is to have a folder .cursor/rules with rule files – here we might name the file something like enterprise-project.mdc as the rule set name.)
• Mode: We have used two modes here to illustrate the difference. For entries 1 and 3, mode: "overwrite" means knowhub will write the file content directly to those target files (overwriting any existing content). For entry 2, mode: "symlink" means instead of copying the contents of coding-guidelines.yaml into our repo, knowhub will create a symbolic link at docs/coding-guidelines.yaml that points to the source file in ../SharedConfig/coding-guidelines.yaml. We chose symlink for the coding guidelines in this example because perhaps we don’t want to duplicate that large file – and we trust that in our development environment, having a symlink works (developers have the SharedConfig folder available locally). The .windsurfrules, on the other hand, we opted to overwrite, meaning our repo will have an actual copy of the rules file content.

Let’s talk more about modes and multiple targets conceptually:

Overwrite vs. Symlink

knowhub gives you two ways to bring the file into your repo:

• Overwrite (Copy): This mode (as used in the example for the rules files) will make a real copy of the file’s contents in your repository. If the file already exists, it will replace the contents (hence “overwrite”). Use this when you want each repository to truly hold its own copy of the file — for instance, if contributors might edit it locally or if you want to commit that file to version control for each project. Overwrite mode is straightforward and works everywhere (including Windows, which has some restrictions with symlinks). The downside is you now have multiple copies of the file (one in each repo), but knowhub makes it easy to update them all by just running the sync again whenever the source changes.
• Symlink: This mode creates a symbolic link in the filesystem pointing to the source file’s path. The advantage is that you avoid duplication — all those repos effectively refer to one master file. If you update the master file, all the symlinked repos instantly see the change (at least on the local machine). This is fantastic in a controlled environment or for local development, but it has caveats. For example, if someone else pulls your repo on a different machine, that symlink might break if they don’t have the same source path available. Symlinks also may not play well if you archive the repo or on certain OSes without extra steps. Generally, symlink mode is best when the source is on a shared drive or the same machine and you want a single source of truth without copying. Overwrite mode is safer if the repo needs to be self-contained or portable.

Many teams use overwrite mode to periodically update and commit the file in each repo (ensuring everyone gets the update via version control). Others might use symlink mode in development and perhaps exclude the symlink from git. knowhub leaves that choice to you.

Multiple Output Targets

Notice in our example we effectively used the same source URL twice (for .windsurfrules and for a Cursor rules file). We did it by writing two entries (entry 1 and entry 3) with the same source. In knowhub’s config, it’s also possible to streamline this by specifying multiple targets for one source in a single entry, depending on the format. For instance, we could have a single entry for the HTTP plugin with the source URL, and list both .windsurfrules and .cursor/rules/enterprise-project.mdc as the targets. The effect is the same – knowhub will fetch the content once and put a copy in each target location. This “multiple output” feature is handy when one common file should appear in different places or with different names in your repo. It saves you from having to duplicate the plugin configuration for each target.

Another use-case for multiple targets might be if you have, say, a monorepo with several sub-projects and each sub-project needs its own copy of a config file. You could have one knowhub entry that fetches a file and then list each subdirectory’s path as a target. Ultimately, whether you use one entry with multiple targets or multiple entries pointing to the same source is up to readability and convenience — the outcome can be achieved either way.

Example Workflow: Keeping AI Rule Files in Sync

Let’s walk through a concrete example scenario to see how knowhub can be used in practice. Imagine your team has started using AI coding assistants like Cursor and Windsurf across many projects. You’ve developed a comprehensive set of AI guidance rules (maybe a markdown file with instructions for the AI on how to structure code, your internal best practices, what to avoid, etc.). You want every repository to use these rules so that whether a developer is coding in Repo A or Repo B, the AI assistant follows the same guidelines.

Scenario: You have a central repository (or folder) called ai-rules/ which contains the latest global_rules.md (for Cursor) and .windsurfrules file for Windsurf. Repo A and Repo B are two different project repositories that should include these rules.

Approach without knowhub: You would copy global_rules.md into .cursor/rules in each repo, and copy .windsurfrules into each repo. Initially fine, but as soon as you update something (say you found the AI was making a certain mistake, and you add a rule to prevent it), you now have to copy those changes to Repo A, Repo B, and any other repo that uses them. It’s very easy to miss one – for example, Repo B might still have last month’s rules while Repo A has the updated ones.

Approach with knowhub: You decide to use knowhub with the local plugin in this case, since you have the ai-rules directory locally. Here’s how you might set it up:

1. Install knowhub in Repo A and Repo B.

It’s a Node.js-based tool (assuming it’s distributed via npm), so you could add it as a devDependency:

npm install --save-dev knowhub

Or you can install it globally if you prefer. You’ll need Node.js available to run it.

2. Add a .knowhubrc configuration in each repo.

In Repo A, for example, you create a file .knowhubrc.json (or YAML or TS, format doesn’t matter as long as knowhub can parse it). Let’s use JSON for variety:

{
  "files": [
    {
      "plugin": "local",
       "source": "/Users/yourname/ai-rules/.windsurfrules",
      "target": ".windsurfrules",
      "mode": "overwrite"
    },
    {
      "plugin": "local",
      "source": "/Users/yourname/ai-rules/global_rules.md",
      "target": ".cursor/rules/global_rules.md",
      "mode": "overwrite"
    }
  ]
}

This config says: use the local plugin to grab .windsurfrules from the ai-rules directory and put it in the current repo’s root as .windsurfrules. Also grab global_rules.md from that folder and put it in .cursor/rules/ here. (We choose overwrite mode because we want actual copies in the repo that we will commit.)

3. Run knowhub to sync the files.

If installed locally, you can run it via npx:

npx knowhub

This will read .knowhubrc.json, execute each entry, and output logs of what it’s doing. Now your Repo A has the latest rules. You commit the updated .windsurfrules and ./cursor/rules/global_rules.md to version control.

4. Do the same in Repo B (and any other repos).

Each repo’s .knowhubrc can be identical if they use the same sources. In fact, you could script this or template it. Once knowhub runs, all repos have identical content for those files.

5. Next time

You update the rules in the ai-rules source, just rerun npx knowhub in each repo (or as part of a CI job or a pre-commit hook) to pull in the changes. You’ll get diff updates if anything changed, and you commit them. This ensures no repo falls behind on the shared knowledge.

This is one common approach — teams can also adopt other methods (custom plugins, alternative registries, etc.) depending on where they store their files. For example, if the ai-rules were on an internal Git server or a cloud storage, a custom plugin or the HTTP plugin could be used. The key point is that knowhub provides a structured, automated way to pull updates, rather than relying on each developer to manually remember to update files.

In our example, we used knowhub for AI assistant rule files, but you can use it for any kind of config or asset that is meant to be reused. Maybe it’s a coding-guidelines.yaml that defines linting or formatting standards. Or a common README snippet that should be present in all repos. Or license text files, CI pipeline YAML templates, you name it. knowhub can help distribute those uniformly.

Parallels with AI Coding Assistants and Project Rules

You might be wondering why we’ve focused on examples like .windsurfrules or .cursor/rules so much. It’s because the rise of AI coding assistants has made the need for shared configuration files more pronounced than ever. Tools like Cursor, GitHub Copilot, Windsurf, and Devin all allow (or encourage) you to include project-specific rule files in your repository to guide the AI. These files essentially give context or instructions to the AI model about your project’s conventions and expectations.

For instance:

• GitHub Copilot now supports a repository-wide custom instructions file (.github/copilot-instructions.md). This file can contain text telling Copilot about your preferences (like “we use tabs, not spaces” or architectural notes). Copilot will use this invisible context to tailor its suggestions. According to GitHub’s docs, adding such a file provides the AI with context so it can “generate higher quality responses” that match your project’s needs.
• Cursor (an AI-powered IDE) previously used a single .cursorrules file and now uses a “Project Rules” system (often storing rules under a .cursor/rules/ directory). These rules can dictate how Cursor’s agent behaves, what it should avoid, and even allow it to take autonomous actions safely. Developers have started sharing sophisticated .cursorrules configurations to push Cursor’s capabilities.
• Windsurf (another AI coding environment, especially for JetBrains IDEs) uses a file named .windsurfrules in the project root. Similar to Cursor’s rules, this file provides guidance to Windsurf’s AI assistant about how to handle your codebase or what style to follow.
• Devin is a more advanced AI software engineer agent. While Devin itself is a paid service that comes with its own managed environment, the interesting thing is that the community discovered you can emulate a lot of Devin’s behavior in other tools by using carefully crafted rule files. For example, a project called devin.cursorrules provides a template .cursorrules and .windsurfrules that make Cursor or Windsurf behave more like Devin. In their setup instructions, they highlight how different AI platforms each use their own config file: “Windsurf users need both .windsurfrules ... Cursor users only need the .cursorrules file. GitHub Copilot users need the .github/copilot-instructions.md file.”. In other words, each tool has its own spot for these instructions.

These AI configuration files have quickly become “must-haves” in modern repos — they’re like the new README or the new linters, in terms of being ubiquitous in certain circles. And because they serve a similar purpose across projects (telling the AI how to help us), teams naturally want to standardize them. If your organization has certain coding standards or patterns, you’d encode those in the AI rules so that the AI helps enforce them. But without a tool like knowhub, ensuring that all your AI config files are up-to-date in every repository is a nightmare (as we saw in the problem discussion).

knowhub’s design was influenced by this very trend. The idea of a “knowledge hub” for your projects is in line with how AI assistants gather context. We can think of knowhub as doing for your config files what an AI does for its prompt context: it centralizes knowledge so it can be applied wherever needed. Instead of copying rules manually, knowhub automates it. In effect, knowhub complements these AI tools — as you update your AI rules to improve the assistance (maybe you learned something new that all projects should follow), knowhub ensures that improvement propagates to all projects. This synergy means better AI suggestions across the board, with minimal manual overhead.

It’s worth noting that the concept isn’t limited to AI rules. The same applies to any config file that encodes institutional knowledge. But the explosion of .windsurfrules and friends provided a clear use case that pushed the development of knowhub. Just as developers share “awesome rules” repositories to improve their AI coding experience, knowhub helps teams share their own rules internally without hassle.

How difficult is it to write a plugin? Likely not very — the knowhub plugin interface would handle details like where to put the files; your plugin mostly just needs to fetch data. The example above is ~15 lines of code to fetch from S3. Many plugins could be that small. Also, consider that not everyone on your team needs to write these — you can create a plugin and share it (even publish it as an npm package, e.g. knowhub-plugin-github), then everyone can use it by simply including it in their config.

Where knowhub Fits in Your Development Workflow

knowhub is meant to be a lightweight addition to your multi-repo workflow. It doesn’t replace version control or package managers — instead, it supplements them by managing those pieces of information that normally fall through the cracks. Think of knowhub as sitting in between your documentation and your code. It takes the documentation/config that you want to reuse and makes sure it actually lands in your code repositories.

Here are some practical tips on integrating knowhub into your process:

• Run knowhub regularly: A common practice is to run knowhub as part of a project’s setup or CI pipeline. For example, you might add a script in package.json like "sync:knowhub": "knowhub" and have developers run npm run sync:knowhub whenever they pull the latest changes. This ensures any updates to shared files are fetched before they start working. In CI, you could run knowhub to make sure the repo’s config is up-to-date (and maybe even fail the build if a required file is missing or outdated – though implementing that would require some additional checking logic).
• Centralize what makes sense: You don’t have to use knowhub for everything — identify the files that genuinely benefit from central maintenance. Good candidates are: AI assistant rules, coding standard docs, license text, contributing guidelines, template config files, etc. Things that change and should uniformly change everywhere. If some repos diverge intentionally, then they might not be suitable for this.
• Version your shared files: One challenge when sharing files is knowing which version each repo has. You might include a comment or version number in the file content itself (for example, inside coding-guidelines.yaml have a field version: 2.1). This way, even if someone forgets to sync, it’s apparent in a code review that “oh, this repo is on v2.0 of the guidelines, we should run knowhub.” Alternatively, if your sources are on the web or in a repo, you can manage versions there (like tagging releases of a central config). knowhub will always pull the latest unless you specify a versioned URL, so you have control over when to update by choosing which URL or path to point to.
• Alternative approaches: knowhub is one approach to the multi-repo config problem. Some teams solve this by using mono-repos (everything in one repository) or by creating internal libraries/packages that encapsulate certain configurations. Those approaches have their merits but also trade-offs — mono-repos can be heavy and all-or-nothing, and internal libraries work for code but not always for plain text config that tools expect in specific locations. knowhub fills a niche by focusing on file-level sharing rather than package-level. It’s a tool in your toolbox alongside others. You can still keep code dependencies via packages; use knowhub for the bits that don’t fit neatly into a package.

In the grand scheme, knowhub is a small utility, but it can have an impact on consistency and developer happiness. It removes a source of friction (“Did we update that file everywhere?”) and lets you focus on the actual content of those shared files rather than the mechanics of copying them around. Teams adopting knowhub can evolve their shared guidelines faster, because they know distribution is a solved problem.

Finally, as developer-to-developer: if you’ve ever groaned at a production issue caused by one service not getting the memo about a config change, or if you’ve spent half a day updating boilerplate text across 18 repositories, you’ll appreciate what knowhub brings. It’s straightforward by design — no fancy infrastructure needed, just a config file and a command. But through that simplicity, it addresses a very real day-to-day problem.

In summary, knowhub helps you maintain a single source of truth for the files that matter across repos. It was inspired by real needs (like keeping AI assistant rules consistent) and built to be extensible for various environments. By using knowhub, you ensure that every repository in your ecosystem speaks the same language when it comes to guidelines and config. In an age where AI tools are reading those guidelines to assist us, having them uniformly applied can even improve the AI’s output across your projects — a nice synergy between tool and content. Give knowhub a try on one of your shared files, and you might find it quickly becomes an indispensable part of your multi-repo development workflow. Happy syncing!

TL;DR

After 15 years in engineering (12 as an EM, VPoE, or CTO), I’ve learned that managing teams is a lot like designing software systems. Treat your organizational design like system architecture: define clear interfaces between teams like microservices APIs to reduce tight coupling; use checklists as validation tests to catch mistakes early; rely on 1-on-1s as monitoring alerts to surface “organizational bugs” (issues) before they escalate; apply dependency inversion — leaders set high-level policies (the “what”) and empower teams to design the implementation (the “how”); use OKRs as dependency injection for goals, cascading top-level objectives down through layers; build redundancy in skills across teams (avoiding a single point of failure, as in RAID 0) to improve resilience; load-balance workloads via roles like a PMO to prevent any team from overloading; cache knowledge with FAQs and documentation so answers are quick and don’t always hit “expensive” expert time; create a containerized onboarding process that packages your team’s knowledge and environment to get new engineers productive fast; and continuously refactor your organization, using metrics like DORA and SPACE as your guide to find bottlenecks and areas to improve. In short, thinking like an architect can make you a better engineering manager.

Microservices and Team Communication

In software, a microservices architecture breaks a large application into small, loosely coupled services, each with a clear interface. This modularity allows services to be developed and deployed independently. Similarly, in an organization, each team should have clear ownership of a domain and well-defined communication channels — essentially an API for how others interact with that team. When teams communicate via well-specified interfaces (e.g. documented processes, API contracts, regular syncs), they avoid stepping on each other’s toes, just as services avoid unintended side-effects through encapsulation.

There’s even a famous principle, Conway’s Law, which states: “organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations”. In practice, this means if your teams are structured with clear, decoupled communication pathways, your software is likely to be modular as well. Loosely coupled teams build loosely coupled systems. For example, if you have a frontend team and a backend team with a defined API between them, that boundary will reflect in the software architecture. Conversely, if your org is a chaotic web of cross-team interactions with no clear ownership, you often end up with a monolithic, tangled codebase.

Defining teams like microservices also means aligning teams to business capabilities, not technical layers. Amazon famously uses “two-pizza teams” (small autonomous teams) each owning a microservice or feature area. They communicate through APIs and well-defined protocols, not via ad-hoc tight integration. The result is faster deployment and troubleshooting — a team can update their service without needing a full-company coordination. As a manager, think of your teams as services: design the org chart as you would a system diagram, with explicit interfaces and minimal necessary coupling.

The Organizational API Gateway: Controlled Intake of Requests

Even with microservice-like teams, you need a good request routing mechanism. In software, an API Gateway is a single entry point that routes external requests to the correct internal services. In organizations, this translates to having a structured intake process for new work so that stakeholders aren’t directly hitting individual developers with feature requests or random tasks. A well-defined intake acts as the API gateway between the outside world (customers, executives, other departments) and the engineering teams.

For example, many companies funnel all incoming project requests through a Product Manager or a Project Management Office (PMO). These roles triage and translate requests into the team’s backlog, ensuring the request is well-formed and goes to the right team at the right time. This prevents the “random exec asks developer X to quickly hack something” scenario — which is analogous to someone bypassing your API gateway and directly querying a database, causing chaos. Instead, all requests go through a unified interface (backlog or ticket system) where they can be prioritized and assigned systematically.

A good intake process means developers can focus on execution while product and project roles handle routing and prioritization. It’s essentially decoupling the source of work from the implementation, just like an API decouples clients from internal service implementations. The product backlog serves as the bridge between discovery (what needs to be done) and development (implementing it). This also lets you apply “load balancing” at the intake — if one team is at capacity, new requests can be held or routed to another team if appropriate, rather than overwhelming the first available engineer.

In short, don’t let every request hit your teams unfiltered. Implement an organizational API gateway — whether it’s a formal PMO or simply a strong product owner role — to ensure the flow of work is controlled, preventing overload and context-switch thrash. This keeps the system (your org) stable under high inbound demand, much like an API gateway protects internal services from being overwhelmed or misused.

Checklists as Validation Tests

In engineering, we validate our code with unit tests, integration tests, and code reviews — essentially checklists to ensure everything works as expected. In the same way, managers can use checklists to validate processes and outputs. A checklist might sound simplistic, but it’s one of the most powerful tools to ensure quality and consistency. As Atul Gawande famously noted, “Checklists seem to provide protection against failures. They remind us of the minimum necessary steps and make them explicit… instill a kind of discipline of higher performance.”.

Consider inter-team handoff checklists: before the sales team passes a request to engineering, they might go through items like “Is the PRD complete?”, “Has a kickoff meeting been scheduled with all stakeholders?”, “Is the due date realistic given current engineering capacity?”. This resembles a pre-flight clearance where one unit verifies all prerequisites before handing control to another. It’s a simple validation layer to catch gaps in requirements or misaligned expectations. In my experience, having a standardized checklist at team boundaries (e.g., sales → engineering, product → design) dramatically reduces “We never knew X was missing” moments. The checklist functions as a low-tech “integration test” for handoffs, ensuring nothing falls through the cracks when one team’s deliverable becomes another team’s input.

The key is to apply checklists where consistency matters. For instance, a hiring process might have a checklist so that every candidate experience is uniform (same set of interviews, same evaluation steps) ensuring fairness and thoroughness. This is akin to test cases for specific failure modes: they force consideration of things that might otherwise be overlooked under pressure.

However, just like tests, checklists must be maintained — an outdated checklist that no one follows is like a brittle test suite. Encourage the team to evolve their checklists based on past “bugs” (e.g., if an outage post-mortem finds a missing step, add it to the deploy checklist). When done right, checklists reduce variance and increase reliability in execution, much as a solid test suite gives developers confidence to ship code.

Monitoring and Alerts through 1-on-1s

No matter how good your validation, issues will arise — call them bugs in the organization. In software, we set up monitoring dashboards and alerting systems to catch problems in production. In management, your monitoring system is the regular 1-on-1 meeting with team members. These conversations are early-warning sensors for morale issues, miscommunication, bottlenecks, or any brewing “incident” in your team’s health.

A well-run weekly or bi-weekly 1on1 can surface things that don’t show up in metrics — an engineer feeling burnt out, conflict between teams, confusion about priorities. Think of each 1:1 as log data and an APM trace from the internal workings of your organization. Skipping them is equivalent to running a production system blind to errors. In fact, if a report frequently cancels 1-on-1s or doesn’t open up, that itself is a red flag — much like an alert that your monitoring agent has stopped reporting. As one leadership coach noted, 1on1 are where you get the first early warning signs that you are failing as a boss. If someone suddenly starts canceling or shortening these meetings, it’s like an alarm bell: something’s off.

By keeping these communication channels open, a manager can catch small issues before they explode. For example, an engineer might mention in a 1:1 that they’re getting stuck waiting on another team’s API — that’s a sign of an architectural bottleneck you might address by aligning roadmaps or adding integration tests. Or perhaps a team member expresses career growth concerns — an indicator to adjust roles or provide mentorship before you unexpectedly lose a “node” from your system.

Moreover, just as we set thresholds for alerts (CPU > 90%, error rate > X), as a manager you develop a sense of thresholds in 1:1s. If two or three team members mention being overworked, that’s an alert to investigate workload distribution. If multiple people seem confused about product direction, that’s an alert to clarify vision. These human signals are qualitative but invaluable. Frequent 1:1s lead to higher engagement and catch issues early, preventing costly turnover or project failures.

Bottom line: don’t fly blind. Use 1:1 meetings as your monitoring system, actively listen for anomalies or error conditions in team dynamics, and respond to those alerts. It’s far better to tune your org with minor adjustments (a conversation, a redirected priority) than to only find out about problems at the equivalent of a system outage — like a key engineer quitting or a project derailing.

Dependency Inversion: Context, Not Control

One of the SOLID principles in software design is Dependency Inversion: high-level modules should not depend on low-level modules; both should depend on abstractions. In organizational terms, this means leadership (high-level) shouldn’t micromanage the implementation details (low-level). Instead, leaders provide an abstract interface — context, goals, constraints — and teams provide the implementation. This is essentially the management version of “inversion of control”: teams drive the how, leaders define the what and why.

I learned (sometimes the hard way) that if I, as a leader, tried to dictate exactly how a team should solve a problem, it almost always led to frustration and suboptimal solutions. It’s like writing a library and also trying to hard-code how every application uses it — you end up with a rigid system. Modern leadership at companies like Netflix encapsulates this as “lead with context, not control.” Provide teams with the necessary context (purpose, goals, business rationale) so they can make good decisions, instead of trying to make every decision for them.

Concretely, this might mean as a CTO you set a policy: “We need to improve reliability and meet a 99.9% uptime goal this quarter.” That’s the abstract requirement. How the teams do it — whether through better testing, redundancy, or changing architecture — you leave to them. You’ve defined the interface (uptime target); the implementation is up to the engineering teams’ ingenuity. This not only empowers those closest to the problem to solve it (often they have the best information), but it also increases buy-in and creativity. People take ownership when they design the solution.

When leaders practice this dependency inversion, the flow of dependency is inverted: leadership depends on teams to execute well on goals, and teams depend on leadership for clear objectives and support, not step-by-step instructions. As one Netflix leader put it, “Great leaders provide the what and not the how… folks closest to the problem determine a much better solution (the how) which aligns with the goals (the what) set by leaders.”. In other words, leadership creates the abstract class, and teams write the concrete class that extends it.

This approach also flips accountability in a healthy way. If a team fails to meet a goal, instead of immediately blaming the team, a context-driven leader asks, Did I provide the right context? If not, that’s akin to providing a bad interface specification. Perhaps requirements were unclear or constraints were hidden. Fix the interface and the abstraction, rather than micromanaging the implementation. Over time, this builds a culture of trust and autonomy, much like well-designed software modules trust each other’s contracts without peeking inside.

OKRs: Injecting Goals as Dependencies Across Layers

If dependency inversion is the principle, OKRs (Objectives and Key Results) are the practical mechanism to implement it across an organization. Think of an Objective as an abstraction of a goal — a high-level outcome you want — and Key Results as specific measures or implementations to achieve that outcome. The beauty of OKRs is how they can cascade: a high-level Objective at the company level becomes context for departments, whose own OKRs provide context for teams, and so on. This creates a chain of “dependency injection” for goals.

Here’s how it typically works: leadership defines a top-level Objective (for example, “Improve Product Stability”). They also define a few Key Results, say “Reduce outage minutes by 50%” and “Automated test coverage to 80%”. Now, each engineering team (or department) takes those as input — like an injected dependency — and formulates their own OKRs that align with it. One team might set an Objective “Harden our payment service” with KRs like “Reduce payment errors by X%” or “Implement chaos testing by Q3” that contribute to the company goal. Another team might target improving deployment processes to reduce outages. Each team essentially implements the interface defined by the higher objective, but in their own context.

In formal OKR terms, this is often called cascading OKRs. For example, a team could inherit a Key Result from above as their Objective, then devise their own Key Results to achieve it. If the department’s KR was “Improve uptime to 99.9%”, the team takes that as their Objective “Achieve 99.9% uptime for Service X” and outlines specific KRs like “Implement failover for database” or “Upgrade monitoring to detect issues within 1 min”. This cascade ensures alignment without direct micromanagement: leadership’s intent is injected downward, but the implementation details are defined at each level.

A crucial rule in this cascade is not to write the team’s KRs for them. That would violate dependency inversion and feel like control. Instead, leadership and teams collaborate to agree on what the right KRs are. John Doerr (who popularized OKRs) highlights that innovation happens when teams have ownership of how to hit the targets. In our analogy, top management provides the “constructor parameters” to the class (the constraints and targets), but doesn’t write the internal methods — the team does.

When done well, OKRs act like an API contract between layers of the org. They provide clarity on what each layer expects from the other. Engineers often appreciate this because it translates the abstract company strategy into concrete things they can build or measure, while still giving them freedom in approach. Also, OKRs create a natural feedback loop (via quarterly results) to see if the implementation met the abstract goal, akin to writing tests for an interface.

To illustrate, if the Objective was to “Improve user engagement by 20%”, a front-end team might inject that by focusing on “Decrease page load time to under 2s” as a KR (assuming faster loads improve engagement). Another team might work on “Add 5 new engaging features”. Each team picks what they believe moves the needle, aligned with the common objective. This is like different services each improving their part of a system to collectively meet a global SLA.

In summary, OKRs allow distributed decision-making with alignment. They are the management equivalent of providing an interface (Objective) and letting each component implement it in its own way (Key Results), ensuring that dependencies (goals) are consistently injected across all levels without dictating the inner workings of each team.

Redundancy and RAID Levels in Team Design

In engineering systems, we build redundancy to avoid single points of failure. Think of disk arrays: RAID 0 stripes data with no redundancy (fast but if one drive fails, all data is lost), RAID 1 mirrors data on two drives (one fails, you still have an exact copy), RAID 5/6 spreads data with parity across drives (you can lose one or two drives and still recover). We can apply this thinking to team skills and responsibilities to improve organizational resilience.

A RAID 0 organization is where each person or team is a unique specialist with no backup. It might be efficient in the short term — everyone has their niche — but it’s risky. If “Alice” is the only one who knows System X (bus factor = 1), when Alice is on vacation or leaves the company, that system is effectively down or unmaintainable. High performance with zero redundancy: one failure and you’re in trouble. Sadly, early in my career I saw teams that celebrated individual heroes who owned massive areas of knowledge. It works until it catastrophically doesn’t.

Moving to RAID 1 in org design means having at least two people capable of any critical task or possessing critical knowledge. It’s like pair programming at the team level or having a “shadow” for every key role. For example, ensure that two engineers are familiar with the deployment process, two people understand the legacy billing system, etc. This mirroring provides immediate backup. The downside, as in RAID 1, is cost — you’re effectively paying two people to be capable of one role’s output at any time. But for truly mission-critical areas, this cost is justified to prevent downtime. I’ve found that rotating people through different components (shared code ownership, code reviews, documentation) naturally builds RAID 1-like coverage over time.

RAID 5/6 analogy would mean distributing knowledge among a group such that any one (or two) team members could be lost and the team still operates. This could be achieved by cross-training and collaborative project work. For instance, on a team of 5, not everyone knows everything, but each critical piece of knowledge is known by at least 2–3 people (with overlaps, like parity). You might have one front-end specialist, one back-end specialist, but they pair enough that each can cover basic needs of the other’s area in a pinch. This way, if someone is out or leaves (one “drive” fails), others can fill in the gaps using the shared knowledge parity. It’s a balance of specialization and backup.

From a management perspective, assess your team’s bus factor: where are you at risk with only one person deep in knowledge? Increase redundancy there. This could involve documentation (a form of knowledge backup), or proactively having people share responsibilities. On a larger scale, if only one team in the company can do a certain function (say only one team knows the mobile codebase, or only one SRE team for all ops), think about how to introduce redundancy — maybe cross-train some developers from another team, or form a secondary on-call rotation.

Keep in mind, redundancy isn’t just about continuity if someone leaves — it also reduces burnout. If only one person can do a certain on-call task, they carry a heavy burden. Spread the knowledge and you spread the load. Yes, like RAID, there’s a write overhead (time spent sharing info, not all focus on new features), but the resiliency gains are worth it. A well-designed system or org can survive the failure of any single component; that’s what we should aim for in team design.

Load Balancing Work Across Teams

Just as a server cluster needs load balancing to ensure no single server gets overwhelmed with traffic, an organization needs to balance workload across teams and functions. In projects, there are times one team is swamped while others are idle — that’s akin to one server at 100% CPU and others at 5%. Management’s job is to redistribute work or resources to even that out, keeping everyone efficiently utilized and avoiding burnout on the hot spots.

One way to do this at scale is through a Project Management Office (PMO) or similar oversight function, which can act as a load balancer. The PMO monitors all incoming projects and the capacity of each team. If Team A is at capacity, new projects are queued or routed to Team B if they have slack and appropriate skills. They’ll also make sure deadlines are realistic given the team’s load. Essentially, they ensure the “requests per team” are within healthy limits — no one team or individual becomes a bottleneck due to overload.

On a smaller scale, line managers or tech leads perform load balancing within a team. They look at the Kanban or sprint board and make sure tasks are distributed, reassigning work if one engineer has 10 tasks and another has 2. This is analogous to how a load balancer might shift requests from an overloaded instance to an underutilized one to optimize throughput and latency. Transparent task allocation and regular check-ins can reveal imbalance early. For example, in sprint planning if one person ends up with all the complex tickets, a good scrum master or manager will notice and spread them out.

Another aspect is prioritization as load shedding. In extreme cases, if every team is overloaded (say lots of urgent projects), management might have to drop lower-priority work — similar to a server shedding load or returning 503 errors when overwhelmed. A healthy organization will signal when it’s at capacity (through the intake process and honest communication) so that either more capacity (hiring, outsourcing) is added, or some work is postponed. As the PMI guidance notes, a good intake process “limits the amount of work getting to development to the maximum they can handle efficiently”, making the reality visible to stakeholders. In other words, don’t accept more work into the system than it can process — this avoids thrashing and maintain quality.

When load balancing is effective, teams maintain a steady, sustainable pace, and the overall system (company) achieves higher throughput. If you ignore it, you get intermittent outages: teams burning out (human downtime) or quality issues (“dropped requests”). As one workload management guide puts it, balancing workload “helps keep the entire system healthy.” No single part should operate at redline for too long.

In summary, keep an eye on the distribution of tasks and projects. Use processes and roles (like PMO, scrum masters) to act as load balancers. And remember, sometimes the solution to overload is to say “not now” to some work — just as a server will refuse connections when it can’t handle more, an org must also manage incoming demand to protect its people and output quality.

Caching Knowledge with FAQs and Docs

We’ve all seen what happens to a database when it’s hit with the same query repeatedly — performance suffers. The solution in software is caching: store the result of expensive operations so subsequent requests are served quickly from cache. In an organization, repeated questions and common problems are the “queries” that hit expensive resources (usually senior experts or busy teams). The cache equivalent is a well-maintained FAQ or internal knowledge base that provides quick answers and resources without always interrupting someone’s workflow.

For instance, if new engineers often ask “How do I get access to X system?” or “What’s the process to deploy to production?”, these are prime candidates for an internal FAQ or wiki page. Instead of each person having to find and ask a specific teammate (expensive call to a busy human), they can self-serve from documentation (fast lookup in cache). An internal knowledge base becomes the single source of truth for information your team needs. It dramatically improves productivity by reducing the time to find answers.

Consider also the role of automation — we can set up bots or search tools that function like a cache server. A Slack bot that answers “@bot how do I request PTO?” with a link to the policy page is essentially an automated cache hit. The information was pre-recorded and is now served on-demand instantly. Without it, someone would ask HR or a manager directly (cache miss, going to slow path). Over time, this can save countless hours and interruptions.

It’s not just for Q&A. Think of runbooks and playbooks for on-call engineers — those are caches of solutions for known issues. When an alert goes off at 3 AM, having a playbook entry is like a cached response for that incident, so the engineer doesn’t have to derive a solution from scratch (and doesn’t need to wake someone else up). This speeds up resolution and lowers cognitive load under pressure.

To effectively cache organizational knowledge, you need to invest in creating and curating documentation. Encourage teams to document decisions (Architecture Decision Records, for example), write post-mortems and make them searchable, maintain onboarding docs. It’s an upfront cost, but it pays off by preventing repetitive questions and mistakes. Imagine a new hire able to follow a step-by-step onboarding guide (cache hit) versus floundering around asking ten different people how to set up their dev environment.

A caution: just like caches can become stale, documentation can too. There should be owners or periodic reviews of key docs to keep them up to date. Also, make the knowledge base easily accessible (intranet search is your friend). If it’s easier to ask a person than find a doc, people will default to bypassing the cache. So invest in good tools (Confluence, Notion, wikis, etc.) and a culture where the first response to a common question is, “It’s in the wiki — let’s make sure that page is easy to find and up to date.”

In essence, document liberally and automate retrieval. By caching knowledge, you free up your experts to do new, creative work instead of answering the same question for the 20th time. Your organization’s “cache hits” will increase — and everyone gets answers faster, with less disruption to the flow of work.

Containerization of Onboarding and Knowledge

Containerization in software bundles an application with all its dependencies so it can run reliably in any environment. It’s about packaging everything needed so a system can be moved or replicated easily. In an organization, the analogous challenge is onboarding new team members or transferring people between teams. We should strive to containerize the onboarding process and institutional knowledge, so that bringing a new engineer up to speed is as smooth as running docker run new-engineer with all necessary context pre-loaded.

In practice, containerizing onboarding means creating a standardized environment for newcomers: documentation, training, access, and tools all set up in a repeatable way. For example, you might maintain a script or checklist that automatically gives a new hire access to repos, sets up their dev environment (maybe even using Docker to avoid “it works on my machine” issues), and provides them with a curated reading list of design docs and team norms. The more this process is automated or at least templatized, the more consistent and fast the ramp-up. One team I led prepared a “new hire container” — essentially a one-stop onboarding guide with links to everything (architecture overview, key contacts, how to set up the dev environment, first-week tiny tasks to commit code). The feedback from new hires was fantastic, because they could self-serve a lot and felt productive within days.

This approach mirrors what happens when you use Docker for development: the environment differences disappear and everyone is up and running quickly. In fact, many engineering orgs now literally use dev containers or virtual machines to distribute a working dev setup. That’s a direct technical application of this principle — no more “spent two weeks installing libraries and debugging environment issues.” Instead, “run this container and you have a fully working local stack in 10 minutes”. The time saved is huge, and it frees the new engineer to focus on learning the system’s behavior and code, not wrestling with setup. It also frees your existing team from spending excessive time helping with setup issues.

Beyond onboarding, think of knowledge packaging for roles. If someone goes on extended leave, did they containerize their responsibilities? This might mean they created a document or brief for their stand-in outlining ongoing tasks, contacts, and tips — effectively handing off a “container” of their role. When teams transfer ownership of a service, do they throw code over the wall or do they hand over a container of context (architecture docs, runbooks, design rationale)? The latter makes transitions much smoother.

Another facet is cultural onboarding — packaging “how we work here” in an easily digestible format. Many companies have an internal handbook (like GitLab’s public handbook approach) that acts as a container of the company culture and processes. A new hire reading that gets a uniform understanding of values and procedures, which might take months to absorb organically. This is similar to how container images encode not just the app code but also environment variables and config that represent how the container behaves.

Ultimately, containerizing in an org is about making knowledge and setup portable and consistent. When you achieve this, scaling the team (adding new people) or moving people around becomes far less painful. Each new person doesn’t create a ton of custom work for others; they plug in more seamlessly. Your organization can thus replicate its “runtime” environment reliably as it grows, maintaining efficiency and developer happiness.

Refactoring the Organization with Metrics (DORA/SPACE)

Good software systems don’t stay static — we need to refactor code to improve maintainability, reduce tech debt, and adapt to new requirements. Likewise, organizations need periodic refactoring. This could mean restructuring teams, changing processes, or updating roles/responsibilities as the product and people evolve. The trick is knowing when and where to refactor an org, and ensuring those changes actually improve things rather than cause chaos. Here’s where metrics like DORA and SPACE come in, as the organization’s monitoring and profiling tools.

DORA metrics (popularized by the DevOps Research and Assessment group) measure software delivery performance across four key dimensions: Deployment Frequency, Lead Time for changes, Change Failure Rate, and Mean Time to Restore. These have become industry benchmarks for engineering team efficiency and quality. If you track these over time, they can highlight when your development process is bogging down. For example, if deployment frequency plummets or lead time skyrockets for a particular team, that’s like seeing a hotspot in a code profiler — something in the team’s structure or process needs refactoring. Maybe the team took on too broad a scope and needs to be split (like breaking a large class into two). Or perhaps a heavy approval process is gating releases, suggesting a need to streamline (refactor the process).

Similarly, a rising Change Failure Rate (more bugs or rollbacks) might indicate quality issues or burnout, possibly stemming from organizational problems (like lack of QA support, or rushed timelines). Mean Time to Restore being high could highlight gaps in on-call structure or tooling. These metrics are leading indicators for organizational performance and even team well-being — they connect to how the team is structured and operating. I’ve used DORA metrics after reorgs to see if the changes improved our velocity and stability, much like running tests after refactoring code to ensure we didn’t break anything.

The SPACE framework complements this by encouraging a multidimensional look at developer productivity: Satisfaction, Performance, Activity, Communication/Collaboration, and Efficiency/Flow. It reminds us not to focus on just one number (like lines of code or hours worked) because productivity is complex. For organizational tuning, SPACE can guide what to measure beyond pure output. For instance, Satisfaction and well-being: if your surveys or attrition rates show declining morale, that’s a sign the org design might need refactoring — perhaps management layers are too heavy, or people don’t have clear career growth (a structure issue). Communication and collaboration: if cross-team projects are consistently delayed due to miscommunications, maybe the org boundaries need adjusting (like aligning teams into a value stream, reducing dependencies — essentially re-architecting modules to reduce chatty interfaces).

When you identify an issue via metrics, treat potential org changes like code refactorings: have a hypothesis, involve the team (just as you’d get input on a significant code change), and iterate carefully. For example, your metrics show Team X has very slow lead time. You hypothesize it’s because they are juggling two distinct product areas — so you propose splitting the team into two focused groups. Measure after the change: did deployment frequency improve? Did people feel better focus (satisfaction up)? If yes, your refactor worked. If not, maybe the problem was elsewhere (perhaps they needed better dev tools rather than a reorg). Like code optimization, don’t jump to conclusions — use data and feedback to guide you.

One caution: metrics should inform, not dictate. Just as we wouldn’t optimize purely for one microbenchmark at the cost of overall design, we shouldn’t chase a metric target without context. The goal is a healthy, high-performing engineering organization, not just pretty numbers. That said, ignoring metrics is flying blind. Combining qualitative insights (from 1:1s, retrospectives) with quantitative metrics (DORA, SPACE) gives the fullest picture. Maybe your deployment frequency is fine, but developers complain in surveys about too many meetings (impacting “Efficiency/Flow” in SPACE) — that might prompt a process refactor to cut status meetings in favor of async updates.

In summary, continuously monitor your organizational design like you monitor a system in production. When you see signs of strain — be it slower delivery, quality issues, or unhappy developers — consider an organizational refactor or tune-up. It might be small (change who approves PRs) or large (realign teams, update roles), but it should be done thoughtfully and tested. A culture of continuous improvement in management, analogous to continuous refactoring in code, will keep your engineering org effective and adaptable as you scale or face new challenges.

Conclusion

The parallels between software architecture and organizational design are surprisingly strong. As an engineering leader, when I encountered a management challenge, I often found clarity by asking: How would I solve this if it were a technical system? Many times, the answer followed an analogous pattern to a well-known system design principle. By treating your org as a living system — with components, interfaces, flows, and feedback loops — you can reason about improvements in a concrete way that makes sense to engineers.

For hands-on engineers moving into management, this perspective is empowering. It demystifies management by leveraging the mental models you already use daily. Designing a reliable, scalable service is not unlike designing a reliable, scalable team. Both require forethought, clarity, and continuous refinement. You wouldn’t deploy a mission-critical service without monitoring, tests, and clear interfaces — so don’t run an organization without 1:1s, checklists, and defined team APIs. You wouldn’t architect a system with single points of failure or no plan for growth — so build team redundancies and be ready to refactor as you scale.

In the end, good engineering management isn’t about fancy jargon or abstract HR theories; it’s often about applying solid engineering principles to human systems. By seeing an org chart as akin to a system diagram, we ensure that our teams are robust, our communication efficient, and our goals aligned. This systems-thinking approach turns management into an extension of the engineering discipline — and it’s a perspective I hope more engineers-turned-managers will embrace to bridge the gap between technology and people. The result can be a healthier organization and better software, hand in hand.

TL;DR

• 2018–2020 — Early Prompting: NLP researchers began casting tasks as text prompts instead of task-specific models, enabling zero-shot and few-shot learning. GPT-3’s 2020 debut showed that giving a large model the right instructions and examples in plain text could outperform many fine-tuned models. This era established basic prompt engineering: provide clear instructions, optionally with a few examples, instead of retraining the model.
• Foundational Principles Emerge: Early guidance from OpenAI and others emphasized clarity, specificity, and context in prompts. Good prompts explicitly describe the task, format, and desired style, use delimiters to separate instructions from data, and frame requests positively (what to do rather than not to do). These principles have remained vital even as models improved.
• 2022 — The Reasoning Breakthrough: Researchers discovered that prompting models to “think step by step” dramatically improved complex reasoning. With Chain-of-Thought (CoT) prompting, models like PaLM 540B could solve multi-step math problems far better by outputting intermediate reasoning before the answer. Simply adding a trigger phrase (“Let’s think step by step”) to a question made even zero-shot prompts much more effective on reasoning tasks, an early example of a “magic prompt”.
• Beyond Chains — Tools and Actions: Late 2022 saw prompts driving actions as well as thoughts. The ReAct paradigm combined CoT-style reasoning with tool use. By interleaving thought steps and action commands (e.g. searches), a prompt could let an LLM not only reason but also fetch new information mid-answer. This reduced factual errors and enabled interactive decision-making beyond the static prompt.
• 2023 — Structured Thought Exploration: Researchers generalized prompting to non-linear reasoning. Tree-of-Thought (ToT) prompting let the model explore multiple reasoning paths like branches, with backtracking and lookahead. Graph-of-Thought frameworks extended this further, modeling each idea as a node in a graph of partial solutions. These techniques address tasks requiring planning or search by evaluating many possibilities instead of one chain of reasoning.
• Fading Tricks, Stronger Models: As models like ChatGPT and GPT-4 (2022–2023) were trained to follow instructions better, the need for elaborate “prompt hacks” diminished. Simple, well-structured prompts often work as well as convoluted ones for these models. For example, GPT-4 can often reason correctly without an explicit “step by step” cue (it does so internally), making the basic CoT trigger less of a secret weapon. Many one-off “magic prompts” or role-play personas from early days are now unnecessary or even counter-productive on modern aligned LLMs.
• 2024+ — Prompt Automation and Agents: Recent efforts focus on automating prompt engineering itself. Techniques like Automatic Prompt Engineer (APE) use LLMs or evolutionary algorithms to generate and refine prompts for a given goal, instead of manual trial-and-error. Meanwhile, LLM-based agents (e.g. AutoGPT-style systems) string together multiple prompts and model outputs to handle complex, multi-step tasks with minimal human intervention. Prompt engineering now includes designing high-level instructions, roles, and iterative loops so that AI systems can break down tasks, call tools, self-correct, and decide on next actions autonomously.
• Key Takeaway: Prompting has evolved from a simple art of asking a question to a structured engineering discipline. Many early tricks have given way to robust frameworks (CoT, ReAct, ToT) and the core lesson is clear — clarity and reasoning structure matter. Today’s practitioners focus more on decomposing tasks and giving precise instructions, while leveraging advanced prompting methods (or letting the AI help generate them) for the best results.

Early Era (2018–2020): From Zero-Shot to Few-Shot Learning

The concept of prompting as we know it arose alongside the first large pre-trained language models. Instead of fine-tuning a model for each task, researchers wondered: what if we just asked the model to perform the task via text input? This idea was hinted at in 2018, when the Natural Language Decathlon (decaNLP) project showed multiple NLP tasks could be reformulated as a single question-answering model. Around the same time, OpenAI’s GPT-2 (2019) demonstrated surprising zero-shot abilities: for example, GPT-2 could generate a decent summary of an article just by appending TL;DR: at the end of the text. This was a revelation – the model had implicitly learned the concept “TL;DR means summarize” from its training data. Such results hinted that language models can follow prompts without additional training, given the right phrasing.

The real breakthrough came with GPT-3 in 2020. The GPT-3 paper “Language Models are Few-Shot Learners” showed that a 175B-parameter model could perform new tasks with just a well-crafted prompt containing a handful of examples. For instance, to get GPT-3 to translate English to French, you could simply prompt it with:

Translate English to French:
English: I have a pen. 
French: J'ai un stylo.

English: How are you?
French: Comment ça va?
English: This is a beautiful day.
French:

And GPT-3 would complete the pattern with the correct French translation. This few-shot prompting worked for translation, question answering, arithmetic, and more — all without updating the model’s weights. It became clear that in-context learning was a powerful alternative to explicit training. GPT-3 also handled zero-shot prompts surprisingly well for many tasks (just an instruction with no examples), though performance was usually a bit lower than with a few examples.

Along with these discoveries, the term “prompt engineering” started entering the lexicon around 2020. Practitioners realized that how you phrase the input can make a huge difference in output quality. Early GPT-3 users swapped tips on getting the model to follow instructions: e.g. start with a clear command ("Summerize the following text"), provide necessary context, ask for output in a specific format, etc. By late 2020 and 2021, community blogs and OpenAI’s own guidance had compiled best practices for effective prompts:

• Be specific about the task and format: Don’t assume the model knows what you want. Explicitly tell it your expectations for content, style, length, format, etc. (e.g. “Write a one-paragraph executive summary outlining three key points.”). Specific prompts narrow down the model’s output path and improve reliability.
• Provide context or examples: If the task input needs context, include it in the prompt with clear delimiters. For example, prepend a passage with Text: and then ask a question about it. Or give a couple of QA pairs as demonstrations (few-shot) before the real query. High-quality examples can significantly steer the model.
• Use delimiters and structure: Early on, users found that separating instructions from data with symbols (XML tags, triple quotes, etc.) helped prevent confusion. For instance:
  Instruction: Summarize the text below in one sentence.
  Text: """ {article text here} """
  Summary:
  Such structure makes it obvious what is prompt and what is content.
• Prefer positive phrasing: Instead of only saying what not to do (“Don’t produce long output” or “Don’t mention X”), it worked better to say what to do (“Keep the answer concise”). Models tended to be less obedient to negations; guiding them toward the correct behavior was more effective.
• Iterate and refine: Prompt design was acknowledged as an iterative process. If the first attempt isn’t ideal, one should analyze the output and adjust wording, add an example, or break the task into smaller parts. Practitioners treated prompts as living instructions that could be tuned over multiple tries.

By 2021, these foundational techniques allowed users to coax impressive results from models like GPT-3, which were otherwise unaltered from their pre-training. Still, there were clear limits. While you could prompt GPT-3 to do many things, certain complex tasks (e.g. multi-step logic puzzles, math word problems) revealed a shortcoming: the model often jumped straight to an answer and made mistakes, because the prompt didn’t force it to work through the problem. Few-shot prompts that only showed input-output pairs weren’t teaching it to think, just to mimic answers. This realization set the stage for the next era of prompt engineering, where the focus shifted to eliciting the reasoning process, not just the final answer.

2022: Chain-of-Thought — Teaching Models to Think Out Loud

2022 was a pivotal year. Researchers at Google discovered that large LLMs could perform much better on reasoning tasks if prompted to “think out loud”. The technique, called Chain-of-Thought (CoT) prompting, was first reported in early 2022. Instead of asking the model for just the answer, the prompt includes a few examples where the solution is arrived at step-by-step. For example, a CoT prompt for a math word problem might look like:

Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now?
A: Roger started with 5 balls. 2 cans of 3 tennis balls each is 2 * 3 = 6 balls. So he has 5 + 6 = 11 balls. The answer is 11.

Q: The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many apples do they have now?
A:

Here, the example answer explicitly walks through each step of the calculation (highlighting the arithmetic and reasoning) before giving the final answer. Seeing this chain-of-thought in the prompt encourages the model to produce a similar step-by-step solution for the next question. Crucially, this approach was found to massively improve accuracy on multi-step problems compared to standard prompting.

Google’s researchers demonstrated CoT’s impact with the 540-billion parameter PaLM model. With normal prompting, PaLM’s performance on a math word problem benchmark (GSM8K) barely improved with scale. But with chain-of-thought prompting, PaLM at 540B solved 58% of the problems — outperforming even a fine-tuned 175B GPT-3 model that had specialized training for the task. CoT essentially unlocked an emergent reasoning capability that only became reliable in the largest models. It showed that beyond a certain model size (~100B+ parameters), prompts could induce surprisingly advanced behaviors.

Moreover, an interesting hack was discovered mid-2022: you might not even need example chains of thought — just a trigger phrase. Kojima et al. found that by appending “Let’s think step by step” to a question, you can often activate the model’s latent reasoning ability in zero-shot mode. This Zero-Shot CoT technique boosted accuracy dramatically on many tasks (e.g. solving a simple math test went from 17.7% to 78.7% accuracy by adding that prompt to an InstructGPT model). Essentially, the phrase caused the model to output a chain of reasoning and arrive at a correct answer, even without any worked examples in the prompt. It became a widely cited “magic phrase” in prompt engineering circles — a prime example of how a small wording change can tap into an LLM’s hidden capabilities.

The success of CoT led to a flurry of related ideas to improve reliability and performance:

• Self-Consistency: Instead of trusting a single chain-of-thought, Wang et al. (2022) had the model generate multiple reasoning paths and then took a majority vote on the final answer. This method, called self-consistency, dramatically improved accuracy by averaging out reasoning mistakes. For GSM8K, self-consistency with CoT raised PaLM 540B’s score from 58% to 74%. The intuition is that while any single chain might have an error, the most common answer across many independent chains is likely correct.
• Least-to-Most Prompting: Another 2022 innovation was to explicitly break down a hard problem into a sequence of simpler sub-problems — and prompt the model to solve those one by one. In Least-to-Most prompting, the model is first asked to solve the easiest part of a problem, then incorporate that result to tackle the next part, and so on. This was shown to help with tasks that require compositional reasoning or problems where a direct chain-of-thought might get stuck. It forces the model to progressively build up to the full answer, mimicking how a human might tackle a complex problem by splitting it into steps.
• Better Prompting for Arithmetic & Logic: Researchers also explored variations like giving the model a “scratch pad” or asking it to produce structured reasoning (e.g. list out assumptions, do calculation, then conclude). All these are essentially prompt patterns designed to guide the model’s internal thought process. They underscore a key point: prompt engineering had expanded from what the answer should be to how the model should derive it.

By the end of 2022, Chain-of-Thought prompting had become a cornerstone technique. It was clear that for any task involving reasoning — math, logic puzzles, commonsense questions — instructing the model to show its work was hugely beneficial. That transparency also had a side benefit: users could now see the model’s intermediate thinking (printed out as part of the answer), which made it easier to debug or trust the results. CoT transformed the prompt engineer’s role from just crafting queries to also scaffolding the model’s reasoning process.

However, one limitation of basic CoT was that the model’s knowledge was still limited to what it had seen in training. It would diligently reason step by step, but if it needed an external fact (say, “Who is the president of X country?” in a multi-step question), it might either make up a fact or get stuck. Also, CoT by itself didn’t allow interacting with the world or taking actions — it was purely internal reasoning. This led to the next development: prompts that not only think, but also act.

Late 2022: ReAct — Reasoning + Acting with Tools

One of the landmark techniques bridging reasoning with action was ReAct, introduced by Yao et al. in late 2022. ReAct stands for “Reason + Act”, and it extends the chain-of-thought idea by interleaving reasoning steps with the ability to use tools or take actions. In a ReAct prompt, the model is shown examples of a thought → action → observation cycle. For instance, given a question, the model might output a thought like “Hmm, I should look up X,” then an action like calling a search API, then get an observation result, then continue reasoning based on that.

The key insight of ReAct is that an LLM can function like an agent in a larger system: it can decide what action to take (e.g. query a knowledge base, run a calculator, etc.) as part of its output, not just produce a final answer. The prompt demonstrates the format for this, typically something like:

Thought 1: <the model’s first thought on how to solve the task>
Action 1: <an action command, e.g. SEARCH[Apple Remote]>
Observation 1: <the result of the action, e.g. content from Wikipedia>
Thought 2: <model’s thought after seeing the result>
Action 2: <next action...>
...
Thought N: <final thought leading to the answer>
Action N: <Finish>
Answer: <the final answer>

The model, following this ReAct prompt pattern, generates a sequence of interleaved thoughts and tool uses. This allowed it to overcome two big issues of plain CoT: hallucination and limited knowledge. If the model isn’t sure about a fact, the ReAct format encourages it to explicitly take an action (like a lookup) to gather evidence, rather than guessing. And by examining the chain of thoughts and actions, users can trace how the model arrived at an answer, increasing transparency.

For example, suppose the question is: “Aside from the Apple Remote, what other devices can control the program that the Apple Remote was designed to interact with?” A ReAct interaction might go like:

• Thought 1: “I need to find what program the Apple Remote was designed for.”
• Action 1: Search["Apple Remote originally designed to control"]
• Observation 1: Returns info: “The Apple Remote … originally designed to control the Front Row media center program…”
• Thought 2: “The program is Front Row. The question asks: aside from the Apple Remote, what other devices can control Front Row?”
• Action 2: Search["Front Row other devices control"]
• Observation 2: Returns info about Front Row being controllable via keyboard, etc.
• Thought 3: “It appears Front Row can be controlled by an Apple Remote or the keyboard’s function keys.”
• Action 3: Finish
• Answer: “It can also be controlled by keyboard function keys.”

In essence, ReAct turns the prompt into a script for interaction, not just a one-shot query. It was a stepping stone towards autonomous AI “agents”. Empirical results showed ReAct outperformed standard CoT on tasks like factual question answering (where external lookup helps) and game environments that require decision making. For instance, on a HotpotQA fact-finding task, a ReAct-prompted model could search Wikipedia for each clue and achieve higher accuracy with fewer hallucinations.

ReAct also highlighted a more general trend: prompt engineering was now about designing entire process flows, not just single-turn prompts. A prompt could establish a loop where the model’s output is fed back in (as tool input) repeatedly. This idea of looping prompts laid groundwork for later automation.

By early 2023, the community had embraced both CoT and ReAct patterns (often combining them). Open-source frameworks emerged to implement these prompting patterns easily (e.g. LangChain provided templates for ReAct-style agents). It became common to refer to “agentic” use of LLMs — meaning treating the model as an agent that can think (CoT) and act (ReAct) iteratively to accomplish a task. The prompt engineer’s role expanded to writing meta-prompts that instruct the model when to use tools, how to format thoughts, when to stop, etc., effectively programming the model’s behavior through exemplars.

One consequence of these advances is that some earlier prompt tricks became less relevant. If you have an agent that can look things up, you no longer need to front-load as much knowledge into the prompt itself. And as models got better at not hallucinating trivial facts (due to instruction tuning and RLHF), the need to always force a tool use diminished for simpler queries. The balance between trusting the model’s internal knowledge vs. invoking an external tool became something a prompter could control via the ReAct pattern.

2023: Beyond Linear Chains — Tree-of-Thought and Graph-of-Thought

Chain-of-thought prompting proved that a linear sequence of reasoning steps can greatly enhance an LLM’s performance on complex tasks. But humans don’t always think in single straight lines — we consider alternatives, backtrack, and explore different approaches, especially for hard problems. In 2023, researchers proposed new prompting frameworks to enable non-linear reasoning in LLMs:

• Tree-of-Thought (ToT): Introduced by Shunyu Yao et al. in early 2023, ToT generalizes CoT by allowing the model to branch out into multiple possible thoughts at each step. Instead of a single chain, the model generates a tree of thought sequences. A search algorithm (like breadth-first search or depth-first search) then explores this tree to find a promising path to the solution. The prompt for ToT might involve instructions like “consider multiple approaches” and even examples where the model enumerates Options A, B, C at a decision point. By evaluating partial solutions (pruning branches that look unpromising and expanding fruitful ones), ToT lets the model do strategic lookahead. For example, in the Game of 24 puzzle (where you must reach 24 with arithmetic operations on given numbers), ToT prompting enabled GPT-4 to try different sequences of operations; one paper reported GPT-4’s success rate jumped from near-zero to 74% with ToT on that task. The model could effectively say “If I do this, then that… does it get me closer? If not, backtrack.” — a capability emergent from the way the prompt and search procedure were set up.
• Graph-of-Thought (GoT): Pushing the idea even further, later in 2023 another framework posited that we can model the LLM’s reasoning as a general graph structure (not just a tree). In Graph-of-Thought prompting, each thought is a node and edges represent dependencies or transitions between thoughts. This allows for even more flexible exploration — for instance, the model might revisit a prior idea and take a different route, forming loops or merging two lines of reasoning. The GoT approach treats the prompt and the model’s outputs as constructing a network of connected thoughts, which can be traversed in various ways to reach an answer. In practice, implementing GoT might involve multiple rounds of prompting: e.g. first prompt the model to generate a bunch of candidate thoughts/nodes, then prompt it to suggest connections or next steps from those nodes, and so on, orchestrated by an external algorithm. The researchers behind GoT argued it offers a better trade-off between the “breadth” of exploration and the “depth” of reasoning within a fixed number of model calls. In plain terms, a graph-based approach could find solutions with fewer attempts than either a single chain or an exponentially growing tree, by smartly reusing partial results.

These advanced prompting strategies (ToT, GoT) remained largely in the research realm in 2023, but they signaled an important direction: prompt engineering was no longer confined to writing a single prompt and getting one answer. It was evolving into designing algorithms that call the model multiple times with carefully constructed prompts at each step. The prompts themselves might contain instructions for the model to critique or evaluate intermediate answers (“Judge if this partial solution is on the right track: …”) — effectively turning the model into a participant in a search procedure.

Another benefit of tree/graph-based prompting is robustness. Instead of relying on one chain-of-thought which could go wrong, exploring many thoughts increases the chance of finding a correct solution. This is analogous to ensemble methods in traditional AI. Even though it’s more computationally expensive (multiple model queries), it can substantially improve reliability on very hard tasks (e.g. puzzles, planning, coding challenges).

By the end of 2023, we had a spectrum of prompting techniques for reasoning:

• Zero-shot vs Few-shot, for straightforward tasks depending on whether examples are needed.
• Chain-of-Thought, to force stepwise reasoning in a single answer.
• Self-consistency, to sample many CoTs and take a consensus.
• ReAct and tool-use, to allow looking up information or interacting with environments.
• Tree-of-Thought, to systematically explore different solution paths.
• Graph-of-Thought, to generalize and optimize the exploration.

Not every technique was needed for every problem — simpler tasks still did fine with basic prompting, and very powerful models (like GPT-4) could handle many reasoning tasks with just a direct prompt. But these innovations armed prompt engineers with a toolkit of approaches to push the limits of what LLMs could do.

It’s worth noting that many of these techniques overlap or can be combined. For instance, one could do a tree-of-thought search where each node’s content is generated using a chain-of-thought prompt. Or use ReAct within a graph-of-thought framework so that the model can fetch facts during the search. The lines between “prompt engineering” and “algorithm design” blurred — prompting became a centerpiece in building complex AI systems without modifying the LLM’s weights.

2024–2025: Meta-Prompting and Automation

As we moved into 2024, two things became evident: (1) models were getting even more capable (e.g. new versions like GPT-4, Claude 2, etc.), often handling instructions with ease, and (2) the manual craft of prompt writing was turning into a bottleneck — people wondered if AI can help us prompt AI. This has led to a focus on meta-prompting and automated prompt engineering:

Instruction-Following and Simpler Prompts: The release of ChatGPT (GPT-3.5) in late 2022 and GPT-4 in 2023 changed everyday prompting. These models were fine-tuned on instructions and dialogues, making them much better at following plain-English directions without needing clever phrasing or examples. Tasks that used to require a few-shot prompt on GPT-3 could often be done zero-shot with a single instruction on GPT-4. For practitioners, this meant that the emphasis shifted back to clear and precise language, rather than finding obscure trigger phrases. In fact, some “over-engineered” prompts from earlier might even confuse the newer models. For example, whereas GPT-3 might have needed a lengthy role-play setup (“You are an expert chatbot…”) to produce a helpful answer, ChatGPT already behaves helpfully by default, so a direct question often suffices. The core principles — clarity, context, and specificity — remained as important as ever, but the need for workaround tricks diminished. Many in the field began cautioning against prompt “cargo culting” (using elaborate presets that aren’t well understood); instead, they advised focusing on the basics: tell the model what you want clearly and it will likely do it. In short, models got better at understanding us, so we could speak more naturally to them.

Automatic Prompt Optimization: A fascinating development has been the rise of Automatic Prompt Engineering (APE) tools. The idea is to use algorithms (including the LLM itself) to search for an optimal prompt for a given task. For instance, one can use a genetic algorithm to evolve a prompt: generate a bunch of random prompt variants, test them on a validation query, keep the ones that perform best, mutate and repeat. Alternatively, one can turn the model on itself: ask the LLM to propose prompts and evaluate them. Research projects have found that LLMs can iteratively refine prompts if given a goal, a few test cases, and some feedback signals. There was even work showing an automated method found a better prompt for arithmetic reasoning than “Let’s think step by step” — essentially discovering new trigger phrases or instructions through trial and error. Companies like Anthropic started offering a “prompt improver” feature, where their AI (Claude) suggests ways to rephrase your prompt for better results. All this points toward a future where machines help us communicate with machines. Instead of a human manually trying dozens of prompt variants, we might have an agent that does this and comes back with the best-performing prompt.

Multi-Prompt Orchestration and Agents: By 2024, the community was abuzz with “AI agents” — systems like AutoGPT, BabyAGI, etc., which string together many prompts and actions to accomplish user goals. These are essentially an extension of the ReAct idea, scaled up: the model not only reasons and uses tools, but also can spawn new goals or sub-tasks and prompt itself again and again until a final objective is met. For prompt engineering, this means designing a robust meta-prompt (or “controller” prompt) that tells the agent how to operate. Typically, a system prompt might define roles (e.g. “You are a project manager AI that can break down tasks. You have access to these tools…”) and the agent maintains a loop of planning → executing (via prompts to itself or another model) → reviewing results → updating the plan. This is still an emerging area, but it shows how prompts can be used to establish entire automated workflows. The human’s job shifts to writing high-level instructions that guide the agent’s behavior over multiple steps, rather than micromanaging each step.

One implication of agentic prompting is the need for managing memory and context over long interactions. Prompt engineers now consider how to feed the model relevant information from previous steps (often by summarizing or using a moving window of dialogue). Concepts like a “scratchpad” or dedicated memory slots in the prompt have become more common. For instance, a prompt may have a section like:

## Plan:
{the agent’s current plan}

## Actions History:
- Thought: ...
- Action: ...
- Observation: ...
(repeated)
## Current Thought:
...

This is effectively a structured prompt that grows over time. Designing it well is crucial so that the model doesn’t run out of context or get confused. We also see tool libraries being defined in prompts (telling the model what actions it can take and the format to use), which is a new kind of prompt engineering challenge (it’s almost like giving the model an API spec within the prompt).

Personalization and Dynamic Prompts: Another frontier is making prompts that adapt to the user or situation. Instead of a static prompt template, developers use code to assemble prompts on the fly, pulling in user profile data, context from earlier interactions, etc. For example, a customer support bot might have a prompt that includes placeholders for the user’s name, their purchase history, the last thing they asked, and so on, before the main instruction. Ensuring the prompt stays coherent and doesn’t overflow length becomes a design task. There’s also research on having the model itself manage the prompt — for instance, letting the model decide to summarize earlier dialogue when context gets long, effectively learning to write its own condensed prompt. These “self-reflection” or “self-summary” techniques blur the line between model and prompt, but they are all part of handling complex interactions.

Lastly, with models extending beyond text (vision, audio), prompt engineering is expanding too — though that’s beyond our scope here. Multimodal prompting introduces its own techniques (like combining text prompts with image embeddings or instructing the model to describe an image then answer a question). But the text-based prompting principles largely carry over: clarity, providing context (which could be an image description), and step-by-step guidance remain key.

Enduring Principles: What Has Stayed Important

Despite the rapid evolution of prompt engineering methods, some fundamentals have remained constant from 2018 to 2025:

• Clarity and Specificity: A well-specified prompt is always better than a vague one. This was true in 2018 and is true now. Models are giant pattern-matchers — if your prompt is ambiguous, the model has too many patterns it could follow, leading to unpredictable outputs. Clear instructions narrow down the possibilities. For example, saying “List 3 key takeaways from the article in bullet points” is far more likely to produce a useful result than “Summarize the article”, which could output anything from one sentence to five paragraphs. All the fancy techniques (CoT, ToT, etc.) still require the prompt to precisely communicate the task.
• Context Provision: Providing the right context — whether it’s relevant facts, conversation history, or examples — is crucial. Early few-shot prompts literally embed context as examples. Today, even with powerful models, if the question depends on specific knowledge (say details from a user’s document), you need to include that text or have the model retrieve it. The method of delivery might differ (manual copy-paste in 2020 prompts vs. tool-assisted retrieval in 2025 agents), but the principle is the same: the model can only work with what it’s given. Prompt engineers spend a lot of effort deciding what information to put into or alongside the prompt.
• Step-by-Step Decomposition: Tackling complex tasks by breaking them into steps has proven consistently effective. Initially we did this implicitly via few-shot examples. Later we did it explicitly with Chain-of-Thought, least-to-most, etc. The names differ, but the idea is enduring: guide the model through the task one step at a time. Even if we don’t always have to say “Think step by step” anymore, under the hood the model (especially large ones) often does multi-step reasoning. When it fails, a common remedy is still to prompt it to organize its approach (e.g., “First, outline a plan. Then solve each part.”).
• Format and Structure Cues: Models respond well to formatting cues. This was seen early with things like ```markdown or HTML in prompts to extract info. Today, if you want JSON output, you can literally say “Output JSON” and the model will (usually) comply. Or you can provide a template in the prompt. The importance of showing the desired output format (with either examples or clear markup) is a lasting lesson. Structured prompts lead to structured outputs.
• Iterative Refinement: From the beginning, practitioners treated prompt engineering as iterative — try something, see what the model does, and refine. This remains true, though tools have improved to help the cycle. Even with automation, a human often needs to review outputs and adjust the approach. The model might misunderstand an instruction in a subtle way; only by analyzing its output can you pinpoint how to clarify the next prompt. Essentially, debugging prompts is still a core skill, much like debugging code.

In summary, a prompt that is clear about what to do, provides any needed context or examples, and perhaps guides the model’s thinking process, has been the gold standard throughout the years. New techniques have augmented this, but not replaced it. When people ask “What’s the secret to prompting?”, the answer has stayed quite stable: be clear, be specific, and show the model what you want. All the evolutions we’ve seen build on that foundation.

Evolving Strategies: What Changed and What Faded

While core principles stayed, many strategies have been refined or replaced as LLMs advanced:

• Few-Shot Prompting vs. Instruction Tuning: In 2020, few-shot examples were the primary way to get models to perform tasks. By 2023, models like GPT-4 were so well instruction-tuned that they often didn’t require examples at all for straightforward tasks. Providing 5 examples sometimes made GPT-4 worse (due to over-specifying or hitting context limits) where GPT-3 would have needed them. Thus, the art shifted to knowing when a model needs examples and when it’s better to just instruct plainly. Few-shot prompting is still useful for very open-ended generation or when you have a specific style to mimic, but it’s less universally needed for correctness now.
• “Magic” Phrases and Prompts: The era of discovering one-size-fits-all prompts (like the famous “Let’s think step by step”) was relatively short-lived. These cues were effective on certain model generations (especially those not explicitly trained for following instructions), but as soon as models incorporated those capabilities through training, the magic phrases lost their magic. In fact, developers started hard-coding filters so that models wouldn’t reveal internal chain-of-thought unless asked, which made the simple “think step by step” less useful unless you structure the prompt to expect a chain-of-thought format. Similarly, elaborate personas or boilerplate like “You are a helpful assistant…” became unnecessary for models already aligned to be helpful. The field matured past secret incantations towards understanding why a prompt works. Many early prompt “tricks” were essentially exploiting quirks of a model’s training data; once architecture and training improved, those quirks either vanished or became built-in features.
• Reduced Prompt Length/Complexity: In the early GPT-3 days, one might prep a huge prompt with detailed role, multiple examples, and so on, to maximize performance. Now, with larger context windows and smarter models, prompt engineers learned that conciseness can be an advantage. Unnecessary verbosity can confuse the model or introduce more room for error. A concise but precise prompt often outperforms a long-winded one. There’s been a move toward minimal effective prompts — giving exactly the information needed and nothing more. This doesn’t contradict providing context; it means provide relevant context but avoid extraneous instructions. For example, instead of a 500-word prompt coaxing the model gently into an answer, today one might use a 50-word prompt that gets straight to the point (especially with GPT-4 class models). The output quality doesn’t suffer, and it’s easier to maintain.
• Prompt Fragility and Robustness: Earlier models could be very sensitive to prompt wording (a single word change might swing the output). This led to some fragile prompts that only worked in one form. Newer models are more robust — they catch synonyms or rephrasings better. As a result, prompt engineering is a bit more forgiving now. You don’t need to hunt for the exact perfect phrasing as much; a range of phrasings will yield similar results. That said, for tricky tasks, slight rewording can still matter, so testing variants isn’t obsolete — it’s just that the variance is lower with advanced models.
• Integration of Tools and Memory: Where once prompt engineering dealt with a single static input->output transaction, now it often deals with a sequence. Designing how the prompt changes after each step (e.g. appending the last Q&A to the context, or updating a “knowledge base” section of the prompt) is a new skill set. This was not relevant when using GPT-3 via the API in single turns, but with conversational and agent systems it’s become key. In effect, the prompt is now a living object that evolves, and engineering it includes deciding what to keep or discard at each turn. Techniques like summarizing earlier dialogue to save space, or re-injecting important facts into each new prompt, have become standard in building long-running conversations or multi-step workflows.
• Emergence of Prompt Standards and Taxonomies: In 2021, “prompt engineering” felt like folk wisdom and scattered tricks. By 2025, it’s a more systematized field. We see formal taxonomies of prompt techniques in academic surveys, and even books dedicated to prompt engineering best practices. This maturity means practitioners today have a clearer map of options. For example, if the task is a reasoning problem, the taxonomy might suggest: try standard CoT, if not sufficient try self-consistency, if external info needed try ReAct, etc. The trial-and-error is guided by collective knowledge now, whereas early prompt engineers often had to invent solutions from scratch for each new challenge. In essence, strategies have been codified, and some have been phased out (you rarely see advice to use the “**” trick or other early GPT-3 oddities anymore) while others have been codified as recommended patterns.

Practical Implications for Today’s Practitioners

Fast-forward to 2025: what does all this mean for someone working with LLMs? In practical terms:

• Start Simple: Given how capable models are, often the best initial prompt is a straightforward one. Don’t over-engineer from the get-go. Clearly state the task in plain language, provide context, and maybe the desired format. You might be surprised that this “naive” prompt already works on a modern model. If it doesn’t, then incrementally add complexity (perhaps break the task into steps, or add an example). This approach prevents unnecessary prompt bloat and keeps outputs more predictable.
• Use Advanced Techniques Judiciously: Techniques like CoT, ReAct, ToT are powerful but not always needed. They also introduce complexity in parsing the output (e.g., if the model is spitting out thoughts and actions, you need to handle that). So, identify if the task truly requires it. If the task is straightforward — e.g., summarization, classification, straightforward Q&A — a direct prompt is usually enough. If the task involves reasoning or multi-step answers, consider CoT or self-consistency. If it requires external knowledge beyond the model’s training, use a ReAct/tool approach or provide the knowledge in the prompt. Essentially, match the method to the problem.
• Maintain Control of the Format: One thing that hasn’t changed is that LLMs will follow the path of least resistance in output unless guided. So always specify the format you need, especially if the output will be consumed by a program. Whether it’s JSON, bullet points, or a formal report style, telling the model how to output is as important as telling it what to output. Fortunately, models now are quite good at adhering to format instructions (even complex ones) if you clearly delineate them.
• Be Mindful of Token Limits and Cost: With very large prompts (from including many examples or long histories), remember that context is finite and tokens are money (for API usage). Part of prompt engineering today is actually prompt trimming — deciding what not to include. The best prompt is concise but complete. Techniques like summary of context or dynamically retrieving just the relevant pieces of information for inclusion can help manage this. The trend towards retrieval-augmented prompting (though we haven’t discussed it in depth here) is one way to keep prompts focused by injecting only pertinent facts. Even without fancy retrieval, you should curate your prompt content to avoid overloading the model with irrelevant text.
• Leverage System and Role Prompts: In chat-oriented models, use the system message (or equivalent) to set the high-level behavior, and use role-playing or few-shot dialogue in the prompt if it helps the model’s performance. For example, if the model tends to be too verbose, a system instruction like “You are a concise assistant” can globally affect its style. These features didn’t exist in earlier models, so take advantage of them — they are basically an easier way to enforce certain prompt conditions without repeating them every turn.
• Continuously Evaluate and Adapt: The field is still moving quickly. New models might render some techniques obsolete or introduce new possibilities. For instance, if a future model can internally browse the web, you might not need to prompt it with explicit search actions (it might do so on its own from a high-level ask). Staying informed through literature and community resources is now part of a prompt engineer’s job — what worked last year might not be optimal this year. That said, having a strong grasp of the why behind each technique will allow you to adapt those concepts to any new prompting paradigm.

In conclusion, prompt engineering evolved from a niche art into a critical skill for harnessing LLMs. We journeyed from simple one-shot queries to complex, multi-turn orchestrations. Yet, the heart of it remains using language effectively to communicate with the model. As LLMs get ever closer to understanding our intent, the “engineering” may gradually give way to just asking AI systems for what we want in intuitive ways. But until that day, understanding the history and toolbox from 2018–2025 gives us a rich foundation to craft prompts that truly unlock these models’ potential — reliably, transparently, and efficiently.

• AI assistance has reshaped coding: Tools like GitHub Copilot and new AI coding agents are accelerating software development, but they also change the nature of a developer’s work and output. Traditional metrics (e.g. lines of code or hours logged) do not tell the full story when an AI can generate chunks of code on demand.
• Old metrics fall short: Relying on a single metric for productivity is misleading. For example, a developer can write less code by hand yet deliver features faster with AI support — or conversely produce heaps of AI-generated code that don’t translate to real progress. We need a more nuanced approach beyond “one metric that matters.”
• Many AI IDEs now include metrics dashboards: Tools like GitHub Copilot, Cursor and Windsurf expose usage stats like suggestion acceptance rate, percentage of code written by AI, and tab completions accepted per day. Devin also enables task-level success tracking via PR merge rates or internal benchmarks. These built-in metrics — while still evolving — are key to measuring not just developer performance, but AI agent performance itself.
• The SPACE framework offers a holistic lens: It defines five dimensions of developer productivity — Satisfaction (developer happiness), Performance (outcomes/quality), Activity (volume of work), Communication & Collaboration, and Efficiency & Flow. It reminds us that productivity isn’t just output; it’s also how developers feel and work together.
• DORA metrics focus on delivery outcomes: The DevOps Research and Assessment (DORA) framework tracks four key metrics of software delivery performance — Deployment Frequency, Lead Time for Changes, Change Failure Rate, and Time to Restore Service. These “four keys” measure how quickly and reliably teams ship code, which remains crucial in the age of AI-assisted development.
• AI can improve multiple metrics: Research shows AI coding tools can speed up coding tasks by 20–55% and boost developer satisfaction (e.g. reducing frustration and preserving “flow” state). But to truly measure productivity gains from AI, organizations are combining metrics — for example, tracking faster delivery and higher developer satisfaction rather than just counting lines of AI-generated code.
• Adopt a balanced measurement approach: In practice, the most successful teams use a mix of quantitative data and qualitative insight. They monitor objective outcomes like deploy frequency or bug rates alongside developer experience. This balanced scorecard prevents skewed incentives and ensures that “AI-powered” productivity means better software and happier engineers, not burnout or technical debt.

From Assistance to Autonomy

In early 2022, I remember experimenting with GitHub Copilot. It felt like a superpower, yet it was clearly an assistive tool: I was still driving, and the AI was the co-pilot. Fast forward to today: my development workflow is teeming with far more powerful AI aides. At work I use Cursor as my daily editor (a VS Code fork with robust AI capabilities), and for personal projects I’ve switched to RooCode (an open-source AI code assistant, forked from Cline). I occasionally summon Devin, an “AI software engineer” that lives in Slack, just to push the boundaries of what autonomous coding feels like. In short, we’ve gone from simple auto-complete assistance to partially autonomous coding agents.

The spectrum of AI coding tools: This evolution from assistance to autonomy is stunning. With Cursor or RooCode, I can highlight a chunk of code and ask the AI to refactor it, or generate a new file from a description. Devin takes it a step further: I can tag a bot in a Slack thread and ask it to implement a feature or fix a bug, and it will spin up a remote environment to write and test code for me. These AI agents are not perfect — I still review their output and often guide them — but they’ve changed my role. I’m no longer writing every line from scratch; I’m describing intent, validating AI outputs, and coordinating high-level tasks. In other words, the developer’s job is shifting toward supervising coding done in tandem with AI.

Productivity in this context is both amplified and complicated. On one hand, I can deliver features faster. For example, GitHub’s own studies found that developers using GitHub Copilot completed a task 55% faster on average than those without it. And McKinsey reported that tools like GitHub Copilot and ChatGPT have the potential to help engineers complete tasks up to 2× faster. When Google’s CEO says that over 25% of new code at Google is now generated by AI, it signals just how much more output each developer can achieve with machine help. But does more code equal higher productivity? Not necessarily. If an AI writes 1,000 lines that I then spend all day debugging, was that productive? If a task used to take me a week and now takes 2 days with AI, what do I do with the “saved” time — deliver twice as much, or spend the extra time polishing quality? These questions show why measuring productivity in the LLM era isn’t straightforward.

The Limits of Traditional Metrics

For years, managers and engineers have relied on metrics like lines of code written, number of commits, or story points completed to quantify developer productivity. These metrics are still useful — especially for spotting patterns or establishing baselines — but relying on them alone can be misleading.

The research community has also cautioned against one-dimensional metrics. Nicole Forsgren et al. (the experts behind the SPACE framework) note that productivity cannot be reduced to a single dimension — too often teams fixate on one proxy and miss the bigger picture. In fact, one of the most persistent myths is “productivity = activity”. Sure, AI can boost activity (more lines, more PRs). But as Forsgren explains, a spike in commit count might mean a developer is stuck wrestling with a problem, or conversely might reflect great tools that enable rapid iteration. The point is, activity alone doesn’t guarantee useful outcomes.

So how should we measure developer productivity in the LLM-assisted world? The consensus emerging in industry is to use multiple measures that capture both the process and outcomes of development. In particular, two frameworks — SPACE and DORA — still remain invaluable today. SPACE provides a holistic, human-centric view, while DORA focuses on the engineering outcomes that teams deliver.

The SPACE Framework: A Holistic View of Productivity

SPACE is a framework introduced by researchers at GitHub and Microsoft in 2021. The name is an acronym for five dimensions:

• Satisfaction & Well-Being: How happy, fulfilled, and healthy developers feel in their work. This can be measured through surveys, self-reported morale, or qualitative feedback. It recognizes that a productive team is one where developers aren’t burned out or frustrated; in fact, studies show satisfaction and productivity go hand-in-hand. In the context of AI, this dimension asks: Does the AI tool make developers enjoy their work more or less? Interestingly, GitHub’s research found that 60–75% of Copilot users reported feeling more fulfilled and less frustrated when coding with AI assistance — an example of improved satisfaction that might not show up in code metrics but is critical for long-term productivity. However, as tools become more autonomous, the satisfaction equation becomes more delicate. For example, Atlassian’s internal HULA study found 59% of the AI-generated pull requests were merged into the codebase. These numbers highlight both promise and limitation: the agent offloaded some work, but if developers must frequently review or revise autonomous output, it can create new frustrations and diminish perceived value.
• Performance (Outcomes): The actual outcomes of the work — such as feature completeness, code quality, bug counts, or business value delivered. It’s essentially “productivity as results.” For an individual, this could mean tasks completed or user stories delivered; for a team, it could mean hitting project goals or quality standards. With AI, one might look at whether using an LLM leads to better code or faster delivery of new capabilities. For example, a coding trial by GitHub showed that the Copilot group not only finished faster but also had a slightly higher task completion success rate (78% vs 70%) than the non-AI group — a performance boost attributable to AI help. Performance is ultimately what matters to end-users and the business: if AI lets us ship more value, that’s a win, but if it just produces sloppy code faster, that’s a problem.
• Activity: The volume of actions or output — commits made, lines of code written, pull requests opened, comments in code reviews, etc. This is the easiest to measure (and thus often overemphasized), but by itself it says little about productivity. A developer could generate thousands of lines with an AI’s help (high activity) that later get refactored or thrown away. On the flip side, someone who finds a one-line solution and writes just a few lines might solve a problem very efficiently (low activity, high impact). The SPACE framework warns not to use activity in isolation. However, activity metrics can still be useful in context. In our team, we do keep an eye on things like “How many AI suggestions are accepted” or “code churn” — not to judge productivity, but to inform whether the AI is being used and how. (On average, developers accept about 30% of Copilot’s code suggestions, so this metric gives a sense of how much code the AI is contributing.) For instance, a sudden drop in commit activity after introducing an AI tool might signal that developers are integrating it into larger commits or focusing more on design work; a spike might mean the AI is generating a lot of code that requires multiple revisions. The key is interpreting activity with other metrics, never alone.
• Communication & Collaboration: How developers work together and share knowledge. This includes interactions like code review feedback, design discussions, pair programming sessions, or even helping others on forums. Productivity is a team sport in software, so a great individual output means little if the team isn’t communicating. AI’s role here is fascinating: does having an AI “pair programmer” reduce the need for a human buddy, or free up time to increase collaboration? I’ve found that with AI handling some routine Q&A, I actually ask my teammates deeper questions and have more high-level design conversations. On the other hand, if a developer quietly relies on AI and never engages in code reviews, that could hurt team coherence. We haven’t cracked a metric for this in our AI-rich environment yet, but we look at things like PR review turnaround time and knowledge-sharing activity (Slack discussions, etc.) to ensure AI isn’t isolating people. SPACE suggests measuring things like feedback quality in code reviews or the network of communication in the team, which remain as important as ever.
• Efficiency & Flow: The state of smooth progress, minimal friction, and the ability to maintain “flow” (deep focus). Efficiency can refer to how much output or outcome is achieved per unit of input (time, effort). AI assistants can dramatically improve efficiency by automating tedious tasks and providing instant answers. They can also enhance flow by reducing context-switching. Indeed, 73% of developers in GitHub’s survey said Copilot helped them stay in flow, and 87% said it reduced mental effort on repetitive work. However, efficiency and flow can suffer if the AI leads you down rabbit holes (e.g. generating wrong code that you then debug for hours). So we track things like cycle time for tasks, and even use internal surveys to ask, “Did the tools help you stay in flow this sprint?” A high efficiency & flow score, combined with high performance, is a strong sign the AI is paying off.

The power of SPACE is in combining multiple dimensions. It explicitly pushes you to measure more than one thing — ideally at least one metric in each category — rather than saying “we increased output 30%, we’re done.” For example, if AI coding yields high Activity but low Satisfaction, SPACE would flag a problem. Or if developers are happy and coding a lot but product quality tanks, that’s a different problem. By looking at a constellation of metrics in tension, you get a fuller picture. In my own workflow, I informally rate each coding session: Did I feel happy/productive? Did I create something that works well? Did I write an unreasonable amount of code or do an unreasonable number of edits? — these correspond to S, P, and A/E dimensions. It’s not rigorous, but it reminds me that a great day is when I feel good, produce value, and don’t thrash around too much.

Notably, GitHub’s research team explicitly used SPACE to evaluate Copilot’s impact. They looked at satisfaction (job fulfillment, frustration levels), efficiency/flow (staying in the zone, mental energy), and other aspects to holistically assess the AI tool. This kind of approach is now influencing how companies roll out AI: they run surveys and gather telemetry, not just to see if coding got faster, but also if developers are happier and collaborating well. The takeaway is that SPACE provides a balanced scorecard for developer productivity, which is extremely relevant when AI is changing some metrics but not others. However, SPACE by itself doesn’t directly measure team delivery outcomes like how fast features reach users or how stable the system is. For that, in parallel, we turn to the DORA metrics.

The DORA Metrics: Measuring Delivery Performance

While SPACE zooms in on developer experience and process, DORA metrics focus on the outcomes of the engineering team as a whole. DORA (DevOps Research & Assessment) was a research program (led in part by Dr. Forsgren as well, coincidentally) that identified four key metrics which high-performing software teams excel at:

• Deployment Frequency: How often the team deploys code to production or releases to end-users. Elite teams deploy far more frequently (on-demand or daily) than low performers (who might deploy only monthly or less). AI could affect this if it speeds up development; for example, if code is written faster, we might choose to deploy more often in smaller increments. However, deployment cadence also depends on business context and risk tolerance.
• Lead Time for Changes: The time from committing code to having it successfully running in production. This metric gauges how quickly code moves through the pipeline. Automation (CI/CD) and efficient review processes usually shorten lead time. If AI reduces the coding phase (say by generating code or tests faster), lead time might drop — but if the review or fix cycles increase due to AI errors, lead time might not improve. In one study, developers using AI pair-programming cut average coding time dramatically, which effectively halved the overall lead time for a task. DORA uses lead time as a crucial indicator of delivering efficiently.
• Change Failure Rate: What percentage of deployments result in a failure? For example, if out of 100 releases, 5 cause an incident, the change failure rate is 5%. Elite teams keep this rate low even as they deploy fast, showing they can move quickly and maintain quality. AI’s impact here is complex: if AI generates more bugs, it could worsen failure rates; if it helps catch bugs (via generating tests or performing smart checks), it could improve them. At the moment, we rely on our normal testing and monitoring to track failures — AI assistance hasn’t fundamentally changed how we measure this, other than potentially influencing the raw number of issues.
• Time to Restore Service (MTTR): How long it takes to recover from a failure (e.g. a production incident). A lower MTTR means the team can quickly diagnose and fix issues. AI could assist in rapid diagnosis (perhaps by analyzing logs or suggesting fixes), but it might also introduce new kinds of failures that are harder to debug unless the AI can explain itself. DORA emphasizes MTTR as a stability metric — fast recovery means resilience.

Together, these four metrics (often called the “Four Keys”) paint a picture of delivery performance. The reason we care about DORA in the AI era is that no matter how much AI we throw into coding, what ultimately counts is delivering value to users quickly and reliably. These metrics will show if AI is helping on that front. If AI makes developers twice as fast at writing code, but our deployment frequency or lead time doesn’t budge, the bottleneck might be elsewhere (or the AI-generated code is causing delays down the line). DORA metrics serve as a reality check on productivity — they tie it to real outcomes like shipping and stability.

It’s worth noting that DORA metrics mainly cover the “outer loop” of development — build, deploy, operations — rather than the “inner loop” of coding and debugging itself. That’s why they complement SPACE. One recent analysis pointed out that while DORA is great for DevOps outcomes, it doesn’t tell you if developers are struggling internally (e.g. writing overly complex code, staying in flow), which SPACE captures. So, using SPACE and DORA together gives you both an inside-out and an outside-in view of productivity.

Balancing Metrics in the LLM Age: How to Measure What Matters

With both SPACE and DORA in our toolbox, how do we actually measure productivity when AI is part of the team? The key is balancing these frameworks to get a full picture. In my experience, this means instrumenting our process with a mix of metrics and constantly cross-checking them.

After we adopted AI pair-programming tools, we took a step back and defined a few metrics in each SPACE category and all four DORA metrics as our “dashboard.” For instance, we monitor developer satisfaction via quarterly surveys and simple pulse checks (“On a scale of 1–5, how productive do you feel with our tooling?”). We track efficiency/flow by looking at cycle times for tasks and pull request sizes. For activity, we glance at commits and code review counts (making sure any changes align with outcomes). For communication/collaboration, we might look at something like PR review participation or intra-team mentorship hours. None of these alone is definitive, but together they start to flag where things are improving or need attention.

One interesting insight I’ve gained is that productivity gains from AI often show up in indirect ways. For example, developers saving time on rote coding (thanks to AI) might invest that time in writing more unit tests or doing more code review for peers. That doesn’t immediately translate to “more features delivered,” but it improves code quality and team knowledge — a benefit captured in SPACE under collaboration or performance quality. A narrow metric like sprint velocity could miss that, but a broader set of metrics can capture it (e.g. higher code coverage, better satisfaction scores). In our case, we noticed an uptick in our internal quality metrics post-AI adoption: fewer late-night emergency fixes, more documentation being written. Developers seemed to have a bit more breathing room, which they channeled into stability tasks. So in a sense, AI reallocated some effort from coding to improving robustness. That’s a win, but you only see it if you measure things beyond speed.

Maintaining a narrative with the metrics is important. We regularly discuss as a team: “AI helped us do X, which we see reflected in metric Y.” For instance, “Our code review turnaround is faster (Communication) — possibly because AI-generated code is a bit easier to review in parts, or folks have more time to review since coding took less time.” Or, “Our deployment frequency didn’t jump as much as we expected; maybe our bottleneck is actually testing or approval processes, not coding — AI or not, we need to streamline that.” This qualitative interpretation alongside the numbers is crucial. It keeps us from misattributing cause and effect. AI is a new factor, and it can be tempting to blame or credit it for every uptick or downtick. The reality is software development remains a complex system of people, practices, and tools.

By using SPACE and DORA together, we guard against the extremes. We won’t declare “victory” just because lines of code went up or task time went down in the short term. We’ll look at whether those lines of code led to more deploys, better customer feedback, and a happier team. We also avoid panicking if, say, developers are writing less code — if our outcomes and satisfaction are steady or better, it could just mean efficiency is up (less code for the same result). In essence, these frameworks encourage continuous improvement. We can iteratively adjust how we use AI and see how it impacts various metrics, then adjust again.

Measuring LLM’s performance

One more thing to consider is the new metrics or signals that AI tools themselves can provide. For example, some AI coding tools provide their own telemetry: how many suggestions are accepted vs. rejected, how often the AI’s code passes tests, etc. These can be useful supplements to our traditional measures. If your AI is generating a lot of suggestions that developers discard, that might indicate mismatched expectations or trust issues. If an AI agent takes 20 minutes to complete a task that a human would take 5 minutes, that’s also insight — maybe it frees the human’s time, but maybe it’s not efficient. At this stage, we’re still exploring these AI-specific metrics, but they’re quickly becoming part of the conversation.

GitHub Copilot — one of the first widely adopted AI pair programmers — has started offering such metrics to help measure its impact. Copilot for Business provides an admin dashboard and even a REST metrics API for organizations to track usage data like the total number of code suggestions and how many of those were accepted. In practice, this enables a metric for suggestion acceptance rate (accepted suggestions divided by total suggestions). In GitHub’s studies, developers accepted roughly 30% of Copilot’s code suggestions on average, so that figure can serve as a baseline for what “normal” AI utilization looks like. Copilot’s tooling doesn’t directly tell us higher-level outcomes like our PR merge rate or bug rate, but we can correlate its telemetry with external data. For instance, in one experiment with Accenture developers, introducing Copilot correlated with an 8.8% increase in pull requests opened and a 15% higher pull request merge rate for the team.

Another tool I rely on, Cursor, similarly tracks how we interact with its AI suggestions. Cursor actually distinguishes between two types of AI help: “Tab” completions (its inline code suggestions as you type) and larger AI-generated edits initiated via chat or command. It records how many Tab suggestions the developer accepted, and how many lines of AI-generated code were applied from the chat/composer. This gives immediate feedback on how much code Cursor is contributing to our codebase. However, as of now, Cursor’s analytics are fairly bare-bones for teams — the business plan only provides very basic usage info like each user’s last active date, lacking a rich organization-wide metrics dashboard. There isn’t yet a public metrics API for Cursor. So if we wanted to aggregate something like “team-wide suggestion acceptance rate in Cursor,” we’d likely have to parse local log files or rely on manual observation. In short, Cursor provides some raw data on AI usage (at least to individual users), but turning that into actionable productivity insights for a team currently requires extra effort. This is a case where we have to infer the impact (e.g. by looking at changes in commit patterns or PR sizes) rather than having a turnkey solution from the tool.

On the other hand, Windsurf has embraced built-in productivity analytics more than most. It not only shows usage stats to individual developers (for example, how many completions you accepted per day and in which languages), but also offers team-wide dashboards in its enterprise version. Team administrators can see metrics like the percentage of code written by AI versus by humans, as well as estimates of time saved and even money saved thanks to AI assistance. In fact, the Codeium team has proposed new composite metrics such as “Characters per Opportunity” and “Percentage Code Written” to quantify the AI’s contribution. Their enterprise deployment includes integration with standard monitoring tools — it comes with a Prometheus metrics endpoint and a pre-built Grafana dashboard — so organizations can directly track AI usage over time and correlate it with other engineering data. This means key KPIs like suggestion acceptance or coverage of AI-generated code are available at a glance, and we can overlay them on our DORA metrics or bug rates. For example, Windsurf could show that 30% of a release’s code was AI-generated (via its “Percent Code Written by AI” metric) and that might tie into a higher deployment frequency if everything is going well. Among today’s tools, Windsurf provides some of the most native support for productivity measurement, which shows how important this has become. It essentially builds the measurement of AI’s impact into the development environment, rather than leaving us to guess.

Cline and RooCode are indeed Visual Studio Code extensions that embed autonomous AI coding agents into the editor. As for telemetry and analytics, neither extension provides a user-facing dashboard for productivity metrics. There’s no in-IDE display of “time saved” or coding statistics.

Devin provides a cloud-based project environment for AI-driven software development. It does allow you to export data about tasks and pull requests from that environment. Devin can also be instructed to analyze repository statistics such as pull request metrics. Because Devin operates in a sandbox with a full development toolkit (shell access, Git integration, etc.), you can ask it high-level questions about your codebase or workflow. For instance, one could prompt Devin: “Analyze our repository’s pull request history and tell me the average PR merge rate and time to merge.” Devin has the ability to fetch repository data (via the GitHub API or git logs in its environment) and compute such statistics.

The overarching theme is that each AI tool brings its own data and requires its own KPIs. We leverage whatever metrics the tool provides natively — suggestion acceptance rates, time saved dashboards, test pass rates, etc. — and we integrate those with our SPACE and DORA metrics to get the full picture. For instance, if GitHub Copilot’s telemetry shows a high suggestion acceptance rate and Windsurf reports that 30% of our code this month was AI-generated, we’d expect to see some effect in our DORA outcomes (maybe faster lead times or higher deploy frequency). If we don’t, that discrepancy is telling: perhaps code is being generated quickly but getting hung up in review or QA, meaning we need to improve that part of the process. Conversely, if an autonomous agent like Devin is closing a lot of trivial tickets, we might not immediately see a spike in feature throughput, but we could see developers spending more time on complex tasks (which might later show up as improved quality or bigger features delivered). In practice, we’re starting to treat these AI tool metrics as first-class signals. And the industry is heading that way: it’s safe to assume companies like Microsoft are already crunching this telemetry to refine their models and features. As AI assistants become a normal part of development work, our productivity dashboards will likely incorporate these AI-specific metrics alongside the usual DevOps and agile metrics.

Conclusion: Productivity, Redefined (and Measured)

The rise of LLMs in programming is often compared to giving developers “superpowers” or a “10× boost.” But realizing those gains in practice — and proving them — requires rethinking how we define and measure productivity. The days of measuring a programmer by keystrokes or commit counts are fading. In the LLM era, a truly productive developer might be one who writes less code themselves but orchestrates AI and tools to achieve the same outcome more efficiently. Or they might be someone who leverages AI to generate loads of code and thus can tackle bigger problems solo than before. How do we ensure in either case that we’re actually better off? By measuring the things that matter: Are we delivering value faster and more reliably? Are developers happier and more efficient?

By combining frameworks like SPACE and DORA, we can answer those questions. SPACE keeps us grounded in the human and process aspects — if an AI assistant causes developer frustration or hurts team collaboration, that’s a red flag even if output is high. DORA keeps us focused on actual delivery — if all the AI-generated code isn’t making it to production successfully, did it really help? Together, they encourage a balance: speed and quality, output and well-being.

My journey with AI coding tools has been eye-opening. I’ve experienced days where I felt like an unstoppable force, gliding through tasks with an AI pair programmer — and other days where I wrestled with weird AI suggestions that I had to unravel. Measuring my productivity on those days requires more nuance than “I wrote X commits.” It involves reflecting: Did the AI free me up to solve higher-level problems? Did I end up with a solid piece of software? These qualitative reflections align with the SPACE dimensions. And later, when that code is deployed, the truth shows in our DORA metrics — perhaps that feature indeed went out faster or with fewer issues.

As we enter this new era of AI-augmented development, it’s likely that our productivity metrics will continue to evolve. We might incorporate direct measures of AI’s contributions or even new frameworks to assess human–AI team effectiveness. But right now, SPACE and DORA (with a healthy dose of experimentation and common sense) are our best guides. They ensure we don’t get lost in either hype or fear, but instead keep an empirical, balanced view of progress.

In summary, measuring developer productivity in the LLM era means embracing multidimensional metrics. It means celebrating that your AI pair programmer helped deploy a feature in half the time and asking if the team is happier and the software is robust. It’s about treating AI as a new part of the process whose impact we verify with data, not assume. If we do this well, we can truly harness these “developer jetpacks” to not only go faster, but to build a better developer experience and superior software. And the next time someone asks “So, did that AI actually make your team more productive?”, we’ll have a confident, well-rounded answer — backed by facts and focused on what truly matters.

TL;DR

• Bayesian Marketing Mix Modeling (Bayesian MMMs) like Meridian and PyMC-Marketing struggle to model indirect brand effects like media → search → sales.
• These models lack native support for campaign-level or creative-level data, collapsing all inputs into channel-level spend and impressions.
• Neural Nested Networks (NNN), proposed in April 2025, address these limitations with a nested architecture and support for rich input embeddings.
• NNN models direct and indirect effects with residual fusion and multi-task outputs for sales and branded search.
• Factored self-attention and channel mixing allow NNN to capture cross-channel synergy, not just independent effects.
• The model can incorporate qualitative inputs such as creatives, campaign metadata, and brand intent as embeddings.
• Bayesian MMMs remain useful for uncertainty quantification, but NNN unlocks modern ML for more expressive causal modeling.

Problems with Bayesian MMM

Until recently, I only knew Bayesian Marketing Mix Models (Bayesian MMM) like Meridian and PyMC-Marketing. These hierarchical Bayesian models were powerful, quantifying uncertainty and leveraging priors. But as I applied them to real-world campaigns, I kept hitting the same two limitations:

1. It was hard to properly model brand recognition effects like media → branded search → sales. These required custom multi-layer models, carefully tuned priors, and a lot of patience. And they weren’t supported natively by the open-source tools.
2. I couldn’t use campaign-level context — things like creatives, targeting strategies, or operational metadata — as inputs. Everything had to be collapsed into weekly channel-level spend and impressions. That lost a lot of signal, and broke the consistency assumption when creatives or formats changed mid-flight.

In short, Bayesian MMMs assumed a lot of additivity and aggregation. Channels’ effects just added up, and any nuance had to be shoehorned in manually.

Enter Neural Nested Networks (NNN)

Fast-forward to April 2025. A thesis paper introduced something called Neural Nested Networks (NNN). It immediately caught my eye because it promised to address exactly those pain points. NNN is essentially a Transformer-based neural network for MMM — a completely different beast from Bayesian regression. Instead of explicit priors and additive terms, it uses deep learning to learn the relationships, with an architecture designed to reflect the nested funnel of media effects.

According to the paper, NNN brings a few key innovations:

• It uses rich embeddings to represent inputs, so it can incorporate qualitative factors like search queries or ad creative attributes (not just numeric spends).
• A factored self-attention mechanism lets it capture temporal patterns and cross-channel interactions — meaning it can model long-term carryovers and synergy between channels that traditional additive models might miss.
• It directly models multiple outcomes (e.g. both sales and anbrand awareness metrics like search volume) in one unified network. This was described as a nested architecture: one part of the model predicts the intermediate (search volume), which then feeds into the final sales prediction.

Reading this, I had a flashback to all those hours tweaking my Bayesian models. Could NNN really make these things easier? Only one way to find out — I decided to implement an NNN myself and compare.

You can find Colab notebook in nnn_for_mmm.ipynb.

How NNN Works: Key Innovations

Before diving into code, let’s unpack what makes NNN different from a classic Bayesian MMM. Here are the key technical innovations that I found especially exciting:

• Factored Self-Attention (Time & Channel): NNN’s architecture uses a Transformer-style attention that is factored by time and channel. In practice, this means the model can learn temporal patterns for each channel separately, and then mix information across channels. For example, it can learn the weekly seasonality or adstock effect within TV, within YouTube, etc., and also how those channels interact. This two-step attention lets NNN capture long-term effects and cross-channel influences efficiently.
• Residual Path Fusion (Direct + Indirect Effects): A defining feature of NNN is that it has two prediction paths: one for direct “media → sales” effects, and one for indirect “media → search → sales” effects. The model learns to fuse these paths with residual connections. This residual fusion means the model itself decides how much of a channel’s effect is direct vs. through brand search.
• Qualitative Feature Embeddings: Unlike regression-based MMM, which only handles numeric inputs, NNN can ingest embedding vectors for categorical or qualitative data. In the paper’s words, it can capture “qualitative aspects of marketing… (e.g., search queries, ad creatives)”. In practice, this means we can embed things like campaign creatives, messaging themes, etc., and let the model learn their contribution.
• Modeling Synergy (Non-Additive Effects): Traditional MMMs often assume each channel’s effect is independent (add them up to get total sales). NNN breaks this barrier. Through its attention mechanism and non-linear layers, it can learn non-additive interactions — e.g., maybe TV and YouTube together drive more sales than the sum of each alone, due to combined awareness effects. In other words, it doesn’t force the assumption that “1 + 1 = 2” for channel contributions; sometimes 1 + 1 could equal 1.5 or 3, depending on synergy or saturation patterns.

Those are the core ideas. In summary, NNN is a neural network that natively represents a marketing funnel (with intermediate outcomes) and allows rich inputs, using attention to handle time dynamics and interactions. As an engineer, this sounded like a more expressive and modular way to do MMM — if it could be made to work. I set out to build a prototype to see these concepts in action.

Building an NNN Model Step-by-Step (in Code)

To test NNN, I built a Colab notebook from scratch, following the paper’s description. I wanted to simulate a scenario close to a real marketing funnel and see if NNN could learn it. Here’s a walkthrough of what I did, with key code snippets and explanations:

1. Simulating a marketing funnel dataset

I generated a synthetic dataset representing 2 years of weekly data for 47 regions (to mimic a geo-level MMM dataset). It includes four paid media channels (let’s call them TV, YouTube, Taxi Ads, and Display), an organic search volume metric, and sales. The simulation followed a causal DAG: each channel’s spend influences sales directly and also indirectly via search.

I incorporated standard MMM effects like adstock carryover and saturation (Hill function) to make it realistic. I also built in some seasonality for organic search (e.g., sinusoidal yearly trend) and added a bit of noise. In code, it looked roughly like this:

# --- Adstock and Hill effect functions ---
def adstock_transform(x, retention=0.5):
    carry = 0.0
    y = []
    for t in range(len(x)):
        carry = x[t] + retention * carry
        y.append(carry)
    return np.array(y)

def hill_transform(x, half_sat=1.0, slope=1.0):
    return x**slope / (x**slope + half_sat**slope)

# For each channel and geo, simulate spend, then apply adstock + saturation
# Direct contribution to sales and indirect contribution via search
for channel in channels:
    ad_effect = hill_transform(adstock_transform(spend[channel]), half_sat=...)
    sales_direct_contrib = direct_weight[channel] * ad_effect  # direct path
    search_lift = indirect_scale[channel] * ad_effect          # indirect path (to search)

I generated random spend patterns per channel, apply an adstock filter (adstock_transform) to simulate delayed impact of media, then a Hill saturation (hill_transform) to simulate diminishing returns. Each channel’s adstocked, saturated spend contributes to sales directly (sales_direct_contrib) and also boosts the search volume (search_lift). The search volume for each region = a baseline organic trend + the sum of all media-driven lifts. Finally, sales = a base conversion of that search volume + the sum of direct media contributions (plus noise). This mirrors a funnel: media → search → sales. We end up with a DataFrame of week, geo, spend_tv, spend_youtube, ..., search, sales.

2. Encoding “brand intent” as an embedding

One piece I added was a latent “brand intent” factor that varies over time. The idea is that some weeks the brand is “hotter” (so media converts better and drives more search), other weeks it’s colder. I generated a random series intent_factor[t] that oscillates between 0.5 and 1.5. Rather than directly use this number, I encoded it as a vector embedding that gradually shifts from a “low-intent” state to a “high-intent” state. Concretely, I created two random anchor vectors: low_embed and high_embed (each 256-dimensional). For each week, I interpolated between them based on the intent factor value (nnn_for_mmm.ipynb):

min_factor, max_factor = 0.5, 1.5
intent_factors = np.random.uniform(min_factor, max_factor, size=TIME_STEPS)
low_embed  = np.random.normal(size=EMBED_DIM)
high_embed = np.random.normal(size=EMBED_DIM)
creative_embeddings = [
    (1 - (f - min_factor)/(max_factor - min_factor)) * low_embed 
  + ((f - min_factor)/(max_factor - min_factor)) * high_embed
    for f in intent_factors
]

This way, each week t gets a 256-dim vector summarizing the brand’s latent intent or awareness at that time. When intent is low (0.5), the vector is near low_embed; when intent is high (1.5), it’s near high_embed. I fed this into the simulation by making the media effects scale with intent_factor. Essentially, this embedding represents things like brand health or creative effectiveness that change over time.

3. Structuring the model inputs (channel × week × embedding)

With data simulated, I prepared the inputs for the NNN model. NNN expects the input as a sequence over time, with multiple channels/features at each time step. I constructed an input tensor X of shape (geo, week, channel, embed_dim). For each week and channel, the input vector includes that channel’s information for the week. In this simple case, I embedded each channel’s spend as a vector of length 256. How? I took the easy route: each channel got its own learned linear projection from the scalar spend to a 256-dim vector. I also appended the brand intent embedding as an additional “channel” for each week. So effectively, for each week we have 5 channel vectors: TV, YouTube, Taxi, Display, and a Brand-Intent vector (same across geos for a given week). This (week × channel × features) structure is fed into the neural network. It’s quite different from a regression matrix for MMM – we’re feeding the model a rich panel of time-series vectors.

4. Defining the NNN architecture

I implemented the Neural Nested Network in PyTorch. The model has three main parts — (a) a temporal self-attention stack, (b) a channel mixing layer, and (c) two output heads (for sales and search) with a residual fusion. Here’s a simplified breakdown of the forward pass:

1. First, the input X (shape [B, T, C, D] for batch, time, channels, embedding_dim) goes through several layers of multi-head self-attention along the time axis. Crucially, I applied attention within each channel separately. I flattened the tensor to shape [B*C, T, D] and ran a multi-head attention with a causal mask (so each week looks at past weeks up to a certain lookback, here 52 weeks). This means each channel’s time series learns its own temporal patterns (trends, seasonality, lag effects). After attention, I unflatten back to [B, T, C, D] and add a residual connection of the original input (so the model keeps the raw input info too).
2. Next comes channel mixing: for each time step (and each embedding dimension), the model mixes information across channels via a small feed-forward network. Essentially, at week t, it looks at the vector of all channels [TV_t, YT_t, Taxi_t, Display_t, BrandEmbed_t] and passes that through an MLP that outputs the same number of channels (nnn_for_mmm.ipynb). This is like a learned cross-channel interaction at each time. After this, we have an output tensor X_mixed of shape [B, T, C, D] that now contains information both along time and across channels.
3. Multi-task output heads: I set up two linear heads that take the final mixed representation and predict two series: weekly sales and weekly search volume. Each head flattens the channel & embedding dimensions and outputs a single value per time (so it’s akin to a linear combination of all channel features). We get sales_pred and search_pred as the model’s outputs.
4. Residual fusion: We combine the two outputs to reflect the nested structure. We want the final sales prediction to incorporate both the direct effect and the indirect-via-search effect. I implemented a learnable parameter w_search that scales the search prediction’s contribution to sales. The formula (during training) was sales_combined = σ(w_direct) * sales_pred + σ(w_search) * (search_pred * search_to_sales_factor) where σ() is a sigmoid to keep weights in [0,1], and search_to_sales_factor is a scaling factor. The end result: the model learns to balance how much sales come directly from media vs. indirectly via search. For example, it might learn w_search = 0.7, meaning 70% of the sales impact goes through the search channel. This kind of interpretation is normally hard-wired in a Bayesian model via hierarchical coefficients, but here it’s learned as a network parameter.
5. Finally, I included a geo-level calibration: since each region had different scale of sales, I added a learned per-geo scaling and bias (basically a simple linear regression layer per geography) to adjust the model’s outputs (nnn_for_mmm.ipynb). This ensured that the sales and search predictions for each region were on the right scale (since the network itself was predicting some normalized value).

Here is the model definition:

class NNNModel(nn.Module):
    def __init__(
        self,
        n_channels,
        embed_dim,
        n_heads=N_HEADS,
        lookback_window=LOOKBACK_WINDOW,
        hidden_channels=HIDDEN_CHANNELS,
        n_layers=N_LAYERS,
        n_geos=GEO_COUNT,
        time_steps=TIME_STEPS,
    ):
        super(NNNModel, self).__init__()
        self.n_channels = n_channels
        self.embed_dim = embed_dim
        self.lookback = lookback_window
        self.n_layers = n_layers

        # === Transformer Stack ===
        self.time_attn_layers = nn.ModuleList([
            nn.MultiheadAttention(embed_dim, num_heads=n_heads, batch_first=True)
            for _ in range(n_layers)
        ])
        self.time_ln_layers = nn.ModuleList([
            nn.LayerNorm(embed_dim) for _ in range(n_layers)
        ])

        # === Channel Mixing MLP ===
        self.channel_fc1 = nn.Linear(n_channels, hidden_channels)
        self.channel_fc2 = nn.Linear(hidden_channels, hidden_channels)
        self.channel_fc3 = nn.Linear(hidden_channels, n_channels)
        self.channel_ln = nn.LayerNorm(n_channels)

        self.dropout = nn.Dropout(p=0.1)

        # === Output heads ===
        self.head_sales = nn.Linear(n_channels * embed_dim, 1)
        self.head_search = nn.Linear(n_channels * embed_dim, 1)

        # === Residual path weights ===
        self.search2sales = nn.Parameter(torch.tensor(1.0))
        self.w_direct = nn.Parameter(torch.tensor(0.5))
        self.w_search = nn.Parameter(torch.tensor(0.5))

        # === Geo-based denormalization ===
        self.geo_scaling = nn.Embedding(n_geos, 2)
        self.geo_bias = nn.Embedding(n_geos, 2)
        nn.init.constant_(self.geo_scaling.weight, 1.0)
        nn.init.constant_(self.geo_bias.weight, 0.0)

        # === Causal attention mask ===
        full_mask = torch.ones((time_steps, time_steps), dtype=torch.bool)
        for t in range(time_steps):
            start_idx = max(0, t - lookback_window)
            full_mask[t, start_idx : t + 1] = False
        self.register_buffer("attn_mask", full_mask)

    def forward(self, x, geo_idx):
        B, seq_len, n_chan, d = x.shape
        attn_mask = self.attn_mask[:seq_len, :seq_len]

        x_reshaped = x.view(B * n_chan, seq_len, d)
        for attn, ln in zip(self.time_attn_layers, self.time_ln_layers):
            attn_out, _ = attn(x_reshaped, x_reshaped, x_reshaped, attn_mask=attn_mask)
            x_reshaped = ln(attn_out) + self.dropout(x_reshaped)

        attn_out = x_reshaped.view(B, n_chan, seq_len, d).permute(0, 2, 1, 3)
        x_attended = attn_out + x

        # Channel mixing
        x_ct = x_attended.permute(0, 1, 3, 2).reshape(-1, n_chan)
        x_ct = F.relu(self.channel_fc1(x_ct))
        x_ct = F.relu(self.channel_fc2(x_ct))
        x_ct = self.channel_ln(self.channel_fc3(x_ct))
        x_ct = x_ct.view(B, seq_len, d, n_chan).permute(0, 1, 3, 2)
        x_mixed = self.dropout(x_ct) + x_attended

        # Output heads
        x_flat = x_mixed.reshape(B, seq_len, n_chan * d)
        sales_direct = self.head_sales(x_flat).squeeze(-1)
        search_pred = self.head_search(x_flat).squeeze(-1)

        # Residual fusion
        w_d = torch.sigmoid(self.w_direct)
        w_s = torch.sigmoid(self.w_search)
        sales_combined = w_d * sales_direct + w_s * self.search2sales * search_pred

        # Denormalize output
        y = torch.stack([sales_combined, search_pred], dim=-1)
        scales = self.geo_scaling(geo_idx)
        biases = self.geo_bias(geo_idx)
        y_scaled = y * scales.unsqueeze(1) + biases.unsqueeze(1)
        return y_scaled

5. Training with enhancements and regularization

I trained the NNN model on the synthetic data using Adam optimizer. One thing about neural nets — they can overfit, especially with many parameters and relatively little data (2 years of weekly data isn’t huge). So I incorporated a few regularization techniques:

• Dropout: I applied dropout in the attention layers and channel mixing MLP with p=0.1, to avoid relying too heavily on any one feature.
• L1 regularization: The authors noted that using an L1 penalty on the weights can help such a rich model not overfit in data-sparse situations. I added a small L1 penalty on the output head weights, encouraging the model to use fewer features (effectively to push insignificant channel contributions towards zero).
• Multiple attention layers: Instead of a single layer of attention, I stacked 3 layers (“Transformer blocks”) to give the model more capacity to learn complex patterns (with layer normalization and residual connections at each layer). This is an enhancement beyond the original paper which primarily proposed the concept — stacking deeper can improve accuracy at the cost of more training data needed (nnn_for_mmm.ipynb).

Here’s the model initialization:

model = NNNModel(
    n_channels=CHANNEL_COUNT_INPUT,
    embed_dim=EMBED_DIM,
    n_heads=N_HEADS,
    lookback_window=LOOKBACK_WINDOW,
    hidden_channels=HIDDEN_CHANNELS,
    n_layers=N_LAYERS,
    n_geos=GEO_COUNT,
    time_steps=TIME_STEPS,
).to(DEVICE)

criterion = nn.SmoothL1Loss()
optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=NUM_EPOCHS)

Training was surprisingly smooth — no divergent chains or R-hat diagnostics to worry about. After ~500 epochs, the model converged to a low error on both sales and search. Importantly, it learned to predict the search volume series very well (basically nailing the seasonal trend plus media bumps), which in turn helped it predict sales accurately. This multitask learning seemed to regularize the model: by learning the intermediate metric, the model’s internal representation had to make sense in terms of a funnel, not just fit sales arbitrarily.

Results: validating baseline vs. incremental effects

The ultimate test for me was: did the NNN model correctly learn the baseline (what sales would be with no media) versus the incremental media-driven sales? In MMM, we often plot the model’s predicted baseline vs actuals to ensure the model isn’t attributing baseline trend to media or vice versa. I did just that. I took the trained model, fed in a scenario with all media turned off (zero spend), and got a predicted sales series — this is the model’s learned baseline. Then I compared: actual sales vs. model-predicted sales vs. model-predicted baseline.

The blue line is actual sales, the orange line is NNN’s predicted sales, and the green line is NNN’s predicted baseline if there were no media. The NNN model cleanly learned the seasonal baseline and the incremental lift from media.

As shown above, the NNN’s predictions closely track the actual sales. The gap between the green and orange lines at any point is the model’s estimated incremental impact of media.In classic Bayesian MMM, we enforce this separation via priors (e.g., a prior on the intercept or trend). Here the neural net figured it out from data. It was quite satisfying to see that the indirect path via search was utilized: for instance, when I looked at the learned w_search weight, it was about 0.65, meaning ~65% of media’s effect was going through search in the model – not far from the 70% I actually simulated.

In summary, the NNN implementation was able to recover the true patterns of the simulation: it understood the funnel dynamics (media→search→sales) and even handled the “creative effectiveness” variability via that time-varying embedding. Now that I had a working model, it was time to reflect on how this approach compares to the traditional Bayesian MMM workflow.

NNN vs. Bayesian MMM: Key Differences in Practice

Having worked with both approaches, I want to highlight some practical differences I observed between Neural Nested Networks and classical Bayesian MMMs:

• Additive Assumptions vs. Learned Synergies: Bayesian MMMs (like PyMC-Marketing’s implementations or Google’s Meridian) typically assume the contribution of each channel is additive — you fit a separate curve (with saturation) for each channel and sum them up. If you want any interaction (say TV × YouTube synergy), you have to explicitly include interaction terms, which quickly becomes impractical beyond a couple of channels. NNN, on the other hand, does not assume additivity. The Transformer attention and channel mixing layers can learn interactions implicitly. The flip side is interpretability — additive models make it easy to say “channel X contributed Y sales” because of the strict partitioning. NNN gives a more holistic view, but you might need to do extra work (like SHAP values or counterfactual experiments) to attribute exact credit to each channel when interactions are at play.
• Limited Context vs. Rich Feature Embeddings: In my Bayesian models, input data was a table of spends or impressions (or maybe reaches and frequencies). If I had multiple campaigns on one channel, their spends would be aggregated, losing any info about differences between campaigns or creatives. NNN treats inputs as high-dimensional vectors, so I can include campaign metadata, creative type, audience segment, etc., as part of the input embedding. In short, NNN opens the door for high-dimensional, multi-modal data in MMM, bringing techniques like word embeddings or image embeddings into the model.
• Hierarchical Complexity vs. Nested Simplicity: To capture the media→search→sales funnel in Bayesian models, one might construct a two-stage hierarchical model: first stage for search as a function of media, second stage for sales as function of search (and media). I’ve tried before; It was too difficult for me to implement. There are lots of parameters and the MCMC can be slow to converge, plus you have to ensure the search data is good quality, etc. In NNN, that structure is just part of the architecture — we simply output search and feed it into sales. It felt simpler to set up and especially to tweak. I didn’t have to specify priors for the intermediate process or worry about identifiability between direct vs indirect effects; the model learned to divide the effect because it was given the explicit opportunity to do so with the residual fusion. This nested modeling is baked into NNN’s design, which made it very convenient for encoding the causal assumptions. If I wanted to add another layer (say media→site traffic→sales, with search as well), I could extend the network to have another head for traffic. You can imagine building out a whole funnel network — something that would be extremely cumbersome in a Bayesian framework.
• Probabilistic Inference vs. Deterministic Learning: This is a fundamental difference — Bayesian MMM provides a probability distribution of outcomes (posterior), whereas NNN gives point estimates. In practice, when I deliver MMM results to stakeholders, they do care about uncertainty (e.g. “90% credible interval for TV ROAS is…”). With NNN, you don’t get that for free. We’d have to bootstrap or use techniques like Monte Carlo dropout to get error bars. That’s a potential downside if uncertainty quantification is a priority. On the upside, training NNN was fast — no sampling, just backpropagation — and I can transfer learn. For instance, I could pretrain the model on a large industry dataset, then fine-tune on a specific brand, which is not something you can easily do with Bayesian models.

Final Thoughts: Causality Meets Representation Learning

Adopting Neural Nested Networks for MMM has been an eye-opening experience. It’s not often that a new modeling approach comes along and directly targets your daily pain points. With NNN, modeling a causal funnel felt natural — I was literally building the funnel into the network. Incorporating qualitative features was as simple as feeding in embeddings. And capturing channel synergy didn’t require an explosion of manual interaction terms; the model learned some on its own.

From an engineering perspective, NNN has unlocked capabilities we didn’t have before in MMM. The ability to leverage high-dimensional data (text, images, metadata) and transfer learn across datasets is huge in marketing analytics, where data can be siloed and scarce for individual brands but abundant in aggregate. We’re essentially bringing techniques from modern deep learning into MMM, which has historically been a relatively small-data discipline.

My real-world takeaway: Neural Nested Networks made certain causal assumptions easier to encode and interpret than the Bayesian approach I was used to. Instead of wrestling with complex hierarchical formulas, I spent that time designing the architecture and features. When I saw the model correctly split out the branded search contribution, I’ll admit — I geeked out a bit. It was a validation that our intuitions about the marketing funnel can be learned by a neural net.

Moving forward, I’m excited to apply NNN on actual marketing datasets and see if the promises hold up. Will it attribute better than classical MMM? Will those creative embeddings reveal new insights about what kind of ads work? Time (and plenty of hyperparameter tuning) will tell. But one thing is certain: the toolkit for MMM is expanding, and for those of us trying to quantify marketing impact, that’s a very good thing.

The full code for my NNN implementation is available in the accompanying Colab notebook (see link). If you’re an MMM practitioner or data scientist, I encourage you to experiment with this approach. It’s a refreshing blend of causal modeling and representation learning — and it just might save you from a few more Bayesian headaches on your next marketing analysis!

Have you ever built a machine learning model to estimate causal effects and then wondered how to act on those insights? For example, say you can predict how likely each person is to respond to a treatment — how do you decide who should get that treatment under limited resources? This tutorial tackles that problem by combining machine learning and mathematical optimization. We’ll walk through a demo from the GitHub project “Causal Inference Demo with Gurobi Machine Learning” to show how to turn predictions into optimal decisions, step by step.

Causal inference is about understanding cause-and-effect relationships — for instance, how offering an incentive (treatment) causes a change in outcome. A common machine learning approach for causal inference is uplift modeling, which splits a population into groups like Persuadables (those who respond only if treated) versus Sure Things (respond regardless), etc. However, uplift models alone don’t tell you what to do when you have constraints (like a limited budget for incentives). They identify who might respond to treatment, but not which individuals to treat when you can’t treat everyone.

Enter mathematical optimization. By formulating the decision as an optimization problem, we can directly compute the best treatment plan under our constraints. In this tutorial, we use Gurobi’s Machine Learning integration to embed a causal prediction model (a logistic regression) into an optimization model. This approach lets us find, for example, the optimal set of incentives to offer to maximize the number of positive outcomes, given budget limits.

Gurobi Machine Learning (gurobi-ml) is a library introduced in Gurobi 10.0 (Nov 2022) that allows us to incorporate trained ML models into optimization (specifically as mixed-integer programming constraints). In practice, it translates the predictive model into mathematical expressions that the Gurobi solver can handle (For certain models, like logistic regression, this involves creating a piecewise-linear approximation of the model’s prediction function).

In the example below, we’ll tackle a policy decision problem from an economics study. The dataset comes from Thornton (2008), where individuals in rural Malawi were offered random monetary incentives to learn their HIV test results. The question we’ll answer is: Given a limited incentive budget, which people should we pay (and how much) to maximize the number of individuals who return for their HIV results? By the end, you’ll see how a machine learning model (predicting the probability someone returns for their results) can be combined with optimization to yield an optimal incentive allocation strategy.

Let’s dive into the step-by-step implementation.

Step 1: Setup and Data Loading

First, we import the necessary libraries and load the dataset. We’ll use pandas and NumPy for data handling, scikit-learn for our ML model, and Gurobi (with its pandas integration) for optimization. We also import Gurobi’s machine learning helper add_predictor_constr to integrate the ML model into the optimization model.

# Import necessary packages
import gurobipy as gp
import gurobipy_pandas as gppd
import numpy as np
import pandas as pd
from gurobi_ml import add_predictor_constr
from causaldata import thornton_hiv  # dataset
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

Next, we load the HIV incentives dataset from the causaldata package and take a quick look:

# Load the Thornton (2008) HIV dataset
data = thornton_hiv.load_pandas().data
print(data.head())

This dataset contains the following columns:

• villnum – Village ID
• got – Indicator if the person got their HIV test results (this is the outcome we want to maximize)
• distvct – Distance (in km) to the testing center
• tinc – Total incentive amount offered (in local currency)
• any – Indicator if any incentive was offered (tinc > 0)
• age – Age of the individual
• hiv2004 – The person’s HIV status result (1 if positive, 0 if negative)

For context, in the original study without any incentive, only about 34% of people learned their HIV status, but even a small incentive (worth ~one-tenth of a day’s wage) doubled that share. In other words, incentives have a big impact on the likelihood of got=1. Our goal is to exploit this by allocating a fixed incentive budget optimally.

Before modeling, we’ll do a train-test split on the data. We’ll train the ML model on one portion and reserve another portion to simulate a “deployment” scenario where we decide incentives for a new set of individuals (the test set). This isn’t strictly necessary for optimization, but it mirrors a realistic situation where you train on historical data and then optimize for a new group.

# Define feature and target columns
features = ['tinc', 'distvct', 'age']
target = 'got'

# Split data into training and test sets
train, test = train_test_split(data, test_size=0.2, random_state=0)
# For optimization, we'll treat 'tinc' (incentive) as a decision variable.
# Remove the actual 'tinc' and outcome 'got' from the test feature set (they will be decided/predicted).
test = test.drop(columns=['tinc', 'got'])
print(test.shape[0], "individuals in test set.")

After this step, train contains the data (including incentives and outcomes) we’ll use to fit our model, while test contains the features of individuals for whom we need to decide incentives. We dropped tinc and got from test because in that set tinc will be determined by our optimization and got will be predicted by our model. The test set size is printed (for example, 200 individuals).

Step 2: Training a Causal Prediction Model

Now we train a machine learning model that predicts the probability that an individual will get their HIV results (got=1) given their features. We choose a logistic regression model, since it’s appropriate for binary outcomes and, importantly, it’s supported by Gurobi’s ML integration. We’ll include distvct (distance) and age as predictive features, and tinc (incentive amount) as well, since offering a higher incentive should increase the probability of retrieval.

Before fitting, we scale the features using StandardScaler (a common practice to improve logistic regression convergence). We’ll create a scikit-learn pipeline for convenience.

To gain insight into how Gurobi-ML works, we instrument the scaler and logistic regression classes to print messages whenever their attributes or methods are accessed. This is a neat trick to peek under the hood later. (We subclass StandardScaler and LogisticRegression and override __getattribute__ to log calls.) This isn’t required for functionality, but it will let us see which model parameters Gurobi reads when formulating the optimization constraints.

# (Optional) Wrap StandardScaler to log attribute access for demonstration
class LoggingStandardScaler(StandardScaler):
    def __getattribute__(self, name):
        attr = super().__getattribute__(name)
        if callable(attr):
            def new_func(*args, **kwargs):
                print(f'Calling StandardScaler.{name}()')
                return attr(*args, **kwargs)
            return new_func
        else:
            print(f'Accessing StandardScaler.{name} attribute')
            return attr

# Wrap LogisticRegression similarly
class LoggingLogisticRegression(LogisticRegression):
    def __getattribute__(self, name):
        attr = super().__getattribute__(name)
        if callable(attr):
            def new_func(*args, **kwargs):
                print(f'Calling LogisticRegression.{name}()')
                return attr(*args, **kwargs)
            return new_func
        else:
            print(f'Accessing LogisticRegression.{name} attribute')
            return attr
# Use the wrapped classes for transparency
scaler = LoggingStandardScaler()
logreg = LoggingLogisticRegression(random_state=1)

Now we set up the pipeline with our logging scaler and logistic regressor, and fit it to the training data:

# Create a pipeline and train the model on the training set
pipe = make_pipeline(scaler, logreg)
pipe.fit(X=train[features], y=train[target])

After fitting, we have a trained logistic model pipe that can predict the probability got=1 for a person, given their incentive tinc, distance, and age. This model will serve as our predictive component in the optimization.

Technical note: In this dataset, the logistic regression will learn, for example, that greater incentives (tinc) increase the probability of uptake (got), while greater distance (distvct) likely decreases it (since it’s harder to travel to get results), and perhaps age has some effect. We won’t focus on the exact model coefficients here, but rather on how to use the model in optimization.

Step 3: Formulating the Optimization Problem

With the predictive model in hand, we turn to formulating the optimization problem. The goal is to choose incentive amounts for each individual in our test set to maximize the total number of people who get their results, subject to a budget constraint.

Let’s denote:

• x_i as the incentive amount we give to person i (our decision variables, corresponding to tinc).
• y_i as the predicted probability that person i gets their result (which is output by our ML model given x_i, distance, and age).

We want to maximize the sum of y_i (expected number of people who get results). The constraints are: (1) we have a limited total budget for incentives, and (2) each individual’s incentive is capped at a maximum (in the study, incentives were at most 3 units of currency).

In equation form, our optimization model is:

where Bis the total incentive budget and f(...) is the prediction function given by our logistic regression model (mapping incentive, distance, age to the probability of got=1).

Now, let’s implement this with Gurobi. We’ll create a new optimization model and add the decision variables:

# Create a new Gurobi model
m = gp.Model()

# Add a decision variable y_i for each test instance to represent probability of outcome (got=1)
y = gppd.add_vars(m, test, name="probability")
# Add a decision variable x_i (incentive) for each test instance, with bounds 0 <= x_i <= 3
test = test.gppd.add_vars(m, lb=0.0, ub=3.0, name="tinc")
x = test["tinc"]
# Ensure the DataFrame `test` now has columns [tinc, distvct, age] in the correct order
test = test[["tinc", "distvct", "age"]]
print(test.head())

The gppd.add_vars utility from gurobipy_pandas helps create variables aligned with the indices of our test DataFrame. We first added y variables (one per individual) and then added a new column "tinc" of variables to the test DataFrame for the incentives. After this, test now includes the tinc variable column along with each person’s fixed features (distvct and age).

Printing test.head() would show something like:

tinc                    distvct      age
993 <gurobi.Var *Awaiting>  2.144576   30.0
859 <gurobi.Var *Awaiting>  3.905001   25.0
298 <gurobi.Var *Awaiting>  2.306510   33.6
553 <gurobi.Var *Awaiting>  0.725098   23.0
672 <gurobi.Var *Awaiting>  3.821342   50.0

Here tinc entries are Gurobi decision variables (initially awaiting model update), and distvct and age are the data for those individuals. The indices (993, 859, ...) come from the original dataset’s indexing.

Now we add the objective and budget constraint. Let’s say the budget $B$ is 0.2 times the number of people (this was mentioned as 0.2n in the project, meaning on average we can spend 0.2 units per person – if $n=200$, $B=40$). We’ll compute that and add the constraint $\sum_i x_i \le B$:

# Set the total budget B as 0.2 * number of test individuals
budget = 0.2 * test.shape[0]

# Set objective: maximize sum of y_i probabilities
m.setObjective(y.sum(), gp.GRB.MAXIMIZE)
# Add budget constraint: total incentive sum <= budget
m.addConstr(x.sum() <= budget, name="budget")
m.update()

At this point, we have an optimization model with:

• one continuous variable x_i per person (incentive amount),
• one continuous variable y_iper person (predicted outcome probability),
• an objective to maximize sum(y_i),
• and a linear constraint on sum(x_i).

What’s missing is the link between x_i and y_i — i.e., the constraints that force y_i to equal the logistic model’s prediction given x_i, distance, and age. We handle that next.

Step 4: Integrating the ML Prediction Constraints

Gurobi’s add_predictor_constr function is the key to integrating our trained pipeline pipe into the optimization model. It will add all necessary constraints to relate the input variables (x, distvct, age) to the output variable (y) according to the machine learning model’s equations.

# Add constraints from the trained ML model (pipeline) to link x, distvct, age to predicted y
pred_constr = add_predictor_constr(m, pipe, test, y, output_type="probability_1")
pred_constr.print_stats()

A couple of things to note in the call above:

• We pass pipe (our trained sklearn pipeline) and test (the DataFrame of input variables: now containing Gurobi vars for tinc and the fixed distvct, age values).
• We also pass y (the Gurobi vars for the outputs).
• output_type="probability_1" specifies that for a binary classifier, we want the constraint to produce the probability of class 1 (in our case the probability got=1). Gurobi-ML supports getting either raw prediction, class, or probability; here we need the probability.

When add_predictor_constr runs, it effectively creates a bunch of new internal decision variables and constraints that represent the computations of the pipeline. This includes the scaling transformation and the logistic regression piecewise linear approximation. We used our logging subclasses, so during this process you would see messages like:

Accessing StandardScaler.scale_ attribute  
Accessing StandardScaler.mean_ attribute  
Accessing LogisticRegression.coef_ attribute  
Accessing LogisticRegression.intercept_ attribute

These indicate Gurobi-ML is reading the trained model parameters to formulate the constraints.

The pred_constr.print_stats() call then prints a summary of the added model:

Model for pipe1:
1200 variables
800 constraints
200 general constraints
Input has shape (200, 3)
Output has shape (200, 1)

Pipeline has 2 steps:
-----------------------------------------------------
Step                Output Shape    Variables  Constraints   Linear   Quadratic   General
=====================================================
std_scaler1         (200, 3)        1000       600           0        0           0
log_reg1            (200, 1)        200        200           0        0           200
-----------------------------------------------------

What does this mean? The pipeline had 2 steps (the scaler and the logistic regressor):

• The scaler step introduced 1000 new variables and 600 constraints (these come from how scaling is applied to each of the 3 features for 200 data points : essentially linear equations enforcing
  z_i_dist = (distvct_i - mean_dist) / sd_dist,
  and similarly for age).
• The logistic regression step introduced 200 variables and 200 general constraints. Those “general constraints” are Gurobi’s way of handling the piecewise‑linear approximation of the logistic function (since a logistic curve isn’t linear, it’s approximated by SOS2 constraints or similar). In total, 1200 extra variables and 1,000 constraints were added to represent the whole pipeline across 200 individuals.

The result is an optimization model that fully encodes the relationship
y_i ≈ f(x_i, distvct_i, age_i)
as defined by our trained ML model. Now we can let Gurobi do the heavy lifting to find the optimal incentive allocation.

Step 5: Optimizing and Examining Results

With everything in place, we optimize the model:

# Optimize the model
m.optimize()

Once you run this, Gurobi will iterate and find the optimal solution. The solver log (truncated) might look like:

Optimize a model with 801 rows, 1600 columns and 2200 nonzeros
Model has 200 general constraints
...
Presolved model has 200 SOS constraint(s)
Root relaxation: objective 5.941394e+01
...
Solution count 1: 59.4139 
Optimal solution found (tolerance 1.00e-4)
Warning: max constraint violation (6.53e-03) exceeds tolerance

The optimal objective value is about 59.4139. Since our objective was the sum of predicted probabilities, this suggests the model expects about 59 out of the 200 people in the test set to get their results under the optimal incentive allocation. (For comparison, if no incentives were given, the logistic model would predict a much lower number — recall only ~34% might follow up with no incentive on average, which would be ~68 people; our budget isn’t enough to treat everyone, but it tries to maximize that count.)

The warnings about constraint violation (~6.5e-3) are due to the piecewise linear approximation of the logistic function. Essentially, the $y_i$ values might be off by up to 0.0065 (about 0.65%) from the “true” logistic curve. This is a minor approximation error. We can check the maximum approximation error directly:

# Check the maximum approximation error in the logistic regression constraint
max_error = np.max(pred_constr.get_error())
print(f"Maximum error in approximating the regression {max_error:.6f}")

This prints something like Maximum error in approximating the regression 0.006531, confirming the ~0.0065 max deviation. This error is small, but if needed, one could tighten the approximation (e.g., by adjusting Gurobi parameters for piecewise linear functions).

Now, let’s retrieve the optimized incentive values x_i. We expect many of them to be zero (not everyone gets paid) and some to be at the maximum of 3 (the best use of budget for the most “persuadable” individuals). We also ensure none are negative (they shouldn’t be, but due to the tiny approximation, some variables might come out as very small negative numbers like -1e-6, which we can treat as 0):

# Get the optimized incentive values and round/floor small numerical artifacts
tinc_solution = pred_constr.input_values['tinc']  # pandas Series of optimal x_i
tinc_solution = tinc_solution.apply(lambda x: 0 if x < 0 else np.floor(x * 1e5) / 1e5)
print(tinc_solution.head())

After this, tinc_solution holds each person’s incentive in the optimal plan (rounded down to 5 decimal places to clean up float quirks). Finally, let’s verify that the budget constraint is satisfied:

print(tinc_solution.sum() <= budget)  # This should output True

Indeed, the total sum of incentives used will be within the budget (likely exactly equal or very close to it, since we expect the budget to be fully utilized for maximum objective).

At this point, one could further analyze the solution — for example, how many people got the max incentive versus none, or which characteristics made someone more likely to receive an incentive. Those with moderate distances and ages that made them “persuadable” likely got the incentive, whereas those very close (who might go anyway) or very far (who might not go even with incentive) might not get any in the optimal plan. Such analysis can be done by examining tinc_solution values and the corresponding features.

For our purposes, we have demonstrated the core idea: using Gurobi to optimally allocate treatment based on a learned causal model. We’ve effectively solved a constrained uplift optimization problem — something uplift modeling alone cannot do directly.

Conclusion

The integration of machine learning predictions and mathematical optimization shows great promise for causal decision-making in specific use cases. Traditional uplift modeling can identify who is influenced by a treatment, but it doesn’t tell you how to optimally deploy that treatment under real-world constraints (e.g., limited budget, capacity, etc.). In our example, instead of just predicting who would respond to an incentive, we formulated an optimization to decide exactly who should receive an incentive to maximize overall success. This addresses questions like, “Who should we treat given we can only afford to treat X people?” which uplift models alone leave unanswered.

By leveraging Gurobi Machine Learning, we were able to incorporate a complex prediction (logistic regression) directly into an optimization model. The solver then figured out the best treatment plan — something like deciding which students to offer scholarships to, or which patients to give an intervention — while respecting constraints. This approach provides a more comprehensive solution to causal inference questions that involve decision-making, beyond just analysis.

To sum up, this tutorial demonstrated how to turn predictive insights into optimal decisions. We showed how a causal inference problem (maximizing outcomes under constraints) can be solved by embedding an ML model into an optimization problem. This approach can be applied in many domains: marketing (who to target for a campaign), healthcare (who to treat or screen), economics (who to subsidize), and beyond. As tools like Gurobi Machine Learning mature, they open the door for data scientists and operations researchers to collaborate in solving decision problems that lie at the intersection of AI and optimization.

The full code for this demo is available in the project’s repository, and we encourage you to explore it and experiment with your own scenarios. With a bit of modeling creativity, you can start answering “What should we do?” — not just “What do we predict?” — using the power of optimization.

TL;DR

While Web Components are hyped as the future — promising portability and longevity, the reality is more nuanced. In my experience after building my own UI library, flow‑modal, Web Components (especially when built with Lit) shine for small, isolated widgets but introduce challenges when used as the backbone of a full‑scale application. This post explores bundle size, templating performance, renderless components, styling concerns, UI flashing, and more — with plenty of code examples.

The Hype

According to “2023 State of Web Components”, the web community has witnessed impressive adoption:

• YouTube: The YouTube app has long relied on custom elements (e.g., <ytd-video-preview>) to deliver a fluid user experience.
• Adobe Photoshop for Web: Adobe’s web‑based Photoshop uses Lit and Web Components to create a rich, interactive interface.
• Microsoft: From the MSN experience to the UI Toolkit in VS Code and even the new OpenAI‑powered Bing, Microsoft has embraced FAST Web Components, reporting performance gains of 30%–50% compared to previous React implementations.

Proponents claim that, because Web Components rely solely on native browser APIs, they’ll remain future‑proof. Blog posts like “Web Components Will Outlive Your JavaScript Framework” have even suggested that these standards are so robust that they outlast popular frameworks.

Yet, not everyone is sold. Influential voices such as Rich Harris (the creator of Svelte) and Ryan Carniato (Solid’s author) have published thoughtful pieces — “Why I don’t use web components” and “Web Components Are Not the Future” — highlighting concerns ranging from performance to developer ergonomics. Although Rich’s article was published in 2020 and became a bit outdated, their critiques remind us that while Web Components are powerful, they’re not a one‑size‑fits‑all solution.

Shadow DOM

One of the major advantages that often comes up when discussing Web Components is the use of Shadow DOM. Although Shadow DOM is not part of the Web Components specification per se — it’s its own standard — it works in perfect harmony with Web Components.

• Encapsulation: Shadow DOM encapsulates the component’s internal structure and styles, preventing unwanted style leakage in or out. This makes components more predictable and easier to manage.
• Styling Isolation: By isolating CSS, Shadow DOM allows developers to design components that look and behave consistently, regardless of where they’re used.

My Skepticism

Despite these exciting features, my own journey has taught me that Web Components are best suited for small, independent widgets rather than as the foundation for an entire website. Issues such as extra DOM nodes, templating overhead, UI flashing before component upgrade, and styling challenges (despite Shadow DOM’s benefits) reveal that the “magic” comes with trade‑offs.

In what follows, I’ll detail these challenges, starting with the underlying technology and moving through templating, styling, and even the user experience.

The Allure of Web Components

At their core, Web Components are a set of W3C standards that let you define custom HTML elements with encapsulated behavior, style, and markup. In theory, you write one component, register a custom tag (like <canvas-demo>), and—once its script is loaded—it just works anywhere in your HTML. For example:

<!-- Pure HTML in Markdown -->
<canvas-demo resolution="20" debug></canvas-demo>

Behind the scenes, the component might look like this:

class CanvasDemo extends HTMLElement {
  constypestructor() {
    super();
    this.attachShadow();
    this.render();
  }

  render() {
    this.shadowRoot.innerHTML = `
      <div class="canvas">
        <input type="color" class="color" value="#000000" />
      </div>
      <style>
        .canvas { /* input styling */ }
      </style>
    `;
  }
}

customElements.define("canvas-demo", CanvasDemo);

This simplicity and portability are compelling. However, when scaling up to full‑scale websites, complexities arise.

Lit and Bundle Sizes

Many developers — myself included — use a framework like Lit to build Web Components. Lit’s templating system, reactive properties, and built‑in support for Shadow DOM enable developers to write concise, maintainable code. And speaking of bundles, here’s a quick comparison from minimal projects built with Vite:

• React: ~140 KB
• Vue: ~57 KB
• Lit: ~17 KB

While Lit’s footprint is impressively small (around 17 KB in a minimal setup), it’s important to understand how dependencies are managed when you build multiple independent components. Often, developers assume that using Web Components is as simple as importing the library. However, if each component individually imports Lit as a direct dependency, it creates a separate copy of Lit for every component.

To avoid this, it’s essential to mark Lit as a peer dependency. By doing so, you ensure that all your components rely on a single, shared instance of Lit, and your build system (or bundler) will include Lit only once in the final bundle.

However, there’s a trade-off. While using Lit as a peer dependency minimizes the size overhead, it also means that the seamless, plug‑and‑play experience — where you simply drop in <canvas-demo> and expect it to work independently—is not entirely realized.

Template Diffing vs. Reactive Rendering

Lit leverages tagged template literals to create HTML templates, and when data changes, it re‑evaluates the entire template and computes a diff against the previous version. For example:

import { LitElement, html } from 'lit';

class MyElement extends LitElement {
  render() {
    return html`<div>Hello, ${this.greeting}!</div>`;
  }
}

customElements.define('my-element', MyElement);

When this.greeting changes, Lit re‑generates the template and uses its diffing algorithm to update only the parts of the DOM that have changed. This approach fits naturally with the Web Components model—but it comes with a runtime cost due to the diffing process.

In contrast, Solid takes a different approach by compiling templates into direct, fine‑grained reactive bindings. Consider this Solid component:

import { createSignal, createEffect } from 'solid-js';

function App() {
  const [name, setName] = createSignal('World');
  
  return (
    <div>
      Hello, {name()}!
    </div>
  );
}

Conceptually, Solid compiles this component into something similar to:

function AppCompiled() {
  const [name, setName] = createSignal('World');
  
  // Create DOM nodes manually:
  const div = document.createElement('div');
  const prefixText = document.createTextNode('Hello, ');
  const nameText = document.createTextNode(name());
  
  div.appendChild(prefixText);
  div.appendChild(nameText);
  
  // Direct reactive binding:
  createEffect(() => {
    nameText.data = name();
  });
  
  return div;
}

Here’s what distinguishes the two approaches:

Lit

• Re‑renders the Template: When data changes, Lit re‑evaluates the entire template.
• Diffing Process: It then computes the difference between the new and old templates and updates only the changed parts in the DOM.
• Runtime Overhead: While efficient for many scenarios, this diffing process incurs a runtime cost.

Solid

• Direct Reactive Bindings: Solid’s compiler creates direct associations between reactive signals and the specific DOM nodes that depend on them.
• No Diffing Required: When the data updates, only the affected DOM node is updated immediately, with no diffing step.
• Performance Edge: This minimal approach reduces overhead, making updates faster — especially for frequent or granular changes.

It’s important to stress that this performance trade-off is a limitation of Lit’s design, not an inherent limitation of Web Components. Lit’s approach was chosen to provide a flexible, declarative developer experience that aligns with Web Components’ lifecycle.

SSR Limitations

Server‑Side Rendering (SSR) is another area where Web Components face challenges. Frameworks like React and Vue have mature SSR ecosystems largely because they use a Virtual DOM — a plain JavaScript representation of the UI — that can be rendered to an HTML string without needing a full browser environment. This abstraction decouples them from the actual DOM API, making SSR both realistic and efficient. When React or Vue renders on the server, they generate a lightweight, tree‑structured object that represents the UI and then convert it to HTML without invoking any browser‑specific features.

In contrast, Web Components (and thus Lit) are designed to work with the real DOM and depend on browser‑specific APIs and lifecycles. Simulating the actual DOM on the server is a non‑trivial task — it requires polyfilling or emulating many complex APIs that browsers provide natively. Although experimental projects such as @lit-lab/ssr are making progress in this area, full‑featured, production‑ready SSR for Web Components remains a work in progress.

Renderless Components

A recurring challenge with Web Components (and Lit) is the inability to create truly renderless components. In frameworks like React or Vue, you can return null or an empty fragment, thus creating no extra DOM node for purely behavioral components.

React Example:

function MouseTracker({ onMouseMove }) {
  React.useEffect(() => {
    window.addEventListener('mousemove', onMouseMove);
    return () => window.removeEventListener('mousemove', onMouseMove);
  }, [onMouseMove]);
  return null; // No DOM element is created
}

Lit/Web Component Example:

import { LitElement, html } from 'lit';

class MouseTracker extends LitElement {
  connectedCallback() {
    super.connectedCallback();
    window.addEventListener('mousemove', this.handleMouseMove);
  }

  disconnectedCallback() {
    window.removeEventListener('mousemove', this.handleMouseMove);
    super.disconnectedCallback();
  }

  handleMouseMove(e) {
    this.dispatchEvent(new CustomEvent('mouse-move', { detail: e }));
  }

  render() {
    // Even an empty template results in a <mouse-tracker> element in the DOM.
    return html``;
  }
}

customElements.define('mouse-tracker', MouseTracker);

This extra DOM node may be trivial in isolation, but when building many orchestration components, the inability to be renderless results in unnecessary DOM bloat.

Composition and Rendering Inefficiencies

My experience with Web Components was building flow‑modal — a UI library for orchestrating multi‑step modal flows. Consider the markup:

<flow-modal>
  <flow-step ready timeout-duration="1m">
    <flow-badge button-count-down-format="mm:ss"></flow-badge>
    <h2>Welcome to Our Presentation!</h2>
    <p>This is the first step of our presentation flow. You have one minute to review this announcement.</p>
  </flow-step>

  <flow-step ready>
    <flow-badge button-text="Click"></flow-badge>
    <h2>Presentation Details</h2>
    <p>Thank you for viewing the introduction. This is the second step of the presentation.</p>
  </flow-step>
</flow-modal>

With this markup, you can have the stateful modals and badges as follow:

While the concept is to use slotting to compose content , the reality is that all <flow-step> and <flow-badge> components render eagerly, even when inactive. In a large‑scale application with hundreds of components, this eager rendering can lead to inefficiencies that frameworks offering lazy‑loading or renderless components inherently avoid.

The Flash of Unstyled Content (FOUC)

One particularly vexing issue while working with Web Components is UI flashing. Since the browser renders slot content immediately — even before the Web Component’s definition is loaded — the user may see unstyled or default content briefly before the component upgrades. Here’s the common workaround:

flow-modal:not(:defined),
flow-step:not(:defined) {
  display: none;
}

While this CSS rule hides the <flow-modal> and <flow-step> until it’s defined, it must be applied at the host level. This requirement forces every consumer of your Web Component to remember the tip, which can lead to a fragmented and error‑prone developer experience.

The Challenge of Passing Data

One subtle — but significant — limitation of Web Components is how they handle attributes and properties. In frameworks like React or Vue, you can pass complex objects or functions directly as props. For example, in React you might write:

// React example: Passing an object and a callback
<FlowBadge 
  buttonOptions={{ text: 'Hello', ariaLabel: 'Greeting' }} 
  buttonCountDownFormatter={buttonCountDownFormatter} 
/>

With Web Components, every attribute is passed as a string. If you need to pass complex data like objects or arrays, you must serialize them (typically with JSON) and then parse them inside your component:

<flow-badge button-options='{"text":"Hello", "ariaLabel": "Greeting"}'>

A particularly challenging example from my own experience involves passing a function to format the remaining time in my widget. In a typical DOM element, you might pass a function via the onclick attribute. Browsers treat onclick specially—they can resolve a function even if it’s not explicitly bound. However, in Web Components, attributes are always strings. For instance, I tried to do something like:

<!-- Intended usage: -->
<flow-badge button-count-down-formatter="buttonCountDownFormatter"></flow-badge>

buttonCountDownFormatter is intended to be a function that receives remaining time as its argument and returns the formatter string. But with Web Components, every attribute should be a string.

You may think of dispatching a custom event when it needs to perform the formatting:

// Inside the web component:
this.dispatchEvent(new CustomEvent('format-request', { detail: timeRemaining }));

However, the event system in the DOM is one‑way: even if the listener computes a formatted string, that return value isn’t captured by the component that dispatched the event.

CSS in JavaScript

Another important aspect of working with Web Components is how CSS is handled. With Lit, styles are written directly in JavaScript using tagged template literals. For example:

static styles = css`
  :host {
    display: block;
    padding: 16px;
    color: #333;
  }
`;

This approach offers encapsulation, ensuring that the component’s styles are tightly bound to its Shadow DOM. But it also means that designers who prefer to work with standalone CSS files may feel constrained.

It’s important to note that this isn’t a limitation of Lit. Rather, it’s an inherent limitation by the current Web Components specification. There are emerging solutions, such as CSS Module Scripts, which would allow you to import a CSS file as a JavaScript module.

However, these solutions generally require extra build steps and depend on browser support that is still evolving. As such, for the plug‑and‑play simplicity many users expect from Web Components, working with CSS directly in JavaScript remains the norm for now.

My Experience with flow‑modal

After wrestling with these trade‑offs, I decided to put Web Components to the test by building flow‑modal.

GitHub - yujiosaka/flow-modal: A guided, multi-step modal using customizable web components and state persistence.

I published a Storybook so you can explore the components interactively.

Single Step:

Storybook

Multi Step:

@storybook/core - Storybook

Despite the robust feature set, my conclusion remains clear: Web Components (and Lit) are ideal for small widgets and isolated UI modules — but not for building an entire website. In my experiments:

• Complexity: The quirks of attribute parsing, inability of passing functions and eager slot rendering lead to increased technical debt as projects scale.
• Render Overhead: Inability to create renderless components forces extra DOM nodes into your page.
• Developer Collaboration: The coupling of JavaScript and CSS creates friction.
• UI Flashing: Without host‑side CSS, users experience a flash of unstyled content before components upgrade.

Conclusion

Don’t get me wrong — Web Components and Lit offer a modern, powerful development experience. Their minimal bundle size (as low as 17 KB in a minimal Lit setup), built‑in Shadow DOM, and straightforward API make them excellent for building small, self‑contained widgets like chat badges, modals, or custom form elements.

However, if you’re planning to build a complex website with hundreds of interactive components and intricate data flows, the benefits of a full‑featured framework with a fine‑grained reactivity (like React, Vue, or Solid) can outweigh those of a pure Web Components approach. Moreover, the extra DOM nodes mandated by Web Components, along with issues like UI flashing, make them less ideal for tightly coupled applications.

In the end, it’s all about choosing the right tool for the job.