<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pietro Bolcato on Medium]]></title>
        <description><![CDATA[Stories by Pietro Bolcato on Medium]]></description>
        <link>https://medium.com/@pietrobolcato?source=rss-52732991c759------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*crYYaioxqeBJR9relgoKGQ.jpeg</url>
            <title>Stories by Pietro Bolcato on Medium</title>
            <link>https://medium.com/@pietrobolcato?source=rss-52732991c759------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 01:15:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pietrobolcato/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Spec-Driven Development: Code is the New Assembly]]></title>
            <link>https://towardsdev.com/spec-driven-development-code-is-the-new-assembly-58ef4a9bfe52?source=rss-52732991c759------2</link>
            <guid isPermaLink="false">https://medium.com/p/58ef4a9bfe52</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[vibe-coding]]></category>
            <dc:creator><![CDATA[Pietro Bolcato]]></dc:creator>
            <pubDate>Thu, 22 Jan 2026 07:30:10 GMT</pubDate>
            <atom:updated>2026-01-23T07:31:27.364Z</atom:updated>
            <content:encoded><![CDATA[<h4>The natural next steps in Software Engineering</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fIHguUZuN1jrc4rfdkoq0A.png" /></figure><h3>Introduction</h3><p>I’ve been an early adopter of AI tooling and agents; I started experimenting with autocompletion at the time when GPT-2 was released and it was possible to finetune for this purpose. We are living through a massive compression of the abstraction stack. Just think about this: in the 90s, we stopped managing memory manually because compilers got good enough. In the 2000s, we stopped writing low-level data structures because standard libraries handled the boilerplate.</p><p>Today, we are reaching the next logical step: <strong>Spec-Driven Development (SDD)</strong>.</p><p>With the maturity of coding agents (latest Opus and Gemini) and context-aware IDEs like <strong>Cursor</strong> and <strong>Windsurf</strong>, the actual syntax (the for loops, the API calls, the Pydantic models) is becoming an obvious implementation detail, almost the new Assembly language. We shouldn’t be hand-writing it; we should be compiling it from a higher-level source.</p><p>That source is the <strong>Specification</strong>.</p><h3>The Agent as is your Junior Partner</h3><p>The biggest mistake engineers make is treating AI as a “code completion” tool. Sure this is powerful, but the real leverage comes when you treat the agent as a capable, tireless Junior Engineer, while you act as the <strong>Principal Architect</strong>.</p><p>In SDD, you don’t start by writing code. You don’t even start by writing the formal spec. You start with a <strong>Brain dump</strong>.</p><p>You describe your architectural constraints, your throughput goals, and your failure modes in natural language. The Agent takes that unstructured intent and compiles it into a structured <strong>Plan</strong>. Then, the real work begins: a negotiation loop where you review the agent’s proposal, spot architectural bottlenecks, and refine the requirements until the design is resilient.</p><h4>A Practical Example: The Synthetic Data Flywheel</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/1*zeZtk50AcgVsDVyRe2XMog.png" /></figure><p>Let’s look at a real-world scenario. We need to build a <strong>Synthetic Data Pipeline</strong> to distill a large reasoning model into a smaller, domain-specific SLM (Small Language Model). We need to generate 100k+ unique, high-quality samples across a complex taxonomy.</p><h4>The Old Way (Code-First)</h4><p>You’d open an editor, install langchain, and start hacking a loop. You’d pick the first vector store you find in the docs. Four hours later, you have a script that OOMs (Out of Memory) on large datasets, has no resume capability, and burns through API credits because it re-generates duplicates. You’ve built technical debt.</p><h4>The SDD Way (Spec-First)</h4><p><strong>Step 1: The Context Dump (The Senior/Architect View)</strong><br>You open a chat with the agent and define the system boundaries:</p><blockquote><em>We need a robust pipeline to generate synthetic SFT (Supervised Fine-Tuning) data. Inputs are a deep taxonomy tree in YAML, see example in </em>/path/to/file</blockquote><blockquote><em>Requirements:<br>- </em><strong>Idempotency:</strong> I need to be able to kill the process and restart it without losing progress or duplicating costs.<br>- <strong>Diversity:</strong> We need semantic deduplication<br>- <strong>Output:</strong> Data must be stored in Parquet or Arrow for efficient downstream training ingestion, and compatible wit HF datasets<br>- <strong>Cost Control:</strong> We need strict concurrency limits and a token budget per node.</blockquote><blockquote>Draft a technical specification.</blockquote><p><strong>Step 2: The Agent Drafts the Spec</strong><br>The agent analyzes the request and generates a <em>plan.md</em>. It suggests a standard Python script using Pinecone for deduplication and Postgres for state tracking.</p><p><strong>Step 3: The Human Review (The Staff Engineer Intervention)</strong><br>You read the plan and immediately spot operational complexity.</p><blockquote>You:<em> “Reject the Pinecone and Postgres dependency. This is a batch job, not a microservice. We don’t want to manage external infrastructure for a data generation script.</em></blockquote><blockquote>- <em>State tracking should use a local DuckDB instance or a WAL file.<br>- Deduplication should use a localized HNSW index (via </em>usearch<em> or </em>faiss<em>) that persists to disk.<br>- Add a requirement for a ‘Dry Run’ mode that estimates token costs before execution.<br>- Ensure the schema enforces strict JSON structure via Pydantic.</em></blockquote><p><strong>Step 4: Refinement </strong><br>The agent rewrites the plan. It creates a cleaner, self-contained architecture that requires zero external infra setup, uses pydantic for schema validation, and implements a robust checkpointing system.</p><p><strong>Step 5: Repeat</strong><br>At this point, you’d check the plan again, ask for clarification, improvements, ideas. And the cycle continue until the specification is clear, actionable and robust.</p><p><strong>Step 6: Design Review <em>(optional)</em> <br></strong>Finally, before instructing the agent to actually implement the plan — I have seen very helpful to take it, and out-of-context ask another agent to act as a design reviewer. It’s quite helpful to round the last details and make sure the plan is even more robust.</p><p>The tradeoff is that this can be time consuming; therefore, I typically do it only for larger, non-obvious features.</p><h3>Commit Your Plans</h3><p>Here is an important shift in workflow: <strong>The Spec is a build artifact.</strong></p><p>Files like <em>.cursor/plans/data-pipeline-architecture.md </em>should not be ephemeral context, they should be committed to the repository, for two very important reasons:</p><ol><li><strong>Documentation as Decision Records:</strong> The code tells you <em>how</em> it works. The Plan tells you <em>why</em> we chose DuckDB over Postgres. It captures the constraints at the time of decision.</li><li><strong>Context for the Future:</strong> When a new engineer joins (or you return 6 months later), the agent reads the plan and understands the <em>invariants</em> of the system.</li></ol><h3>The Spec-First PR Review</h3><p>Code reviews often devolve into “bikeshedding”, i.e arguing over variable names or indentation. This is a waste of senior talent, and this is something I actively coach in my team on. Let’s focus on what matters.</p><p>I love that SDD enables a <strong>Architecture-First PR Workflow</strong>: by committing the plan alongside the code, it becomes incredibly easier to understand the logic and review a PR effectively.</p><p>This moves the intellectual labor from “finding bugs” to “designing systems.”</p><h3>Conclusion</h3><p>For years I’ve known that the difference between a Junior and a Senior engineer isn’t code speed, but design clarity. AI has validated that belief.</p><p>SDD forces a necessary evolution. It demands that we maintain a higher level of abstraction, treating natural language specifications with the same rigor we once reserved for code. We are no longer translators of logic; we are architects of systems. We always were, actually!</p><p>The prompt is becoming the new program, with the code being just the binary. The abstraction layer has moved up; I’d strongly recommend to make sure you move with it too.</p><h3>References</h3><ul><li><a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Ftom.preston-werner.com%2F2010%2F08%2F23%2Freadme-driven-development.html">Readme Driven Development</a> — <em>The spiritual predecessor to SDD.</em></li><li><a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FThe_Mythical_Man-Month">The Mythical Man-Month (Brooks)</a> — <em>Specifically the chapters on design vs. implementation.</em></li><li><a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Fadr.github.io%2F">Architecture Decision Records (ADRs)</a> — <em>Why capturing the “why” matters more than the “how”.</em></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58ef4a9bfe52" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/spec-driven-development-code-is-the-new-assembly-58ef4a9bfe52">Spec-Driven Development: Code is the New Assembly</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recursive Language Models: Infinite Context that works]]></title>
            <link>https://medium.com/@pietrobolcato/recursive-language-models-infinite-context-that-works-174da45412ab?source=rss-52732991c759------2</link>
            <guid isPermaLink="false">https://medium.com/p/174da45412ab</guid>
            <category><![CDATA[scalability]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Pietro Bolcato]]></dc:creator>
            <pubDate>Thu, 08 Jan 2026 07:35:58 GMT</pubDate>
            <atom:updated>2026-01-08T07:35:58.678Z</atom:updated>
            <content:encoded><![CDATA[<h4>How treating the prompt as an external environment enables O(log N) retrieval and solves the attention bottleneck</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4eAZQHqIww0TvQoSxss43A.png" /></figure><h3>Introduction</h3><p>Lots of effort in Large Language Model research has been the pursuit of the “infinite context window.” Through techniques like Ring Attention, Rope scaling, and linearized attention mechanisms, we have pushed input limits from 4k to 1M+ tokens. However, empirical evidence (the “Lost-in-the-Middle” phenomenon) suggests that <strong>attention is not understanding</strong>. As sequence length <em>N</em> grows, the signal-to-noise ratio in the key-value (KV) cache degrades, limiting reasoning capabilities over massive documents.</p><p>The paper <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2512.24601">Recursive Language Models</a> introduces a new idea which is quite incredible because of how simple it is: treating context not as a tensor input to the transformer’s forward pass, but as an <strong>external state variable</strong> in a Turing-complete environment. By combining an LLM with a Python REPL and a recursive self-query primitive, Recursive LMs transform long-context processing from an architectural problem into a <strong>program synthesis</strong> problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wLG43es73BrzCMeGEhYfrA.png" /></figure><h3>Formalizing the Approach: Context-as-State</h3><p>In a standard Long-Context LLM, the generation <em>y </em>is conditioned on the entire prompt <em>P</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/1*xBA3LsKFWTpLBMRS2tPVKQ.png" /></figure><p>Computational complexity for attention is typically <em>O(N²) </em>(or <em>O(N)</em> for linear attention variants), and memory usage for the KV cache scales linearly or quadratically.</p><p>In an <strong>RLM</strong>, the prompt <em>P </em>is stored in external memory <em>M</em> (e.g., a string variable in a Python environment). The model does not see <em>P</em>. Instead, the model generates an executable program <em>C </em>(actions) based on a task description <em>T</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/342/1*n_nT6E1OQGk7G9KlVjb-Pw.png" /></figure><p>The program <em>C </em>interacts with <em>M</em> via standard I/O operations (slicing, string manipulation) and, crucially, a recursive primitive:</p><pre>output = llm.query(prompt=instruction, context=chunk)</pre><p>This effectively decouples the <strong>Task Context</strong> (the high-level goal) from the <strong>Data Context</strong> (the massive document).</p><h3>Algorithmic Decomposition</h3><p>The RLM framework allows the model to synthesize algorithms to solve retrieval and reasoning tasks. This shifts the inductive bias from the Transformer’s attention mechanism to the control flow logic of Python.</p><h4>1. Divide and Conquer (Map-Reduce)</h4><p>For global reasoning tasks (e.g., “Summarize the timeline of events across this 10M token log file”), the RLM synthesizes a map-reduce strategy:</p><ul><li><strong>Map:</strong> It writes a loop to chunk the variable data into segments of size <em>K </em>(where <em>K </em>fits comfortably in the effective context window, e.g., 4k tokens). It calls llm.queryon each chunk to extract local timelines</li><li><strong>Reduce:</strong> It aggregates the intermediate outputs and recursively calls itself to synthesize the final timeline</li></ul><h4>2. Semantic Binary Search</h4><p>For “Needle in a Haystack” retrieval tasks, standard models must attend to all tokens. An RLM can write a semantic search algorithm:</p><ul><li>Divide the text into 10 large sections.</li><li>Query: “Does the event X happen in this section?”</li><li>Recursively drill down into the positive section.<br>This reduces the computational complexity from linear (scanning all tokens) to logarithmic <em>O(log N)</em></li></ul><h3>Inference-Time Compute vs. Input Length</h3><p>A key contribution of the paper is the analysis of <strong>Test-Time Compute (TTC)</strong>.</p><ul><li><strong>Standard LLMs:</strong> Compute is constant for a given input length <em>N </em>(one forward pass per generated token). It pays the compute cost for irrelevant noise tokens in every layer</li><li><strong>RLMs:</strong> Compute is dynamic. The model trades “tokens read” for “steps of reasoning.”</li><li>For sparse tasks (needle retrieval), RLM compute is negligible compared to standard LLMs because it discards irrelevant chunks via code logic before the LLM ever processes them.</li><li>For dense tasks (global summarization), RLM compute scales linearly but maintains higher accuracy because the <strong>effective context</strong> for any single inference step remains small and clean.</li></ul><p>This essentially moves the complexity burden from the <strong>Attention Mechanism</strong> (hardware/memory bound) to the <strong>Agentic Control Loop</strong> (software/logic bound).</p><h3>Evaluation: OOLONG and S-NIAH</h3><p>The authors validate RLMs on <strong>OOLONG</strong> (Order-Oriented Long-Context), a benchmark designed to punish simple RAG and approximate attention methods. They got:</p><ul><li><strong>SoTA Performance:</strong> An RLM based on <strong>GPT-4o-mini</strong> outperforms the much larger <strong>GPT-4o</strong> and <strong>Claude 3.5 Sonnet</strong> on tasks requiring reasoning over 1M+ tokens.</li><li><strong>Scaling Laws:</strong> While standard models exhibit performance collapse as context length increases (due to attention dispersion), RLMs show <strong>flat scaling</strong>. Performance does not degrade with sequence length, provided the decomposed chunk size remains constant.</li><li><strong>Cost Inversion:</strong> For massive contexts, RLMs are often cheaper than standard long-context calls. Even though multiple API calls are made, the total number of tokens processed through the transformer layers is significantly lower because irrelevant text is filtered out via string manipulation in Python.</li></ul><h3>Technical Limitations</h3><p>Despite the advantages, RLMs introduce specific failure modes:</p><ol><li><strong>Code Fragility:</strong> The system relies on the model’s ability to write syntactically correct and logically sound Python code. A bug in the loop indexing or string slicing crashes the inference.</li><li><strong>Error Propagation:</strong> In a recursive stack, a hallucination in a leaf node (a sub-query) can propagate up to the root node. Unlike attention, which effectively ensembles information, a discrete function call returns a hard decision.</li><li><strong>Latency:</strong> Sequential processing (loops) prevents the massive parallelism that Transformers excel at. Processing a book chapter-by-chapter sequentially is significantly slower than a single parallel forward pass, even if total FLOPs are lower.</li></ol><h3>Conclusion</h3><p>Recursive Language Models suggest that the solution to the “Infinite Context” problem isn’t larger KV caches, but better <strong>memory management</strong>. By granting LLMs a Turing-complete interface to manage their own context, we enable <em>O(N)</em> or even <em>O(log N)</em> scaling behaviors that are fundamentally impossible with fixed-graph neural networks.</p><p>It’s a great idea and a great paper, very much recommended to read!</p><h3>References</h3><ul><li><a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2512.24601">Recursive Language Models</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=174da45412ab" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The critical importance of designing incentives]]></title>
            <link>https://medium.com/@pietrobolcato/the-critical-importance-of-designing-incentives-bff91eb89852?source=rss-52732991c759------2</link>
            <guid isPermaLink="false">https://medium.com/p/bff91eb89852</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[design]]></category>
            <category><![CDATA[systems-thinking]]></category>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[psychology]]></category>
            <dc:creator><![CDATA[Pietro Bolcato]]></dc:creator>
            <pubDate>Wed, 09 Jul 2025 06:27:15 GMT</pubDate>
            <atom:updated>2025-07-09T06:31:43.699Z</atom:updated>
            <content:encoded><![CDATA[<h4>From teams to society, from families to ourselves: a guide to seeing (and reshaping) the systems we live</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R_ZA2kItEiX_6KxoC_ip-g.png" /></figure><h3>Introduction</h3><p>For many years I have worked, seen, and helped scale organisations. Over time, I noticed that the biggest blockers weren’t always technical, and they weren’t even about people, most of the time. They were about incentives.</p><p>Incentives are the underlying forces – economic, social, cultural, or political – that shape how individuals, organisations, and even entire societies behave. They can be embedded in systems like capitalism, education, governance, or technology.</p><p>Often invisible, incentives drive choices and priorities – not always in ways that align with long-term well-being or collective progress. When incentives are misaligned, they can lead to dysfunction, short-termism, or harm, even when everyone believes they’re doing the right thing.</p><p>On the other hand, when incentives are well-designed, people align quickly, work flows smoothly, and productivity and clarity increase. Society might even create long term, sustainable and positive impact!</p><p>That’s why incentives aren’t just a “management problem” – they’re a foundational design challenge for everyone, and it doesn’t get enough recognition.</p><p>Take, for example, tech companies. Many reward speed of delivery – shipping fast, hitting deadlines, showing “velocity.” But what happens when that incentive becomes dominant? Teams might cut corners, pile on technical debt, or launch poorly thought-out features. The system didn’t <em>ask</em> for bad work. It just rewarded speed more than sustainability.</p><p>Or look at social media platforms. Their business models reward engagement – clicks, likes, shares. So algorithms are tuned to maximize time on screen. The side effect? Sensationalism spreads faster than truth. Polarization increases. Mental health declines. None of this was the intended outcome – it was the byproduct of incentives designed to optimize ad revenue.</p><p>Zoom out further, and you’ll see the same pattern in capitalism itself. When profit is the primary incentive and externalities (like environmental impact or social harm) aren’t priced in, companies may act rationally within the system – while producing irrational, even catastrophic, outcomes for society and the planet.</p><p>And in government? Elected officials are often incentivized to win the next election cycle rather than invest in long-term solutions. Infrastructure, education, or climate policy all suffer because the payoff doesn’t fit within a 4-year term.</p><p>These are not failures of intelligence. They are failures of <em>incentive design</em>.</p><h3>This Applies Everywhere</h3><p>You don’t need to be a CEO, a Technical Lead Manager or a policy maker to use this lens. Incentives show up everywhere, in families, friendships, communities, even in how we treat ourselves. Whether we realise it or not, we are always shaping and reacting to incentives through our expectations, our reactions, our habits, and our language.</p><p>Take a moment, and ask yourself:</p><ul><li><strong>At home</strong>: Are you implicitly or explicitly rewarding honesty with your partner, your children, your family? If someone tells the truth and gets punished, shut down, or ignored: what incentive are you really reinforcing? Do your reactions encourage vulnerability, or make people hide things to avoid discomfort?</li><li><strong>In friendships</strong>: Are you rewarding availability over boundaries? Do you only show appreciation when someone says yes, or also when they set healthy limits? That difference shapes long-term trust.</li><li><strong>In self-care</strong>: Are you driven by guilt, fear, or quick rewards, pushing yourself out of pressure or avoiding discomfort with distractions? Or do you create small, sustainable incentives for rest, reflection, and consistency; ones that actually nourish you over time?</li><li><strong>In communities</strong>: Are contributions recognized only when they’re loud or visible? Or do you create space to celebrate quiet work: listening, supporting, resolving conflict, building trust?</li></ul><p>Incentive design is not just a business tool. It’s a fundamental lens for understanding how cultures form, how trust builds, and how relationships last. Once you see it, you can’t unsee it. And once you begin to shape it intentionally, you start to unlock a different kind of clarity: one that doesn’t just optimize behavior, but honors it.</p><h3>Designing the Right Incentives</h3><p>Designing effective incentives is undoubtedly complex. The best incentives don’t just produce results; they <em>sustain</em> them. They encourage the right behaviors today without causing unintended damage tomorrow, and for this it’s fundamental to consider not only first order consequences, but also second and third.</p><p>Here are a few principles to guide better incentive design, drawn from systems thinking, behavioral psychology, product strategy and my own experience:</p><h4><strong>1. Start with behavior, not outcomes</strong></h4><ul><li>Rather than asking <em>what</em> you want to achieve, ask <em>what behaviors</em> will naturally lead there, what is the path forward and what might get in their way.</li><li><em>Example</em>: If your goal is team innovation, don’t just reward ideas. Reward <em>risk-taking</em>, <em>cross-functional collaboration</em>, and <em>learning from failure</em>. Google’s famous “20% time” was an implicit incentive: it gave employees permission to explore. That freedom led to products like Gmail and AdSense.</li></ul><h4><strong>2. Think in time horizons</strong></h4><ul><li>Short-term incentives can be motivating, but they often backfire when they ignore long-term consequences. Always ask: <em>What happens if this works? And what happens after that?</em></li><li><em>Example</em>: Social media companies once celebrated “time spent” as a key metric. It worked — people stayed longer. But the second-order effects were addiction, misinformation, and mental fatigue. If the incentive had been “net positive engagement” or “well-being per session,” for example measured through sentiment analysis — the algorithm might have evolved very differently.</li></ul><h4><strong>3. Understand second- and third-order effects</strong></h4><ul><li>Every incentive sets off a chain reaction. The first effect is usually obvious. The second, how people adapt, game, or extend the system, is less so. And the third, how those adaptations shape culture, trust, and broader systems, is where real existential risks and surprises lie.</li><li><em>Example</em>: A gig economy app might incentivize more hours worked by offering bonuses. First-order effect: workers put in longer shifts. Second-order: they start skipping breaks. Third-order: fatigue sets in, accidents increase, and turnover rises. What looked like higher productivity was actually extracting future costs.</li></ul><p>Designing incentives means thinking not just about what <em>happens</em>, but what <em>happens next</em>, and <em>what that makes possible</em> after.</p><h4><strong>4. Reward collective success, not just individual performance</strong></h4><ul><li>Individual bonuses often lead to local optimization: people maximize <em>their</em> numbers, sometimes at the expense of the whole system,</li><li><em>Example</em>: In sales teams, moving from commission-only to a mix of individual and team performance metrics can reduce toxic competition and increase collaboration, especially between sales and support. It also helps prevent “land and leave” behavior, where deals are signed with no regard for customer success.</li></ul><h3>How to put this into practice</h3><p>Incentives can feel abstract, but intentional design often begins with small, human choices: how you respond, what you praise, what you highlight or ignore. These micro-incentives stack up and shape culture faster than policies do.</p><p>Here are a few simple ways to start:</p><ul><li><strong>In your team</strong>: Create a norm where speaking up is rewarded. That doesn’t mean every idea gets implemented, but rather every contribution should be met with openness. Thank people for speaking up, sharing ideas, and disagreeing thoughtfully. Model curiosity. If the incentive is <em>“raise what others might have missed,”</em> you’ll build trust and depth.</li><li><strong>In meetings</strong>: Notice whose voices are missing. Intentionally create space for quieter team members to contribute, and recognize when they do. The incentive becomes: <em>your input matters, even if you’re not the loudest in the room</em>.</li><li><strong>In feedback</strong>: Don’t just reward results. Reward process. Did someone loop in another team early? Push for clarity? Clean up old documentation without being asked? These are foundational behaviors, but only if you make them visible and valued.</li><li><strong>In product culture</strong>: Track and celebrate “prevented fires,” not just “shipped features.” Who raised a user concern that stopped a misstep? Who asked the awkward question that saved a whole product development cycle? Design the incentive to <em>question, notice foresight — </em>not just output.</li><li><strong>In friendships</strong>: Consider how you respond when a friend is honest, sets a boundary, or shares something vulnerable. Do you reward them with presence, understanding, and space — or change the subject, judge, or pull away? Small reactions teach people what’s safe to share and what’s not. Over time, those reactions <em>are</em> the incentive structure of your relationship.</li><li><strong>In personal life</strong>: Ask yourself: what behaviors do I reward in myself? Am I only proud when I’m productive, or also when I rest? Do I give myself credit for small acts of care or only big wins? Often, we internalize systems that reward burnout, overachievement, or people-pleasing. Redesigning those incentives can change your relationship with yourself.</li></ul><h3>Conclusions</h3><p>The more I’ve worked with teams, founders, and systems, the more I’ve come to see incentive design as a kind of quiet architecture. It doesn’t scream. It rarely gets noticed in the moment. But over time, it shapes everything: what gets built, what gets ignored, how people feel, and how long-term relationships are formed or fractured.</p><p>I’ve made the mistake of advocating for strategy, tooling, and features without pausing to ask: <em>What happens if this really works? And what happens after that?</em></p><p>It’s incredibly important — and humbling — to start thinking in second- and third-order horizons. Good intentions alone are often no match for misaligned incentives.</p><p>But what gives me hope is this: once you see the incentives, you can start to shape them. You can redesign how your team works, how your relationships feel, and how you treat yourself. You can reward what matters, not just what’s visible. And over time this brings real culture shift. It doesn’t take a title — just intention.</p><h3>Further Reading</h3><p>If you want to dive deeper into incentive design, systems thinking, and behavioral nudges, here are a few great starting points:</p><ul><li><strong>Thinking in Systems</strong> — Donella Meadows<br> A foundational read for understanding feedback loops, leverage points, and long-term effects.</li><li><strong>Nudge</strong> — Richard Thaler &amp; Cass Sunstein<br> On how small design tweaks can steer behavior without restricting choice.</li><li><strong>Drive</strong> — Daniel Pink<br> Explores why intrinsic motivation often beats external rewards, especially in creative work.</li><li><strong>The Tyranny of Metrics</strong> — Jerry Z. Muller<br> A sharp look at how over-relying on metrics can distort incentives and damage systems.</li><li><strong>Obliquity</strong> — John Kay<br> Shows how pursuing goals indirectly often leads to better outcomes — especially in complex systems.</li><li><strong>What We Owe the Future</strong> — William MacAskill<br> A powerful perspective on second- and third-order effects across generations.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bff91eb89852" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NoProp: Training Neural Networks Without Back-Propagation or Forward-Propagation]]></title>
            <link>https://medium.com/@pietrobolcato/noprop-training-neural-networks-without-back-propagation-or-forward-propagation-920ebe8cb1af?source=rss-52732991c759------2</link>
            <guid isPermaLink="false">https://medium.com/p/920ebe8cb1af</guid>
            <dc:creator><![CDATA[Pietro Bolcato]]></dc:creator>
            <pubDate>Fri, 25 Apr 2025 10:31:01 GMT</pubDate>
            <atom:updated>2025-04-25T10:31:01.915Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Gradient-Free Alternative that Beats Prior “No-Backprop” Methods</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*smvtk3cro0cWBrfRsPw49w.png" /><figcaption>NoProp architecture — Image courtesy of the paper.</figcaption></figure><h3>Introduction</h3><p>Back-propagation has powered deep learning for almost forty years, yet it still comes with two major issues:</p><ul><li>High memory cost, because activations must be saved for the backward pass.</li><li>Sequential gradient flow, which blocks easy multi-GPU scaling.</li></ul><p>The new <strong>NoProp</strong> paper from Oxford, Mila and Google DeepMind shows that we can train image classifiers <strong>without any forward or backward propagation at all</strong>. Instead, each layer learns <strong>to denoise a noisy version of the target label</strong> — an idea borrowed from diffusion models.</p><p>It’s really fascinating to challenge such established fundamentals! Let’s dive right in!</p><h3>Why other gradient-free methods still struggle</h3><p>Most “no-backprop” ideas fall into one of three categories and all have some downsides:</p><ul><li><strong>Forward-Forward and Target Propagation</strong><br>Use local losses but often plateau on modern datasets and need tricky hyper-tuning.</li><li><strong>Zero-order gradient estimates</strong><br>Replace gradients with finite differences; hard to scale with parameter count.</li><li><strong>Evolution strategies</strong><br>Remove gradients entirely but require millions of network evaluations.</li></ul><p>None of these options give a clean drop-in replacement for back-prop with competitive accuracy <em>and</em> reasonable compute. NoProp closes that gap.</p><h3>The core idea</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MZM7JaQyQ2N2orXTIDjW6w.png" /></figure><ol><li><strong>Fix a label embedding space — </strong>Each class has a learned vector.</li><li><strong>Add Gaussian noise to the true label vector — </strong>The noise level is controlled by a schedule similar to that used in diffusion models.</li><li><strong>Train layer t to predict the clean label from the noisy label produced by layer t-1 — </strong>The loss is a simple squared distance.</li><li><strong>At inference: </strong>Starting from pure noise, each layer performs one denoising step; after T steps the last layer is fed to a soft-max for classification.</li></ol><p>Because layers never need signals from later layers, <strong>there is no global forward or backward pass</strong>. Training can even shuffle layers or place them on different devices.</p><h3>Three variants</h3><p>NoProp has three versions. The first is <strong>NoProp-DT</strong>. It makes ten quick denoise steps and gives the highest accuracy in the tests. The second is <strong>NoProp-CT</strong>. It treats time as a smooth line and lets the model learn how fast to lower the noise. It uses less GPU memory than DT and still trains faster than a standard neural-ODE solver. The third is <strong>NoProp-FM</strong>. It swaps diffusion for a flow-matching ODE. After you let the class vectors learn together with the model, FM reaches accuracy close to the other two.</p><h3>Stability tricks that make it work</h3><p>The authors use three small ideas to keep training steady. First, every class starts with a <em>prototype</em> image — an example picture that best shows that class — so the model begins with sensible vectors. Second, in the continuous version the amount of noise is not fixed; the model learns the right schedule during training, so noise is never too high or too low. Third, an extra loss keeps the class vectors apart, stopping them from clumping into one spot.</p><h3>Results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H1V7aMK3lQtvcSwXacc2FA.png" /></figure><p>On CIFAR-10 with an A100 GPU, classic back-prop training needs about <strong>1.17 GB</strong> of memory. NoProp-DT drops this to <strong>0.64 GB</strong> and NoProp-CT to <strong>0.45 GB</strong>. NoProp-CT also reaches 70 % test accuracy roughly <strong>twice as fast</strong> as the neural-ODE adjoint method. For accuracy, NoProp hits <strong>99.5 % on MNIST</strong> and <strong>80.5 % on CIFAR-10</strong>, matching back-prop while using less compute.</p><h3>Why this matters</h3><p>NoProp shows we can train deep networks without sending gradients through every layer. Each layer learns on its own, so you can spread layers across many GPUs easily. The method cuts memory use, shortens training time, and may lower the risk of catastrophic forgetting.</p><h3>Conclusion</h3><p>NoProp re-thinks deep network training as <strong>plain denoising</strong>. By discarding the global gradient chain, it reduces memory use, opens new parallelism routes and matches back-prop accuracy on classic vision benchmarks.</p><p>If back-prop feels like a heavyweight tool for your edge device, keep an eye on this line of work — gradient-free deep learning just took a big step forward.</p><h3>Resources</h3><ul><li><strong>Paper</strong>: <a href="https://arxiv.org/abs/2503.24322">“NoProp: Training Neural Networks Without Back-Propagation or Forward-Propagation”</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=920ebe8cb1af" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The State of Vibe Coding — April 2025 Edition]]></title>
            <link>https://medium.com/@pietrobolcato/the-state-of-vibe-coding-april-2025-edition-29cc839fb587?source=rss-52732991c759------2</link>
            <guid isPermaLink="false">https://medium.com/p/29cc839fb587</guid>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[app-development]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Pietro Bolcato]]></dc:creator>
            <pubDate>Tue, 08 Apr 2025 15:52:50 GMT</pubDate>
            <atom:updated>2025-04-08T15:52:50.055Z</atom:updated>
            <content:encoded><![CDATA[<h3>The State of Vibe Coding — April 2025 Edition</h3><h4>Building a real, personalized productivity app with nothing but prompts</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ugSeu92T8g5bYFqAiU3Wmg.png" /></figure><h3>Introduction</h3><p>Thanks to my job and passions I am using, working and enjoying LLMs and VLMs on a daily basis. I had time this weekend, and I wanted to try the status of fully vibe coding as of April 2025 — purposely, as an experiment! So I asked myself:</p><blockquote><strong>Could I build a complete, useful app — incredibly tailored to my specific needs — without writing a single line of code myself?</strong></blockquote><p>Turns out… yes. Mostly. Here’s how it went.</p><h4>The Experiment Setup</h4><p>For this experiment, I gave myself a few <strong>strict</strong> <strong>constraints:</strong></p><ul><li>Use a programming language that is <strong><em>not</em></strong> my first (so: no Python).</li><li>Write literally <strong>zero code</strong> manually — only interact with the system via <strong>natural language prompts</strong>.</li><li>It had to be <strong>useful</strong>, at least for me.</li><li>It had to be a <strong>deployable artifact</strong>.</li></ul><p>No cheating. No tweaks in the editor. Just prompting. Let’s go!</p><h4>The App I Decided to Build</h4><p>I can’t deny that while I am careful not to obsess over it, I always been a productivity geek. I mean, we are in this life for so relatively little; if we can optimize <em>what</em> we do and <em>how</em> we do it, how much more can we make of our incredibly precious time! I’ve tried many frameworks, but hands on — the <strong>12 Week Year</strong> has been the most impactful system I’ve ever used, and my daily driver for quite a while.</p><h4>What’s the 12 Week Year?</h4><p>If you don’t know, this alone might change your life, more than any AI vibe coding. It’s a productivity method that breaks the year into 12-week “cycles” instead of following the traditional 12-month calendar. Each cycle is treated as its own year — with goals, execution plans, and weekly reviews. The idea is that by compressing time, you create urgency, focus, and better follow-through.</p><p>It has helped me <strong>10x my output</strong> since I adopted it, and never looked for any other methods. You can read more about it in the reference section below.</p><h3>My Old Setup</h3><p>Until now, I tracked my cycles with a custom Excel sheet. It had all the formulas, weekly rows, task tables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*87oX_3uUPAcZj4nexHds0Q.png" /></figure><p>It worked — but had<strong> a few issues:</strong></p><ul><li>Editing from mobile was <strong>very painful.</strong></li><li>Adding new goals, counting weeks, and archiving old ones took <strong>manual effort.</strong></li><li>Not <strong>very interactive</strong> or “fun” to use.</li></ul><p>So for this weekend project, I had a choice:</p><ol><li>Automate the Excel sheet further, or</li><li><strong>Vibe code a full replacement</strong>.</li></ol><p>You know already what I picked.</p><h3>What I Built — Just with Prompts</h3><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DSEJ9qN6_-9U&amp;type=text%2Fhtml&amp;schema=youtube&amp;display_name=YouTube&amp;src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FSEJ9qN6_-9U%3Ffeature%3Doembed" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/78c5462cea626df5b8da5def68337c7b/href">https://medium.com/media/78c5462cea626df5b8da5def68337c7b/href</a></iframe><p>Using <strong>React</strong>, <strong>Next.js</strong>, and <strong>Tailwind CSS</strong>, here’s what I built using only prompts to Gemini 2.5, in Cursor:</p><ul><li>Authentication and user registration</li><li>Manage multiple 12-week cycles, with automatic week detection</li><li>Add, remove, eidt goals per cycle</li><li>Add, remove, edit, duplicate tasks</li><li>Show progress and completion rate</li><li>Weekly and cycle-wide statistics (e.g., tasks per week, streaks, completion rate)</li><li>Full backend syncing across devices</li><li>Optimized mobile and desktop layouts</li></ul><p><strong>Total build time: </strong>~10 hours (over a weekend). It included the GPT-4o generated logo too.</p><h3>What I Learned</h3><h4>Personalization is the superpower</h4><p>Because I was building this <em>for myself</em>, I could add tiny quality-of-life features that would be impossible in an off-the-shelf tool:</p><ul><li>Auto-scrolling to the current day on mobile</li><li>Urgency tags via ! and !!, with automatic styling</li><li>Stats I actually care about — like “when during the day am I most productive?”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*LUV703eqS4xCQUNQK-ANuA.png" /><figcaption><em>Urgency feature</em></figcaption></figure><p>This is what really sold me. You’re not building for “users” — you’re building for <em>you</em>.</p><h4>Iteration speed is awesome</h4><p>In retrospective, probably I could have coded this better alone in a similar amount of time. But being able to work on a secondary language at this speed, was quite incredible.</p><h3>What Didn’t Work So Well</h3><ul><li>Sometimes, the models got stuck in loops or hallucinated file names. It took some <strong>painfully verbose prompts</strong> to get back on track.</li><li>As the app became more complex, generating clean and modular code got harder.</li><li>The code quality is quite terrible. Naming conventions break. Files get messy. No tests. Definitely not production-level. Very OCD triggering!</li></ul><p>Still — <strong>no code written by hand</strong>, and yet the app works, syncs, and helps me run my 12 Week Year better than before.</p><h3>Conclusion</h3><p>This little project is a <strong>memento of where vibe coding stands today</strong>.</p><p>And honestly? It’s pretty wild.</p><ul><li>I built a full, useful, multi-device app <strong>without writing a single line of code myself</strong>.</li><li>It’s not ready for serious production workloads, but it’s amazing for <strong>prototyping and personal tooling</strong>.</li><li>The fact that I could tailor everything to my own needs made the result <em>way</em> more useful than any generic app out there.</li></ul><p>It’s clear that we’re entering a phase where <strong>everyone can create their own software</strong>. The apps we use will increasingly reflect <strong>our own processes, preferences, and needs — </strong>not the lowest common denominator.</p><p>It was a lot of fun. A lot of frustration. Some OCD-triggering moments when the code got too messy. But all in all, pretty incredible.</p><blockquote>The world is about to become incredibly more tailored to individual needs — and drastically more productive.</blockquote><p>I’ll keep repeating this experiment every few months to track how far things go. This was <strong>April 2025</strong>.</p><p>What a time to be alive!</p><h3>References</h3><ul><li><a href="https://www.amazon.de/-/en/12-Week-Year-Others-Months/dp/1118509234">12 Week Year</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=29cc839fb587" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Generating Images Like LLMs: The Autoregressive Approach]]></title>
            <link>https://medium.com/@pietrobolcato/generating-images-like-llms-the-autoregressive-approach-052d80c5823c?source=rss-52732991c759------2</link>
            <guid isPermaLink="false">https://medium.com/p/052d80c5823c</guid>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[generative-art]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Pietro Bolcato]]></dc:creator>
            <pubDate>Wed, 02 Apr 2025 05:15:08 GMT</pubDate>
            <atom:updated>2025-04-02T05:31:36.903Z</atom:updated>
            <content:encoded><![CDATA[<h4>A look at how models like GPT-4o might generate images, from tokenization to scaling laws</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-gnq0NMz7bzjHoXfbpT6Vg.png" /></figure><h3>Introduction</h3><p>I have worked with generative models for a long time, and the strong performance of <strong>GPT-4o</strong> really intrigued me. They shared in the model card that is an <strong>autoregressive (AR)</strong> method, nothing more. Unlike dominant diffusion models, AR models generate images <strong>token by token</strong>, much like large language models (LLMs) generate text. This sequential approach is fascinating, particularly for multimodal systems, as it offers a unified way to model both text and image tokens in the input <em>and</em> output space.</p><p>This post dives into the mechanics of autoregressive image generation. We’ll break down:</p><ul><li>The <strong>core concept</strong> of AR image modeling.</li><li>How images are <strong>transformed into sequences of tokens</strong>.</li><li>Key <strong>architectural choices</strong> and <strong>training strategies</strong>.</li><li>Evidence for <strong>scaling laws</strong> similar to LLMs.</li><li>The <strong>emergent capabilities</strong> and <strong>current limitations</strong> of these models.</li></ul><p>While OpenAI hasn’t released a technical report for GPT-4o’s image generation, recent research like <strong>Liquid</strong> likely shares similarities in architecture and training, giving us valuable insights into this very interesting approach.</p><h3>What is Autoregressive Image Generation?</h3><p>At its heart, autoregressive modeling is about <strong>predicting the next element in a sequence based only on the elements that came before it</strong>. For text, this means predicting the next word or sub-word token. For images, the core idea is the same, but we predict the next <strong>image token</strong> xᵢ based on the preceding sequence x&lt;ᵢ.</p><p>The challenge? Images are inherently 2D grids, not 1D sequences. To apply AR models, we need to impose a 1D order:</p><ol><li><strong>Raster Scan:</strong> The classic approach. Flatten the image (or its tokenized representation) row-by-row or column-by-column, then predict tokens sequentially, like reading text. <strong>ImageGPT</strong> pioneered this.</li><li><strong>Next-Scale Prediction:</strong> A more recent approach, coming from <strong>VAR</strong>. Instead of predicting single tokens, the model predicts <em>entire token maps</em> at increasing resolutions. Generation happens sequentially <em>across scales</em> (coarse-to-fine) but often in parallel <em>within</em> a scale.</li></ol><p>These sequential predictions are typically handled by <strong>Transformer decoders</strong> using <strong>causal attention masks</strong>, ensuring a model only attends to previous tokens, identical to how LLMs operate. This architectural similarity allows adapting pre-trained LLMs for image generation and jointly modeling text and image tokens.</p><h3>Turning Pixels into Tokens: Image Tokenization</h3><p>LLMs operate on discrete vocabularies (e.g., words or sub-words defined by a text tokenizer). To make images compatible with these AR transformers, we need to convert them into sequences of “image tokens.” This process, <strong>image tokenization</strong>, primarily falls into two categories:</p><h3>1. Discrete Tokenization (e.g., VQ-VAE, VQGAN)</h3><p>This is the most common approach, analogous to text tokenization.</p><ul><li><strong>Encoder:</strong> Maps image patches to continuous latent vectors f.</li><li><strong>Quantizer:</strong> The crucial step. Maps each vector f to the <strong>nearest vector in a finite, learned codebook</strong> (like a dictionary of visual patterns). It outputs discrete integer indices (token IDs).</li><li><strong>Decoder:</strong> Reconstructs the image from the sequence of chosen codebook vectors.</li></ul><p><strong>Pros:</strong> Output tokens are discrete integers, directly usable with standard LLM cross-entropy loss. Models like <strong>Liquid</strong> use this.</p><p><strong>Cons:</strong></p><ul><li><strong>Information Loss:</strong> Quantization inevitably discards some information, potentially hurting reconstruction fidelity and fine details.</li><li><strong>Training Instability:</strong> Learning the codebook can be challenging.</li><li><strong>Representational Bottleneck:</strong> Standard VQ codebooks might lack the capacity for both fine-grained reconstruction and semantic understanding, a problem addressed by methods like <strong>UniTok</strong>’s Multi-Codebook Quantization (MCQ).</li></ul><h3>2. Continuous Tokenization (e.g., MAR)</h3><p>This approach avoids discrete codebooks to potentially retain more information.</p><ul><li><strong>Encoder:</strong> Maps image patches to continuous latent vectors x. These vectors <em>are</em> the tokens.</li><li><strong>Next Token Prediction:</strong> The AR model predicts a conditioning vector z based on previous continuous tokens.</li><li><strong>Modeling Continuous Distributions:</strong> Since tokens aren’t discrete indices, we can’t use simple cross-entropy. We need to model the probability distribution p(x|z) for the continuous next token x. <strong>MAR</strong> cleverly uses a small <strong>Diffusion Model head</strong> conditioned on z to define the loss and enable sampling. <strong>MetaMorph</strong> predicts continuous SigLIP embeddings directly.</li><li><strong>Decoder:</strong> A separate network reconstructs the image from the sequence of continuous tokens.</li></ul><p><strong>Pros:</strong> Avoids quantization loss, potentially higher fidelity.<br><strong>Cons:</strong> Requires more complex modeling (like diffusion heads) for the probability distribution.</p><p><strong>Current Trend → Discrete tokenization</strong> remains dominant, especially for unified models leveraging existing LLM architectures. However, innovations like <strong>UniTok</strong> (improving discrete VQ) and <strong>Infinity</strong> (bitwise modeling) try to bridge the fidelity gap.</p><h3>Architecture &amp; Training: Scaling</h3><p>AR image generation models often leverage Transformer architectures, sometimes adapting pre-trained LLMs (<strong>Liquid</strong>, <strong>MetaMorph</strong>) or training large Transformers from scratch (<strong>VAR</strong>, <strong>Infinity</strong>). Some approaches remain purely autoregressive, while others incorporate diffusion elements, perhaps for decoding continuous tokens (<strong>MetaMorph</strong> visualizer) or as part of the token prediction process (<strong>MAR</strong>).</p><p>A key promise of AR visual models is their potential to exhibit <strong>scaling laws</strong> similar to LLMs:</p><ul><li><strong>Predictable Improvement:</strong> Papers like <strong>Liquid</strong>, <strong>VAR</strong>, and <strong>MetaMorph</strong> show strong correlations between increased model size/compute and improved performance metrics (Validation Loss, FID, IS, VQA Scores).</li><li><strong>Unified Capacity:</strong> As models scale, the trade-off between visual and linguistic performance diminishes (<strong>Liquid</strong>), suggesting large models can effectively handle both modalities.</li></ul><p>If AR visual generation scales as predictably as LLMs, it makes large-scale training efforts more efficient and justifiable.</p><h3>The LLM Connection: Emergent Capabilities</h3><p>Because visual AR models often share architectures and training paradigms with LLMs (especially when fine-tuning pre-trained LLMs), they can inherit powerful capabilities:</p><ul><li><strong>Zero/Few-Shot Generalization:</strong> Models can perform tasks not explicitly seen during training, like in-painting, out-painting, or spatial conditioning, often guided by natural language instructions.</li><li><strong>Emergent Reasoning:</strong> The reasoning abilities of the underlying LLM can transfer to image generation. <strong>MetaMorph</strong> showed models could generate an image of a <em>violin</em> when prompted with <em>“A musical instrument, this instrument is often played by the scientist who formulated the theory of special relativity”</em> by implicitly reasoning Einstein → Violin. <strong>Liquid</strong> highlights similar abilities, leveraging world knowledge absent in standard text encoders.</li><li><strong>Connection between Understanding and Generation:</strong> Jointly training on visual understanding and generation tasks often benefits both. Better image understanding seems to lead to better image generation, and vice-versa, especially in unified architectures.</li></ul><h3>Challenges and Limitations</h3><p>Despite the promise, AR image generation faces hurdles:</p><h4><strong>High-Resolution Scaling</strong></h4><p>Achieving diffusion-level quality at very high resolutions (e.g., 1024x1024 and beyond) is still an active area of research for AR models, although recent work (<strong>Infinity</strong>) shows significant progress.</p><h4><strong>Generation Speed</strong></h4><ul><li>Traditional <strong>raster-scan AR</strong> is inherently slow due to its sequential nature.</li><li>Faster methods like <strong>VAR</strong> (sequential across scales) and <strong>MAR</strong> (parallel prediction) have trade-offs. MAR can lose KV-caching benefits and Diffusion Loss adds per-token sampling steps. While potentially faster than multi-step diffusion, direct comparisons need careful consideration of optimizations.</li></ul><h4>Training Complexity</h4><p>VQ-based methods involve tricky codebook training (<strong>UniTok</strong> aims to improve this). Continuous methods might require joint Diffusion Loss training (<strong>MAR</strong>). <strong>VAR</strong> requires multi-stage tokenizer pre-training.</p><h4><strong>Controllability</strong></h4><p>The rich ecosystem for fine-grained control seen in diffusion models (e.g., ControlNets) is less developed for AR models currently.</p><h4><strong>Component Dependency</strong></h4><p>Performance can be bottlenecked by the quality of the pre-trained image tokenizer or the base LLM used.</p><h4><strong>Global Context</strong></h4><p>Predicting pixel-by-pixel can make capturing long-range dependencies and global image coherence challenging compared to methods with more global conditioning, such as Diffusion models. Next-scale prediction (VAR, Infinity) mitigates this.</p><h3>Conclusion</h3><p>Autoregressive models represent a interesting and increasingly competitive paradigm for image generation, offering an alternative or complement to diffusion. Their <strong>native compatibility with sequential data</strong> makes them ideal for <strong>unified multimodal systems</strong> that seamlessly process and generate interleaved text and images, as hinted at by GPT-4o.</p><p>Key advantages lie in their potential for <strong>LLM-like scaling</strong>, <strong>emergent reasoning capabilities</strong>, and <strong>architectural simplicity</strong> (often leveraging standard Transformers). However, challenges remain in <strong>generation speed</strong>, <strong>high-resolution fidelity</strong>, <strong>training complexity</strong>, and developing fine-grained <strong>controllability</strong>.</p><p>With novel methods like next-scale prediction (<strong>VAR</strong>), advanced quantization (<strong>UniTok</strong>), bitwise modeling (<strong>Infinity</strong>), and direct continuous token prediction (<strong>MAR, MetaMorph</strong>), the gap between AR and diffusion models is rapidly closing.</p><p>It’s likely that we will witness a surge of research in this direction — what a time to be alive!</p><h3>References</h3><ul><li><a href="https://arxiv.org/abs/2412.04332">Liquid: Language Models are Scalable and Unified Multi-modal Generators</a></li><li><a href="https://arxiv.org/abs/2502.20321">UniTok: A Unified Tokenizer for Visual Generation and Understanding</a></li><li><a href="https://arxiv.org/abs/2412.14164v1">MetaMorph: Multimodal Understanding and Generation via Instruction Tuning</a></li><li><a href="https://arxiv.org/abs/2412.04431">Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis</a></li><li><a href="https://arxiv.org/abs/2404.02905">Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction</a></li><li><a href="https://arxiv.org/abs/2406.11838">Autoregressive Image Generation without Vector Quantization</a></li><li><a href="https://openai.com/index/image-gpt/">Generative Pretraining from Pixels (ImageGPT)</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=052d80c5823c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Vision-Language Models (VLMs): A Practical Guide]]></title>
            <link>https://medium.com/@pietrobolcato/understanding-vision-language-models-vlms-a-practical-guide-8da18e9f0e0c?source=rss-52732991c759------2</link>
            <guid isPermaLink="false">https://medium.com/p/8da18e9f0e0c</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Pietro Bolcato]]></dc:creator>
            <pubDate>Wed, 26 Mar 2025 14:05:36 GMT</pubDate>
            <atom:updated>2025-03-26T14:05:36.896Z</atom:updated>
            <content:encoded><![CDATA[<h4>From CLIP to Gemma 3: Deep Dive into Modern Multimodal Models</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cW0pHB4IJul_lMtsegJLkQ.png" /></figure><h3>Introduction</h3><p>Vision-Language Models (VLMs) have surged in popularity, allowing powerful interactions and reasoning capabilities between images and text. In this blog post I aim to provide a structured, technical introduction to them: what they are, how they work, notable architectures, how to effectively prompt and fine-tune them — and how to use them through most popular frameworks!</p><p>Let’s dive right in!</p><h3>Understanding Vision-Language Models (VLMs)</h3><p>Vision-Language Models (<strong>VLMs</strong>) are models specifically designed to jointly understand and reason about visual and textual information simultaneously. Over the last few years, these models have rapidly evolved, transforming how neural networks perceive images, interpret complex visual contexts, and generate coherent, relevant textual responses.</p><p><strong>Typical use-cases include:</strong></p><ul><li><strong>Image Captioning:</strong> Generating accurate, context-aware descriptions of images or video frames.</li><li><strong>Visual Question Answering (VQA):</strong> Answering detailed, open-ended questions based solely on visual content.</li><li><strong>Image-based Reasoning:</strong> Providing explanations or logical reasoning about visual scenes or complex visual data.</li><li><strong>Multimodal Dialogues:</strong> Engaging in conversational interactions involving visual inputs, such as describing, reasoning, or making decisions based on images or video clips.</li></ul><h3>How Do Vision-Language Models Actually Work?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/0*qMgnOpCXe0u5pnGs" /><figcaption>Image copyright of Huggingface.</figcaption></figure><p>To clearly understand the workflow of a VLM, consider this simplified, step-by-step process (<strong>common to most modern VLMs</strong>):</p><p><strong>Step 1. Multimodal Inputs</strong></p><p>A VLM typically receives two distinct modalities as inputs:</p><ul><li><strong>Visual Input:</strong> Usually images, or even videos.</li><li><strong>Textual Input (Prompt):</strong> Natural language instructions, questions, or commands guiding the desired output.</li></ul><p>For example, an input might look like:</p><ul><li><strong>Image:</strong> A picture of a busy city intersection.</li><li><strong>Prompt:</strong> <em>“What kind of vehicles are present, and how many pedestrians are crossing?”</em></li></ul><p><strong>Step 2. Encoding Modalities</strong></p><p>These inputs go through separate initial processing steps:</p><ul><li><strong>Vision Encoder:</strong> Transforms the visual input into structured numerical representations, known as <strong>visual embeddings</strong> (<em>Z(i)</em>). This is commonly done by Vision Transformers (<strong>ViTs</strong>), Convolutional Neural Networks (<strong>CNNs</strong>), or hybrid architectures like <strong>ConvNeXt</strong>.</li><li><strong>Language Encoder (or Text Processor):</strong> Converts textual prompts into dense vector representations called <strong>textual embeddings</strong> (<em>Z(t)</em>). Typically done by Transformer-based Large Language Models (<strong>LLMs</strong>) such as LLaMA, Mistral, Qwen.</li></ul><p>These embeddings (<em>Z(i)</em> for images and <em>Z(t)</em> for text) represent structured semantic information suitable for further multimodal interaction.</p><p><strong>Step 3. Projection and Fusion into a Joint Latent Space</strong></p><p>Once independently encoded, visual and textual embeddings must be aligned into a <strong>shared multimodal embedding space</strong>. This is typically done by a smaller module called the <strong>Projection or Fusion Layer</strong>:</p><ul><li>Usually implemented as a small Transformer block or a simple fully-connected neural network (MLP).</li><li>The goal is to merge visual and textual embeddings into a unified embedding space where both modalities can seamlessly interact.</li></ul><p>After projection, the embeddings form a joint representation, enabling the model to reason simultaneously across images and language.</p><p><strong>Step 4. Autoregressive Multimodal Decoding</strong></p><p>In the final step, the combined multimodal embeddings serve as an input prefix for an <strong>autoregressive language model (Decoder)</strong>. This decoder generates a text-based response one token at a time:</p><ul><li>Each generated token is conditioned on previously generated tokens and multimodal embeddings.</li><li>The decoder continues generating tokens until reaching a predefined stop token or maximum output length.</li></ul><p>For instance, the previous example’s output might be:</p><blockquote>“In the image, there are several vehicles, including 2 buses, 4 cars, and 1 motorcycle. Additionally, there are approximately 6 pedestrians crossing at the intersection.”</blockquote><h3>Why is the Joint Multimodal Embedding Space Crucial?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DAJtDrYUdcfHV9Gi.png" /><figcaption><em>Image copyright of Weaviate</em></figcaption></figure><p>The multimodal embedding space is effectively what enables VLM’s reasoning abilities. By representing both visual and textual information within the <strong>same latent space</strong>, VLMs gain significant advantages:</p><ul><li><strong>Contextual Awareness: </strong>enables rich interaction between modalities, meaning the model can accurately link textual concepts (e.g., “bus,” “intersection”) with visual evidence (bus position, color, intersection layout).</li><li><strong>Semantic Grounding: </strong>aligns abstract textual concepts with concrete visual examples. For instance, the model understands “pedestrian” not just as a word but as a visually identifiable entity within images.</li><li><strong>Cross-Modality Reasoning: </strong>allows the model to reason across modalities, answering complex visual questions, making logical inferences, or detecting nuanced visual-textual discrepancies.</li></ul><p>This multimodal alignment is precisely what differentiates modern VLMs like GPT-4 Vision or Gemini from earlier, less sophisticated image-captioning systems.</p><h3>Architectural Evolution of VLMs (A Brief Timeline)</h3><p>To contextualize current models, here’s a short timeline highlighting how VLMs evolved to their current form:</p><h4><strong>~2015–2019 (Early VLMs):</strong></h4><ul><li>CNN+RNN approaches (e.g., “Show &amp; Tell”)</li><li>Initial Transformer-based architectures (ViLBERT, UNITER)</li></ul><h4><strong>~2021 (CLIP &amp; SimVLM Era):</strong></h4><ul><li>CLIP introduced effective zero-shot vision-language alignment using contrastive learning</li><li>SimVLM simplified training, treating images as language model prefixes</li></ul><h4><strong>~2022 (Flamingo &amp; BLIP-1):</strong></h4><ul><li>Flamingo introduced cross-attention layers for efficient few-shot multimodal prompting</li><li>BLIP improved visual-language tasks efficiency with minimal model complexity</li></ul><h4><strong>~2023–2024 (GPT-4 Vision, Gemini, BLIP-2, LLaVA):</strong></h4><ul><li>GPT-4 Vision &amp; Gemini integrated fully multimodal architectures, greatly enhancing reasoning and zero-shot capabilities; in general, the field achieved significant multimodal reasoning advancements</li></ul><h4><strong>2025+ (Emerging Models: Gemini 2.0, Mistral 3.1, Gemma-3, Phi-4):</strong></h4><ul><li>Models achieve state-of-the-art reasoning, visual comprehension, and fine-grained multimodal understanding at improved efficiency levels</li></ul><h4>Current State-of-the-Art</h4><p>As of early 2025, some leading VLM architectures include:</p><ul><li><strong>Gemini 2.5</strong> (Google): Advanced reasoning with “Flash” reasoning modules, multimodal agentic capabilities</li><li><strong>Gemma-3</strong> (Google/DeepMind): High efficiency, state-of-the-art visual understanding at lower compute</li><li><strong>Mistral 3.1 Small</strong> (Mistral AI): Excellent performance in open-source, efficient multimodal capabilities</li></ul><p>You can find the current rankings and capabilities in detail on the <a href="https://huggingface.co/spaces/opencompass/open_vlm_leaderboard">Open VLM Leaderboard</a>.</p><h3>Architectural Patterns &amp; Fusion Strategies</h3><p>Today, VLM architectures fall broadly into three categories:</p><ul><li><strong>Fully Integrated Multimodal Transformers</strong> (<strong>Gemini, GPT-4 Vision</strong>):<br> Deeply combine modalities early, powerful but computationally expensive.</li><li><strong>LLMs with Visual Adapters</strong> (<strong>BLIP-2, LLaVA</strong>):<br> Connect existing large language models and vision encoders via adapter modules, balancing performance and efficiency.</li><li><strong>Parameter-Efficient Models</strong> (<strong>SmolVLM, Phi-4 Multimodal</strong>):<br> Use minimal parameters to achieve excellent performance for edge devices and resource-limited settings.</li></ul><p>These models differ mainly in how and when they fuse visual and textual features:</p><ul><li><strong>Early Fusion</strong>: Combines vision and text from the start (deep multimodal understanding).</li><li><strong>Intermediate Fusion</strong>: Integrates visual features into specific Transformer layers via cross-attention (balances complexity and performance).</li><li><strong>Late Fusion</strong>: Combines modalities only at the end (simpler, but less effective for nuanced tasks).</li></ul><h3>Prompting Strategies to Guide VLMs</h3><p>Effective prompting is critical for getting the best performance from VLMs. Similar to LLMs, common prompting techniques include:</p><ul><li><strong>Zero-shot prompts</strong>: Direct questions (“Describe this image”).</li><li><strong>Few-shot prompts</strong>: Provide a small number of examples within the prompt to guide the model.</li><li><strong>Chain-of-Thought (CoT)</strong>: Encourage models to explicitly explain their reasoning before giving an answer, improving accuracy and reducing hallucinations.</li></ul><p>Prompt engineering involves careful iterative experimentation and evaluation, significantly affecting the model’s quality and relevance of responses.</p><h3>Fine-tuning VLMs for Custom Applications</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*3JRLuakybGRiwLjw.png" /><figcaption>Visualization of LoRA — image copyright of run.ai</figcaption></figure><p>To tailor VLMs to specific tasks or domains, generally the way to go is fine-tuning. Goals of fine-tuning typically include:</p><ul><li>Updating the model’s knowledge to domain-specific contexts.</li><li>Adapting the response style to match desired behaviors or brand voice.</li><li>Optimizing accuracy for specialized tasks like customer support, product recognition, or image-based QA.</li></ul><p>Techniques like <strong>LoRA (Low-Rank Adaptation)</strong> and <strong>QLoRA</strong> enable efficient fine-tuning of large VLMs, reducing computational demands.</p><p>Fine-tuning generally involves:</p><ul><li>Creating a small multimodal dataset (100–300 samples initially, the more the better).</li><li>Formatting the dataset in conversational or instruct-style JSON (text-image pairs).</li><li>Training the model briefly (1–3 epochs) with careful parameter tuning (learning rate, LoRA settings).</li></ul><h3>Emerging Trends &amp; Best Practices</h3><p>Current trends in the VLM community include:</p><ul><li><strong>Synthetic Data &amp; Instruction Tuning</strong>: Leveraging larger models (like GPT or diffusion models) to generate rich multimodal datasets cheaply.</li><li><strong>Data Curation &amp; Augmentation</strong>: Ensuring balanced and safe datasets through rigorous filtering and augmentation.</li><li><strong>Correct Scaling</strong>: Matching vision and language model sizes carefully to maintain performance without wasting resources.</li></ul><h4>Tools &amp; Frameworks</h4><p>Useful frameworks and libraries include:</p><ul><li><a href="https://github.com/huggingface/transformers"><strong>Hugging Face Transformers</strong></a>: Quick integration, fine-tuning, and evaluation.</li><li><a href="https://github.com/vllm-project/vllm"><strong>vLLM</strong></a>: Efficient serving of large multimodal models.</li><li><a href="https://github.com/unslothai/unsloth"><strong>Unsloth</strong></a>: Simplifies fine-tuning and accelerates multimodal experiments.</li></ul><h3>Conclusion</h3><p>Vision-Language Models (VLMs) have dramatically improved how models understands both images and language. Early examples used simpler approaches, combining CNNs and RNNs for tasks like basic image captioning. Today’s VLMs, such as GPT-4 Vision, Gemini, and Gemma-3, are far more advanced, able to understand and explain complex visual information clearly and accurately.</p><p>In this article, we covered:</p><ul><li><strong>How VLMs evolved over time</strong>, highlighting major architectures, training methods, and ways they combine text and images (fusion strategies).</li><li><strong>Main architecture types, </strong>focusing on autoregressive approaches</li><li><strong>Evaluation methods, </strong>rigorous testing on tasks such as captioning, visual question answering, and advanced multimodal reasoning to measure how well models understand combined visual and textual information</li><li><strong>Personalization methods,</strong> such as fine-tuning with LoRA and QLoRA</li></ul><p>Looking ahead, future VLM developments will likely focus on:</p><ul><li><strong>Improving efficiency and performance</strong>, making advanced multimodal AI accessible even in low-resource environments. Models like Phi are extremely exciting for this very reason!</li><li><strong>Better alignment with human expectations</strong>, using synthetic data and human-feedback methods to ensure safe, accurate, and useful outputs.</li><li><strong>Real-world applications (Agentic AI)</strong>: Allowing VLMs to not just understand but actively interact with their environment, opening opportunities in fields like healthcare, robotics, education, and creativity.</li></ul><p>Now it’s quite an exciting time to explore VLMs, as they’re rapidly becoming a key method for so many applications. The frameworks, architectures and models are also becoming increasingly mature, making adoption easier.</p><p>Happy exploring!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8da18e9f0e0c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SANA-Sprint: One-Step Diffusion with Continuous-Time Consistency Distillation]]></title>
            <link>https://medium.com/@pietrobolcato/sana-sprint-one-step-diffusion-with-continuous-time-consistency-distillation-ded84dcbed42?source=rss-52732991c759------2</link>
            <guid isPermaLink="false">https://medium.com/p/ded84dcbed42</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[Pietro Bolcato]]></dc:creator>
            <pubDate>Wed, 19 Mar 2025 15:58:04 GMT</pubDate>
            <atom:updated>2025-03-19T15:58:04.837Z</atom:updated>
            <content:encoded><![CDATA[<h4>Fast Text-to-Image Generation in 0.1s with Diffusion Transformers</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZSOVlhWLtHmyp9Pwip3aqA.png" /><figcaption>Example generations with Sana Sprint, in 0.1s — courtesy of the original paper.</figcaption></figure><h3>Introduction</h3><p>Diffusion models have become the foundation of modern text-to-image (T2I) generation, producing high-quality results but at the cost of <strong>slow, iterative inference</strong>. Most models require <strong>20–50 denoising steps</strong>, making real-time applications hard.</p><p>Existing <strong>distillation techniques</strong> attempt to accelerate diffusion sampling but introduce <strong>stability issues, quality loss in ultra-low-step regimes, and high memory requirements</strong>.</p><p>SANA-Sprint by Nvidia introduces a <strong>hybrid distillation framework</strong> that integrates <strong>continuous-time consistency models (sCM)</strong> with <strong>latent adversarial diffusion distillation (LADD)</strong> to achieve:</p><ul><li><strong>Step-free training</strong>, allowing flexible <strong>1–4 step inference</strong></li><li><strong>State-of-the-art speed vs. quality</strong>, reaching <strong>FID 7.59</strong> and <strong>GenEval 0.74</strong> in a single step</li><li><strong>1024×1024 image generation in 0.1s on an H100 GPU</strong>, running <strong>10× faster than FLUX-Schnell</strong> while maintaining superior quality</li></ul><p>Let’s dive into how SANA-Sprint achieves this!</p><h3>Why Traditional Distillation Fails at Ultra-Low Steps</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wJSglrP30mnE3tJr17bT7g.png" /><figcaption>Latency and memory requirements comparison — courtesy of the original paper.</figcaption></figure><p>Diffusion models rely on <strong>stochastic differential equations (SDEs) or ordinary differential equations (ODEs)</strong> to generate images across multiple steps. Various <strong>step-reduction methods</strong> exist, but they each come with limitations:</p><ul><li><strong>GAN-based distillation (e.g., </strong><a href="https://medium.com/@pietrobolcato/from-add-to-padd-to-ladd-advancing-fast-diffusion-model-distillation-0a52c6dcb101"><strong>LADD</strong></a><strong>)</strong> speeds up inference but suffers from <strong>mode collapse</strong> and poor generalization</li><li><strong>Consistency models (CMs)</strong> enable fast sampling but <strong>degrade semantic alignment</strong> in <strong>ultra-low-step settings (&lt;4 steps)</strong> due to trajectory truncation errors</li><li><strong>Variational Score Distillation (VSD)</strong> requires training an auxiliary diffusion model, increasing <strong>GPU memory and computational overhead</strong></li></ul><p>SANA-Sprint overcomes these issues by <strong>combining sCM and LADD into a unified framework</strong>, ensuring both <strong>fast inference and high image quality</strong>.</p><h3>Reusing Pre-Trained Models with a Training-Free Consistency Transformation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fTK9_GsTgR4JQVpoQwcdgA.png" /><figcaption>Architecture — courtesy of the original paper.</figcaption></figure><p>Diffusion models typically use <strong>flow-matching</strong> or <strong>score-based learning</strong>, while <strong>consistency models (CMs)</strong> operate under the <strong>TrigFlow parameterization</strong>. To enable <strong>fast distillation without retraining</strong>, SANA-Sprint introduces a <strong>mathematical transformation</strong> that converts <strong>pre-trained flow-matching models into TrigFlow models</strong>.</p><p>This transformation ensures:</p><ul><li><strong>Seamless time-domain mapping</strong> from the <strong>flow-matching interval [0,1]</strong> to <strong>TrigFlow’s [0, π/2]</strong></li><li><strong>Signal-to-noise ratio (SNR) consistency</strong>, maintaining fidelity during adaptation</li><li><strong>Correct parameterization of model outputs</strong>, ensuring that learned velocity fields remain aligned with TrigFlow’s formulation</li></ul><p>This allows <strong>pre-trained models to be used directly</strong> in the SANA-Sprint framework <strong>without retraining</strong>, making it highly efficient!</p><h3>Solving Training Instability in Large-Scale Consistency Models</h3><p>Scaling consistency models to <strong>higher resolutions and larger architectures</strong> often leads to <strong>unstable training</strong> due to exploding gradients. SANA-Sprint stabilizes training with two key contributions:</p><h4>Dense Time Embeddings to prevent Gradient Explosion</h4><ul><li>Standard diffusion models scale time embeddings using <strong>multiplicative factors (e.g., </strong><strong>1000 * t)</strong>, which amplify <strong>time-derivative gradients</strong>, leading to training collapse</li><li>SANA-Sprint <strong>normalizes time embeddings</strong>, ensuring <strong>evenly distributed timestep representations</strong> that improve <strong>stability and sample quality</strong></li><li>This allows the model to <strong>converge faster and produce sharper generations</strong></li></ul><h4>QK-Normalization for Stable Self- and Cross-Attention</h4><ul><li>Increasing model size (<strong>0.6B → 1.6B parameters</strong>) leads to <strong>unstable gradient norms (&gt;¹⁰³)</strong>, causing training failures</li><li>SANA-Sprint introduces <strong>RMS normalization in Query and Key (QK) components of attention layers</strong>, stabilizing gradients <strong>without modifying the architecture</strong></li><li>Just <strong>5,000 fine-tuning iterations</strong> significantly reduce instability, enabling <strong>stable distillation for large-scale diffusion models</strong></li></ul><h3>Combining Consistency Models with Adversarial Supervision</h3><p>Consistency models traditionally rely on <strong>local trajectory learning</strong>, making them <strong>slow to converge</strong> and <strong>prone to detail loss</strong> in <strong>single-step generation</strong>. SANA-Sprint enhances these models by incorporating <strong>GAN-based adversarial supervision via LADD </strong><a href="https://medium.com/@pietrobolcato/from-add-to-padd-to-ladd-advancing-fast-diffusion-model-distillation-0a52c6dcb101">(Latent Adversarial Diffusion Distillation)</a>:</p><ul><li>A <strong>frozen teacher model</strong> extracts high-level <strong>latent-space representations</strong> to enforce consistency</li><li>A <strong>multi-head discriminator</strong> learns <strong>feature-based differences</strong>, avoiding pixel-space mismatches</li><li>A <strong>hinge loss formulation</strong> improves <strong>training stability and sample realism</strong></li></ul><p>This technique <strong>enhances single-step image quality</strong>, preserving <strong>high-frequency details</strong> that traditional consistency models struggle with.</p><h3>Evaluation and results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3WaA_8uUBJ2f5W_CQkLY-g.png" /></figure><p>SANA-Sprint sets a <strong>new state-of-the-art</strong> in both <strong>speed and quality</strong>;<strong> </strong>it is <strong>10× faster than FLUX-Schnell</strong> while generating <strong>better-quality images</strong>. It achieves <strong>7.59 FID and 0.74 GenEval</strong> in just <strong>one step</strong>, outperforming models that need multiple steps. Even on <strong>consumer GPUs like the RTX 4090</strong>, it generates <strong>1024×1024 images in just 0.31 seconds</strong>, making high-quality AI generation more accessible. On an <strong>H100 GPU</strong>, it reaches <strong>0.1s for text-to-image</strong> and <strong>0.25s for ControlNet tasks</strong>, enabling near-instant visual feedback. So cool!</p><h3>Conclusion</h3><p>Unlike traditional diffusion models that need <strong>20+ steps</strong>, SANA-Sprint delivers <strong>high-quality results in just 1–4 steps</strong> without extra training. <strong>One-step inference is nearly instant</strong>, ideal for real-time applications. <strong>Two-step generation sharpens details while keeping speed under 0.25s</strong>. <strong>Four steps offer the best balance of quality and efficiency.</strong></p><p>The paper is quite mathematically involved, but it’s a <strong>technical masterpiece</strong> and therefore a <strong>very much recommended read</strong>. This line of work will open lots of <strong>downstream optimization to Flow Matching DiT models</strong>, which will in turn allow for faster, cheaper generation.</p><p>Distilled inference is <strong>democratized</strong> inference!</p><h3>Resources</h3><ul><li><strong>GitHub</strong>: <a href="https://github.com/NVlabs/Sana">SANA-Sprint Code</a> <em>(coming soon!)</em></li><li><strong>Project Page</strong>: <a href="https://nvlabs.github.io/Sana/Sprint/">SANA-Sprint</a></li><li><strong>Paper</strong>: <a href="https://arxiv.org/pdf/2503.09641">ArXiv</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ded84dcbed42" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Paper deepdive: Stable Flow]]></title>
            <link>https://medium.com/@pietrobolcato/paper-deepdive-stable-flow-c70bd627bca3?source=rss-52732991c759------2</link>
            <guid isPermaLink="false">https://medium.com/p/c70bd627bca3</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[diffusion-models]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[image-editing]]></category>
            <dc:creator><![CDATA[Pietro Bolcato]]></dc:creator>
            <pubDate>Thu, 13 Mar 2025 10:25:46 GMT</pubDate>
            <atom:updated>2025-03-13T10:25:46.781Z</atom:updated>
            <content:encoded><![CDATA[<h4>Training-free image editing using Diffusion Transformers</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QMTIQAR2ZOt7QptPIf1Gbg.png" /><figcaption>Example edits that the method allows — courtesy of the original paper.</figcaption></figure><h3>Introduction</h3><p>Diffusion models have transformed image generation and editing, but most methods require fine-tuning or per-instance optimization, making them computationally expensive and impractical for real-world applications.</p><p>Until recently, most diffusion-based models used UNet architectures, and many text-based image editing techniques (e.g., SDEdit, Instruct-P2P, MagicBrush) were designed specifically for this kind of architecture.</p><p>However, state-of-the-art diffusion models are shifting toward Diffusion Transformers (DiT), which outperform UNets in scalability, efficiency, and high-resolution image generation. DiT-based architectures power SOTA models like FLUX and Stable Diffusion 3, making them the new standard for image synthesis and editing.</p><p>Stable Flow introduces a training-free image editing framework that works with Diffusion Transformers (DiT) to:</p><ul><li>Enable text-guided image modifications (pose changes, object addition/removal, scene edits).</li><li>Ensure identity and structure preservation in the edited image.</li><li>Work directly with existing diffusion pipelines (e.g., FLUX, Stable Diffusion 3) without retraining.</li><li>Leverage flow-based diffusion models’ lower diversity to produce consistent and stable edits.</li></ul><p>Instead of requiring fine-tuned diffusion models, <em>Stable Flow</em> finds the right subset of layers in DiT for injecting control signals, making it training free, lightweight, stable, and efficient.</p><p>Also — it’s open source, and conveniently built on top of Diffusers! Let’s dive in.</p><h3>The Challenge of Editing Images in DiT</h3><p>Most image editing frameworks rely on UNet-based diffusion models, which have a coarse-to-fine hierarchical structure. This structure makes it easy to inject control signals at specific layers — early layers for structural changes, later layers for fine details.</p><p>DiT models, however, do not follow this hierarchical pattern. Every transformer layer processes a mixture of text and image embeddings through self-attention, making it unclear where edits should be injected.</p><p>The two key challenges are:</p><ol><li>Where in DiT should we inject information to enable stable, localized edits?</li><li>How do we invert real images into the latent space without introducing distortions?</li></ol><p>Stable Flow solves both problems using a training-free approach.</p><h3>Finding the Right Layers: Identifying Vital Layers in DiT</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QMvmGr0RHPJsmHtsimnsDQ.png" /></figure><p>Since DiT does not have a coarse-to-fine structure, Stable Flow needs to automatically determine which layers are critical for image formation. Instead of making manual choices, it systematically identifies these vital layers using a layer ablation study.</p><h4>How are vital layers identified?</h4><ol><li>Each DiT layer is removed one at a time, and an image is generated without that layer.</li><li>The difference between the original and modified image is measured using perceptual similarity metrics (e.g., CLIP similarity, DINOv2, LPIPS).</li><li>If removing a layer significantly alters the image, that layer is marked as “vital”.</li><li>If removing a layer has minimal effect, it is ignored for editing purposes.</li></ol><h4>Key Findings</h4><ul><li>Vital layers are not grouped together but are spread across the transformer stack.</li><li>Injecting edits into all layers causes global changes that distort identity.</li><li>Injecting only into vital layers leads to stable, localized modifications.</li></ul><p>This fully automated layer selection process allows Stable Flow to generalize across different DiT architectures without manual tuning.</p><h3>Parallel Feature Injection in Vital Layers</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xctqx3z_-dJIINc69lrmgg.png" /><figcaption>Multimodal attention distribution — courtesy of the original paper.</figcaption></figure><p>Once the vital layers are identified, the editing process involves injecting attention features only into these layers rather than modifying the entire transformer.</p><h4>How does this work?</h4><p><strong>1. Two generation processes run in parallel:</strong></p><ul><li>One process generates the source/reference image.</li><li>The second process generates the edited image based on the target prompt.</li></ul><p><strong>2. Selective Feature Injection:</strong></p><ul><li>Instead of injecting across all layers, which causes instability,</li><li>Stable Flow only modifies the identified vital layers.</li></ul><p><strong>3. Blending Reference Features into Edited Image Generation:</strong></p><ul><li>During inference, some features from the reference image are injected into the corresponding layers of the edited image.</li><li>This ensures that the structure and identity remain unchanged, while only the requested edits (e.g., pose change, object replacement) are applied.</li></ul><h4>Why is this effective?</h4><ul><li>It preserves identity and structure while making only necessary modifications.</li><li>It ensures localized changes, avoiding unintended global distortions.</li><li>It works across different types of edits (rigid &amp; non-rigid modifications).</li></ul><p>This is why <em>Stable Flow</em> can handle complex edits without additional fine-tuning.</p><h3>Improving Real-Image Editing with Latent Nudging</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*fLONTVuO7lUWC1fS7e5Nbw.png" /><figcaption>Latent nudging — courtesy of the original paper.</figcaption></figure><p>For real-world image editing, an existing image must first be inverted into the model’s latent space. However, standard inversion techniques introduce distortions, making the edited output significantly different from the input.</p><p>Stable Flow introduces latent nudging, an improved inversion method that enhances real-image editing.</p><h4>What’s wrong with standard image inversion?</h4><p>Most inversion methods use Inverse Euler ODE solvers to map an image into the model’s latent space. However:</p><ul><li>These methods often fail to perfectly reconstruct the original image.</li><li>Small inconsistencies in the model’s learned vector field cause unintended changes to the image during editing.</li><li>As a result, identity and fine details are often lost.</li></ul><h4>How does latent nudging fix this?</h4><ul><li>Instead of directly inverting an image, a small perturbation is applied to the latent representation before inversion.</li><li>This nudges the image slightly away from the model’s training distribution, helping it stay faithful during reconstruction.</li><li>The difference is visually imperceptible but significantly improves identity preservation during editing.</li></ul><h4>Why does this matter?</h4><ul><li>Better real-image editing: The final image remains closer to the original input.</li><li>Prevents unintended changes: Non-edited regions stay intact.</li><li>Higher fidelity in edits: Objects remain more recognizable even after modifications.</li></ul><h3>Experimental Results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ozpxb9bf8ULubfdGuwvTyw.png" /><figcaption>Results comparison with other models.</figcaption></figure><h4>How well does Stable Flow perform?</h4><p>Stable Flow was evaluated against existing state-of-the-art text-based image editing methods:</p><ul><li>CLIPimg (measures how similar the edited image is to the input).</li><li>CLIPtxt (measures how well the edited image follows the text prompt).</li><li>CLIPdir (measures how closely the direction of image change follows the prompt change).</li></ul><h4>Results</h4><p>On quantitative metrics, Stable Flow outperforms all baselines, achieving higher image fidelity, prompt adherence, and realism.</p><p>Furthermore, a user study on Amazon Mechanical Turk compared Stable Flow against baselines like: SDEdit, Instruct-P2P, MagicBrush and more.</p><p>Users ranked edits based on:</p><ol><li>How well the result matched the prompt</li><li>How well the identity of the original image was preserved</li><li>How realistic the final output looked</li><li>Overall edit quality</li></ol><p>Results showed that Stable Flow was preferred over all competing methods, with users consistently ranking its edits as more stable, natural, and faithful to the input.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JJH8Rjs7iVpog95qKRga2w.png" /></figure><h3>Conclusion</h3><p>Stable Flow introduces a training-free, diffusion-based image editing method that enables localized, high-fidelity edits without retraining. Its layer selection, feature injection, and latent nudging techniques make it a lightweight and scalable alternative to existing methods, allowing DiT-based architectures like FLUX and Stable Diffusion 3 to perform precise, identity-preserving modifications.</p><p>Future work could integrate spatial control mechanisms to expand its flexibility, making it even more powerful for a wider range of image-editing applications. Kudos to the authors for such an amazing work!</p><h3>References</h3><ul><li><strong>Paper:</strong> <a href="https://omriavrahami.com/stable-flow/static/paper/StableFlow.pdf">https://omriavrahami.com/stable-flow/static/paper/StableFlow.pdf</a></li><li><strong>Project page:</strong> <a href="https://omriavrahami.com/stable-flow/">https://omriavrahami.com/stable-flow/</a></li><li><strong>Repository:</strong> <a href="https://github.com/snap-research/stable-flow">https://github.com/snap-research/stable-flow</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c70bd627bca3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Improving Diffusion Models by Optimizing Noise at Inference Time]]></title>
            <link>https://medium.com/@pietrobolcato/improving-diffusion-models-by-optimizing-noise-at-inference-time-0d8f5cfb13f6?source=rss-52732991c759------2</link>
            <guid isPermaLink="false">https://medium.com/p/0d8f5cfb13f6</guid>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[image-generation]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[transformers]]></category>
            <dc:creator><![CDATA[Pietro Bolcato]]></dc:creator>
            <pubDate>Thu, 06 Mar 2025 07:59:34 GMT</pubDate>
            <atom:updated>2025-03-06T07:59:34.540Z</atom:updated>
            <content:encoded><![CDATA[<h4>Similar to LLMs, this method improves image generation by searching for the best noise — scaling inference-time computation beyond just adding denoising steps</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DkLEiH2i6J84FkNe8WPASw.png" /><figcaption>Example of using more time during inference, searching for a better noise — the image quality clearly improves.</figcaption></figure><h3>Introduction</h3><p>Diffusion models are awesome. The latest models, such as <strong>FLUX</strong>, based on the <strong>DiT architecture</strong>, combine transformers with diffusion processes and rectified flow, delivering incredible results. However, achieving even better results usually requires performing more denoising steps during sampling.</p><p>A recent approach explores a different way of scaling inference-time computation: instead of just adding more denoising steps, it <strong>optimizes the injected noise itself through a structured search</strong>. By carefully selecting the <strong>initial noise</strong> fed into the diffusion process, this method achieves significant improvements in generation quality <strong>without additional training</strong>.</p><p>It’s a really cool idea. Let’s dive deeper into the details.</p><h3>Optimizing Noise Selection as a Search Problem</h3><p>The core idea of this approach is <strong>framing the selection of initial sampling noise as a search problem</strong>. Specifically, the method explores two main components:</p><ul><li><strong>Verifiers</strong>: Functions used to assess the quality of generated samples.</li><li><strong>Algorithms</strong>: Techniques that iteratively improve candidate noise samples based on feedback from the verifiers.</li></ul><p>Instead of performing a single inference pass, this approach evaluates <strong>multiple candidate noises</strong> and selects the best one based on verifier feedback.</p><p>This <strong>search process adds computational costs</strong> at inference time, measured in <strong>number of function evaluations (NFEs)</strong>. The key trade-off is clear: <strong>higher NFEs lead to better generation quality</strong>, but also increase computational overhead. The challenge is to find an optimal balance where the additional compute is justified by the improvements in sample quality.</p><h3>Verifiers: Evaluating Sample Quality</h3><p>Verifiers play a crucial role in this search process. They determine how the noise candidates are ranked, directly influencing the final output quality. Different types of verifiers offer different advantages and drawbacks:</p><h4>1. Oracle Verifiers</h4><p>Oracle verifiers use <strong>privileged evaluation metrics</strong> that are typically unavailable during inference. For example, using <strong>FID (Frechet Inception Distance)</strong> directly as a selection criterion ensures optimal sample selection. However, in real-world scenarios, these verifiers are impractical because they require a reference dataset, which is <strong>not available at generation time</strong>.</p><h4>2. Supervised Verifiers</h4><p>Supervised verifiers rely on <strong>pretrained models</strong> to estimate quality metrics. Examples include:</p><ul><li><strong>CLIP-based verifiers</strong>: Measure how well the generated image aligns with its text prompt.</li><li><strong>DINO-based verifiers</strong>: Evaluate the image’s feature similarity to other high-quality samples.</li></ul><p>These verifiers are much more practical than oracle verifiers since they don’t require access to a reference dataset. However, they introduce biases — <strong>favoring certain attributes over others</strong>, which may limit diversity in generations.</p><h4>3. Self-Supervised Verifiers</h4><p>Self-supervised verifiers attempt to assess quality without external classifiers, often by measuring <strong>feature similarity</strong> within the generated set itself. However, experiments show that <strong>self-supervised approaches struggle in text-to-image generation</strong>, making them less effective than supervised verifiers.</p><h4>Bias in Verifier Selection</h4><p>Each verifier introduces a different bias:</p><ul><li><strong>Aesthetic-based verifiers</strong> prefer visually pleasing images, but this can come at the cost of text alignment.</li><li><strong>CLIP-based verifiers</strong> improve text-image alignment but may degrade visual quality.</li></ul><p>To mitigate these biases, <strong>ensemble verifiers</strong> — which combine multiple evaluation metrics — tend to produce the best overall results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w-7Y1vJG1MNscYAFs1ZMfQ.png" /></figure><h3>Algorithms: Optimizing Noise Candidates</h3><p>Once a verifier is chosen, the next step is to <strong>search for the optimal noise</strong> using different algorithms. Three main approaches are considered:</p><h4>1. Random Search</h4><ul><li>Generates multiple noise samples.</li><li>Selects the best one based on the verifier’s score.</li><li>Simple but limited in efficiency.</li></ul><h4>2. Zero-Order Search (ZO-S)</h4><ul><li>Iteratively refines noise candidates by introducing small perturbations.</li><li>Uses verifier feedback to improve selection over multiple steps.</li><li>More computationally expensive but leads to better sample quality.</li></ul><h4>3. Search Over Paths</h4><ul><li>Adjusts sampling trajectories by modifying noise <strong>at multiple diffusion steps</strong>, rather than just the initial step.</li><li>Allows for deeper optimization of the entire sampling process.</li></ul><h4>Key Findings</h4><ul><li>No single <strong>verifier-algorithm</strong> combination is universally optimal — task-specific choices matter.</li><li><strong>Verifier hacking</strong> can occur if the search <strong>overfits to a biased verifier</strong>, degrading other quality aspects. Effectively, the <strong>search algorithm excessively optimizes for a single verifier’s score</strong>, leading to unintended degradations in other aspects of image quality. This happens because verifiers — whether they are CLIP-based, aesthetic-based, or feature-similarity-based — are <strong>not perfect representations of true image quality</strong>. They introduce <strong>biases</strong>, and when the optimization process exploits these biases too aggressively, it can <strong>overfit to the verifier rather than improving overall image quality</strong>.</li><li><strong>Ensemble verifiers</strong> mitigate bias and improve generalization.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H_7bj_sFwFVGzsFEOGG2_g.png" /></figure><h3>Evaluation</h3><p>The method was tested on two key tasks:</p><ul><li><strong>Class-conditioned image generation </strong>on ImageNet</li><li><strong>Text-to-image generation </strong>using the FLUX.1-dev model</li></ul><p>To measure performance, the evaluation focused on <strong>generation quality, semantic consistency, and controllability</strong>.</p><h4>1. Generation Quality</h4><ul><li><strong>FID (Frechet Inception Distance)</strong>: Measures realism by comparing generated images to real images.</li><li><strong>IS (Inception Score)</strong>: Evaluates diversity and quality in class-conditioned settings.</li><li><strong>Aesthetic Score, CLIPScore, ImageReward</strong>: Metrics for text-to-image quality.</li><li><strong>LLM Grader</strong>: A language model-based evaluation for human-like assessment.</li></ul><h4>2. Semantic Consistency</h4><ul><li><strong>CLIP-based evaluation</strong>: Measures how well the generated image aligns with the text prompt.</li></ul><h4>3. Controllability</h4><ul><li><strong>T2I-CompBench</strong>: Evaluates attribute binding, object relationships, and compositional correctness.</li></ul><h4>Results</h4><ul><li><strong>Search significantly improves generation quality</strong> compared to standard diffusion sampling.</li><li><strong>Verifier choice impacts the final results</strong>, and <strong>ensemble verifiers perform best</strong> in general tasks.</li><li>Achieves <strong>state-of-the-art (SOTA) performance</strong> relative to baseline methods that only increase denoising steps.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m6fl3TwP4E-Abj_4xPdq_g.png" /></figure><h3>Practical Implications</h3><p>Optimizing noise selection has important practical takeaways:</p><ul><li><strong>Better quality at a cost</strong>: High-quality generation is achievable, but <strong>computational costs increase</strong>.</li><li><strong>Trade-offs for real-time applications</strong>: If low latency is required, search-based noise optimization <strong>may be impractical</strong>.</li><li><strong>Smaller models can outperform larger ones</strong>: With noise optimization, <strong>smaller models with search can surpass larger models without search</strong>, making it valuable for applications where model size is a constraint but time is not.</li></ul><h3>Conclusion</h3><p>This approach provides a novel way to improve <strong>diffusion model sampling</strong> by optimizing <strong>the initial noise through structured search</strong>.</p><p>Rather than relying on more denoising steps, this method <strong>selects the best initial noise</strong> using verifiers and search algorithms, leading to <strong>higher-quality generations without additional training</strong>.</p><p>While this method introduces <strong>increased inference costs</strong>, it presents a promising strategy for applications where <strong>sample quality is more important than generation speed</strong>.</p><p>Some experiments also showed that noise optimization is <strong>compatible with fine-tuning techniques</strong>, such as <strong>Diffusion-DPO</strong>. When both approaches are combined, further improvements in image quality are observed.</p><p>This suggests that <strong>fine-tuning and inference-time search are not mutually exclusive</strong>, but rather complementary strategies for improving diffusion model performance.</p><p>This a really cool direction for future research!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0d8f5cfb13f6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>