<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Leo Wossnig on Medium]]></title>
        <description><![CDATA[Stories by Leo Wossnig on Medium]]></description>
        <link>https://medium.com/@leowossnig?source=rss-d2e17e79f2e0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*HPRfnYyK-WxI1BG1L26Q1w.jpeg</url>
            <title>Stories by Leo Wossnig on Medium</title>
            <link>https://medium.com/@leowossnig?source=rss-d2e17e79f2e0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 02:50:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@leowossnig/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The coverage gap]]></title>
            <link>https://medium.com/@leowossnig/the-coverage-gap-555c7eb19a93?source=rss-d2e17e79f2e0------2</link>
            <guid isPermaLink="false">https://medium.com/p/555c7eb19a93</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[product-management]]></category>
            <dc:creator><![CDATA[Leo Wossnig]]></dc:creator>
            <pubDate>Sun, 20 Sep 2026 11:46:13 GMT</pubDate>
            <atom:updated>2026-09-20T13:11:39.180Z</atom:updated>
            <content:encoded><![CDATA[<p>Date: 20.09.2026</p><p>2026 changed how companies build software. AI collapsed the cost of engineering: what used to take months of engineering now takes a few days to a few weeks. The rest of the product development cycle, however, did not accelerate in the same way. Customer discovery, thoughtful design, and judging whether an output is actually good remained slow, and mainly manual human work. So the conventional Engineer : PM : Designer ratio stopped working, and many teams at leading AI companies proposed an obvious fix: if there are too many engineers relative to PMs and designers, so let the engineers do more of the product and design work. The AI product engineer was born and promoted.</p><p>Six months into running a product-engineering-centered org, I think that playbook is right about the problem and wrong as a general solution. It is mostly written by teams who are their own users, and it fails in domains where complex domain expertise or deep customer understanding is required. Building stopped being the constraint. Deciding what “good” means and testing it, however, did not. The reality is that not everyone can, or wants to, do that work.</p><h3>1. Engineering-to-product (and design) ratios are off</h3><p>Engineering has shrunk from ~80% of a product development cycle to 10–20% of the actual time, while discovery and testing have become the critical bottleneck. Determining product requirements and acceptance criteria has also become more complex as LLM based products need acceptance criteria in the form of evals. These are human workflow examples that cover, or approximately sample, the true range of inputs the system needs to deal with, ideally with a target output (’ground truth’). We found that people who can define these evals, or who can run good discovery work (and genuinely look for reasons to kill their own idea), are still rare. Not only due to skill, but also since creating evals data is hard.</p><p>For this reason, Andrew Ng argued earlier this year that the Engineering:Product ratios should flip toward <a href="https://medium.com/@michelegalli.pm/andrew-ng-thinks-we-need-2-pms-per-engineer-is-he-right-e160c094934c">2 PMs per 1 engineer</a>. Allstacks also makes the case that the canonical 1 PM : 6 engineers assumed coding was the bottleneck and is <a href="https://www.allstacks.com/blog/rethinking-the-pm-to-engineer-ratio-for-the-ai-era">shifting toward 1:3 or even 1:1</a>. Ant Murphy describes teams that <a href="https://www.antmurphy.me/newsletter/how-product-is-changing-in-2026">build a lot in four days, then spend months on evals and fine-tuning</a>. Amol Avasare, Anthropic’s Head of Growth, <a href="https://www.lennysnewsletter.com/p/anthropics-1b-to-19b-growth-run">said it plainly</a>: with Claude Code, a default team of five engineers, one designer, and one PM now produces the output of 15–20 engineers. Engineering output is therefore up 2–3× while team sizes stay flat. “PM and design is just squeezed.” His conclusion: the PM-to-engineer ratio may need to flip toward more PMs, not fewer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9KFTFBmqZholLK4WR03vpg.png" /></figure><p>The same applies to design: AI can quickly generate many average designs, but good, thoughtful UX still takes craft and a lot of time (as of September 2026). It always reminds me of <a href="https://www.benkuhn.net/thinkrealhard/">this article</a> which I regularly send to new team members. Adobe Design said exactly that: <a href="https://adobe.design/ideas/is-design-dead-in-the-age-of-ai">AI can generate a hundred ideas in seconds, most of them mediocre</a>. Output just isn’t design. Similarly as shipping code today doesn’t mean there is a product. Although I will caveat this by saying: Models keep advancing at an incredible pace. Just today <a href="https://openai.com/index/navier-stokes-solution/">OpenAI claimed they solved on of the Millenia Problems</a>. Of course, one could argue that these are verifiable and hence easier to crack, but I would still bet that models get better even at design and other capabilities.</p><p>This means that team compositions need to change. We found that <strong>the best product teams (versus e.g. our platform teams) at Forgent work in ratios of roughly 1 PM and 1 designer for every 1–2 engineers</strong>. The challenge, however, remains: Most companies can’t hire great PMs fast enough to shift to the new ratio.</p><h3>2. The product engineer role does not work everywhere</h3><p>Since teams at leading AI companies experienced the same challenges with the broken ratio, and hence proposed an obvious solution: if there are too many engineers relative to PMs and designers, let the engineers do more of the product and design work. The AI product engineer was born and promoted.</p><p>Cat Wu (Claude Code) promoted to hire engineers with product taste rather than more PMs. Ryo Lu at Cursor distributes product management across builders. Nan Yu runs <a href="https://www.news.aakashg.com/p/nan-yu-podcast">~2 PMs on Linear’s $1.25B product</a>, with small product-engineer teams owning their roadmaps.</p><p>One thing that got not commented on as much is that most published examples are developer or builder tools or an AI lab building mostly for engineers or technical users. Here, the person shipping can evaluate the output because they are the user. Avasare’s original claim carried a qualifier for this that gets dropped in the retelling: engineers doubling as PMs works especially for highly technical products.</p><p>There is, however, little evidence that the model works for complex products whose users are far from developers. For us, as a B2B software company that is building AI agents for winning tenders, “the user” is the bid manager answering an RFP, the sales person investigating the competition, or the legal reviewer checking the bid before submission. These are users we cannot simply understand based on our own preferences and experiences. This is not because an engineer cannot do product work per se, but because people that are excellent at everything at once are the exception. For example, the <a href="https://newsletter.pragmaticengineer.com/p/tech-jobs-market-in-2026-part-3-hiring">Pragmatic Engineer’s 2026 hiring survey</a> confirms “product engineer” are one of the hardest profiles to find. In particular a decent design eye plus full-stack ability (with taste and trust). The fix for a hiring bottleneck hence ran into another hiring bottleneck.</p><h3>3. Craft excellence is not product excellence</h3><p>Bringing design, product, and engineering craft together is key to any successful product, but just as important are a team’s ways of working. The best teams, in my opinion, operate in very close collaboration from the very beginning. Jenny Wen (Anthropic), for example, rejects the conventional discovery→design→handoff process in favor of pairing with engineers and build prototypes to align everyone. That is what I believe works best in practice: a strong team being in near perfect sync. It only works, however, in small teams, which is fine, because it removes the communication overhead. Fast implementation, high autonomy, and close alignment (since everyone is in the same room so decisions are made immediately) results in incredible iteration speed.</p><p>The other critical part is maintaining a high bar and enforcing it with a good process. AI can generate a lot of code today, and it is easy to end up with codebases that diverge or agents that quietly break the architecture. The best way to protect the team’s ability to iterate quickly and still keep the quality bar high is to have a stringent release process:</p><ul><li>Alpha releases are clearly labelled “experimental” and can get into users hands fast with minimal restrictions. Build prototypes, put them to customers, get feedback, iterate.</li><li>Beta+ comes with strict gates for everything that actually goes to production or GA.</li></ul><p>Anthropic runs this as a ladder: internal dogfooding→ research preview (’experimental’) → public beta → GA, with the bar rising at each step. Experimental features should be easy to kill if they don’t work. And production features shouldn’t be prototypes that someone pushes into Beta/GA.</p><h3>4. PM Work also got more complex since LLMs require evals</h3><p>One of the key reasons LLMs succeeded where conventional automation failed is that they can deal with variable inputs and trajectories and hence unlock real human workflows (’work’). This variability is, however, not incidental to the product but it <em>is</em> the product. The problem with that is that high variability means you need many instances (think inputs/outputs) to test it, and testing requires someone who knows the right answer and can write it down/create it. These instances together form the Evals (short for evaluations) for a product, and these evals are in effect the acceptance criteria an LLM product needs.</p><p>Evals are for that reasons a core product capability as Kevin Weil (OpenAI) said: <a href="https://www.lennysnewsletter.com/p/kevin-weil-open-ai">“Writing evals is going to become a core skill for product managers. It is such a critical part of making a good product with AI.”</a> Mike Krieger at Anthropic has made the same point, and Xavi Amatriain (Expedia) went even further when he said that evals are the new PRD.</p><p>One challenge with that is that the person who knows the right answer is often not on the team but an external subject matter expert. The bottleneck then becomes expert judgment throughput (c.f. <a href="https://knowledge.wharton.upenn.edu/podcast/where-ai-works/judgment-is-the-new-bottleneck/">“judgment is the new bottleneck”</a>). This is which is why companies like Harvey or Legora (Legal AI) embedded many lawyers in their product org (Harvey: 180+ lawyers, Legora <a href="https://jobs.ashbyhq.com/legora/7cbc3e2b-d191-4ddc-9801-9541f2622c65">hires lawyers into the org as legal engineers</a>). The work of those experts shows up in Harvey’s own benchmarks: on <a href="https://www.harvey.ai/blog/introducing-biglaw-bench">BigLaw Bench</a>, which grades models against lawyer-written rubrics from billable work, <a href="https://www.harvey.ai/blog/gpt-5-6-sol-in-harvey">GPT-5.6 Sol scored 92.7% in July 2026</a>. In contrast, on <a href="https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark">LAB</a> (Harvey’s long-horizon legal agent benchmark) when graded on a strict all-pass basis the same model scored 2.5%. Their conclusion is that while short-horizon reasoning is nearly solved, completing end-to-end legal work is far from it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XFGA7Vr16lINa7ehJWzwZw.png" /></figure><p>(Disclaimer: Both are vendor-run benchmarks, so treat the numbers as directional rather than independent evidence.)</p><p>What is important to understand here, is that evals are often tied to the product development stage as well. During the early product development stages, taste, dogfooding, and fast user feedback beat larger eval suites. Braintrust’s founder frames vibe-checking as the “do things that don’t scale” phase. I have also seen too many people invest too early into large evals data, only to find these outdated a few weeks or months later. Evals for mature products, however, is key, and smaller offline-evals are also very helpful during development, in particular during prompt engineering.</p><p>A side note: While LLM-as-judge can help in certain cases, they do not remove the need for experts, <strong>yet (!)</strong>. In the most direct study to date (although a bit outdated!), <a href="https://arxiv.org/abs/2410.20266">Szymanski et al. (IUI ’25)</a> tested LLM judges against registered dietitians and clinical psychologists: experts agreed with a general LLM judge only 60–64% of the time against 72–75% agreement between the experts themselves. Lay users agreed with the same judge 80% of the time (p &lt; 0.0001), “suggesting that the models are more closely aligned with lay user preferences”. Prompting hte models with ‘expert-personas’ also only reduced the gap by ~4 points (to 64–68%). <a href="https://arxiv.org/abs/2508.07827">A COLM 2025 study</a> across finance, law, and biomedicine corroborated this further with the finding that the most capable models could not match expert ground truth on domain-specific annotation. And finally, <a href="https://arxiv.org/abs/2406.18403">Judge-Bench</a> confirmed the same pattern across 20 NLP evaluation tasks, stating that “all models achieve higher correlations with annotations by non-expert human judges compared to expert annotators”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B4IsaTveP2gIhTXzrwAPfg.png" /></figure><p>In complex domains, general purpose LLMs are not enough today. Until they get good enough in such areas, experts are hence a core requirement in every good product org. These experts are still required to determine what ‘good’ looks like. Limitations will likely remain in areas where expert knowledge is not publicly accessible or requires access to proprietary data. However, and that’s important, LLMs and tools will get better at many of the white-collar tasks that are associated with such work (e.g. making slides, filling forms, etc.). The main question is whether they know what to fill these outputs with, i.e. whether expert knowledge is required for the content or not.</p><p>The upside: Evals are your product and your moat. If you are the only one to be able to reliably test whether LLMs perform well on a task you can also optimize your product more than anyone else. At Forgent we have expert-knowledge encoded in many steps in the product and are highly opinionated what ‘good’ means. A lot of this is also encoded in the instructions and the evals we have built across our features. These form a defensible moat that allow us to optimize our model choices and harness to make conscious trade-offs (e.g. speed, cost, quality).</p><h3>5. Eval coverage is a commercial choice, not just a technical one</h3><p>If eval coverage does not match the real diversity of the work, you cannot reliably tell where your product will fail. A PM who does not set up evals for their product ships without knowing whether it works for all customers. So you either build coverage that matches the product scope, or you narrow the scope until it does. Narrow coverage and narrow ICP are the same decision, and it’s important that these go hand in hand with the sales / go-to-market strategy. In most teams that decision is never made explicitly. I think that aligning the teams (GTM &lt;&gt; Product) around the target customer and align evals with that is hence as important as writing a good PRD was in the past.</p><p>To give an example: In our case, we focused with our validation heavily on German IT tenders and while generalization to French IT tenders is likely feasible, we can’t take this for granted without having specific French evals.</p><h3>6. Evals tell you whether you solved the problem, but not whether it is worthwhile</h3><p>Evals tell you whether the product does the job. They do not tell you whether it solves the right problem for the user, or whether anyone is willing to pay for it. There are, in my opinion, four relationships that need to be true in order for a product to be commercially valuable. These are: capability → adoption → (customer) value → willingness to pay. Evals can only establish the first of these and clearing the rest requires (a) a well done discovery, and (b) honest measurement after the product is shipped (adoption, retention, and whatever the specific product needs, e.g. for Forgent, first-pass acceptance, i.e. the share of outputs finalized with minimal editing and rejection rate). These are the numbers that let you decide whether to keep, kill, or evolve a product and complete the standard product development cycle. All of this to say that this is another argument for more PM capacity. Being intentional about which products are worth building at all, and continuing to watch them after launch also takes time (and customer interviews when you can’t measure things).</p><h3>7. Counter-examples exist, but are rare</h3><p>I want to be honest that there are counter-examples to my claims. <a href="https://designerfund.substack.com/p/ai-design-sierra">Sierra</a> (backed by Sequoia, Benchmark, and Tiger Global, among others; ~$200M ARR as of May 2026; <a href="https://www.cnbc.com/2026/05/04/bret-taylor-sierra-fundraise-openai.html">~$1.6B raised at a $15.8B valuation</a>) is building an enterprise, domain-distant product (AI-powered customer service agents for enterprises), and still operating engineering-heavy. Based on public information they roughly use a 20:1 engineering to design ration, where design moved upstream into shared components rather than adding designers. So the product engineer model is not disqualified in enterprise. But notably, at Sierra, their ‘agent’ engineers embed into customer domains, and the whole business is priced on successful outcomes.</p><h3>8. Target state versus where we are</h3><p>For us at Forgent, the target state is clear: 1 PM and 1 designer per 1–2 engineers, plus in-house subject matter experts who create / determine ground truth. However, like for many companies there is a gap between target and where we are today. PMs and designers who can do this work are hard to hire, and the eval work puts even more load on the PMs we already have.</p><p>So the product engineer is still a valid way to ease the transition. Evals are ultimately how you evaluate your product and should be led by the PM with domain expert input; until we get there, some PM and design decisions will need to remain with the product engineers and explicitly defining the scope in which they can decide freely is key to getting through the transition while maintaining velocity.</p><h3>Wrap up</h3><p>Deciding what “good” means and testing whether it works and is a valid product idea have become the main constraints of product development (over building). While the product engineer is a good idea for some teams, we found that product teams need roughly 1 PM and 1 designer per 1–2 engineers. Evals have at the same time become the acceptance criteria for LLM products, and eval coverage is a commercial decision that need to be aligned with the GTM motion, although early on a in a startup this might be limited (since you want to explore).</p><p>In expert domains, because general purpose LLMs cannot reliably define quality themselves yet, product teams still need people with real domain knowledge, since only they can decide what is professionally correct and good. That makes domain experts part of the product org rather than an occasional input to it. For us, the PM owns the eval suite, its coverage, and the scope while the expert owns the ground truth inside it. The exception are verifiable standard tasks and ‘general’ white collar work (’make a presentation’), which foundational models are doing exceptionally well. Founders should stay clear of core product value built on this!</p><p>At Forgent, we do not sell “AI for tenders”, we sell AI with expert quality control for a high-stakes process where almost correct is not good enough: one missed requirement can be enough to lose the opportunity. That is why we follow the research on where general purpose models still fall short and translate those findings back into the product and the evals.</p><p>The one thing I would not bet on is that any of this stays fixed. Expertise is a moving target, and model capability is moving toward it faster than most org designs adapt. So all of this probably will need to be revisited every few months, and shouldn’t be treated as an operating model that can be fixed once.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=555c7eb19a93" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recursive Self-Improvement, Model Advances and Implications for AI Products in 2026]]></title>
            <link>https://medium.com/@leowossnig/recursive-self-improvement-model-advances-and-implications-for-ai-products-in-2026-4a6c4069a052?source=rss-d2e17e79f2e0------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a6c4069a052</guid>
            <category><![CDATA[recursive-language-models]]></category>
            <category><![CDATA[ai-applications]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Leo Wossnig]]></dc:creator>
            <pubDate>Fri, 12 Jun 2026 08:36:58 GMT</pubDate>
            <atom:updated>2026-06-12T13:12:03.598Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CuBK_XXODUw4lMF-EnVNOw.png" /><figcaption>Anthropic releases Claude Fable 5 and Claude Mythos 5</figcaption></figure><p>I usually don’t post unless there is something worth writing about, but the world recently seems to move so fast that a weekly post is needed. This week, Anthropic released Fable 5, a Mythos-class model that beats again all benchmarks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T39cUTrj5rhu3IFsbyVbJA.png" /><figcaption>Fable 5 outperforms other frontier models by a significant amount across several benchmarks.</figcaption></figure><p>The speed at which models are released is getting increasingly faster, perhaps as a result of a phenomenon that Anthropic laid out in their <a href="https://www.anthropic.com/institute/recursive-self-improvement">blog post on recursive self-improvement</a>, i.e. when AI learns to improve itself.</p><p>The topic is not a new one, as a master student in 2012/3 I worked on genetic programming and self-improvement algorithms and back then already read about <a href="https://people.idsia.ch/~juergen/">Schmidhuber</a>’s <a href="https://en.wikipedia.org/wiki/G%C3%B6del_machine">Gödel Machine</a> or Eliezer Yudkowsky’s blog on <a href="https://www.lesswrong.com/posts/JBadX7rwdcRFzGuju/recursive-self-improvement">recursive self-improvement</a>. Building an AI system that can autonomously improve itself is probably simultaneously the dream and an absolute horror of any AI researcher.</p><p>The principle is simple: Recursive self-improvement is AI (or better AGI) systems rewriting their own code to improve themselves. Repeating this again and again ultimately results in an intelligence explosion, that should theoretically result in ‘super’- intelligence.</p><p>While some of this is already happening indirectly (see the example by Anthropic below, taken from their blog post on <a href="https://www.anthropic.com/institute/recursive-self-improvement">recursive self-improvement</a>), we also see on the AI research side first results indicating the progress in this area of research.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TVK8j6BYbsVZZr96QFDVEw.png" /><figcaption>Steps towards recursive self-improvement explained in the <a href="https://www.anthropic.com/institute/recursive-self-improvement">Anthropic blog post</a>.</figcaption></figure><p>One of the big three levers for the 2026 capability improvements in Coding and other areas, that I have <a href="https://forgent.medium.com/building-ai-enabled-engineering-product-design-teams-in-2026-8587e568c0cf">previously written about</a>, were model <a href="https://www.anthropic.com/engineering/harness-design-long-running-apps">harness improvements</a>. The harness is essentially the entire infrastructure around the model that handles how the model accesses tools, gets feedback (responses), etc.</p><p>While harness engineering is still a big topic in the AI engineering community, the end-to-end optimization of the harness by AI, has already been shown to be possible by the research community. For example, the paper ‘<a href="https://arxiv.org/abs/2603.28052">Meta-Harness: End-to-End Optimization of Model Harnesses</a>’ by Lee et <em>al</em>. shows that by using Claude to propose a new harness code, they outperfomed multiple other approaches and models on <a href="https://www.tbench.ai/">TerminalBench-2</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/937/1*odTQZdspOHeDkPHgRllS-A.png" /><figcaption>Optimization loop in the <a href="https://arxiv.org/abs/2603.28052">Meta-Harness paper</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*fDvSJEWKWp0p9nX0WK0ujA.png" /><figcaption>Performance with harness optimization / evaluations.</figcaption></figure><p>While these earlier versions like the Meta-Harness Learning Algorithm or the well-received <a href="https://deepmind.google/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/">AlphaEvolve</a> exist, these have so far still felt far away from a true <a href="https://www.lesswrong.com/posts/JBadX7rwdcRFzGuju/recursive-self-improvement">Seed AI</a>. Given the recent advances, however, we may be closer to an inflection point than expected. Dario Amodei highlighted the risks in <a href="https://darioamodei.com/post/policy-on-the-ai-exponential">his recent essay</a> (although the IPO might also drive a certain narrative).*</p><p>While the idea of AI becoming increasingly powerful can be concerning from a societal perspective (see his essay), there are also real policy considerations that can’t be ignored (see e.g. <a href="https://medium.com/@leowossnig/who-will-own-the-ai-economy-4726ed68c43c">my own recent article on this topic</a>). However, the reality is that a most of the decision making power lies with Anthropic, OpenAI, and the world governments.</p><p>As an entrepreneur, I hence ask what practical implications there are for building AI products today.</p><h3>Implications for AI Products</h3><p>As said above, recursive self-improvement, at the model level, is not something that startups and even most other companies will be able to do. We will never have the compute or the research org to rewrite our own foundation model. The critical question to ask is then how much remains for AI applications to do if the model becomes smarter and progressively will take more of the work that AI products are delivering today.</p><p>Sarah Guo has just written the clearest answer I have recently seen, in an essay called <a href="https://www.linkedin.com/pulse/untrainable-sarah-guo-qr0tc/">The Untrainable</a>. Her starting point is exactly the question above: if the model keeps getting better at everything, then every company built on one is a thin wrapper waiting to be absorbed, and the only value that survives is the compute and the frontier weights.</p><p>She argues that what you can measure, you can train against, which is why coding agents matured first: a compiler and a test suite are free verifiers, so you can grind against the check until you beat it. Devin solved thirteen percent of the standard software benchmark in 2024 and was initially dismissed. Then, eighteen months later the best agents were all in the high eighties. Her argument is that free verifiers make a task trainable and therefore commoditizable. One technology that enabled this is <em>Reinforcement Learning with Verifiable Rewards</em> that I also covered in my recent TUM and ETH talks. She goes on that anything that is legible will also be commoditized and creates a 2x2 matrix with the fourth (bottom right square below) quadrant being the place for startups to play. In a nutshell: intelligence, i.e. the model, will keep getting commoditized. So value will accrue in places a model can’t reach, such as areas with proprietary data, trust, and judgement about what good means (i.e. subject expert knowledge).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zzgaN4CnWCcH4_1OfE6Tow.png" /><figcaption>Sarah’s 2x2 matrix where she believes startup moats will remain.</figcaption></figure><p>One of her most interesting comments here is about token commoditization. Specifically, as model intelligence improves, it will cross specific thresholds to reliably complete certain tasks. Once the threshold is crossed, however, people will care less about which model to use but mainly consider the price for which the task can be completed. This will result in a drop in price and hence commoditization. The same thing applies to the harness.</p><p>This is good news, since it gives the application layer the opportunity to choose the best or cheapest model for a specific task. Combine this with industry-specific evals and benchmarks, it creates a moat where application layer AI companies can offer their customers a good price or optimize their margins. This is far less feasible for foundational model providers who want to sell the most expensive token. Of course, there are certain problems where foundational model providers will have a monopoly. For the hardest problems they might be the only ones that have adequately strong models. However, since other companies and open-source models are usually not that far behind, intelligence is progressively getting cheaper (note the difference between intelligence and token cost: Intelligence = cheapest model to solve a task!) which will enable the application layer.</p><p>Switching models and agents is also becoming progressively easier, since the field is moving further into the direction of separating the intelligence from the action, or as Anthropic’s <a href="https://www.anthropic.com/engineering/managed-agents">Managed Agents</a> post puts it ‘decoupling the brain from the hands’.</p><p>So RSI matters to applications not because we can’t do it, but because it sets the rate at which capability diffuses downward and the floor under each task drops, the faster the brain improves, the faster the legible work commoditizes, and the more value concentrates in the places it can’t reach.</p><p>That gives me four implications for building applied AI products today. Most of these align with my <a href="https://forgent.medium.com/defensibility-in-the-age-of-ai-2026-81dc59566dc8">prior assessment</a> where AI products will remain defensible.</p><p><strong>First, intelligence will keep advancing and the providers will increasingly own it.</strong> We will lose more and more of the brain, and players who build strong hands and inherit the rising intelligence will gain an advantage over the ones that keep trying to build everything on their own. Notably, this has already been the case in the past, and teams who built their tech stack in a way to adapt changes faster often were able to offer better products faster.</p><p><strong>Second, the moats that remain are worth more.</strong> When intelligence is getting cheap and available to everyone, the only variance left is what it is pointed at and what it can touch, what Guo calls the untrainable corner. I see four for us: proprietary or hard-to-acquire data (e.g. data crawled from 100s or 1000s of tender portals), network effects (collaboration and multi-party tender or bidder consortia for us at Forgent), deep workflow understanding and proprietary UIs (the latter is shorter-lived), and reference sales (which count double in a conservative public-sector market).</p><p>One thing worth noting is that the providers are quietly moving the durable substrate onto their side too. In the Managed Agents design the session, the log of everything that happened, lives on their platform (unless you build it yourself), and the same will go for memory and other learning mechanisms. I think here it is critical for companies to be promiscuous about brains and harnesses, but be possessive about the objects, outcomes, and the network. Anything that compounds, like customers’ corrections or feedback, has to live in our application in model-agnostic form.</p><p><strong>Third, we should optimize the whole company for feedback and learning.</strong> Evals, customer interviews, in-product corrections, and memory of those corrections are the flywheel, and it runs on data the providers can’t see. A tender’s outcome, won or lost, attached to the full trace of how it was produced, is about as close to a private benchmark as we will get, and unlike a public one it doesn’t turn worthless the day a new model ships. In fact, I would argue it compounds and creates a growing advantage.</p><p><strong>Fourth, knowing when a cheap model is good enough is itself an edge.</strong> A checklist, a classification, an extraction, none of these need the frontier model. Route them to the cheapest model that passes the eval, charge for the outcome, and keep the margin. This goes back to Sarah’s point.</p><p>None of this is a reason to fear the model getting smarter. The opposite: for everyone who like us built the hands, the data, and the trust, every leap in the brain is one you get to ride for free in the long term. Although you might of course pay in the short term as long as the model capabilities you use are singular and competition hasn’t caught up yet (e.g. Anthropic’s Fable 5 right now).</p><p>*: Nathan Lambert has an interesting counter point to RSI in his <a href="https://www.interconnects.ai/p/lossy-self-improvement">recent blog post</a>. He argues that as a field advances the complexity increases and as a result, progress slows down. At the same time, he highlights how current agents are strong at optimizing single performance metrics but struggle with complex, real world multi-metric optimizations. Finally, Nathan points towards a <a href="https://arxiv.org/abs/2603.08640">March 2026 paper</a> that investigates exactly the ability of agents to conduct post-training, the critical phase that turns base LLMs into useful assistants. The results indicate that AI agents underperform versus official models which were trained using humans. Note: My own take on the paper is actually different: since I believe the results do look promising and indicate already a certain power of the models. It is also unclear how much exploration and resources the ‘leading providers’ have used, andgiven that this was largely an academic collaboration, the performance of the agents is in my view surprisingly strong.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ADopb3yoppdRhL54-a7nNg.png" /><figcaption>Benchmark from the Paper “<a href="https://arxiv.org/pdf/2603.08640">POSTTRAINBENCH: Can LLM Agents Automate LLM Post-Training?</a>”</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a6c4069a052" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Who Will Own the AI Economy?]]></title>
            <link>https://medium.com/@leowossnig/who-will-own-the-ai-economy-4726ed68c43c?source=rss-d2e17e79f2e0------2</link>
            <guid isPermaLink="false">https://medium.com/p/4726ed68c43c</guid>
            <dc:creator><![CDATA[Leo Wossnig]]></dc:creator>
            <pubDate>Sun, 07 Jun 2026 10:19:28 GMT</pubDate>
            <atom:updated>2026-06-20T11:15:25.205Z</atom:updated>
            <content:encoded><![CDATA[<p><em>When AI does the work, which tax actually reaches the money, and why the answer is obviously right in Washington and a lot less useless in London, Paris, and Berlin.</em></p><p><strong>Author: Leo Wossnig, CTO &amp; Co-founder, </strong><a href="https://www.forgent.ai/"><strong>Forgent AI</strong></a></p><h3>Executive summary</h3><p>Recent discussions about taxing AI mainly focus on token tax, a robot tax, or a levy on data-centre electricity. However, these taxes only address the visible activity and mostly miss the durable money. The durable money in AI is not the chips or the robots, since both earn an ordinary return which where margins decrease over time (with more competition). In contrast, the model and the brand can command an intellectual-property rent that leaves the country (if the AI is now owned) every year as a cross-border licence fee.</p><p>I tried to investigate the different scenarios via an agent-based simulation of an automating economy and to understand which fiscal instrument actually reaches that rent. I looked at it from the position of (a) the country that owns the AI and (b) one that imports it. For these, I found almost opposing answers, concretely, that a border levy reaches the rent while a robot or token tax can’t. In addition, wealth (via foreign rent) that is leaving the country untaxed results in the the ability of foreign entities buying up the domestic capital stock.</p><p>I think it is critical to look at digital-tax again, now perhaps with more urgency, because AI does the actual work itself, and is thereby eroding the wage tax base at the same time as the rent leaks abroad. The full paper, with the model and the code, is <a href="https://github.com/lwossnig/post-automation-economy/blob/main/paper_latex/Simulating_a_Post-Automation_Economy__arXiv_full.pdf">available on GitHub</a>. Parts of the work are still in progress.</p><p><em>Disclaimer: I am not a professional economist (my background is in physics and AI), so read this as a modeler’s take on an economics question, with the assumptions spelled out in the </em><a href="https://github.com/lwossnig/post-automation-economy/blob/main/paper_latex/Simulating_a_Post-Automation_Economy__arXiv_full.pdf"><em>paper</em></a><em>. Corrections welcome.</em></p><h3>Recent political context and definition of ‘rent’</h3><p>In the first week of June 2026, Washington started arguing in public about who should own AI. President Trump <a href="https://www.ft.com/content/b1ab6106-77e6-4218-9eb4-e44bd56ca400">floated the US government taking equity stakes</a> in the big AI labs, with a “public wealth fund” that would pay ordinary citizens a share of the upside (an idea that others including OpenAI’s Sam Altman had <a href="https://www.cnbc.com/2026/06/05/trump-open-ai-altman-stake.html">reportedly been pitching since 2025</a>). From the other end of the political spectrum, Bernie Sanders went further, <a href="https://www.notus.org/technology/trump-ai-stake-openai">proposing the government take half of the big AI firms outright</a> and tax their stock into a sovereign fund. While they are on different sides of politics, both agree that the public should own part of the companies that are about to own the automated economy. While I agree with this approach if the AI capabilities are in your country, I want to show in this piece why it does almost nothing for everyone else, and what those countries can do instead.</p><p>Before we start, I want to introduce the idea of economic rent, which not every reader will know. In economics, rent is the part of a price that sits above what the thing would cost to provide in a competitive market: the premium a seller can charge purely because they own something others cannot easily copy, such as proprietary technology or a dominant market position. It is not payment for materials, for labour, or for a normal return on investment but the surplus that comes from scarcity and market power.</p><p>Nvidia chips are a concrete (simplified) example of this. An H100 costs Nvidia a few thousand dollars to make, including a return on the research cost, plus a margin that they make on the sale. The margin is the rent and is determined by the premium Nvidia can charge because no one else makes a comparable chip. When a UK or European company buys an H100, that premium leaves the UK as a payment to the US firm and the UK/EU collects little to no tax on it. The underlying issue is that UK corporation tax falls on the UK buyer’s profits, and the chip is a deductible cost that reduces those profits, while Nvidia books its own profit in the United States or a low-tax jurisdiction (multinationals reinforce this by selling to themselves across borders at inflated internal prices, which shifts where the profit is recorded, i.e. a way of avoiding tax on the rent, not the rent itself.) The result is that the value that is created by UK demand flows out of the country, largely untaxed.</p><p>The same logic applies (one layer up) to AI models. Concretely, when a UK user or firm uses a frontier AI model through an API (license), it pays more than it costs the model provider to answer that query (the compute and electricity). That gap is the rent on the model itself (essentially based on the training, data, and the capability no competitor can easily reproduce). As with the chip, the premium leaves the country as a payment to a US company, and the UK sees little tax on it. The main difference, and the reason the AI case is harder, is that a chip is a physical object that crosses a border, so a government can in principle put a tariff on it. In contrast, a model accessed over the internet crosses no border and is hence much harder to tax. The rent leaves as a digital service payment or a licence fee, which ordinary corporation tax does not reach and only a targeted instrument, such as a digital services tax or a withholding tax on the payment, can capture.</p><h3>Robot tax versus rent taxation</h3><p>Through 2025 and 2026 the question moved from think-tanks into mainstream politics, driven by the fear of displacement. <a href="https://www.axios.com/2025/05/28/ai-jobs-white-collar-unemployment-anthropic">Dario Amodei warned</a> that AI could wipe out a large share of entry-level white-collar work. Simulateously a run of layoffs announced next to record AI spending revived an old worries that fewer taxed workers means a thinner tax base and a stark wealth concentration.</p><p>In light of these event, peole started discussing possible instruments to counter such scenarios. <a href="https://finance.yahoo.com/economy/policy/articles/shark-tank-billionaire-mark-cuban-220000476.html">Mark Cuban</a> floated a charge on every “token” a model produces, while <a href="https://www.axios.com/2026/05/27/elizabeth-warren-tax-ai-companies-benefit-americans">Elizabeth Warren</a> argued for an excise tax on data-centre energy, with the proceeds earmarked for workers displaced by automation. The pushback was just as sharp arguing that a token is an internal accounting unit, not a unit of value, and providers can redefine it. Also per-token prices are falling so fast a fixed rate quickly taxes nothing and a tax your own firms pay simply hands the business to foreign providers you cannot tax at all.</p><p>The underlying question that everyone is trying to answer with different proposals is, how to design a tax base that actually reaches the money. This is a question that you can model with standard economic approaches and what I attempt in the below.</p><p>In the simulation I had to make several modelling assumptions that I want to lay out first: I differentiate between different types of automation, where robots are physical, sit where they operate (and cannot be spirited abroad) and earn an ordinary, competitive return. Once everyone has robots, owning one is not especially lucrative since no durable rent sits on the hardware even if a maker’s margin accrues abroad.</p><p>AI is different. The valuable part is not the rented compute, it is the trained model and the brand behind it, the intellectual property, and that earns a markup: a slice of profit above the cost of the compute and the labour. That markup is a rent in the economist’s sense, money you collect because you own something scarce, not because you did extra work this year. And it is paid, every period, as a licence fee from the local operation to the parent abroad. A British or European customer of a company like OpenAI contracts with its Irish arm or its US parent, so the value leaves the country in a way ordinary corporation tax cannot follow.</p><p>This is not a contrived set-up; it is how the providers already work, and it rhymes with something <a href="https://forgent.medium.com/defensibility-in-the-age-of-ai-2026-81dc59566dc8">I have argued before about defensibility</a>: the scarce, hard-to-reproduce asset is the model and the reputation, not the commodity compute underneath. One honest caveat belongs here. Today the labs lose money and do not expect to be reliably profitable until about 2029 or 2030. The rent in this model is the mature, profitable future the whole sector is investing toward, not a description of today’s accounts. Its existence and size are the model’s central assumption, not its conclusion.</p><h3>The difference to the digital-tax case of the past and why the US treatment is different to the EU and UK</h3><p>Europe has in the past argued at length about taxing the big technology firms. The digital-services taxes that France, the UK, Italy and others introduced were aimed at online advertising and marketplaces, a slice of digital commerce that the old rules could not reach. It was useful, contested, and in the end a fight over a few per cent of revenue.</p><p>AI is a different kind of thing, and that is why the stakes are higher. Advertising and cloud never thinned the workforce. AI, if it does what the sector is building it to do, takes on the work itself, which threatens two parts of the public ledger at once. First, the rent leaks abroad as before, and second, the wage and payroll taxes that fund most of the state shrink as the work is automated. The earlier digital taxes were about catching value that people were still creating somewhere but with AI it becomes about what happens when the value stops being created by people at all.</p><p>And the cost of getting it wrong is not symmetric. A rent left to compound into foreign-held capital for a decade is far harder to claw back than one caught as it forms. So even if you are unsure how fast automation will arrive, and reasonable people are (see the caveat at the end), the asymmetry argues for settling the design early rather than late. You can loosen a tax you set up and turned out not to need, but you cannot easily buy back an economy that was quietly sold while you waited.</p><p>Now turning to the difference between the US and the EU/UK. The difference between owning and importing is simple, and it is the hinge of the whole argument. If you own the labs, the money stays home, and the only question is how to share it, which is exactly what an equity stake or a public wealth fund does. If you import the AI, the money leaves first, as a licence fee, before anyone at home can share anything. A domestic equity stake or a residence-based wealth tax cannot reach money that has already crossed the border.</p><p>So the same proposal is obviously right in Washington but requires a lot more nuance in London. The United States, where the AI companies live, faces a distribution problem: a fortune concentrated at the very top, best addressed with stock-based redistribution. A country like the United Kingdom, or the European Union as a whole, faces a different problem entirely: base erosion, and a slow handover of capital ownership to foreigners. I use the UK as the main example for foreign AI ownership and treat the US as the contrast for domestic ownership.</p><h3>The assumptions and limitations of the simulation</h3><p>Think of it as a very disciplined board game of an economy, played over a long horizon and run hundreds of times to average out luck. It is an agent-based, stock-flow-consistent model, which means two things matter. First, it follows every pound from one pocket to another and tracks who owns what over time, household by household, rather than reporting an average. Second, the accounting has to balance every period: money that leaves one pocket arrives in another, and total wealth always equals the capital that actually exists. I use it only for the things it is reliable at: the direction of travel and the ranking of instruments. It is deliberately not a forecaster, so when a number like “seventy per cent” appears below, read it as an illustration of a direction, not a bet on an outcome. Where I go further and say what a country should do, that is judgment informed by the model, not produced by it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*exNnhqQCwnwYq0mqSaxDBw.png" /><figcaption>The set-up. Inside the dashed box is the home country (taxable at source): workers, the firm, the physical robots, and the home-located share of the AI servers. Outside sits the foreign owner, who owns the model and IP and collects the rent. The four levers each act at a different point, which is the whole reason they give different answers.</figcaption></figure><p>A word on what the simulation is for, because a sceptic could fairly say a lot of this is intuitive. It is. The direction of most of these results, and the central claim that a foreign rent leaks abroad, is something you can reason your way to at a whiteboard, and the existence of the rent is an assumption I put in rather than a discovery the model makes. So the model is not there to surprise you with the headline. It earns its place on four narrower things that argument alone does not give you: it puts magnitudes and a clear ranking on the instruments instead of vague directions; it adjudicates the cases where two plausible arguments point in opposite directions (whether automation lifts or crushes wages, for instance, turns on a single parameter, and in this calibration wages rise in level even as their share falls, though that leans on the remaining human work being scarce rather than something anyone can do); it tracks how small differences compound over a long horizon, where intuition is poor; and the strict accounting forces the whole story to add up, which quietly kills a lot of tempting free lunches.</p><p>One simplification I should flag. I treat AI value as a rent on the models, and the compute beneath them as a commodity. In reality the chip layer (NVIDIA above all) earns a large rent too, and because it is also mostly foreign-owned it reinforces the point here rather than undercutting it. A fuller model would separate the two rents and tax a hardware rent differently, which is for another post. If anything it strengthens the conclusions below, because it means even more of the value is accruing abroad.</p><h3>Results</h3><p>A few results held up across the robustness checks, and they are the reason the policy debate needs reframing.</p><p><strong>The rent is the prize, and most instruments miss it:</strong> In the mature economy the AI layer throws off a permanent rent of around an eighth of output that, untaxed, leaves the country every period. Corporation tax reaches almost none of it, because the rent is a deductible licence fee and the competitive capital profit is competed to near zero. A robot tax cleanly collects on the physical robots but cannot touch the rent. The instruments that actually reach it are the ones levied at source: a digital-services-style levy on the AI value earned locally, and a withholding on the rent as it crosses the border.</p><p><strong>Watch ownership, not just revenue: </strong>This is the result I did not expect, and the one that matters most. Left untaxed, the foreign owner recycles its rent into buying domestic equity and ends up holding the majority of the capital stock, in the illustrative run, around seventy per cent. Taxing the rent slows that buy-up toward roughly half, or lower. A policy can look acceptable on headline revenue while quietly losing the ownership of the productive economy. Revenue is the visible scoreboard, but ownership is the real game.</p><p><strong>Stock beats flow, and timing compounds:</strong> Taxes on the stock (wealth, or the equity itself) compress measured inequality far more than taxes on the flow (income), and intervening early matters disproportionately, because a rent left to compound is much harder to claw back later than to tap as it forms.</p><p><strong>Solvency is about how much you tax, not whether you own:</strong> Here is a case where the model overturns the intuitive answer. You might expect the importer to be the one heading for fiscal trouble. It is not, particularly. What decides whether the government’s debt path stays stable or spirals is whether it taxes the rent adequately, not whether the rent is owned at home or abroad. An owner that under-taxes can drift onto an unstable path where returns outrun growth; a well-taxed importer stays solvent. Ownership and solvency are separate questions, and the model lets you avoid conflating them.</p><p><strong>The cautious version still holds for robots:</strong> A capable robot is, increasingly, AI with hands: the scarce part is again the model, not the metal. When I give the robots their own foreign-held IP rent, it behaves exactly like the AI rent: a hardware robot tax misses it, the same border levy reaches it. So “tax the IP, not the machine” holds for robots too, and the flat, AI-only rent I use elsewhere is, if anything, on the conservative side.</p><p><strong>If the rent rises, every recommendation gets sharper:</strong> If pricing power grows as the sector concentrates, the rent climbs from an eighth of output toward a quarter or more, and the ownership drift accelerates toward the whole capital stock. The ranking of instruments does not change; the stakes do.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TSo1YgJD26Nxhn0oVuYwNg.png" /><figcaption>If the rent rises, the stakes rise with it. In the baseline the durable AI rent sits near an eighth of output (left); if pricing power rises, or AI captures more of the value, it climbs toward a quarter, and past half in the extreme. The right panel is the consequence: the larger the rent, the more of the domestic capital stock the foreign owner ends up holding, from around 70% in the baseline toward nearly all of it. The ranking of instruments does not change; only the stakes do. Illustrative magnitudes, not forecasts.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*xEKVsW5bAA9rN1oJYEWPEQ.png" /><figcaption>Watch ownership, not just revenue. Left untaxed, the foreign owner recycles its rent into domestic equity and ends up holding around 70% of the capital stock. A robot tax barely changes that. The instruments that actually reach the rent, an AI digital levy and a rebated cross-border withholding, bend the path down toward half, and into the low forties for the withholding. The shaded band is the period over which AI is assumed to roll out. Illustrative levels; the ordering is the point.</figcaption></figure><h3>What this means for the US, and how Europe could get there</h3><p>Run the model with the AI owned at home and the picture inverts, in a reassuring way: the mainstream proposals start to work. When the rent never leaves the country, ordinary corporation tax reaches the now-domestic profit, a progressive or wealth tax reaches the fortunes it piles up, and a government equity stake or a public wealth fund (the Trump and Sanders ideas) captures a share of the upside directly. The American problem is not leakage, it is concentration: a vast surplus pooling at the very top. Stock-based redistribution is the right tool for that, and on this point the simulation simply agrees with the people proposing it. The model is not reflexively against taxing AI; it says the popular instruments are well aimed, precisely where the labs live.</p><p>Which points to the deeper lesson, and it is the one Europe should sit up for. What makes those instruments work is domestic ownership, not geography. Any country that hosts a genuinely frontier-competitive model gets the same deal the United States gets: the rent accrues at home, and the standard toolkit reaches it. Europe sits on the importer side of the line today, which is why the harder playbook below applies to it. But that is a consequence of capability, not destiny. If a European champion, Mistral being the obvious candidate, became performance-competitive at the frontier, Europe would flip from importer to owner: the rent would land inside the bloc, and corporation tax, a wealth tax, even a public stake would suddenly reach it.</p><p>That reframes the industrial-policy argument. Funding a competitive domestic model is usually pitched as a matter of sovereignty or security. In this framing it is also tax policy, because it changes which instruments can reach the rent at all. The one caveat is that it has to be real frontier competitiveness, not a national flag on a model that trails badly: a domestic provider nobody chooses still leaves the economy importing the capable models and paying the rent abroad. Build a champion, by all means, but only a champion that actually competes changes the tax picture.</p><h3>What this means for the UK and Europe</h3><p>If you import the AI, a few things follow.</p><ul><li><strong>Tax the rent, not the machine:</strong> The robot tax and the token tax are politically intuitive and economically beside the point. The reachable, durable base is the licence fee. A digital-services-style levy on AI value earned locally, and a withholding on the cross-border royalty, are the instruments that bite. The catch is that today’s digital-services taxes mostly cover advertising and marketplaces and exclude cloud and AI, so this means widening an existing, contested levy to reach the rent, not flipping a switch you already own.</li><li><strong>Treat ownership as a first-class objective:</strong> A levy that funds a transition while the productive base is sold out from under you is half a policy. Measuring success only on revenue collected will flatter instruments that let ownership drift abroad.</li><li><strong>Move early and prefer stock to flow: </strong>The cost of waiting is not linear. A rent that has been compounding into foreign-held capital for a decade is a far harder thing to tax back than one caught as it forms.</li><li><strong>Consider helping the rent commoditise: </strong>There is one move available to an importer that an owner has no reason to make: actively supporting open-weight models, including capable ones from China, so that no single provider holds a durable monopoly. If frontier-level weights are freely available and can be run in-house, the markup that is the whole problem competes toward zero, and a country that cannot realistically win the frontier race neutralises its disadvantage by making sure there is no lasting rent for anyone to collect. The catch is that this only helps if you can actually run the weights yourself. Leaning on someone else’s hosted model, or on their continued willingness to release, just swaps one dependency for another.</li><li><strong>Sovereign compute helps, but it is not the lever:</strong> Onshoring the servers makes more of the physical AI income taxable and helps at the margin, but the IP rent stays mobile wherever the servers sit. Do it for resilience; do not mistake it for the instrument that reaches the rent.</li></ul><p>The binding constraint in practice is not domestic legislation, it is diplomacy: digital taxes and cross-border withholding collide with treaties and trade retaliation. That is the real fight, and it is worth being clear-eyed that this is where the difficulty lives. One way to partly sidestep it is to invest national capital directly into the foreign owners, through a citizens’ fund, as others have proposed. It helps, but with two limits worth being honest about. The return comes home only after the owner country has taxed the profit at source, so you are buying into a rent that someone else’s treasury has already taxed once. And a financial stake is not a strategic one: you own a slice of the upside, not the model, the supply, or the off-switch.</p><p>Taxing the rent (a DST or withholding) and buying into the foreign owners both help, but neither reaches the root. A source levy recaptures part of the rent for the public each year, and ownership brings home part of the return. Yet so long as the models and their IP are owned and domiciled abroad, the surplus is created there and taxed there first, so these tools recapture and redistribute without relocating. Changing where the rent is created, and gaining the strategic autonomy that comes with it, requires a frontier-competitive national capability. Even then, taxation and public ownership remain necessary, now to decide how the rent is shared at home rather than whether any of it stays.</p><p>With all the above said, I should also clearly state the caveats of this simulation and the claims. This is a model, so treat the precise figures as illustrative, not precise forecasts. The biggest assumption of all is that a large, lasting, foreign-owned AI rent exists in the first place. If AI compute and models commoditise, the rent shrinks and the urgency with it. The model has no housing or asset-price sector, no explicit welfare criterion, and it treats near-total automation as a long-run scenario rather than a documented trend. The strongest objections to taxing the rent were put to the model directly, and it survived them, gaining one rider (close the transfer-pricing channel) rather than being overturned. The full list of assumptions and where they could be wrong is in the <a href="https://github.com/lwossnig/post-automation-economy/blob/main/paper_latex/Simulating_a_Post-Automation_Economy__arXiv_full.pdf">paper’s appendix</a>.</p><h3>What should Europe and the UK do?</h3><p>Washington is debating how to share a fortune that stays home. That is a real problem, and equity stakes are a reasonable answer to it. But it is not the problem most of the world has. For a country that imports its AI, the fortune does not stay home to be shared; it leaves as a fee, and it buys the place on the way out.</p><p>The short version, if you remember one thing: a country that notices the rent in time can fund the transition from it and take a stake in it, instead of only paying it. But a share is all that buys. The rent itself is created where the models are built, and taxed there first, so keeping it, and not just a slice, means building the capability at home. The country that does neither will find, quietly and without a single dramatic moment, that the robots are working hard and the profits are leaving.</p><p><em>The full paper, with the model, the code, and an appendix of assumptions and limitations, is </em><a href="https://github.com/lwossnig/post-automation-economy/blob/main/paper_latex/Simulating_a_Post-Automation_Economy__arXiv_full.pdf"><em>available on GitHub</em></a><em>. This piece and the underlying paper were prepared with the assistance of large language models (Claude Opus 4.8 and GPT-5.5) for coding and writing support; I verified the results and take responsibility for the content.</em></p><p><em>A caveat on the premise. This all only bites if AI and robots really do capture a large and lasting share of economic value, which is not guaranteed. A 2024 MIT study (</em><a href="https://futuretech.mit.edu/news/beyond-ai-exposure-which-tasks-are-cost-effective-to-automate-with-computer-vision"><em>Beyond AI Exposure</em></a><em>) found that at today’s costs only about 23% of the wages paid for vision tasks are worth automating with computer vision, with humans cheaper for the rest, and that even with steep cost falls the shift would take decades. The same may well prove true for language models, with human labour staying the cheaper option for many tasks for a long time.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4726ed68c43c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AlphaFold3 — What’s next in computational drug discovery?]]></title>
            <link>https://medium.com/@leowossnig/alphafold3-whats-next-in-computational-drug-discovery-2da534c0845e?source=rss-d2e17e79f2e0------2</link>
            <guid isPermaLink="false">https://medium.com/p/2da534c0845e</guid>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[drug-discovery]]></category>
            <category><![CDATA[alphafold]]></category>
            <category><![CDATA[alphafold-3]]></category>
            <dc:creator><![CDATA[Leo Wossnig]]></dc:creator>
            <pubDate>Mon, 07 Oct 2024 14:45:36 GMT</pubDate>
            <atom:updated>2024-10-22T20:31:13.088Z</atom:updated>
            <content:encoded><![CDATA[<h3>AlphaFold3 — What’s next in computational drug discovery? (<em>Part 1 of 2)</em></h3><p>Co-authored with <a href="https://alexandermath.github.io/">Alexander Mathiasen</a>.</p><p>AlphaFold3 (AF3) was released earlier this year, and since then, everyone has been trying to understand what this means for drug discovery. While the news is filled with exciting results achieved by AF3, we take a more measured approach and use this blog to examine its impact on computational drug discovery by highlighting limitations and open problems. For anyone trying to understand AF3 we highly recommend this <a href="https://research.dimensioncap.com/p/an-opinionated-alphafold3-field-guide">blog post</a> by <a href="https://www.dimensioncap.com/research">Dimension Research</a> or this <a href="https://elanapearl.github.io/blog/2024/the-illustrated-alphafold/">illustration</a>.</p><p><strong>Specifically, we will cover the following areas:</strong></p><ol><li><strong>The docking baseline was not state-of-the-art</strong></li><li><strong>Train-test set leakage</strong></li><li><strong>Failure modes of protein structure prediction</strong></li><li><strong>Antibody problem</strong></li><li><strong>Lack of data — PDB is done</strong></li><li><strong>Structure isn’t function</strong></li></ol><p>While AF2 has mixed utility for virtual screening (see <a href="https://medium.com/@leowossnig/where-is-generative-design-in-drug-discovery-today-7234945177cf">our previous post </a>or <a href="https://www.science.org/content/blog-post/not-alphafold-s-fault">either</a> <a href="https://www.science.org/content/blog-post/screening-against-alphafold-models-maybe-better-it-looks">one</a> of Derek Lowe’s), AF3 pushed forward prediction accuracy for both protein:protein and protein:ligand complexes. The model is currently closed-source, so there are few benchmarks and thoroughly tested use cases outside the Google sphere.</p><p>That said, Google DeepMind demonstrated AF3 has uses beyond structure prediction, with their model <a href="https://storage.googleapis.com/deepmind-media/DeepMind.com/Blog/alphaproteo-generates-novel-proteins-for-biology-and-health-research/AlphaProteo2024.pdf">AlphaProteo</a> that can design protein binders given only a target sequence (figure 1). Relying on AF3’s improved accuracy and ability to generalise, this latest DeepMind release has shown a dramatic improvement for de novo design of antibody binders (a subset of protein:protein interactions). This means that complete <em>in silico</em> design (‘de novo design’) of binders might soon be feasible (the figure below from the paper shows the experimental success for the designed binders).</p><p>However, since the model relies heavily on AF3 as a filter (see figure 2), it will — like many other models — inherit the limitations of AF3. We’ll take this opportunity to discuss past (AF2) and current (AF3) limitations in this article and lay out where the field needs to go next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CzSSAKIzCQAsYIj7" /><figcaption>Figure 1: AlphaProteo performance across a range of different targets. Source: AlphaProteo paper</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kbs9Ipl4yhXWJyu-" /><figcaption>Figure 2: AlphaProteo likely relied on AlphaFold3.0 as a filter. Source: AlphaProteo paper</figcaption></figure><h3>1. The docking baseline was not state-of-the-art.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/0*A37uT5SP9J2nE6jO" /><figcaption>Figure 3: Inductive.bio benchmarks using a better baseline method.</figcaption></figure><p>AF3 was benchmarked on <a href="https://arxiv.org/abs/2308.05777">PoseBusters</a> (PB). <a href="https://www.inductive.bio/blog/strong-baseline-for-alphafold-3-docking">Inductive.bio</a> improved the <a href="https://arxiv.org/abs/2308.05777">PoseBusters</a> (PB) docking baseline from 59.7% to 79.5% (Figure 3). The PB baseline measures the percentage of ligands that were docked into their protein pocket within 2 angstrom Root Mean Square Distance (RMSD). AF3 maintains its lead with an accuracy of 93.2%. That said, when AF3’s ability to see the pocket is removed, and we only accept PB-valid poses, its accuracy drops to 73.1% (PB-valid means no atomic clashes and no stereochemical violations).</p><p>In other words, AF3 still wins when both methods see the protein pocket. However, in the AF3 paper it appears that pocket-blind AF3 beat the pocket-aware baseline, whereas this is not the case for the updated baseline (if we disallow atomic clashes and stereochemical violations). We also need to mention that Vina is typically not considered to be the state of the art for docking in the biotech industry. Ideally all these would be benchmarked to commercial grade software such as Schroedinger’s GLIDE.</p><h3>2. Train-test set leakage.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/597/0*5fQyaSbytM0pvFd-" /><figcaption>Figure 4: Inductive Bio’s performance assessment based on pocket blindness/awareness.</figcaption></figure><p>The PB benchmark computes performance across their entire test set. To further understand the performance of AF3, the authors split the PB test set into two:</p><ul><li>Common ligands which appear &gt; 100 times in the training data</li><li>Uncommon ligands which appear &lt; 100 times in the training data (<em>It should also be noted that the above definition of ‘uncommon ligand’ is relatively liberal. One might ask if we should really speak about generalisation if the exact same ligand was seen already more than 10 times, let alone 99 times.</em>)</li></ul><p>Intriguingly, AF3 beats the docking baseline on common ligands, and loses on the remaining uncommon ligands. <a href="https://www.inductive.bio/blog/strong-baseline-for-alphafold-3-docking">Inductive.bio</a> argues that performance on uncommon ligands is more valuable to drug discovery than common ligands.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*zg2ppB_9ZGaDH3F9" /><figcaption>Figure 5: Leakage types and frequency</figcaption></figure><p>Perhaps more alarmingly, the PB benchmark has the same ligand appear in both the training and test set (Figure 5). Such train/test set leakage may introduce a community-wide bias which favours models that memorise ligands from the train set over models that generalise to unseen ligands. To prevent such leakage, we need to ensure the same ligand doesn’t appear in the train/test, but we also need to ensure that similar ligands doesn’t. The same goes for proteins, pockets and protein-ligand interactions.</p><p>This is exactly what <a href="http://plinder.sh/">plinder.sh</a> fixes. First, they quantify the extent of the problem: DiffDocks performance increases from 20% to 60% when increasing train/test leakage from 0% to 80%. Concretely, they train DiffDock on six train/test splits, and evaluate the leakage of each train/test split in four different ways to capture different types of leakage (see Figure legend on <a href="https://www.biorxiv.org/content/10.1101/2024.07.17.603955v3">their paper</a> for details). It should be noted that PoseBusters data set was originally introduced for the purpose of showing pose validity. As such it was not designed to be an excellent data split with zero leakage.</p><p>Their most stringent train/test split achieves &lt; 38% leakage on all metrics, much lower than PDBBind which has &gt; 90% leakage on all metrics. For the <a href="https://www.biorxiv.org/content/10.1101/2024.07.17.603955v3">interested reader</a>, they cleverly framed the problem of splitting a train/test set as a graph problem. They construct a graph where each node is an example and each edge is a similarity, the problem of finding a good train/test split then amounts to finding a minimum cut in this graph.</p><p><em>A note on open-source: </em>Several teams are rushing to reproduce AF3, e.g., Chai Discovery (<a href="https://chaiassets.com/chai-1/paper/technical_report_v1.pdf">article</a> and <a href="https://github.com/chaidiscovery/chai-lab">code</a>) and the PaddleHelix team from Baidu (<a href="https://arxiv.org/abs/2408.16975">article</a> and <a href="https://github.com/PaddlePaddle/PaddleHelix/tree/dev">code</a>). While HelixFold approaches the performance of AF3 there is still a small gap. Chai-1 seems 1% better than AF3 on PoseBuster, but was trained on a timesplit with a larger training set. It’s also unclear if this improvement is statistically significant. However, the team should get credit for the excellent code implementation and easy set up. We recommend <a href="https://harrisbio.substack.com/p/0cf545a3-bdf1-4f5f-8480-632c59cd7314?postPreview=paid&amp;updated=2024-09-10T15%3A07%3A09.056Z&amp;audience=everyone&amp;free_preview=false&amp;freemail=true">Charlie’s blog</a> for a more detailed review on the race to reproduce AF3 (see also Figure 6).</p><p>A last comment here: AF and AF-like methods already do outperform some conventional approaches. Since virtual screening is becoming an essential part of any drug discovery project and cost is a factor here, it will be great to have methods that can perform cheap inference on billions of molecules. We believe that AF3 is still a step in the right direction here albeit the cost for antibody:antigen predictions need to be brought down (smaller number of seeds).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-1p_-i9ePyWksuDx" /><figcaption>Figure 6: Protein:Protein interaction performance (% of structures with DockQ score&gt;0.23) of different structure prediction methods.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*sekloP2U6LufzGEu" /><figcaption>Figure 7: Training curves for <a href="https://www.nature.com/articles/s41586-024-07487-w">AlphaFold3</a> with LDDT on the y-axis and training steps on the x-axis. LDDT is visualised for different molecular interactions (protein-ligand, protein-dna, etc.).</figcaption></figure><h3>3. Failure modes of structure prediction.</h3><p>Despite AF3 improving the state-of-the-art for structure prediction, it still makes mistakes, sometimes with high confidence. Looking at the performance in above Figure 7, one can see that the ranking depends on the specific modality. Below we share a few interesting qualitative failure modes / examples which seem to keep in line with the quantitative performance shown above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BX8cwjbyk-Qvp7v4" /><figcaption>Figure 8: AF3 predicted and actual structure for Cas12k-sgRNA-TniQ-S15 and its interaction with the target DNA.</figcaption></figure><p><strong>Example 1:</strong> <a href="https://www.linkedin.com/in/guillermo-montoya-9b57685/">Guillermo Montoya</a> used AF3 to predict the assembly of Cas12k-sgRNA-TniQ-S15 and its interaction with the target DNA (Figure 8). AF3 predicts RNA incorrectly, misplacing S15 and TniQ and an overall inaccurate assembly of nucleic acids. While the protein predictions were accurate, the nucleic acids and their assembly were found to be incorrect. Figure 7 indicates that protein predictions are more accurate than DNA and RNA. Furthermore, LDDT is a local distance measure and does not promise anything with regards to global performance.</p><p><strong>Example 2: </strong>Both AlphaFold2 (AF2) and AF3 contain confidence metrics — both for proteins as well as protein-ligand complexes (i.e. docking). Empirical evaluation on the test sets show that the confidence measure correlates with real performance. For example, see Figure 9 correlating pLDDT to LDDT. As indicated by the quantitative plot, we should expect AF3 to sometimes fail confidently. In particular, this correlation holds for a test set which may have leakage. We would therefore expect the confidence measures to work better on molecules that are similar to what is in the PDB and worse on the ones that are different to what is already in the PDB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/0*FvAwbhUeMvX-dA2k" /><figcaption>Figure 9: Correlation between pLDDT and LDDT scores.</figcaption></figure><p><strong>Example(s) 3:</strong> <a href="https://www.nature.com/articles/s41589-024-01638-w"><em>Agarwal and McShan</em></a> show several interesting failure modes for AlphaFold2 —<strong> it remains unclear today if (and by how much) AF3 improves upon AF2 for these</strong>. We will only know for sure after the community has had the chance to test it rigorously. This highlights the huge importance of open-source attempts to reproduce AF3 (e.g. <a href="https://openfold.io/">OpenFold consortium</a>). We list the failure modes of AF2 below, and invite any reader to explore similar failure modes of AF3. AF2 generally works well for rigid, well-folded globular proteins, but struggles with the below instances:</p><ol><li>Difficulty modelling membrane proteins, especially those with unique features or non-standard membrane thicknesses (Figure 10 f — Example: ChRmine membrane protein).</li><li>Struggles with intrinsically disordered proteins (IDPs) and regions (IDRs) (Figure 10 g — Example: AT2G23090 protein).</li><li>Cannot reliably predict effects of mutations on protein structure/stability (Figure 10 i — Example: L-PGDS K59A/C65A mutant).</li><li>Challenges in modelling novel protein folds not well-represented in existing databases (Figure 10 j — Example: NT-9 de novo designed protein).</li><li>Difficulty in predicting alternative conformations or conformational ensembles (Figure 10 h — Example: SecA open conformation).</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/0*1HAqcnATjQt9ZnRL" /><figcaption><em>Figure 10. Adapted from [</em><a href="https://www.nature.com/articles/s41589-024-01638-w"><em>Agarwal and McShan</em></a><em> — Figure 2]: Superposition of the AF2 model predictions and experimental structure are shown for several classes of peptides and proteins.</em></figcaption></figure><p>As a possible downside, the diffusion module in AF3 may cause hallucination, for example in intrinsically disordered proteins. On the bright side, AF3 already appears to be promising for predicting protein-protein complexes and mutation-induced binding free energy (BFE) changes [<a href="https://pubs.acs.org/doi/full/10.1021/acs.jcim.4c00976">Ref</a>] and there are many other promising results coming out every day (e.g. <a href="https://arxiv.org/abs/2408.12413#">this</a>).</p><p><strong>Example 4: </strong><a href="https://www.biorxiv.org/content/10.1101/2024.07.09.602655v1.full.pdf">Wojciechowska et al.</a> investigated the capabilities of AF3 to predict structures of amyloid proteins (fibrils), which are known for their polymorphism and low representation in protein structure databases. Amyloid proteins often aggregate to form larger structures called fibrils. These fibrils are composed of multiple copies of the same protein stacked or intertwined. To do this, it needs to know how many individual protein molecules (or copies) are involved in the final fibril structure. This number is specified as an input parameter to AF3. They found that the number of protein copies used in AF3 predictions significantly affects the quality and type of structures produced (Figure 11). While the AF3 predictions significantly improve upon AF2, for Aβ-42, models with fewer than five copies often resulted in helical structures rather than amyloid-like fibrils. While the model quality varied, with pLDDT scores indicating that no very high-quality models (pLDDT &gt; 90) were produced, most pLDDT scores were still above 70 (i.e. confident),which highlights challenges in prospective use. Interestingly, the authors also found that AF3’s structure sampling does not follow thermodynamic laws, as structures with the lowest energy were not the most frequently generated (evaluated using the FoldX software, which employs a physics-based force field to calculate the energy of a protein structure). While this is not surprising, it is evidence that more physics will need to be brought into these models in future generations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kQhC5hZS6fHfO3Pi" /><figcaption><em>Figure 11: Distribution of quality metrics for AF3 predictions: Mean and standard deviation of pLDDT and iPTM for different numbers of Aβ-42 protein copies. Notably the pLDDT and iPTM scores improve with increasing no. of protein copies but then decrease after 5 copies. [</em><a href="https://www.biorxiv.org/content/10.1101/2024.07.09.602655v1.full.pdf"><em>SOURCE</em></a><em>]</em></figcaption></figure><h3>4. Antibody problems.</h3><p>AF3 has been hailed as a big improvement for antibody-antigen complex prediction. Three papers lead us to believe further evaluation is needed to fully gauge AF3’s performance. However, due to the restricted access, evaluation will likely require more time:</p><p><strong>Paper 1: </strong><a href="https://doi.org/10.1101/2024.03.15.585121">McCoy et al</a> tested a range of models including AlphaFold-Multimer, RoseTTAFold, IB-ClusPro (ImmunoBuilder + ClusPro: rigid body docking followed clustering of these conformations to identify the most probable binding modes), AF-ClusPro (AlphaFold + ClusPro), SnugDock and AbAdapt. All models were evaluated using DOCKQ score and biassed Random Docking p-value (bRD p-value). They found that AlphaFold-Multimer performed best overall, with 19% of top predictions correct by DOCKQ score and 30% significant by bRD p-value (Figure 12).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*myc3G07DddHJXI5Z" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*56xcNVNVtstriWUN" /><figcaption><em>Figure 12: A Head-to-Head Comparison of Top-Ranked Models Shows AlphaFold-Multimer Performs Best at Antibody-Antigen Complex Prediction. (TOP) The top results of each method are compared side-by-side, showing the distributions of their DOCKQ scores and b RD p-values. (BOTTOM) a heatmap lining up each target model in the benchmark set with its score under each method. Grey indicates a model that was unable to be calculated, either because it could not or failed to produce models (in the case of AbAdapt) or the interaction region was trimmed away (ClusPro-based methods), see Materials &amp; Methods for details. [</em><a href="https://doi.org/10.1101/2024.03.15.585121"><em>Source</em></a><em>]</em></figcaption></figure><p>AlphaFold-Multimer, however, showed a strong bias towards more common structural motifs in its more accurate models, i.e., AlphaFold-Multimer has more accurate predictions / with higher DOCKQ scores when the antibody-antigen combination has interface motifs that are more commonly found in the protein structure database (Figure 13). This confirms (to an extent) the common view that AlphaFold-Multimer is learning and replicating interfacial motifs that are common in protein-protein interactions, rather than generating novel or rare interaction motifs, and therefore has limited ability to generalise.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*P7kv5dVO914_Di4C" /><figcaption><em>Figure 13: Better Quality AlphaFold-Multimer Models Have More PDB-Like Interaction </em>tertiary motifs (<em>TERMs). Interaction TERMs from AlphaFold-Multimer models are split depending on if their original model was incorrect, approximate (significant by bRD p-value, but &lt;0.23 DOCKQ score), acceptable by DOCKQ score, medium by DOCKQ score, or high by DOCKQ score. Then the number of matches below 0.7Å is plotted. The notches represent a 90% confidence interval. This is done for both the TERMs of the model (left) and the TERM of the true structure (right). The number of TERMs evaluated in each category is shown above each box plot in the corresponding color. *P &lt; 0.05, **P &lt; 0.01, ***P &lt; 0.001, ****P &lt; 0.0001; unpaired two-sided t test. [</em><a href="https://doi.org/10.1101/2024.03.15.585121"><em>Source</em></a><em>]</em></figcaption></figure><p><strong>Paper 2: </strong><a href="https://www.biorxiv.org/content/10.1101/2023.11.17.567543v1.full.pdf">Giulini et al.</a> also evaluated different antibody-antigen complex prediction models, in particular:</p><ol><li>Antibody structure prediction: ABodyBuilder2, ABlooper, AF2-Multimer, IgFold</li><li>Antigen structure prediction: AF2</li><li>Complex prediction: AF2-Multimer</li><li>Docking: HADDOCK3, ZDOCK</li></ol><p>They found that AF2-Multimer had limited accuracy in predicting antibody-antigen complexes (T1-acc of 0.17, T10-acc of 0.21). ABodyBuilder2 and AF2-Multimer on the other hand performed best among the ML models for antibody structure prediction (median CDR H3 RMSD of 2.34Å and 2.15Å respectively) and AF2 accurately modelled most antigen structures (median epitope RMSD of 0.78Å). However, HADDOCK3 outperformed ZDOCK in some scenarios and ML-generated antibody and antigen structures with HADDOCK3 docking produces more accurate antibody-antigen complex models compared to AF2-Multimer’s direct complex prediction (see Table 1). This was especially the case when an ensemble of structures were used for docking and accurate interface information was available. Notably, paratope and epitope model accuracy strongly predicted docking success and generally crystal-structure based docking still appears to perform best.</p><p>To be more precise, their results (Figure 8) illustrate the predictive performance of the CDR-EpiVague-AA method versus AF2-multimer for antibody-antigen interactions. For a fair comparison, both approaches utilize the antigen structure predicted by AlphaFold2 (AF2). In the CDR-EpiVague-AA method, they provide HADDOCK with limited interface information: all complementarity-determining region (CDR) residues on the antibody are considered potential binding sites. The antigen’s binding interface is approximated by including residues within 4.5 Å of the CDR residues, expanded by an additional 4 Å neighborhood buffer to create a sizeable interaction region — mimicking a realistic targeted discovery scenario. The results demonstrate that when using the experimental structure of the bound antigen, CDR-EpiVague-AA significantly outperforms AF2-multimer. However, when employing the AF2-predicted antigen and considering only the top-ranked pose, the performance gap narrows. Notably, evaluating the top 10 poses from CDR-EpiVague-AA reveals a marked improvement in prediction accuracy, underscoring the benefit of exploring multiple conformations in docking simulations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*P3HGsb_9lVw5iX_n" /><figcaption><em>Table 1: Docking protocols. The protocols denoted ABBE or ENS use an ensemble of antibody models as input, formed either by four ABodyBuilder2 generated structures (ABBE) or by one structure generated by each ABodyBuilder2, AlphaFold2, IgFold and ABlooper (ENS). [</em><a href="https://www.biorxiv.org/content/10.1101/2023.11.17.567543v1.full.pdf"><em>Source</em></a><em>]</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XVTm7WrmOtDyQ2sIAzItfw.png" /><figcaption><em>Figure 8: </em>Success rate in the top 1 and top 10 poses using CAPRI criteria for defining an acceptable model. Dark blue bars (BOUND T1) refer to the SR of the top one structure for the bound antigen, while dark green bars (AF2 T1) concern the AF2-modelled antigen. Light blue (resp. light green) refer to the success rate (SR) over the top 10 poses for the bound antigen (resp. AF2-modelled antigen) scenario. Top: success rates after the rigid-body docking phase. Bottom: success rates after flexible refinement and energy minimisation. Protocols are briefly described in Table 1. <em>[</em><a href="https://www.biorxiv.org/content/10.1101/2023.11.17.567543v1.full.pdf"><em>Source</em></a><em>]</em></figcaption></figure><p><strong>Combining these insights:</strong> Putting these two results together it becomes clear that while computationally a lot more expensive, running traditional docking approaches like HADDOCK3 on predicted structures performs better than any other previous approach. While, AF3 drastically improved upon AF2-Multimer, it remains to be seen how complex predictions of AF3 would compare to HADDOCK3.</p><p>Oftentimes we hear that AF3 would be a much better tool for virtual screening of a large number of candidates since the costs could be dramatically lower compared to docking. There are, however, two strong arguments against this:</p><ol><li>Combining a HADDOCK3-based (or similar docking method) approach with an active learning based strategy, similar to previous approaches for using FEP with active learning (e.g. [<a href="https://www.sciencedirect.com/science/article/pii/S2667318522000204">Ref</a>, <a href="https://pubs.acs.org/doi/10.1021/acs.jctc.2c00752">Ref</a>]), could still result in the best performance overall since it might generalise better and can be done cheaply via active learning.</li><li>A closer look at the AF3 paper also shows that there is actually an enormous computational cost reported (supplementary material section 5.10) to get the high-accuracy complex predictions. This is because they selected the highest confidence variants from 1000 repeat runs. They report that a single inference run on a complex with 2048 tokens is of the order of a minute on 16 A100 GPUs. A little back of the envelope computation would then put these 1000 runs per antibody complex at 20h (~$40h with current cloud pricing). This is by no means cheap or fast and in order to make this more practically relevant this will also either need to be integrated in an active learning cycle or the computational costs for the inference brought down.</li></ol><p><strong>Paper 3</strong>: <a href="https://www.biorxiv.org/content/10.1101/2024.09.21.614257v1">Another recent paper</a> investigated AF3 specifically for docking on a benchmark dataset of antibody and nanobody structures from SabDab. Docking accuracy was evaluated using DockQ scores, which were categorised into incorrect, acceptable, medium, and high-accuracy predictions based on CAPRI classifications. They showed that AF3 achieves an 8.9% and 13.4% high-accuracy docking success rate for antibodies and nanobodies, respectively. AF3 also improves unbound CDR H3 loop prediction accuracy vs previous methods (different AlphaFold-Multimer versions and AFsample), achieving a median RMSD of 2.04 Å for antibodies and 1.14 Å for nanobodies. The accuracy of CDR H3 loop prediction strongly correlates with overall docking accuracy, especially for high-accuracy predictions. Antigen context further improves CDR H3 loop prediction accuracy, particularly for loops longer than 15 residues. While these improvements are really great, AF3 also has a 60% failure rate for antibody and nanobody docking when using a single seed compared to the higher success rates they present in their paper using 1000 seeds.The utility as a screening tool will ultimately depend on the accuracy, cost, and speed and this implies there is significant room for improvement on the latter two. While the docking accuracy versus previous methods was higher, the point above still remains and it will be interesting to see how AF3 compares to other docking approaches.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ofx6qYMo9aOiQP5Q" /></figure><p>The nice thing, however, about machine learning is that perhaps we will have large enough data sets from docking and other simulations to then again integrate these in a multimodal fashion in future AlphaFold versions or new model.</p><h3>5. But how do we go ahead without more data? PDB is done.</h3><p>Another big question is how much can models like AF3 continue to be improved? A lot of AF3’s limitations are based on the limitations in the PDB (e.g. lack of protein-ligand diversity). We are progressively exhausting existing data and generating more at the same scale will be expensive (recall that the cost to reproduce the <a href="https://cdn.rcsb.org/rcsb-pdb/general_information/about_pdb/Economic%20Impacts%20of%20the%20PDB.pdf">PDB is estimated to be $12B</a>). And nevertheless AF3 has drastically improved the performance — in part by using a huge data set of generated structures (“cross distillation” set from AF2 multimer to avoid hallucination of disordered regions). In fact these structures were considered essential for the training and ensuring that the diffusion process is working appropriately (avoiding model collapse / hallucination). It will be interesting to see where the limits are. For large language models we know that training on generated data can result in catastrophic behaviour and model collapse [<a href="https://www.nature.com/articles/s41586-024-07566-y">Ref</a>] (although this is less studied for models trained on data generated by other models). These problems in fact appear to increase<a href="https://www.nature.com/articles/d41586-024-02355-z"> as the internet gets populated with AI generated content</a>. Perhaps this is where novel data generated from physics-based approaches will also provide a solution in the future. They might also be important to fundamentally enable the prediction of dynamics. Dynamics will be key to enable the prediction of function of proteins, and hence also the effect of drugs on the function.</p><h3>6. Structure isn’t function.</h3><p>Something that has been pointed out by many voices before, but to echo it here as well: Structure is not the same as function. If you want to understand a machine with rotation parts, a picture won’t give you the full picture. You need a video.</p><p>To get a step closer to understanding function and the true behaviour of proteins we still need to better understand how to get the dynamics of the system. <a href="https://www.sciencedirect.com/science/article/pii/S0969212624003708">AF2 ensembles generally do not fully reproduce the free energy surface seen in MD simulations</a> and AF3 is unlikely to improve this significantly. This, however, will be crucial for small molecule drug discovery, since small changes (e.g. a 1Å move of a p-loop in a kinase pocket) might diminish binding and selectivity. Dynamical behaviour of proteins is also essential to uncover or rationalise novel mechanisms to drug a target, as has been demonstrated by Relay<a href="https://aacrjournals.org/cancerdiscovery/article/14/2/240/734065"> who used molecular dynamics simulations</a> and cryo-electron microscopy to identify an allosteric network that provides an explanation for how mutations favour PI3Kα activation. So much of the functional universe of proteins is dependent on dynamics and with the static pictures we get today (even if we use ensemble of structure predictions) we are just scraping the surface. While there were early indications that AlphaFold could predict dynamics to an extent (e.g.<a href="https://www.biorxiv.org/content/biorxiv/early/2022/10/17/2022.10.17.512570.full.pdf"> conformational changes in dynamic proteins</a>) these hypotheses<a href="https://onlinelibrary.wiley.com/doi/full/10.1002/pro.4353"> did not hold up in broader studies</a>. The AlphaFold team noted this already in their ‘limitations’ section. The underlying challenge here is that AF and AF-like methods are trained on static crystal structures which do not sufficiently capture the dynamics. This can and will though likely be rectified in the future.</p><p>Of course, the community is already on it, and there are several emerging approaches that already attempt to better capture the dynamic nature of proteins as for example a preprint of a de novo<em> </em>design approach for <a href="https://www.biorxiv.org/content/10.1101/2024.07.17.603962v1.full.pdf">dynamic changes between intra-domain geometries of proteins</a> or diffusion models and force fields for coarse-grained molecular dynamics that relies on simulated data and can <a href="https://pubs.acs.org/doi/10.1021/acs.jctc.3c00702">reproduce the coarse-grained equilibrium distribution while preserving the dynamics of all-atom simulations such as protein folding events</a> (and <a href="https://www.biorxiv.org/content/10.1101/2024.09.25.614868v1">many</a> <a href="https://www.biorxiv.org/content/10.1101/2024.05.20.594916v2">other</a> <a href="https://arxiv.org/abs/2409.17852">papers</a> come out every day). We are certain there is a lot more to come soon and we will cover more of this in part 2 of the blog post!</p><p>Another recent approach to predicting dynamic protein structures using a 4D diffusion model was introduced in the paper titled “<a href="https://arxiv.org/abs/2408.12419">4D Diffusion for Dynamic Protein Structure Prediction with Reference Guided Motion Alignment</a>”. This model leverages molecular dynamics simulation data to capture the dynamic nature of protein structures using a diffusion model that relies on atomic grouping and side-chain dihedral angle predictions to simulate protein motion efficiently. They integrate a reference network to enhance structural consistency by embedding initial 3D protein structures into the model and a motion alignment module that improves temporal coherence across multiple time steps to aggregate kinetic information, ensuring smooth transitions in protein dynamics. While the accuracy of such models will still require improvement there is clearly progress.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MexuAuAHQLFl8yMh" /></figure><p><em>Figure 9: 4D dynamic protein prediction by </em><a href="https://arxiv.org/pdf/2408.12419"><em>Cheng et al.</em></a><em> — Figure 1: Given an initial 3D structure for reference, their model predicts dynamic proteins at the following 32 time steps simultaneously. They present the predicted 3D protein structures at the intermediate time steps for illustration.</em></p><p>It should be noted that these findings <strong>don’t diminish the improvements of AF3</strong>, especially in its capacity to function with significantly less input data (i.e. predict from protein sequence and smiles string) compared to conventional docking methods reliant on experimental receptor structures. At least for the time being, models such as AF3 do serve as a supplementary tool in our computational arsenal. They may not yet replace existing docking approaches entirely but AF 3 remains a breakthrough for blind docking, in particular for small molecules.</p><h3><strong>So, what’s next?</strong></h3><p>In part 2 of this blog post we will explore how we can deliver systems that are able to reliably predict dynamics using a combination of machine learning and data generation. We will then go on to also highlight something that works already very well today: design, build, test, learn cycles and iterative approaches to combine computational methods with experimental data. We believe that the combination of strong computational models (foundational models for biology?) with fast iterative experiments will enable rapid functional optimization. Furthermore, iterative approaches such as active learning are already available today and can have a massive impact on drug discovery projects.</p><h3><strong>Acknowledgement:</strong></h3><p>We thank Santiago Villalba, Charlie Harris, and Nazim Bouatta for their helpful input and comments on an early version of the post.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2da534c0845e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Importance of Open Innovation in Drug Discovery: Lessons from the CACHE Competition and a call…]]></title>
            <link>https://medium.com/@leowossnig/the-importance-of-open-innovation-in-drug-discovery-lessons-from-the-cache-competition-and-a-call-98d4d5fed839?source=rss-d2e17e79f2e0------2</link>
            <guid isPermaLink="false">https://medium.com/p/98d4d5fed839</guid>
            <category><![CDATA[drug-discovery]]></category>
            <category><![CDATA[generative]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Leo Wossnig]]></dc:creator>
            <pubDate>Mon, 26 Feb 2024 14:14:05 GMT</pubDate>
            <atom:updated>2024-02-26T14:14:05.183Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Importance of Open Innovation in Drug Discovery: Lessons from the CACHE Competition and a call for an antibody CACHE</h3><p>The <a href="https://cache-challenge.org/results-cache-challenge-1">first results</a> of the Critical Assessment of Computational Hit-finding Experiments (<a href="https://www.nature.com/articles/s41570-022-00363-z">CACHE</a>) competition, as detailed in <a href="https://www.nature.com/articles/d41573-024-00036-0">this</a> recent Nature Reviews Drug Discovery article, provides a good glimpse into the current state of artificial intelligence (AI) and computational techniques in drug discovery — specifically for the identification of small molecules capable of binding to difficult targets.</p><p>The competition was targeting the challenging WD40 repeat (WDR) domain of LRRK2 — a multifunctional enzyme associated with Parkinson’s disease. Out of more than 2000 compounds predicted by 23 teams to bind to the LRRK2’s WDR domain, fewer than a dozen displayed any binding affinity in laboratory tests, translating to a hit rate of less than 1%. The teams used a wide range of approaches, from conventional approaches such as ultra-large docking (using e.g. <a href="https://www.nature.com/articles/s41586-020-2117-z">VirtualFlow</a> [see also <a href="https://www.biorxiv.org/content/10.1101/2023.04.25.537981v1">Ref</a>] on a library of 1.4 billion compounds) and pharmacophore-based screening models, to modern machine learning methods such as <a href="https://jcheminf.biomedcentral.com/articles/10.1186/s13321-024-00812-5">REINVENT</a> and other deep learning-based methods (see summary below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LZH9GYfrAZMV9_Zb" /><figcaption><strong>Figure 1</strong>: Overview of the best performing computational approaches used for hit finding in the first CACHE competition [<a href="https://cache-challenge.org/sites/default/files/downloadable/forms/methods_summary_0.pdf">Source</a>].</figcaption></figure><p>Looking at the results, it is clear that AI (or any deep learning based approach) is performing en par with conventional methods but we are far from a distinct ‘AI advantage’. More broadly, the outcome illuminates the gap between computational predictions and experimental results in drug discovery as a &lt;1% hit rate (with mainly 20–70 micromolar binders) is ultimately not that ‘<em>earth-shattering’</em> (to borrow from Pat Walter’s) — although this is of course context dependent as these hits could be great places to start a drug discovery campaign.</p><p>Despite the modest success, the initiative is commendable for its focus on transparency and collaboration in a field with increasing secrecy and competitive reserve. Specifically, an increasing number of teams (e.g. DeepMind’s <a href="https://deepmind.google/discover/blog/a-glimpse-of-the-next-generation-of-alphafold/">latest results</a> for AlphaFold) are claiming high performance without sharing details about the validation process or model architecture.</p><p>The result of this is a lack of transparency and understanding about which models actually work in practice and can ultimately help drug developers find better molecules. The results, again, underscore the critical need for open sharing of methodologies, successes, and failures to foster innovation and improvement in the drug discovery process.</p><p>This necessity for openness and collaboration, however, extends beyond the realm of small molecule drug discovery to areas such as <em>de novo</em> antibody design. The field of computational antibody engineering is similarly experiencing a huge surge of interest, accompanied by large amounts of venture capital, driven by methods such as <a href="https://www.nature.com/articles/s41586-023-06415-8">RFDiffusion</a> (Baker Lab) or <a href="https://www.nature.com/articles/s41586-023-06728-8">Chroma</a> (Generate Biomedicines). However, while some methods are open sourced, many papers still lack proper (independent) experimental validation. In fact, even when code is publicly available, it can be hard to assess the impact of these new methods from a practical point of view.</p><p>These methods hold a lot of promise to unlock the targeted design antibody binders against a specific epitope or against historically hard-to-drug targets. For example, membrane-associated proteins (G protein–coupled receptors and ion channels) often have poor solubility and production of recombinant antigens can be particularly challenging. Understanding the impact and suitability of these generative methods (or structure prediction based ones) for such problems is essential for the appropriate use and to overcome limitations of the current generation of methods. In particular, if we find that these methods ultimately do not perform significantly better than the existing repertoire of tools (as early papers indicate).</p><p>For this reason, I want to advocate for an antibody CACHE competition. I am calling on the academic and industry community to come together and design a public competition that can enable everyone to better understand the performance of existing and novel methods such as <em>de novo</em> protein design in practical drug discovery settings.</p><p>The results of the CACHE competition serve as a powerful argument for the necessity of open innovation and collaboration in computational drug discovery. This lesson is acutely relevant for the field of de novo antibody design, where the complexity of the task and the critical importance of the outcomes demand a collective effort to share, validate, and build upon each other’s work. As we move forward, embracing the spirit of open innovation could significantly accelerate our ability to design novel antibodies and small molecules, ultimately leading to breakthroughs in the treatment of complex diseases.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=98d4d5fed839" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI in DD — where are the success stories?]]></title>
            <link>https://medium.com/@leowossnig/ai-in-dd-where-are-the-success-stories-1a668a145711?source=rss-d2e17e79f2e0------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a668a145711</guid>
            <category><![CDATA[ai-in-drug-discovery]]></category>
            <category><![CDATA[drug-discovery]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Leo Wossnig]]></dc:creator>
            <pubDate>Mon, 23 Oct 2023 16:44:16 GMT</pubDate>
            <atom:updated>2023-10-24T01:30:34.971Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*K3Vd3EoXUA3oNEyh" /></figure><h3>AI in DD — where are the success stories?</h3><p>There are a number of recent comments (<a href="https://endpts.com/first-ai-designed-drugs-fall-short-in-the-clinic-following-years-of-hype/">here</a>,<a href="https://www.nature.com/articles/d41586-023-03172-6"> here</a>, and<a href="https://sifted.eu/articles/ai-drug-discovery-startups"> here</a>) about the failures of AI in drug discovery. While the underlying facts are true, and ‘AI-generated’ drugs have so far not lived up to their hype and expectations in the clinic, the conclusions are not quite right. The articles show that there is a fundamental misunderstanding of how AI (I will call it ‘machine learning’ going forward) is used in the pipeline today and why certain claims on both sides (AI drug discovery companies as well as critics) won’t ever be testable or realised — and hence can’t stand closer scrutiny.</p><p>Machine learning is not a panacea for drug discovery. And will likely never be (at least not in the next 50–100 years). The reason is that the drug discovery process is way too complicated and human biology is way too complex and diverse. This means that we will unlikely be able to collect sufficient amounts of data to computationally design and predict the outcomes of clinical trials, ignoring the fact that the technology of today might not even be able to capture all relevant biological parameters. This is in stark contrast to some claims that have been made over the past few years. It is fair to say that a number of these claims, albeit unfounded, have led to unrealistic expectations. But does that mean that machine learning does not have any impact?</p><p>Currently machine learning is mainly used to bring new drug candidates to the preclinical stage. This means that machine learning helps either with the identification of novel targets, finds better hits, or optimises the hits to turn them into early leads, then late leads and possibly even into a development candidate. This means that it helps us find molecules that satisfy a molecular target profile or perhaps a complex target product profile (if we get lucky and the preclinical tests are all positive, for example through PK/PD or early toxicity predictions). If we optimise for molecular product profiles or target product profiles not fundamentally different to historic ones, then the resulting molecules will unlikely be fundamentally different to historic ones. The question then is why would compounds be more successful?</p><p>Knowing <em>what</em> to optimise for is most often the biggest question. Sometimes we know what we’d like but don’t know how to optimise for it, or (more often) we simply do not yet know <a href="https://substack.com/redirect/72f8959a-1517-4e16-9230-afb6b7d1cd0f?j=eyJ1IjoiNDRyd2YifQ.4URp8ppyGiJsdx-dwK5PAdBGul4R7d48eg9wDpdMTEY">what to optimise for</a> since we don’t have a good understanding of disease biology!</p><p>Some AI-enabled drug discovery companies go after new targets — possibly without the proper (clinical) validation. If targets are not new and have enough evidence to be clinically relevant, then AI can’t help us with that. Even the best compound might not have the desired effect in the clinic.</p><p>In most instances, however, the targets are not new. Looking at Exscientia’s pipeline, specifically their<a href="https://en.wikipedia.org/wiki/5-HT1A_receptor"> 5-HT1a agonist</a>, it’s nothing new at all as<a href="https://www.science.org/content/blog-post/another-ai-generated-drug"> already commented on by Derek Lowe</a>. In fact, a quick search on GlobalData reveals 516 molecules at various stages and a<a href="https://www.cas.org/resources/cas-insights/drug-discovery/ai-designed-drug-candidates"> structural analysis</a> also revealed that the compound shares its shape with haloperidol, a frequently used first-generation antipsychotic agent, which the FDA approved in 1967. This makes sense from a machine learning perspective, since there is a lot of training data available (see the number of<a href="https://en.wikipedia.org/wiki/5-HT1A_receptor#Ligands"> ligands here</a>) and machine learning can hence more likely be used.</p><p>However, while this data can help training machine learning models to quickly find hits (molecules that stick to the target) and potentially some optimised leads, it won’t necessarily increase the overall probability of success in clinical trials — which means there was still a 90–95% chance that the drug would fail (and it did).</p><p>In order to increase the chances of success, we need to optimise the compounds to satisfy requirements that are driven by a better understanding of the disease or due to novel insights that emerged from previous failures. Without that, we cannot expect any new drug to be superior to previous ones other than by chance. To repeat<a href="https://www.science.org/content/blog-post/another-ai-generated-drug"> Derek’s opinion</a>, there is simply not enough reliable information to feed into machine learning models to allow us to predict what will happen against conditions like OCD, depression, anxiety, and many others. And that is the problem. Drugs fail at every stage for some expected or unexpected reasons. Some are due to biology, some due to technical issues, some are due to regulatory changes, and some might be epidemiological. Sometimes we are not picking the right target, because our biochemical understanding of the disease state is wrong and/or incomplete. But they can equally fail due to unexpected toxicity, and the situation with tox is the same as for CNS efficacy: no matter how good the machine learning models we have — we just don’t know enough to use it to tell us whether we are going to run into such problems.</p><p><strong><em>But does this mean that machine learning is useless?</em></strong></p><p>Of course not. While choosing the wrong target is often the problem, resulting in a<a href="https://www.nature.com/articles/nrd.2016.184"> lack of efficacy in clinical trials</a>, there are many reasons for failure. Toxicities, for example, are one of the other major reasons — as was the case in many T-cell engagers. Here on-target off-tumour effects have commonly led to <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7829968/">dose-limiting toxicities</a>.</p><p>By designing molecules that take these learnings into account we can overcome past challenges and design novel, more efficacious, and safer therapeutics. Once we have such novel insights (<em>e.g. </em>a better understanding of the disease biology or nature of toxicity), and we have good (translational) models to early test for this, then we can use machine learning to help us with the generation and optimisation of these new therapeutics. In fact, we can screen much larger (chemical/biological) spaces using machine learning and this might help us find candidates that satisfy the more complex requirements of today’s therapeutics. And perhaps we wouldn’t have been able to identify these candidates only 10, 20, or 30 years ago. It should also be noted that a good fraction of best-in-class drugs <a href="https://www.nature.com/articles/d41573-023-00048-2">are not first-to-launch</a> but take into account learnings from previous generations. The flip side here is, of course, that new modalities, formats, etc., or more broadly speaking any new biology or chemistry might come with new risks — the unknown unknowns. Here, again, machine learning won’t help.</p><p><strong><em>Now, turning to the question of ‘how can we prove that AI works?’.</em></strong></p><p>As Patrick Schwab already<a href="https://www.linkedin.com/feed/update/urn:li:activity:7120081516080025600/?updateEntityUrn=urn%3Ali%3Afs_feedUpdate%3A%28V2%2Curn%3Ali%3Aactivity%3A7120081516080025600%29"> said before</a> — we can only assess the benefits of using machine learning at each stage. The reason for this is that we can establish clear benchmarks that allow for comparison. Requiring a full head-to-head comparison is an impossible task. For example, if we would want to compare clinical trial success rates of AI-enabled programs versus non-AI-enabled ones, it would require at least 150 clinical trials to observe a statistically significant difference, assuming that we have a 5% chance of success for the conventional ones and a 10% chance for the AI-enabled ones. This is under the assumption that these trials are i.i.d., and can be derived through a power analysis with 5% significance and power of 50%; If we increase the power to 80%, which is more common, then this increases to 341 samples and without the i.i.d. assumption this number would be even larger. Long story short, the number of clinical trials we’d need is enormous. This is <a href="https://www.linkedin.com/feed/update/urn:li:activity:7117852211988103169/">not a fundamentally new</a> insight, but it just reinforces that this will take a lot more data than we have seen so far. Further to this, things do not become easier. The low hanging fruits have been plugged, so comparing new trials against historical ones is not necessarily a good metric as well.</p><p>Beyond this, it is even hard to formulate what is AI-enabled and what is not. I would argue that pretty much every drug discovery company today is using some form of machine learning. And the reason for this is that it works for many areas well. Whether we predict binding, ADME properties, T-cell activation, or developability characteristics — both in biologics and small molecules, machine learning (or more broadly speaking statistical modelling) is now commonly used (see Figure 1). And companies who don’t use any form of it are certainly falling behind.</p><p>Now, I agree that finding better candidates is key. However, saving a few months off the process across a large portfolio of say 20–30 programs can still result in massive cumulative savings and benefits. Assuming, for example, that we would be able to reduce the time of going from target to an optimised lead by 50%. From a cost and time perspective, this might enable us to run twice as many programs and then ultimately select the best performing ones to progress into the clinic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zIhK4itta0tNaQ1C" /><figcaption><strong>Figure 1:</strong> Cowen and Company Proprietary AI in Drug Development Survey October 2022 (N=21) on the use of AI in the pipeline by stage. <em>Source: Cowen and Company</em></figcaption></figure><p><strong><em>So where does this leave us with AI in drug discovery today?</em></strong></p><p>Saying a drug needs to be fully AI designed, i.e. all steps need to be done fully automated and autonomously is the same as requiring driving to be fully autonomously. While we still don’t have fully autonomous driving cars today, I would strongly argue that the automation of driving has not been a failure. Most people nowadays use lane assist, cruise control, or parking sensors very happily, and the overall driving experience has become a lot better and safer. The same could be said about machine learning and the different steps in drug discovery. Most companies are using machine learning very productively across all stages of the drug discovery process, and those not using AI (machine learning) are falling behind.</p><p>The vision is that machine learning will enable individual modules within the drug development process to be better or more efficient. These machine learning modules will need to work in unison with traditional approaches — both experimental and computational — in order to deliver programs that are most likely to succeed. And combining this with brilliant scientists and novel insights, will then lead to higher overall probabilities of success in clinical trials.</p><p>Integrating machine learning within the pipeline and existing workflows is key to reducing costs and timelines and being able to find better therapeutics in the future.</p><p>There is no simple panacea like a generative model that spits out a guaranteed drug. But perhaps it’s our expectations that need to be reshaped, because although machine learning cannot guarantee clinical success, it can certainly reduce some of the risks associated with drug failure.</p><p>In the end this shows one thing — which doesn’t come as a surprise: Drug discovery is hard. And it requires strong teams working closely together to enable high quality science. Equipping the best scientists with the best tools is the only way to deliver more and better therapeutics to patients who are waiting for these.</p><h4><strong>Acknowledgements:</strong></h4><p>I’d like to thank Graham Mills, Patrick Schwab, and John Cassidy for feedback on this post and Tonya Frolov for her feedback and proof-reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a668a145711" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Right Data for Good Results: Introducing the 5 ‘V’s of Drug Discovery Data]]></title>
            <link>https://medium.com/@leowossnig/the-right-data-for-good-results-introducing-the-5-vs-of-drug-discovery-data-331e29c683c5?source=rss-d2e17e79f2e0------2</link>
            <guid isPermaLink="false">https://medium.com/p/331e29c683c5</guid>
            <dc:creator><![CDATA[Leo Wossnig]]></dc:creator>
            <pubDate>Thu, 27 Jul 2023 17:33:23 GMT</pubDate>
            <atom:updated>2023-08-01T08:14:55.130Z</atom:updated>
            <content:encoded><![CDATA[<p>Co-authored with Winston Haynes.</p><p>In <a href="https://medium.com/@leowossnig/where-is-generative-design-in-drug-discovery-today-7234945177cf">part 1</a> of this series we outlined the huge variety of machine learning methods that are applied to drug discovery and that there is no single approach that can solve all challenges. Different drug discovery stages and different tasks require different tools — there is no ‘one size fits all’.</p><p>While machine learning holds immense potential in drug discovery and development, the current focus needs to shift from creating more models to acquiring ‘good’ high-quality data that will optimise the functionality of these models. This is an opinion shared by many in the field (<em>e.g. </em><a href="http://practicalcheminformatics.blogspot.com/2023/06/getting-real-with-molecular-property.html">here</a> by Pat Walters, or <a href="https://www.digitalisventures.com/blog/engineering-biology-how-to-build-data-centric-biotech">here</a> by Jacob Oppenheim, or <a href="https://www.benchling.com/blog/data-and-culture-for-AI">here</a> by Benchling). Note that this is fundamentally different to large language models, where we hope that the average information will get it right, and so we rely on large sets of data available on the internet. We have the belief that specialised systems will for a long time prevail in drug discovery, since the amount of data that can be produced relative to the space of candidates is miniscule. General models will unlikely suffice for later stage programs (lead optimisation onwards), and even for early stage programs for entirely new candidates and targets.</p><p>In this post we will first define what good data looks like based on what we call the 5 ‘V’s of drug discovery data. We will then discuss what a tech stack looks like that can generate such data, and the layers that affect the 5 ‘V’s. Since tech is only as good as the teams that leverage it, we go beyond the data and the technology to discuss what mindsets, teams, and culture organisations need to create to enable the real impact of these technologies.</p><h3>What does ‘good’ data look like?</h3><p>‘Good’ data has two key requirements: The relevance of the data, or how well it translates into clinical readouts, and the quality of the data. The latter is a broad term but in reality the devil is in the detail — in the less desirable work of data collection, storage, protocols, reproducibility, and standardisation. This is often the work that <a href="https://www.nature.com/articles/s42256-023-00673-x">no-one really wants to do</a> and, due to a lack of incentive, few people write papers about this. But if this work is not done well, at best the machine learning models won’t work particularly well, and at worst there can be a complete loss of data integrity which can lead to fatal outcomes [<a href="https://www.statnews.com/2023/06/12/laronde-data-integrity-issues-biotech/#comments">ref</a>].</p><p>Exploratory data analysis to build a basic understanding of the data and rigorous model validation are crucial in machine learning, biostatistics, and bioinformatics modelling. These steps strongly influence the predictive accuracy of any model and can counteract data inconsistency or relevance issues, making proper statistical analysis and model validation a discussion worthy of a separate blog post.</p><p>While improper statistical analysis also impacts the accuracy or reliability of<em> in silico </em>approaches, inaccurate chemical and biological data remain the central issue in data-driven methods for drug discovery (<em>e.g.</em> <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7382642/#bib0160">here</a> or <a href="https://pubs.rsc.org/en/content/articlelanding/2020/cs/d0cs00098a">here</a>). There is a long list of articles that discuss how noise can limit the performance of predictive models (see <a href="https://www.sciencedirect.com/science/article/pii/S1359644609000403">here</a>, <a href="https://jcheminf.biomedcentral.com/articles/10.1186/s13321-021-00571-7">here</a> and <a href="https://pubmed.ncbi.nlm.nih.gov/34262264/">here</a>), or more broadly how data quality impacts predictive models (<em>e.g.</em> <a href="https://read.oecd-ilibrary.org/environment/guidance-document-on-the-validation-of-quantitative-structure-activity-relationship-q-sar-models_9789264085442-en#page1">here</a>, <a href="https://onlinelibrary.wiley.com/doi/10.1002/qsar.200810084">here</a>, <a href="https://onlinelibrary.wiley.com/doi/abs/10.1002/minf.201000061">here</a> in context of QSAR models). Reducing noise and improving consistency in the data generation process is hence a key aspect of generating high quality data.</p><p>A decade ago, Fourches <em>et al</em>. outlined important steps for data curation for QSAR modelling (<a href="https://pubs.acs.org/doi/10.1021/ci100176x">part 1</a> and <a href="https://pubs.acs.org/doi/10.1021/acs.jcim.6b00129">part 2</a>). However, with the growth of chemical, genomic, and protein datasets, some steps from the original study, like manual data curation, have become impractical. Further challenges arise because many essential aspects of data preparation and modelling aren’t available in a single standalone program nor are they standardised across the industry. Underlying processes and workflows are standardised to a lesser degree, imposing more fundamental limits. Building data-processing pipelines that allow for the consistent curation and standardisation of collected data is another essential part of generating good quality data.</p><p>In order to achieve consistency and minimise noise in the data and processing, we need to understand the entire stack that is required for machine learning-based predictions. We will come back to this later on, but first we will clearly define the right properties of high quality data. We are next developing a framework to assess the quality of data itself and introducing 5 key properties to evaluate.</p><h3>Introducing the 5 ‘V’s of machine learning-grade data applied to the drug discovery tech stack</h3><p><strong>In this article we repurpose the </strong><a href="https://www.sciencedirect.com/science/article/pii/S1877050915006973"><strong>5 ‘V’s of data science</strong></a><strong> and derive 5 ‘V’s of machine learning-grade data in drug discovery.</strong> These are properties to consider when dealing with data used to train machine learning models in drug discovery. They can be applied in most drug discovery settings and, if satisfied, will result in much better predictive models and meaningful outcomes. Any part of the above tech stack should be continually evaluated based on how it impacts the below principles.</p><p>We define the<strong> 5 ‘V’s of drug discovery </strong>data as follows:</p><ul><li>The right data <strong>veracity</strong>: The quality, integrity, accuracy, and consistency of the data that is generated or available</li><li>The right data <strong>variety</strong>: How balanced is the data? What dynamic range does it cover? What different types of data do we have available? For proteins this could, for example, mean high sequence or structural diversity in the training data</li><li>The right data <strong>volume</strong>: The amount of data that’s available to train the models</li><li>The right data <strong>velocity</strong>: How fast and cheap can data be generated, accumulated, and curated for analysis purposes? This impacts how easily we can validate or retrain our models</li><li>The right data <strong>value</strong>: How translatable is the data to clinical outcome, which is typically related to the biological complexity?</li></ul><p>While this list could be extended to additional properties, in this article we stick to the main 5. In general, data<strong> veracity</strong>, <strong>variety</strong>, and <strong>volume</strong> are qualities of the data itself, while <strong>velocity </strong>helps assess the ease at which we can acquire new data and hence what processes and methods we want to use. The latter is particularly important if there is little data in the public domain, or if we want to use an active learning-based approach. The <strong>value</strong> of the data is based on the biological complexity and its relevance to clinical outcome, which impacts the predictive validity of the data readout. There is also the opportunity to provide the model with more context and introduce conditionality by adding metadata.</p><p>Let’s look more closely at each of the 5 ‘V’s:</p><h4><strong>1. Data veracity</strong></h4><p>Consistent, accurate, good quality, labelled data, generated using biologically relevant functional assays is rare in many areas of drug discovery, or it doesn’t exist at all. Large corporate datasets typically aren’t useful as they aren’t collected in a consistent manner or digitised. Similarly, data from CROs might not include (ML) relevant controls and metadata needed for machine learning. And variability arises between sites and providers as a result of varying protocols, equipment, etc. See Figure 1 for an example of the impact of normalisation via controls. If public data already exists, it usually contains a lot of noise, duplicates and errors, which requires careful analysis, filtering, normalisation, and other preprocessing steps to use it (more examples in the appendix).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l_eFpMUhKTGai_luypRzeQ.png" /><figcaption><strong><em>Figure 1</em></strong><em>: Data from repeated measurements of two different control molecules evaluated in cell-based assays from different campaign pre (left) and post (right) normalisation. The normalisation with respect to a control can dramatically change the picture. Here we clearly see two clusters (control-1 and control-2) emerging (Source: LabGenius internal data).</em></figcaption></figure><p>As we move towards more complex biology, we increasingly have to deal with more noise, more variability, and more confounding variables for each readout. For example, cell-based assays are highly sensitive in comparison to simpler binding assays, and have much higher plate-to-plate and campaign-to-campaign variations. From a translational perspective, more complex assays or systems are preferred because they are more indicative of molecules’ translational value. Increasing the <a href="https://www.nature.com/articles/s41573-022-00552-x">predictive validity</a> by utilising more complex, biologically relevant data is usually better than increasing the quantity of less relevant data. However, increased complexity brings greater variability, amplifying the significance and difficulty of the normalisation and integration process. Similarly, more specific data (<em>e.g.</em> a specific disease-relevant cell line vs a range of different ones) is usually harder to standardise but has higher predictive validity. This is one of the main reasons why standardisation across different programs is also much harder than across a single one.</p><p>Normalisation (<em>e.g.</em> via controls) hasn’t typically been done for historical data because it wasn’t created with the intention of being used for machine learning. This means that a lot of pre-existing data is not suitable for modern machine learning systems. The issue extends beyond a single type of measurement in drug discovery — variability and lack of reproducibility is common across many different areas, arguably anything that touches biology. This ranges from <a href="https://pubs.acs.org/doi/abs/10.1021/acs.jcim.0c00701">solubility</a> to protein kinase assays (<em>e.g.</em><a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2721227/">here</a> or <a href="https://pubmed.ncbi.nlm.nih.gov/23333421/">here</a>) to <a href="https://pubmed.ncbi.nlm.nih.gov/34233495/">toxicology</a> (see also the appendix).</p><p>Even today, most organisations do not pay enough attention to the quality and consistency of the data they produce. This is why machine learning doesn’t have the impact that it could. Perhaps a reason for this is that the collection and curation of data is much less exciting than developing or using a new machine learning model. Experimental and computational scientists, industry, and investors alike need a big shift in mindset here. We’ll come back to this later in the post.</p><p>It is also important to understand that machine learning-grade data requirements are different, and more stringent, compared to conventional drug discovery data. To create machine learning-grade data, we need the right number of controls, repeats, a certain dynamic range, balance in the labels/categories, and <a href="https://www.biorxiv.org/content/10.1101/2023.01.30.526201v1">metadata</a> (context). This is necessary because biological data can often contain huge variances, and even slight changes in protocols or setup dramatically impact the readout. Without the context of these changes, it is unlikely that an algorithm will be able to make sense of the data. Beyond the algorithm, the data scientist or person performing the analysis the data without the larger context might face the challenge of trying to identify an invisible cause.<br>Creating the right data (beyond just <a href="https://www.go-fair.org/fair-principles/">FAIR data principles</a>) is key to enabling the development of models that can truly impact the drug discovery pipeline, move assets into the clinic and ultimately treat patients.</p><p><a href="https://www.sciencedirect.com/science/article/pii/S1359644620305274?via%3Dihub">This article</a> by Andreas Bender and Isidro Cortés-Ciriano has highlighted many of these points. In the context of this discussion, Figure 2 (see below) is especially relevant and highlights a few challenges with biological data, namely <a href="https://www.nature.com/articles/s41586-018-0409-3">cell line drift </a>and <a href="https://www.nature.com/articles/s41588-020-00726-6">response heterogeneity</a>. When we work with biology we suddenly face a whole new set of data quality challenges, and therefore machine learning problems, that are very different to other fields.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c-pCvGdRitslxCHohqK4Hw.png" /><figcaption><strong><em>Figure 2</em></strong><em>: Challenges with biological data in contrast to chemical data. Source: </em><a href="https://www.sciencedirect.com/science/article/pii/S1359644620305274"><em>Artificial intelligence in drug discovery: what is realistic, what are illusions? Part 1: Ways to make an impact, and why we are not there yet; A. Bender and I. Cortés-Ciriano, Table 1</em></a></figcaption></figure><p>When talking about data quality, it’s also important to touch on the lack of reproducibility between studies, assays, and labs. When using such data it is important to understand its limitations. Readouts from different labs have high variability even when measuring the same compound. This is again because experiments generate very different results when performed under different conditions. If this is the case for the data used to train a machine learning model, it is unreasonable to expect the model to provide accurate predictions. This can be significant, as shown in the toxicity measurements in Figure 3. In general, while it’s not always as bad as shown here, data from mixed sources can introduce additional challenges and noise. While this data can still be helpful, for example for pre-training, transfer learning, or multi-task learning, one needs to be aware of the limitations and risks.</p><p>Beyond the variability, public data also contains plenty of errors. A fundamental assumption in most machine learning or chem/bioinformatics papers (and the proposed models) is the correctness of input data. However, error rates in databases can be significant. For example, <a href="https://onlinelibrary.wiley.com/doi/abs/10.1002/3527603743.ch9"><em>Olah et al.</em></a> showed that on average there are two errors per each medicinal chemistry publication, resulting in an overall error rate as high as 8% in some databases as is the case for WOMBAT (also <a href="https://onlinelibrary.wiley.com/doi/10.1002/9783527619375.ch13b">here</a>). Similar results have been found in other databases (<a href="https://pubmed.ncbi.nlm.nih.gov/24160896/">here</a>, <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4607714/">here</a>, or <a href="https://onlinelibrary.wiley.com/doi/10.1002/qsar.200810084">here</a>), and have resulted in efforts to produce <a href="https://jcheminf.biomedcentral.com/articles/10.1186/s13321-020-00456-1">better data pipelines</a> and refined and curated existing databases. As we’ve discussed before, errors within datasets can significantly reduce the predictive ability of any model and require particular care when being used for model building.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V2O7tw-M7kosv3RIrDqDGA.png" /><figcaption><strong><em>Figure 3</em></strong>:<em> This </em><a href="https://pubmed.ncbi.nlm.nih.gov/26541361/"><em>paper</em></a><em> by Cortes-Ciriano and Bender studied the comparability of independent cytotoxicity measurements on a large-scale, in this case in the ChEMBL database. In overlapping compound–cell line systems measured in independent laboratories, one can observe a poor correlation. This is in part due to annotation errors, pointing out the importance of data curation when extracting or accumulating public data.</em></figcaption></figure><p>To make sure we get the best possible data set for training machine learning models we must make sure we have the correct controls and data processing steps to minimise variability and maximise reproducibility. We need to ensure that we have good processes in place to minimise errors and obtain data with high consistency. This requires assessing the entire process, including instruments, cell lines, assay and curve fitting protocols, and other confounding variables. Constant data monitoring can ensure that inconsistencies are immediately detected. To ensure that this is the case, a lot of effort and resources should be spent on setting up the lab processes and data processing pipelines accordingly, which will in turn enable the creation of veracious data.</p><p>For more on this topic, see the publications by Christian Kramer and colleagues (<em>e.g.</em> <a href="https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0061007">here</a>) or this recent <a href="https://greglandrum.github.io/rdkit-blog/posts/2023-06-12-overlapping-ic50-assays1.html">RDKit blog post</a>.</p><h4><strong>2. Data variety</strong></h4><p>Large amounts of balanced data that have a uniform distribution, including negative examples (which <a href="https://pubs.acs.org/doi/10.1021/acs.orglett.3c01282">often don’t exist</a>) with a wide dynamic range (<em>e.g.</em> data with pIC50 across several log orders), are rarely available in a program. For selectivity, this is usually even more challenging, as highly selective compounds are very rare (in particular for small molecule kinase inhibitors). In general, this topic refers to the availability of positive and negative data (if we do binary classification), and to how well the data spans the entire domain that we want to make predictions on (when we perform a regression, <em>i.e.</em> prediction on a continuous variable). A common term for large and diverse sets that is used in the literature is ‘representativeness’. This means that for example all target labels (classes) should be sufficiently present, or that all target values (across the entire range of possible values) should be present and approximately evenly distributed. Beyond this, if the data covers only a smaller dynamic range, we typically achieve not only lower correlation coefficients using predictive models but a lower mean absolute error. Understanding the impact of such data set qualities is hugely important for modelling purposes.</p><p>‘Representative’ data often doesn’t exist since historically (and still today in many instances) scientists have chosen not to take all compounds forward for experimental testing or discarded the data for all failed attempts. Whether prioritised through simulation, machine learning, or human assessment, it is still common practice only to progress the molecules with the best predicted performance, which means that data might not be selected in a fashion that is optimal for learning. Beyond this, even with the best selection approach, we will likely find that most compounds fall within a narrow dynamic range (<em>e.g.</em> within 2–3 logs).</p><p>All of these limitations can drastically limit the model’s ability to learn. A lack of negative examples will limit the ability to predict poorly performing compounds, and a lack of dynamic range will make it hard to extrapolate to regions of interest and limit the model’s applicability domain. Due to lack of variety, a predictive model might miss crucial information and never learn to identify the top performing compounds, for example if they are in an unexplored region of the chemical or biological (sequence) space. The same is true for both antibodies and small molecules, which is why active learning approaches can only truly succeed if there is no final ‘human filter’ in the loop.</p><p>Data variety might also result in higher experimental costs. For example, it is usually easier to synthesise molecules around a specific scaffold, rather than explore a wide range of random structures. But diversity from a sequence/structure, physicochemical and functional perspective can all drive the model’s ability to make better overall predictions. So there is a clear trade-off between cost and speed vs better learning. Being aware of the constraints that the training data might impose on the model is crucial for its successful application. And testing the examples that the model proposes, even if we disagree with the choice, <em>might result in a less successful campaign but in a more successful program.</em></p><p>Beyond variety in the context of one metric, data ‘variety’ also matters across assays. For instance, measurement of activation, selectivity, thermostability, aggregation, and yield for the same protein can be crucial to drive a program forward. For meaningful impact in drug discovery we need to understand the multi-dimensional optimisation space and have a sufficiently high variety in the assays we perform and data we capture.</p><h4><strong>3. Data volume</strong></h4><p>Quantity is of course still important. If it’s of the right quality, the more data the better. But for many problems the data just doesn’t exist yet, <em>e.g.</em> for particular antibody therapeutics such as VHHs or T-cell engagers, or <a href="https://www.nature.com/articles/d41586-023-01612-x?utm_source=substack&amp;utm_medium=email">in chemistry</a>. Even just a quick search for single domain antibody (‘VHH’) structures in the PDB shows that there is very little data available. And here we’re not even looking for functional data — of which there is even less (for example T-cell activation of multispecific or multivalent molecules — please share any data sources if you have them!). The volume of data required to train a machine learning model can vary hugely. Depending on the scope of the project, the model being used, and the endpoint that needs to be predicted, they can vary from 10s to several 100,000s. For instance, <a href="https://www.nature.com/articles/s41586-021-03819-2">AlphaFold</a> was successful, in part, because it was trained on around 100,000 protein structures that had been collected in PDB since 1971 and the data clearly covered all folds (see image below from the PDB). It is less surprising that machine learning can solve this problem since the problem space is comprehensively covered.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iEwVhIgzd3LebZhcQ6RTng.png" /><figcaption><strong><em>Figure 4</em></strong>:<em> Number of unique folds added to the PDB, 1 August 2018, calculated using CATH. The total number of unique folds in each year (red), and the yearly addition of new folds (blue) are shown. Source: </em><a href="https://web.archive.org/web/20170711133210/https:/www.rcsb.org/pdb/statistics/contentGrowthChart.do?content=fold-cath"><em>PDB</em></a></figcaption></figure><p>In contrast, predicting structures for the smaller universe of antibody sequences has been much more <a href="https://www.tandfonline.com/doi/full/10.1080/19420862.2023.2175319">challenging</a>. For small datasets it is harder to cover the problem space, and conventional machine learning models or specialised models that incorporate a lot of prior information are usually better. For larger datasets, neural network / deep learning based approaches are increasingly better to use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*FNkM5xmrDrf1hUEho2p4gg.png" /><figcaption><strong>Figure 5:</strong> Number of crystal structures in the PDB for all proteins (blue), antibodies (red), and single domain antibodies (‘VHHs’) in yellow. Source: <a href="https://www.rcsb.org/">PDB</a>.</figcaption></figure><h4><strong>4. Data</strong> <strong>velocity</strong></h4><p>The speed and cost at which we can acquire high quality data is essential to generate large datasets, use active learning, and fine-tune our models. Simpler assays with less complex biology typically come with a higher data velocity but at the cost of reduced data value (<em>i.e. </em>translatability). Finding the right trade-off between these two is usually determined by which methods a biotech company can use and how effectively they can move its programs forward. Innovation of higher velocity methods with a high relevance to the clinical endpoint can give fundamental advantages in the age of AI and machine learning. Active learning, for example, requires fast cycle times with sufficiently high throughput to complete the design — build — test — learn cycles and to rapidly iterate on the compounds.</p><h4><strong>5. Data value</strong></h4><p>While we can control veracity, volume, and variety of the data (all a matter of resources), we are usually limited by the relevance of the assay to the <em>in vivo</em> outcome (for example in mice for <em>in vivo</em> PoC or ideally in humans). We’ve discussed the challenges of increasing the biological complexity above, but we also need to consider the increases in cost as the complexity of the system increases. But doing so could ultimately save us millions by avoiding later stage failures. For this reason, it is essential that we continue developing methodologies that enable the interrogation of biological systems at a higher resolution and complexity.</p><p>It is worth highlighting again that quality becomes a particular issue when we explore more complex biology. In the past there has been a tendency to scale up systems (high throughput) with low biological complexity — for example, defaulting to large-scale screening for binding affinities. The big challenge with this is that this data rarely translates directly to the desired functional response, which means the candidates fail to deliver the desired response in a more relevant system. This is why we need complex biology data with a higher <a href="https://www.nature.com/articles/s41573-022-00552-x">predictive validity</a>, even if it is challenging to generate from a data quality and quantity perspective.</p><p>For example, in past programs we have repeatedly observed compounds that bind strongly in a biochemical assay but fail to inhibit the same target in a cell-based assay; or high affinity antibodies that don’t result in any functional response in a cellular environment. The reasons for the lack of translatability are varied but always come down to the inadequacies of proxy systems that have little correlation with the desired endpoint. For example, for T-cell engagers we are highly interested in measuring T-cell activation. These multispecific antibodies use different anti-CD3 binders. We observed that the molecules that include a weaker anti-CD3 binder have a consistently higher activation than the ones that include a <strong>30x <em>higher</em> binding affinity</strong> binder (NB: they compete in a binding assay).</p><p>The use of irrelevant proxy assays is a common issue, and while this can often be fixed early in the pipeline, the later you are in the drug development process, the harder, more time consuming and expensive it is to go back and start again. Consequently, the pursuit of superior data encompasses not only more and better quality data, but also novel information and data that has intrinsically higher predictivity for <em>in vivo</em> behaviour. As was recently highlighted in <a href="https://innovationendeavors.substack.com/p/super-evolution-x-biology-2023-edition?utm_source=substack&amp;utm_medium=email">this</a> blog by Nick, Carrie, and Galen at <em>Innovation Endeavours </em>and previously in many other places (<em>e.g.</em><a href="https://www.sciencedirect.com/science/article/pii/S1359644620305274?via%3Dihub">here</a> or <a href="https://www.sciencedirect.com/science/article/pii/S1359644621000428?via%3Dihub">here</a>), emphasis should be placed on approaches that acknowledge complexity and demonstrate the following 3 subcategories:</p><ul><li><strong>Contextualised</strong>: Approaches that facilitate measurements in the most native environments possible, or, more plausibly, an environment more akin to the <em>in vivo</em> context. Native environments are perpetually influenced by various dynamic contexts, for example, the conditions in the tumour microenvironment (TME) are hard to reproduce in 2D or even 3D cell cultures (see <em>e.g.</em> <a href="https://www.cell.com/cell/fulltext/S0092-8674(21)01332-5">here</a> in context of organoid models). Note that the conditions can be so different that one approach for building tumour-selective antibodies is reliant on this premise — specifically mask-cleavage of enzymes that are present at a higher concentration in the TME. This also holds true even earlier in the pipeline and on a smaller scale: proteins are mostly treated as static while we all know of course that protein dynamics play a huge role. Recent efforts try to incorporate more of the dynamics, for example for flexible docking (<a href="https://arxiv.org/abs/2210.01776">DiffDock</a> is indirectly attempting to do this).</li><li><strong>Functional</strong>: Approaches that directly assess activity instead of relying on a proxy. For example, antibody campaigns often pursue high-affinity binders, yet it is evident from the example above, and many others in the literature, that lower affinity is frequently sufficient or even essential for desired function such as activation or selectivity (see also Figure 6). Sometimes, function does not even correlate with binding. So, in order to find molecules that have the potential to overcome current limitations of solid tumour-targeting TCEs in the clinic (<em>e.g.</em> dose-limiting tox — see <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7829968/">here</a>) such approaches will be essential. In these instances, it is more important to find compounds that are not just strong binders but selective, or have other therapeutically valuable properties. Avidity-driven selectivity tries to accomplish this (see <a href="https://pubmed.ncbi.nlm.nih.gov/30333240/">here</a> in the context of HER2).</li><li><strong>Multi-scale:</strong> Approaches that integrate diverse data modalities to deduce causality, stemming from minute biochemical alterations that propagate to functionality at the system level. Large scale initiatives like <a href="https://www.cancer.gov/ccg/research/genome-sequencing/tcga/using-tcga-data/types">TCGA</a> provide great examples of diverse data collection, spanning clinical, genomic, expression, and imaging modalities. Multi-modal clinical-genomic datasets, for example, facilitate the examination of the intricate biology underlying therapeutic efficacy in real-world scenarios.</li><li><strong>Translational:</strong> Approaches that are cognizant of their inherent limitations and strive for translational relevance as a design objective, be it<em> in vivo</em> drug activity or a scaled-up industrial process. Complex<em> in vitro</em> models are progressively demonstrating predictive capabilities (<em>e.g.</em> <a href="https://www.nature.com/articles/s41598-022-23474-5">here</a> or <a href="https://www.nature.com/articles/s43856-022-00209-1">here</a>). That said, there is still a long way to go and many open questions remain (<em>e.g.</em> <a href="https://www.nature.com/articles/s41375-020-0785-1">here</a> and <a href="https://www.frontiersin.org/articles/10.3389/fmolb.2020.00033/full">here</a>). Doing more is not always better, and finding the right proxies with the highest translational probability is key to success. <a href="https://www.nature.com/articles/s41573-022-00552-x">This</a> recent <em>Nature Review drug discovery</em> article makes this point really well.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-PEdug5WPHX2jwtjCa4Itw.png" /><figcaption><strong><em>Figure 6: </em></strong><em>Example of a functional readout that is hard to estimate with simpler assays. Here we see T-cell-mediated killing of a cancer cell, which is a complex mechanism depending on cross-linking of the T-cell and the target receptor on the cancer cell. Stronger binding usually results in lower selectivity which can be detrimental for the safety profile of the compound. (Source: LabGenius)</em></figcaption></figure><h4>Summary of the 5 ‘V’s</h4><p>We have seen that data resources often require extensive curation and preprocessing by experts to extract relevant and informative data and to eliminate noise, errors, and variability that could compromise the utility of implemented machine learning methods.</p><p>The other aspect to consider is the most important part: generating data that is meaningful, <em>i.e.</em> more relevant to our target outcome.</p><p>Being aware of the 5 ‘V’s means that one can actively strike a balance between cost and speed of generation/data availability, quality of the data, and translatability. We can also leverage this framework to assess where our data generation pipeline can be improved to maximise the quality and utility of our data.</p><p>Satisfying the 5 ‘V’s as much as possible is crucial for the successful application of machine learning in drug discovery. Improving any of these will likely translate into improved outcomes. This is key to move medicines from the bench to the patient.</p><p>This thinking is supported by a lot of evidence (<em>e.g.</em> <a href="https://www.sciencedirect.com/science/article/pii/S1359644620305274#bib0155">here</a>, <a href="https://www.sciencedirect.com/science/article/pii/S1359644620305274#bib0155">here</a> and <a href="http://pubmed/10.1371/journal.pone.0147215">here</a>), and yet many companies still get this wrong. For example, generating a million data points from a simplistic binding assay doesn’t necessarily mean you’ll have more success identifying compounds that do something meaningful in a (3D) cell culture, a PDX model, or the complex tumour microenvironment in a human. The good thing is that these insights are slowly arriving and companies are starting to adapt.</p><h3>What do we need to get good data and predictions?</h3><p>Since we now have a good understanding of the properties of the data that we need, we are now turning towards the ‘how’.</p><p>There are <strong>two main ingredients</strong>:</p><ol><li>A technology stack that maximises consistency and reproducibility, and</li><li>A company culture and organisation that allows for the effective use and application of the technology stack.</li></ol><h4>1. The complete tech stack for machine learning-driven drug discovery</h4><p>In general, we need to distinguish between data that is used to train <strong>intra-program machine learning models</strong> (<em>i.e.</em> models that are only used for a specific program), and data that is used to train<strong> inter-program</strong> <strong>machine learning models </strong>(<em>i.e.</em> models used across multiple programs).</p><ul><li>For inter-program models, the conditionality of the data often results in further complexities because differences between the programs and processes make the standardisation of data harder. Examples include different indication- or disease-specific cell lines, or different assay conditions. All factors can influence the data and introduce noise that limits the ability of machine learning systems to learn.</li><li>The steps and requirements for data standardisation in intra-program models are simplified significantly but still require a fair amount of work in practice. This is particularly true for more complex biology.</li></ul><p>The below pyramid captures the full stack that constitutes a machine learning process in drug discovery. Each layer is needed and errors or noise in any will deteriorate the final performance of any data analysis of machine learning. The stack here applies for both cases, but requires more work for cross-program learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5mJXiZ4K5shYD16alAi5PQ.png" /><figcaption><strong><em>Figure 7:</em></strong><em> The entire data stack. The lower layers typically have the biggest impact on the actual outcome of a drug discovery program. Without good foundations (i.e., predictive assays, data generation, data capturing, and data pre-processing steps) the best analysis can only achieve so much. Data analysis &amp; machine learning could be further broken down into data representation and machine learning models.</em></figcaption></figure><p>When designing a data analysis or machine learning pipeline, it is important to understand each layer and carefully assess sources of noise, errors, and inconsistencies. When assessing the data that is generated by a tech stack (example shown in Figure 7), it is important to acknowledge that any changes made at each layer will require a close collaboration between the tech and the science teams. Identifying which layer(s) have the biggest impact on data quality is crucial to maintaining the integrity of the stack, and should therefore be assessed on a continual basis.</p><p>The essential layers in a technology stack are laid out below. We have omitted perhaps the most fundamental layer, which is the valid therapeutic hypothesis, which of course is the most important requirement for any program to succeed. Below we start from the bottom and work our way to the top of the pyramid:</p><ol><li><strong>Data Context And Relevance</strong>: Available models and context of the data that is generated is important for the translatability of the generated results into meaningful (clinical) outcomes. We can optimise the models and capture additional data and metadata to try to obtain more context.</li><li><strong>Data Generation</strong>: Consistent data generation is possible through the use of standardised processes, automation, and standardisation (equipment, etc.). Business rules, standard operating procedures, and automation are key.</li><li><strong>Data Capture And Storage</strong>: Raw data should be captured automatically, with the relevant metadata and stored in a consistent, secure manner according to FAIR data standards. This would ideally happen in a way such that it is immediately accessible to anyone in the company. Version control and provenance for both data and models is helpful to assure the right models are trained on the right set of data and increases resilience to errors and changes.</li><li><strong>Data Processing</strong>: Processing of raw data needs to be standardised across the company and normalised where appropriate (<em>e.g.</em> EC50 values from fitted curves). Automation and business rules can reduce variability at this stage.</li><li><strong>Data Analysis &amp; Machine learning</strong>: Only once all other layers are in place should data analysis and machine learning be performed. This requires careful data curation and model validation. In particular feature choice, data splits, and choice of relevant performance metrics play an important role here.</li></ol><p>Transitioning to a data-driven biotech company requires strategic changes across people, processes, and systems. Here are some potential actions to drive this transformation for the technology stack.</p><p><strong>Creating a good drug discovery tech stack: an action checklist</strong></p><ol><li><strong>Process standardisation:</strong> Establish consistent processes, captured in SOPs and business rules, to be used throughout the organisation. Minimise manual steps through the use of automation and scheduling.</li><li><strong>Automated data capture:</strong> Develop pipelines and scripts to capture, track, and version data and metadata automatically and reliably. It is also important to choose instruments that can be integrated with the existing system which typically requires the availability of suitable drivers.</li><li><strong>Cloud and warehouse adoption:</strong> Leverage services like Google Cloud, AWS, Azure, and data warehouses such as Snowflake and BigQuery for secure data storage of both the raw and processed data. This enables immediate, global access to the data.</li><li><strong>Implement model and data tracking:</strong> Use systems like MLFlow, DVC, or Google Vertex AI pipelines to track models, model versions, and the associated data and data version.</li><li><strong>Track and manage biological variability: </strong>Establish processes for quality control such as automatic tracking of key assay metrics and outlier detection. Establish processes to minimise variability across a program by performing data normalisation (<em>e.g.</em> through various controls).</li><li><strong>Lab automation:</strong> Use automated / robotic labs to maximise process consistency and perform user lab/user acceptance tests when updating a system.</li><li><strong>User-friendly systems:</strong> Ensure systems are easy to navigate for experimental teams and capable of supporting modern data analysis methods. This may require a combination of custom-built and purchased solutions.</li><li><strong>Data accessibility and interaction:</strong> Ensure data accessibility with visualisation and interaction capabilities for all team members, and implement flexible governance systems to manage user permissions.</li></ol><p><strong>So to wrap up</strong>: We always need to ask ourselves the question — are we more likely to achieve our goal of designing better drugs by either predicting complex properties or biology with lower precision <em>or</em> predicting simpler biology with higher precision? Once we have made this decision, which will vary from project to project, we can then optimise all parameters across the entire technology stack, including the experiments, in the best possible way for the training of models that can answer meaningful biological questions.</p><p>The second ingredient is the right company culture and organisation of the team.</p><h4>2. Company culture and team organisations that enable a data driven approach</h4><p>As discussed earlier, the company culture and organisation of the team will play a key role when creating a data-driven organisation. We’ve compiled a list of potential actions that can support the creation of the right culture and teams. We also refer the reader to <a href="https://www.nature.com/articles/d41586-023-02291-4?utm_source=Nature+Briefing&amp;utm_campaign=393bdb7bd3-briefing-dy-20230714&amp;utm_medium=email&amp;utm_term=0_c9dfd39373-393bdb7bd3-45280994">this practical guide</a> on how to work with data scientists.</p><p><strong>Company culture:</strong></p><ol><li><strong>Visionary leadership:</strong> The C-suite should articulate a compelling vision for becoming a data-driven organisation and infuse this vision throughout the company. This vision should be communicated clearly and consistently across all levels. Having strong support from senior leadership is essential and also requires appropriate representation at the senior management level. GSK, for example, has established a Senior Vice President and Global Head of Artificial Intelligence and Machine Learning, and many biotech companies start with a Chief Data Officer in their team.</li><li><strong>Data-centric reward system:</strong> Implement a reward system that incentivises data quality, accessibility, and data-driven innovation, not just short-term milestones. This could be in the form of awards, bonuses, or public recognition but should also be part of annual goal setting and work-acceptance.</li><li><strong>Data literacy:</strong> Prioritise data literacy by establishing regular training programs and workshops. Ensure all team members understand their role in the data value chain and how their contributions impact the overall success of the organisation.</li><li><strong>Shared data ownership:</strong> Encourage a sense of shared ownership of the data. Provide scientists with easy-to-access tools to perform simple data science tasks themselves, and to evaluate and better understand the data and quality of the data that they produce. This fosters a culture of proactive data management.</li><li><strong>End-to-end process focus:</strong> Promote an understanding of the end-to-end data processes instead of focusing only on individual technologies. This helps everyone appreciate the importance of their role in the larger context.</li><li><strong>Shared responsibility and accountability:</strong> Foster a culture of shared responsibility for both short-term and long-term outcomes. The application of machine learning in projects, for example, should be the shared responsibility of data scientists and wet lab teams.</li><li><strong>Continuous learning:</strong> Commit to the ongoing education of staff. Allocate time and resources for continuous training on new systems and technologies. This extends to data scientists and machine learning experts who need to develop a good understanding of the wet lab processes and the data that is generated.</li></ol><p><strong>Organisational structure:</strong></p><ol><li><strong>Cross-functional teams: </strong>Form cross-functional project teams with shared data goals and accountability. This should include data scientists as core team members within drug discovery projects.</li><li><strong>Co-location:</strong> Whenever possible, co-locate teams to facilitate spontaneous interaction and collaboration. In cases where this isn’t feasible, use technology to keep teams connected and promote frequent communication.</li><li><strong>Standardised processes:</strong> Enforce the use of standard data pipelines and systems across all teams throughout the company. As discussed earlier, this consistency helps to minimise errors and facilitates better data analysis.</li><li><strong>Shared success and failure: </strong>Celebrate wins as a joint effort and hold all team members accountable for any shortcomings. This approach fosters a sense of unity and shared commitment.</li><li><strong>Knowledge sharing:</strong> Promote knowledge sharing between different teams and backgrounds. Explain technologies and communicate requirements in regular sessions, and make resources and information readily available across the business. Project teams should maintain clear documentation of all decisions and steps, including models and data used. This information should be easily accessible (upon request) across the business.</li><li><strong>Product teams </strong>(i.e. any team building a computational workflow or analysis system) need to be science or user led — but software developers or data scientists need to be part of the core product team. Working together to define product specifications and perform user acceptance tests is key to building useful tools.</li></ol><h3>Putting it all together</h3><p>By implementing some of the proposed actions and building a good technology stack as well as developing a data-driven culture and organisation, we believe modern biotech companies will be able to produce high quality machine learning-grade data (5 ‘V’s) and get better results using machine learning. This can ultimately lead to the selection of better compounds and improved patient outcomes. We hope that these guidelines will help some of the operators in the field develop their own thinking and strategy for building data-driven organisations, and educate investors and other stakeholders who are less familiar with the space on the need to look at technology and data more holistically.<br>If you have any questions or comments, please do reach out to us.</p><p><strong>Acknowledgements:<br></strong>We’d like to thank Philipp Harbach and Gino Van Heeke for his input on many aspects that we touched on in this article.</p><h3><strong>Appendix</strong></h3><p>More examples.</p><p><a href="https://pubmed.ncbi.nlm.nih.gov/34233495/"><em>Alves et al.</em></a> gave a good example in toxicology regarding the impact of data quality on the confidence in model prediction. As an example, they showed that models generated with un-curated data had a 7–24% higher correct classification rate but the perceived performance inflated owing to the high number of duplicates in the training set. Chemical toxicity mechanisms, like many mechanisms in drug discovery, are complex, involving multiple molecular pathways, cell types, and organ systems, and interpretations of toxicology studies are further complicated by differences in study protocols, exposure conditions, chemical purity, test subject attributes, and dose selection. Other sources of biological and random variability can also impact the concordance of toxicology outcomes from different studies, with one-third of variance in ‘no effect levels’ from rodent studies unaccounted for by obvious study characteristics [<a href="https://pubmed.ncbi.nlm.nih.gov/33426408/">Ref</a>]. <a href="https://pubmed.ncbi.nlm.nih.gov/27746907/">Another study</a> evaluated a corpus of human transcriptomics studies by comparing the provided annotations of sex to the expression levels of sex-specific genes. They identified apparent mislabeled samples in 46% of the datasets studied, yielding a 99% confidence lower-bound estimate for all studies of 33%.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=331e29c683c5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Where is generative design in drug discovery today?]]></title>
            <link>https://medium.com/@leowossnig/where-is-generative-design-in-drug-discovery-today-7234945177cf?source=rss-d2e17e79f2e0------2</link>
            <guid isPermaLink="false">https://medium.com/p/7234945177cf</guid>
            <category><![CDATA[ai-for-good]]></category>
            <category><![CDATA[generative-design]]></category>
            <category><![CDATA[generative-model]]></category>
            <category><![CDATA[drug-discovery]]></category>
            <category><![CDATA[ai-drug-discovery]]></category>
            <dc:creator><![CDATA[Leo Wossnig]]></dc:creator>
            <pubDate>Mon, 03 Jul 2023 15:03:52 GMT</pubDate>
            <atom:updated>2023-07-04T20:37:46.296Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cCZcYRw1596OYgpMjq-8PA.jpeg" /></figure><p><em>Disclaimers</em></p><ul><li><em>The opinions expressed within this article are solely mine and do not reflect the opinions and beliefs of my employer or any affiliate.</em></li><li><em>In the following, I will talk about both generative protein design as well as generative small molecule design. It is important to remember that these two approaches are in fundamentally different in many ways, for example in the way we represent the molecules (e.g., SMILES vs a sequence). I will try to only make statements which are credible for both areas, however, the degree to which this applies may vary.</em></li></ul><h3>Introduction</h3><p>In this blog post, I share the current state of generative molecular design, and offer my perspective on its progress to date. I will endeavour to explain why some past criticisms are no longer relevant and highlight the remaining challenges that need to be overcome to further improve the impact that generative design can have on the drug discovery pipeline.</p><p><a href="https://www.neoleukin.com/">Neoleukin</a>, originally established in 2021 from David Baker’s Lab, and arguably the first company to administer a <a href="https://www.sciencedirect.com/science/article/pii/S1367593120300181"><em>de novo</em> designed protein therapeutic</a> to a patient, is currently <a href="https://www.fiercebiotech.com/biotech/neoleukin-old-ending-de-novo-protein-player-lays-70-staff-searches-strategic-alternative">exploring strategic alternatives</a> such as a sale or merger. This follows the failure of their <em>de novo</em> designed IL-2/IL-15 agonist, NL-201, in phase 1 clinical trials. Failures like this remind us that generative design won’t immediately revolutionise drug discovery and result in better drugs overnight (i.e., ones that end in a patient). However, despite such setbacks, it is important to highlight the promise that this field offers and not forget the advances that have been made over the past few years. There is still substantial opportunity to discover hit and lead molecules more quickly and affordably by adopting principled approaches in molecular design, or find molecules that can address targets that would have once been classified as ‘undruggable.’</p><p>The field of generative molecular design is currently experiencing a huge surge in interest, particularly in the area of generative protein design. Despite the criticism, I believe that some of this interest is justified as these methods have the potential to accelerate the early stages of drug discovery.</p><p><a href="https://arxiv.org/pdf/2303.17615.pdf">Generative or <em>de novo</em> molecular design</a> for both small molecules and proteins has been around for a few years now and following the success of <a href="https://www.nature.com/articles/s41586-021-03819-2">AlphaFold</a>, activity has recently picked up again. A lot of attention is on so-called <a href="https://lilianweng.github.io/posts/2021-07-11-diffusion-models/#reverse-diffusion-process">diffusion models</a> with the most recent versions being able to co-generate protein sequence and structure. Protein diffusion models include <a href="https://www.biorxiv.org/content/biorxiv/early/2022/12/14/2022.12.09.519842.full.pdf">RFDiffusion</a> from the Baker lab or <a href="https://www.biorxiv.org/content/10.1101/2022.12.01.518682v1">Chroma</a> by Generate — the main difference? Chroma has more ML acrobatics (chain co-variation, classifier guidance-based conditioning, a novel low-temperature sampling method for diffusion models that increased sample quality whilst maintaining high diversity, efficient GNNs for modelling long range interactions, etc. ) but RFDiffusion has better experimental validation. Combining all of the ML innovations, however, Chroma was able to train their model from scratch, which didn’t work for RFDiffusion and instead had to use pre-trained RosettaFold.</p><p>These diffusion methods have been used to <a href="https://arxiv.org/pdf/2305.20009.pdf">design antibodies </a>that bind to desired antigens and other proteins. Whilst early results look promising, a lot more validation is necessary to understand how they fit into the wider protein engineering toolbox. From conversations with multiple therapeutic antibody discovery teams, they generally seem to work well and result in hit rates of 2–10% when designing binders targeting a specific antigen, with the rates varying between different targets. A recent <a href="https://arxiv.org/abs/2305.20009">paper by a group of researchers from NYU and prescient design (Genentech)</a> applied these approaches to the optimisation of antibodies for higher expression yield and binding affinity even found a 25% binding rate (log10(KD) values ranging from 6 to 8) against the target and 97% expression yields using <em>in vitro</em> assays (based on 68 designs).</p><p>While more and more models are becoming available (including many open-source ones such as <a href="https://github.com/MolecularAI/Reinvent">REINVENT</a>, <a href="https://github.com/songleee/LS-MolGen">LS-MolGen</a>, or <a href="https://github.com/molecule-generator-collection/ChemTSv2">ChemTSv2</a> just to name a few), my personal take is that <a href="https://www.nature.com/articles/d41586-023-01516-w">more data will be required</a> to build large pre-trained models (for example, a model for antibody binding to any target). Once that data is available, these models could be particularly useful for the antibody lead discovery stage where binding is the main objective. Whether this will translate into functionality is still unclear. For example, T-cell activation and CD3 binding are not the same thing: Not every T-cell engager that binds CD3 will result in T-cell activation, even if cross-linking is achieved. Testing such functional properties also requires more complex functional assays, which by definition require a lot more time and resources when compared to simple binding assays. However, finding more binders will likely increase the chance that one of them achieves the desired function.</p><p>Beyond function, I think there will still be a need for other methods and more data generation to overcome some of the key challenges associated with the current generation of therapeutics. Take for example CD3-targeting bispecifics: We need new therapeutic approaches that can overcome the dose limiting toxicity that appears to be one of the main challenges for their ability to address solid tumours — see <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7829968/">here</a> or <a href="https://www.nature.com/articles/s41570-022-00438-x">here</a>. For example, in the absence of cell-level data (e.g., receptor densities on the cell), the use of generative models for multi-specific T-cell engagers or ADCs is still a long way off.</p><p>Generative design (machine learning for chemistry) has been regularly applied in the small molecule space. Most notably, in one of the biggest cancer target classes, kinase inhibitors. A lot of the early papers and claims have been rightfully scrutinised. See for example, the criticisms by <a href="http://practicalcheminformatics.blogspot.com/2019/09/dissecting-hype-with-cheminformatics.html">Pat Walters</a>, <a href="https://www.science.org/content/blog-post/has-ai-discovered-drug-now-guess">Derek Lowe</a>, or <a href="http://www.drugdiscovery.net/2019/09/03/so-did-ai-just-discover-its-first-drug-comment-on-deep-learning-enables-rapid-identification-of-potent-ddr1-kinase-inhibitors/">Andreas Bender</a> on the early <a href="https://www.nature.com/articles/s41587-019-0224-x">DDR1 paper</a>. This resulted in a number of good recommendations with regards to how we should evaluate such methods (e.g., <a href="https://www.nature.com/articles/s41587-020-0418-2">here</a> and <a href="https://www.nature.com/articles/s41570-022-00391-9">here</a>).</p><p>Since then, however, many of the initial challenges such as <a href="https://www.science.org/content/blog-post/generating-crazy-structures">generating ‘crazy’ structures</a> or the <a href="https://www.sciencedirect.com/science/article/pii/S1740674920300159">exploitation of the scoring functions</a> have been addressed by the community. While some papers still make elementary mistakes (see for example figure 3 in <a href="https://pubs.acs.org/doi/full/10.1021/acs.jcim.3c00562">this paper</a>), more recent papers address many if not most of the initial shortcomings, such as the lack of prospective validation and testing of key properties, such as selectivity for kinase inhibitors. (<em>It’s important to note that there are a lot of challenges with retrospective studies, even using data sets like the </em><a href="https://tdcommons.ai/"><em>therapeutic data commons</em></a><em>; For a discussion see </em><a href="https://arxiv.org/abs/2009.00707"><em>here</em></a><em> and </em><a href="https://chemrxiv.org/engage/api-gateway/chemrxiv/assets/orp/resource/item/648bed3ae64f843f41dba601/original/on-the-difficulty-of-validating-molecular-generative-models-realistically-a-case-study-on-public-and-proprietary-data.pdf"><em>here</em></a><em>; The </em><a href="https://cache-challenge.org/cache-bridging-gap-between-molecule-discovery-and-computational-design"><em>CACHE</em></a><em> challenge is worth highlighting as a new option to test a generative model in a prospective setting.)</em></p><p>Molecules that are generated today can satisfy relevant ranges for physicochemical properties like molecular weight, logP, or logD, and pass through MedChem and PAINS filters. They are selective (where needed), and progressively more examples also include experimental validation. They might not unilaterally be viewed as perfect molecules but <a href="https://pubs.acs.org/doi/abs/10.1021/acs.jmedchem.0c01148">this paper</a> by GSK performed a <em>molecular Turing </em>test and showed that all generated ideas were liked by at least one chemists. This is quite good in my opinion, since if you ask two chemists which molecules are better, they would likely <a href="https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0048476">disagree anyway</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SBsUGIpNpaVGTdpA" /><figcaption><strong>Figure 1</strong>: Turing test for molecular generators. The number of likes per molecule, coloured according to the origin of design: by chemist (yellow), computer (dark green), or both chemist and computer (light green). All ideas were liked by at least one chemist, and most were liked by more than half of the chemists. Source: <a href="https://pubs.acs.org/doi/pdf/10.1021/acs.jmedchem.0c01148">A Turing Test for Molecular Generators</a></figcaption></figure><p>One paper that demonstrates many of the above points is by <a href="https://www.nature.com/articles/s41467-022-34692-w">Li<em> et al.</em></a>, which identified a selective RIPK1 inhibitor with a previously unreported scaffold. They also demonstrated selectivity (experimentally) using a panel of 406 kinases (at 10 µM) including key off-targets. To do this, they generated a targeted virtual library and then screened this library with conventional virtual screening methods to prioritise candidates for testing. <em>Notably they did not actively optimise for selectivity or build this in their design process.</em> Although this is a great example of real validation, it is still ‘<a href="https://pubs.acs.org/doi/10.1021/acs.jmedchem.2c00334">level 1 automatic chemical design</a>’ (thanks to Morgan Thomas for pointing this out). Automatic chemical design level 1 refers to a degree of ideation automation in chemical design where the machine generates ideas for molecular structures, while the chemist still provides varying amounts of guidance and retains the responsibility of final selection and assessment of the synthesisability of the molecules.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CUNvBvaorVLvgnNm" /><figcaption><strong>Figure 2:</strong> Some off-targets for the <a href="https://www.uniprot.org/uniprotkb/Q13546/entry">RIPK1</a> inhibitor tested by <a href="https://www.nature.com/articles/s41467-022-34692-w"><em>Li et al.</em></a></figcaption></figure><p>Over the past few years, I have seen first hand in my own work, and also other <a href="https://pubs.acs.org/doi/full/10.1021/acs.jmedchem.3c00485">research groups</a> and companies (for example the collaboration between Microsoft and Novartis, highlighted on pages 15/16 <a href="https://cdn.eventsforce.net/files/ef-px6ci6a56tzs/website/214/5th_rsc-bmcs-cicag_ai_in_chemistry_meeting_report.pdf">in this report</a>), who have successfully applied these approaches in their internal pipelines. Many new generative approaches have been developed with various different applications such as <a href="https://pubs.rsc.org/en/content/articlelanding/2023/dd/d2dd00115b">linker design</a>, or <a href="https://pubs.acs.org/doi/abs/10.1021/acs.jcim.1c00469">scaffold decoration</a> and <a href="https://chemrxiv.org/engage/api-gateway/chemrxiv/assets/orp/resource/item/647060d7be16ad5c57f1ab9a/original/de-novo-generated-combinatorial-library-design.pdf">combinatorial library design</a> or for specific types of drugs, such as <a href="https://pubmed.ncbi.nlm.nih.gov/35056884/">covalent kinase inhibitors</a>. Generative design can hence be used for <a href="https://jcheminf.biomedcentral.com/articles/10.1186/s13321-021-00565-5">scaffold hopping</a>, <a href="https://pubs.acs.org/doi/full/10.1021/acs.jcim.9b01120">fragment linking </a>(including PROTAC design) and <a href="https://pubs.acs.org/doi/10.1021/acs.jcim.1c01311">fragment growing</a>, <a href="https://pubs.rsc.org/en/content/articlelanding/2020/sc/c9sc04503a">R-group optimisation</a>, and library design.</p><p>Improvements have been made on the efficiency of the methods (<a href="https://www.nature.com/articles/s42256-022-00494-4">here</a> using curriculum learning or <a href="https://arxiv.org/pdf/2210.12458.pdf">here</a> through directly optimising the learning steps) and the way in which they are integrated into <a href="https://jcheminf.biomedcentral.com/articles/10.1186/s13321-022-00667-8">conventional drug discovery</a> programs.<br>It is worth noting that RNN-based generative models are the most prevalent generative small molecule design models in the literature (c.f., Fig. 10 in <a href="https://www.sciencedirect.com/science/article/pii/S0010482522001950">this article</a>). More recently, there has also been a surge of <a href="https://arxiv.org/abs/2306.11768">structure-based ligand design</a> papers, which directly ‘diffuse’ (denoise) a ligand into the 3D protein pocket. For a good introduction (for chemists) on how these models work, I recommend <a href="https://medium.com/@cch57/exploring-the-promise-of-generative-models-in-chemistry-an-introduction-to-diffusion-models-31530e9d1dcb">this recent article</a>. Models, such as <a href="https://arxiv.org/pdf/2210.13695.pdf">DiffSBDD</a>, can be used for a variety of tasks including fragment linking, scaffold hopping, or scaffold elaboration. However, most structure-based papers do not have experimental validation, and generated molecules have significantly more physical violations and fewer key interactions compared to baselines, so it’s unclear how much value they will provide. Look out for a paper by <a href="https://cch1999.github.io/">Charlie Harris</a> and others which investigates this in detail and will become available soon.</p><p>Similarly, generative protein design has moved from hallucination of ‘random’ protein structures to models that <a href="https://arxiv.org/pdf/2305.20009.pdf">generate proteins with well defined properties</a>, for example antibodies that bind a specific antigen or <a href="https://www.biorxiv.org/content/10.1101/2022.07.10.499510v5.abstract">CDR regions targeting a specific epitope</a>.</p><p>While the challenges for generative small molecule design and protein design are somewhat different, some of the open problems are highly related.</p><p>In the next section I will address some of the criticism on <em>de novo</em> small molecule design and then go on to talk more about the real-world challenges in both the fields of generative protein and small molecules design.</p><h3>Common criticisms of de novo (small molecule) design are outdated</h3><p>While I agree with a lot of the initial criticism, some of the more recent comments (for example <a href="https://practicalcheminformatics.blogspot.com/2023/02/generative-molecular-design-we-need-to.html">Pat Walter’s blog post</a> or Derek Lowe’s commentary on <a href="https://www.science.org/content/blog-post/generative-models">generative chemistry</a>) no longer feel appropriate.</p><p>There are two arguments against generative (small molecule) design that are still frequently raised:</p><ol><li>These methods do not generate drugs</li><li>Generative approaches are mainly used to design binders (hits). This means that the molecules are optimised for protein binding, a set of physicochemical properties, and for passing certain MedChem and PAINS filters. This is not as hard as optimising for properties that are truly relevant in drug discovery (e.g., selectivity for kinase inhibitors)</li><li>Generative models do not come up with truly novel molecules (i.e., something entirely unseen) and instead they just design molecules within the known chemical space of existing drugs</li></ol><p>I will try to address these separately:</p><ol><li>An argument that is often made is that these methods do not generate drugs, which is absolutely correct. However, these methods do still add value in helping us find good starting points (e.g., selective hits) or early leads for our programs as fast as possible. If a generative approach can provide us with a large and diverse set of lead-like molecules that fulfil our target molecular profile, then we have a higher chance of progressing into the next stage, due to having a sufficient number of candidates to test. They are ultimately another tool in the large drug discovery toolbox that we should lean on in combination with all of our other methods, either explicitly, or implicitly.</li><li>While I agree that the generation of molecules with more complex profiles (for example, selective, non-promiscuous Kinase inhibitors) has rarely been demonstrated in the literature, the <a href="https://www.nature.com/articles/s41467-022-34692-w">above mentioned publication</a> is a good example of such a result, and there are increasingly more <a href="https://www.sciencedirect.com/science/article/pii/S0223523422009369">papers</a> attempting to perform more rigorous experimental validation. I am also aware of multiple industry run programs that have achieved similar results.<br>It should be noted that generative models will not solve all our problems. In fact, the reinforcement learning based approaches, such as <a href="https://pubmed.ncbi.nlm.nih.gov/33118816/">REINVENT</a> (which continues to be one of the best performing methods — see <a href="https://proceedings.neurips.cc/paper_files/paper/2022/file/8644353f7d307baaf29bc1e56fe8e0ec-Paper-Datasets_and_Benchmarks.pdf">here</a> and <a href="https://pubs.acs.org/doi/full/10.1021/acs.jcim.2c01355">here</a>), can only create molecules for properties that we understand or can predict well. For example, selectivity for small molecule kinase inhibitors is inherently hard to predict and neither conventional methods (docking, FEP, etc.) or machine learning can reliably predict it off-the-shelf. For machine learning, this is largely due to a lack of reliable selectivity data in the right quantities .</li><li>I am not a big fan of the third argument for two reasons.<br>Firstly, a recent paper by <a href="https://pubs.acs.org/doi/10.1021/acs.jmedchem.8b00675">Brown and Boström (Astrazeneca)</a> that analysed the origin of clinical compounds concluded that most clinical candidates (43%) were derived from known starting points (e. g., literature, patents, previous corporate knowledge, etc.), followed by random screening at only 29% (see figure 3 below). <br>Another <a href="https://pubs.acs.org/doi/10.1021/acs.jmedchem.1c02106">recent publication by Roche and Genentech</a> analysed lead-finding trends and the origins of lead series from their organisations between 2009 and 2020 and found that public information made up the majority of development candidates for both organisations (31% and 35% for Genentech and Roche, respectively). This was more recently confirmed by <a href="https://pubs.acs.org/doi/full/10.1021/acs.jmedchem.3c00521">another study</a> by Brown and Boström, which highlighted that the most frequent lead generation strategies resulting in clinical candidates were from known compounds (59%) and random screening approaches (21%). <br>In addition to this, more recent methods all incorporate mechanisms to introduce more diversity and novelty (<a href="https://www.biorxiv.org/content/10.1101/2022.07.15.500218v1.full.pdf">example here</a>). An even more extreme example is that <a href="https://pubmed.ncbi.nlm.nih.gov/29989814/">400 CNS drugs come from just 20 natural product</a> scaffolds.<br>Also, from a drug discovery perspective, it’s irrelevant if the molecule is known or not as long as long as (a) it can potentially help a patient (i.e., it offers a benefit over the current standard of care), and (b) it is novel from a patent perspective, which is typically a requirement for companies to progress their molecule. Beyond this, using a known molecule will typically reduce the development risks because something that has already been in a human is much less likely to result in unexpected side effects.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*VowfEzq2xPvL3fDL" /><figcaption><strong>Figure 3: </strong>Lead generation strategies that resulted in a clinical candidate. The majority of clinical candidates rely on known starting points from the literature, patents, and other sources. Source: <a href="https://pubs.acs.org/doi/10.1021/acs.jmedchem.8b00675">Brown and Boström</a>.</figcaption></figure><p>I do, however, agree with Pat and others, when they say that there is a tendency to overstate the impact of the methods. After all, it’s still unclear whether this will translate to a higher probability of success in the clinical trials of these molecules. While the clinical impact of generative methods is still to be seen, it should be noted that so far the biggest impact of ‘AI’ in the clinic has come from patient stratification: By using machine learning (or bioinformatics), it is possible to double the clinical probability of success (see table 3 <a href="https://academic.oup.com/biostatistics/article/20/2/273/4817524">here</a>), which is huge.</p><p>Putting these criticisms aside, what are the challenges we’re facing in the field of generative design today?</p><h3>Where are the actual challenges of generative molecular design?</h3><p>I see two main limitations for these approaches at the moment:</p><ol><li>Little, and low quality data and slow data generation (e.g., molecular synthesis)</li><li>No data for new targets: A lack of structure, binding sites, and binding data</li><li>Cultural challenges for the adoption of new methods</li></ol><h4>Little, and low quality data and slow data generation</h4><p>Despite advances in the methods/models, <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7382642/">data availability and collection is still the biggest bottleneck</a> in ML/AI for drug discovery. This also applies to generative design approaches. For small molecules, the <a href="https://pubs.acs.org/doi/10.1021/acs.jcim.0c00174">synthesis step</a> is usually the rate limiting factor. While public data or the use of <a href="https://pubs.acs.org/doi/abs/10.1021/acs.jmedchem.8b01048">off-the-shelf libraries</a> can improve the speed of the initial iterations and get you to a hit more rapidly, it usually doesn’t allow you to get to an optimised lead.</p><p><a href="https://www.nature.com/articles/d41586-023-01612-x">Good quality data is usually sparse</a>, in particular for novel targets, and the quality of public data is <a href="https://link.springer.com/article/10.1186/s13321-021-00571-7">rarely good enough</a> to use (see also this recent <a href="https://greglandrum.github.io/rdkit-blog/posts/2023-06-12-overlapping-ic50-assays1.html">rdkit blog post</a>). Using such data helps find hits, but due to the noise, it <a href="https://www.sciencedirect.com/science/article/abs/pii/S1359644609000403?via%3Dihub">has limited use for the optimisation</a> of more complex properties, such as selectivity.</p><p>So the first challenge is the availability of good quality data in sufficient amounts that they can be used by machine learning systems. The underlying causes are often the lack of resources and time requirements to obtain new data. Data generation in the case of small molecules means synthesising the molecule and then testing it. This bespoke synthesis can take several months and yield as little as 10s of molecules. These limited amounts of data are very challenging for machine learning models. For protein engineering this problem is less severe, as there are ways of expediting the build and test steps. For example, using IDT <a href="https://eu.idtdna.com/pages/products/genes-and-gene-fragments">e-blocks</a> (with a length between 300 and 1500 base pairs) for build and testing can be done in as little as 2 weeks when combined with <a href="https://pubs.acs.org/doi/10.1021/acssynbio.2c00392">cell free protein synthesis</a>. While these timelines increase for more complex molecules (e.g., ADCs or multi-specific and multivalent antibodies) or functional read-outs (such as T-cell activation), the cycle times are still much faster. The most advanced companies in this space take <a href="https://labgeni.us/">less than 6 weeks for a full cycle</a> and create hundreds of readouts from disease relevant, cell-based assays. For most companies this impressive turnaround time is unachievable, and even data generation in the protein design space remains comparatively slow. Many companies require 4–6 months to test a few hundred candidates and this challenge only increases with the complexity of the biology being investigated.</p><p>Generating high-quality, program/target-specific data at speed can be approached in two different ways):</p><ol><li>One approach is to use <a href="https://pubs.acs.org/doi/10.1021/acs.jcim.9b00325">pre-existing building blocks (fragments)</a> to create molecules. This imposes restrictions on the design space (i.e., the total number of different molecules defined by all the combinations) that we can explore, but can enable significantly shorter timelines from design to experimental testing. While typically not done in a generative fashion, this strategy is also possible for designing antibodies. For example by <a href="https://static1.squarespace.com/static/627245460748b93cb2c0baf6/t/64610e2edceffe3557b6a976/1684082235754/LabGenius+Scientific+Poster+%28PEGS%2C+Boston%29.pdf">combining pre-existing building blocks</a> such as antigen binders and linkers to design more complex multispecific and multivalent antibodies. This is successfully done by companies like <a href="https://www.harmonicdiscovery.com/">Harmonic Discovery</a> in the small molecules space (for kinase inhibitors, which offers enough data to do so), or by <a href="https://www.labgeni.us/">LabGenius</a> for multispecific and multivalent antibodies</li><li>A second approach is based on modern retrosynthesis tools, which have made significant <a href="https://www.nature.com/articles/nature25978">leaps forward</a> and have found their <a href="https://link.springer.com/article/10.1186/s13321-020-00472-1">way into many applications</a> including generative design (see e.g., <a href="https://arxiv.org/abs/1906.05221">here</a> or <a href="https://arxiv.org/abs/2004.14308">here</a>). They are, however, still limited in their scope. While there are many ongoing efforts to improve both models and data sets, it appears that the <a href="https://www.nature.com/articles/s43588-023-00446-x">availability of high-quality data</a> might also limit our ability to perform retrosynthesis and on-demand-synthesis. Another challenge with retrosynthetically proposed synthesis steps might not match in-house available chemistry capabilities. This can further slow down the process or limit the applicability.<br>Related to this, we can use known synthesis rules to generate molecules that can be synthesised in a small number of steps. For example, a range of relevant scaffolds can be identified using <a href="https://enamine.net/compound-collections/real-compounds/real-database">off-the-shelf libraries</a> and then generative methods then applied to <a href="https://pubs.acs.org/doi/abs/10.1021/acs.jcim.0c01015">decorate these scaffolds</a> to <a href="https://pubs.acs.org/doi/10.1021/acs.jcim.1c00469">design large libraries</a>. This reduces the time to perform the synthesis and can help explore the space much more efficiently, albeit constrained to these known scaffolds</li></ol><p><a href="https://www.sciencedirect.com/science/article/pii/S1359644621002531">This overview</a> by Meyers<em> et al </em>provides more information on the above two approaches<em>. </em>Note that the paper classifies the different design approaches into atom-based, fragment-based, and reaction-based, where the fragment- and reaction-based approaches correspond to the two different approaches discussed here, while atom-based corresponds to the unconstrained design case and the atom-based generative approach corresponds to the unconstrained (but slow to synthesise) generation.</p><p>On the other hand, there is also a need for more structural data. This is challenging because resolving crystal structures can be slow and expensive. We, however, see some companies heavily investing in this space. For example, Generate has <a href="https://generatebiomedicines.com/news/generatebiomedicines-unveils-state-of-the-art-cryoem-laboratory-to-accelerate-generative-ai-drug-discovery-and-development?es_id=29418de17f">just announced</a> the opening of their new Cryo-EM facilities.</p><p>An entirely different approach is taken by Lee Cronin and his team at <a href="https://www.chemify.io/">Chemify</a>. He is attempting to build an universal synthesis machine — the chemputer (see <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6062836/">here</a> or <a href="https://www.science.org/doi/10.1126/science.abo0058">here</a> for more of the science). While potentially a long shot, it is definitely one of the most exciting ideas out there!</p><p>Beyond the synthesis step, the other key bottleneck for all types of machine learning (generative/unsupervised as well as supervised) is the quality and relevance of the data.</p><p>While an immense amount of data is generated in the field of chemical and biological sciences, <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7382642/#bib0160">many existing databases are often noisy</a>, <a href="https://pubs.acs.org/doi/10.1021/acs.joc.3c00844">imbalanced</a>, biassed, or contain incomplete annotations. This lack of high-quality data can result in underperforming models, and even lead to common errors such as impossible valences in chemical structures. Notably, the quality of the data affects small and large models alike (for example <a href="https://huggingface.co/blog/falcon">this large language model</a>, called the Falcon model). However, the smaller the data set, the more important this becomes. In drug discovery, where we mainly deal with small data amounts this is hence much more severe.</p><p>Similarly, for newly collected data, data curation and quality checks are crucial but often underappreciated tasks. However, these non-curated datasets can significantly affect the harmonisation of information and, consequently, model quality and utility. Many companies don’t pay enough attention to these sorts of issues.</p><p>To address these challenges, I think that one of the most important areas to make progress is high-throughput (high-quality) experimentation and building of end-to-end experimental and data pipelines, and <a href="https://www.benchling.com/news/benchling-open-source-library-lab-instrument-data-converters">all the work required at the intersections</a>). To minimise noise and errors such systems should rely as much as possible on robotic automation that promotes consistent experimental and computational workflows, right down to the storage of controls. Companies like <a href="https://www.xtalpi.com/en/AmpChemSyn">Xtalpi</a> have, for example, massively invested in robotic synthesis platforms for these reasons.</p><p>Despite the advantages of automated data creation and collection, accessibility to such platforms is still limited. This is due to the high costs, lack of consistency across providers, and lack of experience in the executing teams. Future data-driven research may be increasingly automated as these technologies become more widely available and teams acquire the know-how, allowing for high-quality, consistent information generation with minimal human intervention.</p><p>Beyond experimental and process accuracy, the importance of conducting highly informative and diverse experiments without human biases is hugely important (e.g., creating negative examples). Methods such as active learning can drastically improve the performance of machine learning by selecting the optimal candidates for training predictive or generative models. See for example here for a <a href="https://medium.com/@winston.haynes/not-one-size-fits-all-4d6cf2395115">recent blog post</a> by Winston Haynes and myself which touches on this. Active learning has been successfully applied to the exploration of large chemical spaces by using deep learning in combination with FEP and docking. There is a long list of examples, and I recommend having a look at some of these [<a href="https://pubs.acs.org/doi/10.1021/acs.jctc.1c00810">Ref1</a>, <a href="https://pubs.acs.org/doi/abs/10.1021/acs.jcim.0c01452">Ref2</a>, <a href="https://pubs.acs.org/doi/10.1021/acs.jctc.2c00752">Ref3</a>, <a href="https://chemrxiv.org/engage/chemrxiv/article-details/62c8ab40e1aaae368320cb17">Ref4</a>, <a href="https://www.sciencedirect.com/science/article/pii/S2667318522000204">Ref5</a>, <a href="https://chemrxiv.org/engage/chemrxiv/article-details/62c8ab40e1aaae368320cb17">Ref6</a>, <a href="https://pubs.acs.org/doi/pdf/10.1021/acs.jcim.2c00554">Ref7</a>, and most recently <a href="https://chemrxiv.org/engage/api-gateway/chemrxiv/assets/orp/resource/item/62c8ab40e1aaae368320cb17/original/active-learning-guided-drug-design-lead-optimization-based-on-relative-binding-free-energy-modeling.pdf">Ref 8</a>]. Active learning has also been successfully used by companies such as <a href="https://labgeni.us/">LabGenius</a> to perform fast antibody optimisation by selecting the optimal candidates to test experimentally. Notably, the fast cycle times get continuous feedback which are crucial to make such approaches work.</p><p>However, most fundamentally we will need to increase predictive validity of the methods we use to generate the data. Better translation into the clinic is only possible by using more complex and more disease relevant biology. Examples here are <a href="https://www.nature.com/articles/s43586-022-00118-6">organs-on-a-chip</a> or <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8752949/">artificial organoids</a>. Many of these fields are still nascent, but hold a lot of promise for the future.</p><h4>No data for new targets: A lack of structure, binding sites, and binding data</h4><p>The second challenge is that many generative models, such as REINVENT, rely on scoring functions. These can be purely ML based but the highest performance is usually achieved by additionally relying on structure-based methods such as docking. For these, we need an input structure for the target. Whether it is a small molecule <em>de novo </em>design algorithm taking in the protein structure (e.g., for docking and binding affinity predictions), or a generative protein design model taking in the structure of the target antigen.</p><p>A lot of progress has been made in predicting protein structures, but it is still not considered a reliable method for the design of new drugs.</p><p>While AlphaFold2-generated protein structures appear to offer a degree of <a href="https://pubs.acs.org/doi/10.1021/acs.jcim.2c00796">efficacy when used in Free Energy Perturbation (FEP</a>) calculations, a result primarily attributed to molecular dynamics simulations facilitating adjustments in sidechain positioning, they appear to be less useful for molecular docking (see for example <a href="https://pdf.sciencedirectassets.com/318494/1-s2.0-S2589004222X00138/1-s2.0-S2589004222021939/main.pdf?X-Amz-Security-Token=IQoJb3JpZ2luX2VjEEgaCXVzLWVhc3QtMSJHMEUCIHW1aRS5wdvtbmX3TkgLZ%2BfrL1LpPOUwAG68Q9j3kr2WAiEA13Hndj41F1eZQajqK9iZbdQyHvckcDuZEeXPOLsKohAqswUIQBAFGgwwNTkwMDM1NDY4NjUiDP8C1br8Zbs%2FEBhdOiqQBX1vmdk9SVxAB3kBWAzOXXxNspJ8q7nX5DN8jEFNxp1dVPWGgoIok8SmP6KuF7bK6ACPkZB%2FqyZ0StdF05OeSRNKGgjCvT4c3938RA%2F4C7vabQ%2FcNfestTAJ%2BHKDWM1gAI5iKkFG8L44VXk8BsdAPEjheRmkcj6M3IFteJcR2zndCHTVLVlTiF8ZZ67IEBR1H6F6VhcLx8Fmpd5MDjXaHxwqauyuwfEHnqM4agBnWdpjDvtvHUe6v4XlWM2bkMLQ6VZKGbI2kDIzP%2FSPvHWSjaNDyiMmxxhgsCzD05zbaBIxU782ZV4OH5bvsdXB%2BtNIcROX6Ymi1wmHpJ0sMPYTkTahgJkl8c4z8kja6gCiFWGTeicmTfuqlsP20%2B5gxGO0gvWl84We5%2BtQX0puBVIGhH6y%2FoFu3Y8rDR19f1gj1ljqz72h%2F1SxG4IVjmTsd8l1DH0jPd%2Fc9LEqDSJsLb9wC3uzEGMiARLnTqNDVqwyUOx%2F6ExaKm6SW1MUy4FdjFdHrqYDSwEtQLkaYvHn6eQnarIsfHVcV68mmAm5H36p3SsCl2Qs6GV6WVWhu%2FJJ%2FneQawYyVmm5XJwR1vLO8vYvN8EqPuJm14X4dFFqJuslBWsv1pgY6Hq5cKOw34xUQQf9zkGPz%2B91vPkFggtYzfrxNKtCWSQVCJBEe4l1LCSCHhn7AuTZmDUJA%2FQ8SUZdWK55LPYMGrMHozvsLLLCv%2Fdd1wWLVEfipkkzVAr%2FybpN5LFTTxGXK0hEC0IXDot9cw5opwaYhbYCi%2FsscLsfxnnqUK8erxJB0z6z7xGitoLnn6Iq1ty5qFonjgsg3DQXc69q%2FvUqRU3Z6XoYstDXMQ%2BnwPLHqzGP8dUESnYRRUKH6ECqMJji86EGOrEBaOfSe%2FA4yqA0jB0DQNF%2FJenLv1bdajorSPBZCO6qinxDlI09ML7D4KF7Apxc3zH6wEud0mYnWk2YnukPb5UUfqaTo9Hp6F%2F%2F12LvjDOIxLmfksykKm1NGEVu0G9UKubtowdNUmEy1PZIMxJzfj5M696ojwMCRerYL0FJWb5Amevp6UoHdlBrNQFztIEnDxWeZmMMF7yb0mUlh3ty95Y7RETmvf8z1n2SsthtIawhdJnc&amp;X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Date=20230417T083314Z&amp;X-Amz-SignedHeaders=host&amp;X-Amz-Expires=300&amp;X-Amz-Credential=ASIAQ3PHCVTY23YRSTXC%2F20230417%2Fus-east-1%2Fs3%2Faws4_request&amp;X-Amz-Signature=929a38e8560735282aa99300b984d353e9e21e05adcc5377810ea02e12f1ac18&amp;hash=370cd41b7a4fe7706de111f108db8227c6860461e04bb758c3dc588bcddf64e9&amp;host=68042c943591013ac2b2430a89b270f6af2c76d8dfd086a07176afe7c76c2c61&amp;pii=S2589004222021939&amp;tid=spdf-d74a6382-a886-41fb-bedf-3f6942e5734f&amp;sid=03b5f5be7975e54b9b0b8e93142c74167196gxrqb&amp;type=client&amp;tsoh=d3d3LnNjaWVuY2VkaXJlY3QuY29t&amp;ua=1d0457045a565b51590507&amp;rr=7b93526cca0071fe&amp;cc=gb">here</a> and <a href="https://onlinelibrary.wiley.com/doi/full/10.1002/pro.4530">here</a> for docking into protein targets and <a href="https://pubmed.ncbi.nlm.nih.gov/36065847/">here</a> for antibiotic discovery).</p><p>For example, <a href="https://onlinelibrary.wiley.com/doi/full/10.1002/pro.4530">the study conducted at the Scripps Institute</a> observed that the resolution of side chains in AlphaFold2 structures is not precise enough to perform accurate docking when compared to protein crystal structures. Consequently, the docking success rate using AlphaFold2 structures was a modest 17%, which is significantly less than the 41% success rate achieved with holo crystal structures. Despite these suboptimal results, it’s worth noting that the docking performance with AlphaFold2 structures was markedly superior to the 10% success rate documented with apo x-ray structures.</p><p>While structural data is still essential for the targeted design of small molecules, in particular to leverage differences between targets and off-targets, it is also used by many generative protein design methods. For example, among other methods, DiffAb, relies on the co-complex of the antibody (the framework) bound to the antigen to fill in (‘graft’) CDR regions that bind. If the co-complex is not available or not of good quality, the performance is likely to drop.</p><p>The availability of high-quality, relevant (e.g., holo) crystal structures will therefore remain a challenge if docking and other methods are used as scoring functions. For novel targets there is a further challenge arising from the need to identify cryptic pockets, allosteric sites, or <a href="https://www.nature.com/articles/s41592-022-01490-7">binding sites</a> more broadly speaking. Methods such as <a href="https://www.sciencedirect.com/science/article/abs/pii/S0022283616000851">CryptoSite</a> or <a href="https://www.nature.com/articles/s41467-023-36699-3.pdf">PocketMiner</a> had some success, and combining these with more conventional approaches, such as <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8091066/">mixed-solvent molecular dynamics </a>simulations can improve the overall performance. It has also been shown that structure predictions for <a href="https://www.nature.com/articles/s41467-022-28865-w">multimers</a> can be useful in identifying <a href="https://www.biorxiv.org/content/10.1101/2021.10.04.463034v2">protein-protein interactions</a> and hence potential binding sites.</p><p>However, while all these methods are promising and have to an extent entered into the toolbox of early drug discovery, they are <a href="https://pubs.acs.org/doi/abs/10.1021/acs.jcim.1c01164">still not reliable enough</a> to pose a simple solution for all problems. Further progress is needed.</p><p>The same holds true for machine learning based binding affinity predictions, which can for example be done using a range of approaches, including proteochemometric or multitask models. Most models however, <a href="https://www.sciencedirect.com/science/article/pii/S2667318523000235">show poor predictive abilities</a> for understudied targets or shifts in chemical space, and limited ability to assess ligand selectivity and promiscuity. In many instances, they still <a href="https://www.nature.com/articles/s42256-022-00581-6">do not outperform baseline models</a> such as Random Forest. The underlying reason here is the lack of high quality (low noise and consistent) binding data for many targets and a wide range of chemistry.</p><h4>Cultural challenges for the adoption of new methods</h4><p>The field often imposes stricter standards on computational methods than it does on human-driven processes. Confidence in these methods can be hard-won but easily lost with just a few failures. Unfortunately, implementing new techniques involves more than simply persuading seasoned medicinal chemists to change their standards.</p><p>Crucially, both sides, computational and experimental scientists, must adapt. They need to assimilate the language of their peers, gain a deeper understanding of the respective topics, and grasp the intricacies of the computational and experimental processes (see also <a href="https://www.jstor.org/stable/2634940">this article</a>).</p><p>To make machine learning truly effective in real-world programs, it’s also necessary to have a complete technology stack where each component seamlessly interacts with the others. This harmony allows for the generation of high-quality, adequate volumes of data, which can ultimately empower machine learning methods to perform optimally and make significant contributions to discovery campaigns. Achieving this synergy necessitates extensive communication among all participating teams. This spans from assay development teams (to ensure proper controls and normalisation) and automation teams (to guarantee process consistency, high throughput, and fast cycles) to software engineers (for the construction of robust data pipelines). The most successful teams today have mastered the tight integration of the entire technology stack, fostering open dialogue and collaboration among experts from diverse backgrounds.</p><p>To achieve this, it is important to have leadership teams that have fundamental understanding of both the experimental and computational work, that understand the drug discovery process, and see the value of combining these fields. It also requires the teams to spend additional time on building a good understanding of the processes and methods their respective counterparts are using. For example, a machine learning scientist needs to understand the possible errors and sources of noise in the data they are dealing with, the ways the data is obtained or processed by the experimental device, and many other details to optimally use that data. At the same time, experimental scientists need to understand fundamentals of algorithms and requirements on the data (e.g., regarding consistency) in order to build their pipelines in a way that generate robust data, or communicate potential challenges and limitations of their experiments and how this could affect the machine learning models.</p><h3>Future directions for generative design: New models and better integration</h3><p>While I believe that progress in the above mentioned areas will have the biggest impact on the field, there are also interesting developments on the model side: Novel approaches such as diffusion based generative design and co-folding (see for example <a href="https://arxiv.org/abs/2209.15171">here</a> or <a href="https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-023-05354-5">here</a>) are being pursued by a number of <a href="https://scholar.harvard.edu/nazimbouatta/research">academic groups</a> and companies such as <a href="https://www.entos.ai/technology">Entos</a> or <a href="https://charmtx.com/">Charm therapeutics</a> (ligand-protein co-folding) and Latent Labs (a UK-based generative protein design company).</p><p><a href="https://arxiv.org/abs/2209.15171">Co-folding</a> could improve on the current generation of small molecule <em>de novo </em>design methods due to it being less dependent on the protein structure as a starting point. As mentioned above, many methods rely on scoring functions such as docking which are inherently limited by the crystal structure that is used (see above)and the lack of conformational flexibility. There are more expensive approaches that do take this into account but they become infeasible in combination with <em>de novo</em> approaches where hundreds of thousands of molecules need to be screened virtually. Alternatively one might aim for better scoring functions and incorporating more of the dynamics/flexibility into the scoring function. For example, this would be possible by relying on methods like <a href="https://arxiv.org/abs/2210.01776">DiffDock</a> (or <a href="https://arxiv.org/abs/2304.03889">DiffDock-PP</a>), instead of conventional docking approaches. While these methods already work really well to identify novel binding sites (e.g., if we want to find a cryptic pocket) and lots of similar data is available (e.g., for kinases), they are <a href="https://arxiv.org/abs/2302.07134v3">yet to outperform conventional approaches</a> when used for docking into established/known pockets (i.e., they don’t seem to work here as well as conventional docking approaches like <a href="https://www.schrodinger.com/products/glide">Glide</a>). I anticipate that in the near future people will likely use combinations of DiffDock (to identify a pocket) and Glide or similar methods (to dock into it).</p><p>A comment here: The fact that there are barely any results for co-folding about 3 years after the release of AlphaFold makes me believe that it might be a harder problem than first thought. I also hope that the introduction of the protein-ligand complex prediction category in <a href="https://onlinelibrary.wiley.com/doi/10.1002/prot.26515?af=R">CASP15</a> will spark more interest and advances in this area and similarly the <a href="https://cache-challenge.org/cache-bridging-gap-between-molecule-discovery-and-computational-design">CACHE</a> challenge for generative models for small molecules.<br>It would be interesting to see companies submitting their results as this would also lead to more transparency on how well these models really work.</p><p>Another direction that has already gained a lot of traction, is the integration of a variety of different approaches into the design process.</p><p>While pure ML based scoring functions can work in isolated cases, it has been shown repeatedly that integrating generative models with conventional physics-based approaches (for example <a href="https://pubs.acs.org/doi/full/10.1021/acs.jcim.1c01334">docking</a>, <a href="https://pubs.rsc.org/en/content/articlehtml/2021/sc/d1sc02436a">pharmacophore</a>, or <a href="https://pubmed.ncbi.nlm.nih.gov/34280849/">shape-based models</a>) outperforms (see e.g., <a href="https://www.sciencedirect.com/science/article/pii/S0959440X23000337">here</a> or <a href="https://chemrxiv.org/engage/api-gateway/chemrxiv/assets/orp/resource/item/60c758a7842e659173db488f/original/de-novo-design-with-deep-generative-models-based-on-3d-similarity-scoring.pdf">here</a>) generative design based on simpler scoring functions or machine learning based scoring functions alone. In particular ensembles of different methods, for example machine learning models combined with physics-based approaches, can result in dramatically improved performance. I have myself observed that ensembles easily outperform other methods in multiple instances. This is unsurprising, since ensemble models have been known to perform better for a while, for example in the context of QSAR methods (see <a href="https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-019-3135-4">here</a> or <a href="https://jcheminf.biomedcentral.com/articles/10.1186/s13321-020-0417-9">here</a>).</p><p>Note that building machine learning models for structure-property relationships will, however, remain challenging. This is due to the fact that compounds, even those slightly modified from a parent molecule, can still possess significantly different properties. For compound activity this is commonly referred to as an ‘activity cliff’ where similar structures exhibit different activity (see examples <a href="https://pubmed.ncbi.nlm.nih.gov/19544191/">here</a>, <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3159128/">here</a>, or <a href="https://pubs.acs.org/doi/pdf/10.1021/ci060117s">here</a>), and results in optimisation landscapes that are very hard to model.</p><p>A final point to make is that a lot can be learned from the closer <a href="https://www.sciencedirect.com/science/article/pii/S0959440X23000490">integration of the models with experiments</a>. This is increasingly being done and the learnings from these programs will inevitably help make the methods more useful in the future. A big part of this will be lab automation, and <a href="https://www.technologynetworks.com/informatics/articles/digitization-the-lab-of-the-future-363356">close integration </a>of data generation, collection, and analysis. I will write more in detail about the best approaches for this in the future.</p><h3>Conclusion:</h3><p>The key challenges in <em>de novo</em> molecular design revolve around data availability and quality, as well as the application and development of innovative methods. Data scarcity and the slow pace of molecule synthesis present formidable hurdles for machine learning in drug discovery. High throughput experimentation, end-to-end experimental and data (collection and management) pipelines, automated systems, and more accessible platforms can help overcome these challenges. Through better selection of compounds we can further enhance our ability to gather insights or improve machine learning model performance.</p><p>Emerging methodologies such as active learning driven experimentation, diffusion-based generative design, and co-folding represent promising avenues for progress. However, further advancements are needed, in particular for novel targets.</p><p>Creating more collaborative teams, with a shared language and understanding of the technologies and requirements (e.g., compound synthesis, compound management, restrictions around logistics such as shipping to and from CROs, assay technology etc.) will ultimately result in the biggest successes.</p><p>In summary, the future of <em>de novo </em>molecular design will be shaped by improvements in data generation and quality, advancements in novel methodologies, and a harmonised integration of machine learning and experimental strategies. These strides will undoubtedly lead to more efficient drug discovery and pave the way for innovative therapeutic interventions.</p><p><em>Acknowledgements<br>I’d like to thank Andreas Bender (PangeAI and Cambridge University) in particular the great examples for CNS drugs and the impact of patient stratification on PoS, Charlie Harris (Cambridge Univ.), Morgan Thomas (Cambridge Univ.), and Justin Grace (LabGenius) for helpful discussions and comments on this post. I’d also like to thank Nathan Brown (HealX), Laksh Aithani (Charm), and Dylan Reid (Zetta Ventures) for additional feedback on the views shared in this post.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7234945177cf" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>