<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Maverick Ventures on Medium]]></title>
        <description><![CDATA[Stories by Maverick Ventures on Medium]]></description>
        <link>https://medium.com/@maverickventures?source=rss-41af1c295e83------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*FOZVJahn335xMs3Kt-r7YQ.png</url>
            <title>Stories by Maverick Ventures on Medium</title>
            <link>https://medium.com/@maverickventures?source=rss-41af1c295e83------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 10:52:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@maverickventures/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Welcome to our newest Managing Director, Ryan Isono!]]></title>
            <link>https://maverickventures.medium.com/welcome-to-our-newest-managing-director-ryan-isono-19fb692c0125?source=rss-41af1c295e83------2</link>
            <guid isPermaLink="false">https://medium.com/p/19fb692c0125</guid>
            <dc:creator><![CDATA[Maverick Ventures]]></dc:creator>
            <pubDate>Tue, 28 Jan 2025 17:01:59 GMT</pubDate>
            <atom:updated>2025-01-28T17:01:59.936Z</atom:updated>
            <content:encoded><![CDATA[<p>We’re thrilled to announce that Ryan Isono has joined Maverick Ventures as a Managing Director. Ryan brings nearly a decade of experience in venture capital and finance, coupled with a stellar reputation as a thoughtful, strategic partner to founders. Based in San Francisco, Ryan will build on our decades of technology investing to accelerate our partnerships with visionary founders building transformative companies.</p><p>The current pace of innovation fueled by AI advancements is unprecedented, evidenced by the abundance of opportunities for disruption and category creation. We are excited to double down on our efforts here with Ryan joining the team. His sharp instincts, intellectual depth, and dedication to founders will support Maverick’s mission of working with visionary companies from inception through IPO and beyond. As we spent more time with Ryan, these traits, in addition to his ability to identify exciting, yet non-consensus areas, stood out and were echoed by founders, investors and operators alike.</p><p>Ryan graduated from Harvard College and began his career in investment banking at Credit Suisse. Since then, Ryan has invested in all forms of B2B software, most recently as a Partner at Felicis, where he worked with rapidly scaling companies across sectors including PermitFlow, n8n and more.</p><p>We couldn’t be more excited for Ryan to join the Maverick team and partner with the next generation of founders reinventing industries and creating new ones. If you’re building, especially in overlooked and non-obvious areas, Ryan would love to hear from you.</p><p>Welcome, Ryan!</p><p><em>The views expressed herein are solely the views of the author(s) and are not necessarily the views of Maverick Capital, Ltd. or any of its affiliates. They are not intended to provide, and should not be relied upon for, investment advice.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=19fb692c0125" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bringing Nutrition Care to the Masses: Our Partnership with Nourish]]></title>
            <link>https://maverickventures.medium.com/bringing-nutrition-care-to-the-masses-our-partnership-with-nourish-395a500c8ac4?source=rss-41af1c295e83------2</link>
            <guid isPermaLink="false">https://medium.com/p/395a500c8ac4</guid>
            <dc:creator><![CDATA[Maverick Ventures]]></dc:creator>
            <pubDate>Thu, 28 Mar 2024 22:59:21 GMT</pubDate>
            <atom:updated>2024-04-01T15:14:46.060Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://www.businesswire.com/news/home/20240327148762/en/Nourish-Raises-35M-Series-A-to-Expand-Access-to-Registered-Dietitians"><strong>Nourish Raises $35M Series A to Expand Access to Registered Dietitians</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*LuxOKTK8Wf92BZ39.jpg" /><figcaption>Nourish founding team (left to right): Sam Perkins, Stephanie Liu, Aidan Dewar</figcaption></figure><p><a href="https://www.usenourish.com/home">Nourish</a> is making nutrition care accessible to the 150 million Americans who need it — and we are beyond excited to be part of that journey!</p><p>50 percent of Americans struggle with nutrition-related chronic diseases, but most of these patients are unable to access nutrition care. <em>Why?</em> Like so many other parts of healthcare, it is prohibitively expensive.</p><p>Despite clinical evidence proving that registered dietitians (RDs) can effectively prevent and treat the most prevalent chronic diseases, RDs are typically not covered by insurance. As a result, patients have no choice but to pay out-of-pocket for care. This prevents most people who need care from accessing it, while disproportionately impacting lower-income patients who are more likely to have poor nutrition.</p><p><strong>Enter </strong><a href="https://www.usenourish.com/home"><strong>Nourish</strong></a><strong>: a telemedicine platform that connects registered dietitians (RDs) to people who need care — all covered by insurance.</strong></p><p><em>How does it work?</em></p><p>With its proprietary matching algorithm, Nourish connects patients with RDs whose expertise best meets their needs. Nourish enables patients and RDs engage in virtual and asynchronous care. RDs can focus on delivering care, while Nourish handles heavy administrative lifts like contracting with insurance plans to bring them in-network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/1*rOkCjnidgYv2-egjhoUfVA.png" /></figure><p><em>Why are we excited?</em></p><p><strong>1. Healthcare is broken, and the only path forward is technology-driven, consumer-centric solutions</strong></p><p>The U.S. spends twice as much per person on healthcare compared to the average among our peer nations. Total healthcare spending reached $4.7 trillion in 2023, which accounted for over 17 percent of GDP. According to the CDC, 75 percent of that spending went towards chronic disease. Despite this massive spending, <a href="https://time.com/6279937/us-health-care-system-attitudes/">73 percent of consumers</a> feel failed by the healthcare system.</p><blockquote><strong><em>We believe technology solutions that prioritize consumer experience are key to solving healthcare costs and patient dissatisfaction.</em></strong></blockquote><p>Covid-19 opened the floodgates to digital-first, virtual care delivery methods. Though virtual care isn’t right for all patients and all conditions, there is no doubt that it is often a better alternative to traditional brick &amp; mortar healthcare. A J.D. Power consumer <a href="https://www.jdpower.com/business/press-releases/2022-us-telehealth-satisfaction-study">survey</a> found that 67 percent of patients received virtual care in 2022 and 94 percent of those patients indicated they would likely use telehealth medical services in the future.</p><p><strong>2. Nutrition care is at an inflection point</strong></p><p>There are 19 concurrent phase 3 GLP-1 trials. The market is gearing up for an influx of new drug launches over the next few years. The bottleneck is quickly becoming access and cost. Will payers and employers cover GLP-1s, and under which circumstances?</p><p>Furthermore, a recent report published by <a href="https://www.reuters.com/business/healthcare-pharmaceuticals/most-patients-using-weight-loss-drugs-like-wegovy-stop-within-year-data-show-2023-07-11/">Reuters</a> found that 2 out of 3 people stop taking GLP-1s after one year. And once patients stop treatment, they gain back weight more quickly than they lost it because of the extreme drop in muscle mass.</p><blockquote>“Payers will likely soon require nutrition support ahead of or in conjunction with GLP-1 prescriptions in order to mitigate costs of long-term medication usage.” — Sam Perkins, co-founder, President &amp; COO of Nourish</blockquote><p>We therefore believe that solutions addressing 1) GLP-1 retention and 2) post-GLP-1 care for patients who discontinue treatment will be winners in this space.</p><p>Nourish is uniquely positioned to win by helping patients implement diet and lifestyle changes whether on a GLP-1 or not.</p><p><strong>3. Standout growth and unit economics</strong></p><ul><li>Nourish’s business model hits a rare sweet-spot: efficient D2C go-to-market coupled with $0 out-of-pocket spend for consumers. In other words, Nourish offers a consumer-grade product and experience at a $0 cost for most users.</li><li>This $0 cash-pay model, coupled with a delightful tech-enabled consumer experience, leads to best-of-breed retention and engagement statistics. Both patients and RDs are extremely loyal to Nourish. This bodes well for unit economics — Nourish experiences unnaturally high provider and patient retention, acquisition payback periods, and LTV/CAC ratios.</li><li>Nourish enjoys a virtuous cycle: attracting high quality RDs leads to strong consumer engagement and therefore more payer contracts…which in turn leads to the ability to attract more high quality RDs and patients for them to treat…you get the gist. With an existing network of dietitians and patients across all 50 states, Nourish is situated for rapid growth in 2024. We are confident in their ability to do this because of the flywheel they’ve ignited:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/356/1*PcF6F3C6RkXeDqgO3N8CNQ.png" /></figure><p><strong>4. Poised for AI-powered growth</strong></p><p>Nourish has built a solid foundation from which to grow. In 2024 and beyond, the team will continue investing in AI-powered solutions to enhance its personalized care plans and reduce administrative burden for RDs. In addition, Nourish is integrating “food as medicine” into its program by empowering RDs to “prescribe” diets that are personalized to each patient’s needs and preferences. In addition, Nourish will turn to partners — including insurance companies and grocery delivery stores, to make an impact in food insecurity and social determinants of health.</p><p><strong>5. Best-in-class team</strong></p><p><a href="https://www.linkedin.com/in/davidsamperkins/">Sam</a>, <a href="https://www.linkedin.com/in/aidandewar/">Aidan</a>, and <a href="https://www.linkedin.com/in/stephaniehliu/">Stephanie</a>, co-founders of Nourish, exhibit the many traits we look for in founders. They are scrappy, driven, and have complementary skills — ranging from technical to commercial. We have been incredibly impressed with their strategic intuition, their grasp of the nuances in this space and in their model, and their proven ability to execute. Three years into their journey building Nourish, they have scaled the business rapidly while delighting their customers and building an exceptional team in both corporate and clinical roles.</p><p>Welcome to the Maverick family, Nourish team — we are honored to work with you!</p><p><em>The views expressed herein are solely the views of the author(s) and are not necessarily the views of Maverick Capital, Ltd. or any of its affiliates. They are not intended to provide, and should not be relied upon for, investment advice.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=395a500c8ac4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Funding the New Standard in Maternity Care]]></title>
            <link>https://maverickventures.medium.com/funding-the-new-standard-in-maternity-care-96fd47390565?source=rss-41af1c295e83------2</link>
            <guid isPermaLink="false">https://medium.com/p/96fd47390565</guid>
            <category><![CDATA[healthcare]]></category>
            <category><![CDATA[healthcare-technology]]></category>
            <category><![CDATA[venture-capital]]></category>
            <category><![CDATA[womens-health]]></category>
            <dc:creator><![CDATA[Maverick Ventures]]></dc:creator>
            <pubDate>Thu, 22 Feb 2024 21:38:17 GMT</pubDate>
            <atom:updated>2024-02-22T21:55:15.061Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Maverick Ventures co-leads $28M Series B investment in Oula Health</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*TYxm9C7HNyLKJX4ELwPMKQ.jpeg" /><figcaption>Oula’s Co-Founders (and the next generation): Adrianne Nickerson (CEO), Elaine Purcell (COO), and Joanne Schneider Demeireles (CXO)</figcaption></figure><h3>Maverick Ventures and Oula Health</h3><p>Our team at Maverick could not be more excited to partner with <a href="https://oulahealth.com/">Oula</a>, the New York-based startup reimagining maternity care. Oula operates maternity clinics and offers tech-driven, team-based services. Unlike many traditional healthcare models, Oula puts patients in the driver’s seat of their own care journey. Oula has proven its model in New York City, having delivered 1,500 babies at Mount Sinai while outperforming benchmark outcomes (including lower rates of caesarean sections and preterm births). As part of this <a href="https://www.prnewswire.com/news-releases/oula-the-modern-maternity-care-clinic-combining-midwifery-and-obstetrics-raises-28m-in-series-b-funding-to-expand-nationally-and-offer-new-services-302066130.html">$28M Series B raise</a>, Oula will expand into new markets and launch additional tech-enabled services.</p><p>Like my colleagues who have known <a href="https://www.linkedin.com/in/adriannenickerson/">Adrianne</a> — Oula’s CEO — since the company’s seed round, I was impressed with the scale of her ambition from our first meeting. Instead of band-aiding a broken system, she wanted to rebuild maternity care from the ground up. After spending time with Adrianne and her co-founders <a href="https://www.linkedin.com/in/elainepurcell/">Elaine</a> and <a href="https://www.linkedin.com/in/schneiderjoanne/">Joanne</a>, we knew they were the right people for the job. We can’t wait to partner with them as Oula sets a new standard in women’s health.</p><h3>What makes Oula different?</h3><p>When I visited Oula’s Soho clinic, I was welcomed into a space that felt more like a homey living room and nothing like a sterile hospital. The waiting room was full of expecting families, and the clinic was staffed with OBs and midwives who were funneling in and out of exam rooms in their scrubs. I met an ultrasound technician who had just finished scanning a patient. The clinic exudes comfort and warmth, but also maintains its identity as a rigorous medical center. Achieving this balance is delicate but critical.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*br-1HM-FOXTd9SOS" /><figcaption><em>Oula’s Soho Clinic at 202 Spring St.</em></figcaption></figure><p>When patients leave Oula’s physical space, they continue to engage with their care team through a consumer app and virtual wraparound services, including maternal health and group support. Oula’s providers spend twice as much time with patients compared to a typical OB.</p><p>Oula provides families with a single “home” — both in-person and virtual — for all prenatal and postpartum care. Oula babies are delivered at hospitals, which ensures families have access to all forms of necessary care during childbirth. There is no membership fee, and patients can pay for Oula with insurance — including most major commercial insurance plans and Medicaid.</p><p><em>The result?</em></p><ol><li><strong>Patients love it.</strong> Oula has a Net Promoter Score (NPS) of 90+ compared to 38 for the average healthcare provider.</li><li><strong>Payors love it. </strong>Oula lowers costs by reducing caesarean sections (C-sections) by 26%, preterm births by 61% and low birth weights by 50%.</li><li><strong>Providers love it.</strong> Oula helps hospitals generate patient loyalty and improve profitability by increasing volumes of maternity patients and decreasing costs.</li></ol><h3>What’s a midwife, anyways?</h3><p>If the word “midwife” sparks an image of a home water birth for you, you’re not alone. I was surprised to learn that 94% of midwife births happen in a hospital (<a href="https://www.ncbi.nlm.nih.gov/books/NBK555484/#pz96-2">Birth Settings in America</a>).</p><p>This perhaps shouldn’t have come as a surprise considering midwives are highly trained medical specialists. It takes 6–8 years to become a Certified Nurse-Midwife (CNN) which includes earning a master’s degree and passing a rigorous Midwifery Certification Board licensing exam. The Commonwealth Fund <a href="https://www.commonwealthfund.org/publications/issue-briefs/2023/may/expanding-role-midwives-address-maternal-health-crisis#:~:text=A%20recent%20analysis%20found%20that,and%2026%20percent%20of%20stillbirths.">reported</a> that midwives can deliver 80% of essential maternal care when integrated into a health system.</p><p>I now understand that the bias against midwives is a uniquely American one. In fact, midwifery is standard in many other developed countries. In wealthy European nations, there are 25–66 midwives per 1,000 births, compared to 4 per 1,000 in the U.S.</p><p>These same countries eat our lunch on outcomes. They outperform the U.S. on 56 — yes, 56 — different outcome measures, including maternal mortality and preterm birth rates. (<a href="https://medicine.yale.edu/news-article/the-value-of-midwives-during-prenatal-care-and-birth/#:~:text=They%20report%20that%20midwifery%20care,and%20reduced%20interventions%20in%20labor.">Yale study</a>)</p><p>Across the pond, the ratio of midwives-to-OB/GYNs is 11 times higher in the U.K. than in the U.S. Perhaps the most telling example: in comparison to the U.S., New Zealand has approximately 11 times more midwives per 1,000 births and a 10 times lower maternal mortality rate.</p><p>The need for more midwives is a global phenomenon and even more pronounced in developing countries. Based on WHO data from 2021, there is a shortage of 900,000 midwives worldwide. Midwives and nurses together make up 50% of the global shortage in health workers. “Fully investing in midwives by 2035 would avert roughly two-thirds of maternal, newborn deaths and stillbirths, saving 4.3 million lives per year,” the WHO <a href="https://www.who.int/news/item/05-05-2021-new-report-sounds-the-alarm-on-global-shortage-of-900-000-midwives#:~:text=An%20analysis%20conducted%20for%20this,4.3%20million%20lives%20per%20year.">reported</a>.</p><p><em>Among the 11 developed countries listed below, the U.S. lags on midwifery supply…</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/983/0*xfyRfb4g8PxxUKSW" /><figcaption>Source: Commonwealth Fund</figcaption></figure><p><em>…while performing worst on maternal mortality rates:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/0*c-6fMuWXVhmd7BDe" /><figcaption>Source: Commonwealth Fund</figcaption></figure><h3>What about OB/GYNs?</h3><p>Midwives work most effectively when they are part of a collaborative care team that also includes obstetricians. Furthermore, high-risk pregnancies — which occur in 6–8% of patients — require an OB and other specialists for prenatal care and delivery.</p><p>Data shows that midwife-assisted care is associated with better outcomes, including <a href="https://time.com/6249274/elaine-welteroth-midwifery-care-maternal-health/">80% lower</a> maternal and newborn mortality rates, and higher patient satisfaction. A recent <a href="https://medicine.yale.edu/news-article/the-value-of-midwives-during-prenatal-care-and-birth/#:~:text=They%20report%20that%20midwifery%20care,and%20reduced%20interventions%20in%20labor.">Yale study</a> found that first-time mothers were 74% less likely to go through induced labor when midwives were on their care team compared to mothers who gave birth at medical centers without midwives. They were also 75% less likely to receive oxytocin augmentation and 12% less likely to deliver by cesarean.</p><p>Oula offers the best of both obstetrics and midwifery and is positioned to become the largest employer of midwives in the U.S.</p><h3>Why now?</h3><p>We believe Oula is the early leader in a space experiencing a confluence of:</p><ol><li><em>Shifting consumer preferences in favor of low-intervention births</em></li><li><em>A shortage of OBs leading to bottlenecks in prenatal care</em></li><li><em>Increased attention from payors on poor maternal outcomes</em></li><li><em>Competition between hospital systems to acquire maternity patients</em></li><li><em>Attractive margins driven by technology and lower fixed costs</em></li></ol><p>I’ll break these down below:</p><p><strong><em>(1) Shifting consumer preferences in favor of low-interventions births</em></strong></p><ul><li>In a 2018 <a href="https://www.chcf.org/wp-content/uploads/2018/09/ListeningMothersCAFullSurveyReport2018.pdf">survey</a>, 74% of women said they wanted a less medicalized approach to birth, up from 45% in 2002.</li><li>Between 2003 to 2018, U.S. hospital births dropped overall by 9%, while midwife-attended hospital births increased by 12% and physician-attended hospital births decreased by 10% (<a href="https://www.ajog.org/article/S0002-9378(20)30382-3/fulltext">study</a>).</li><li>In our diligence, we surveyed pregnant women and new moms who indicated that they are looking for less medicalized, more consumer-friendly options.</li></ul><p><strong><em>(2) A shortage of OBs leading to bottlenecks in prenatal care</em></strong></p><ul><li>Hospitals are losing maternity patients because of labor capacity constraints leading to long wait times for prenatal visits.</li><li>Research indicates that the U.S. faces an 8,000 shortage of OBs, which is expected to grow to 22,000 by 2050 (<a href="https://www.doximity.com/studies/ob-gyn-workforce-study/2018">Doximity)</a>.</li><li>A 2022 March of Dimes <a href="https://www.marchofdimes.org/maternity-care-deserts-report">report</a> classified 36% of states as “maternity deserts”, meaning they have no obstetric hospitals or birth centers and no obstetric providers.</li><li>The aging OB workforce is increasingly opting out of labor &amp; delivery to focus on GYN care. Only 16% of all U.S. OB/GYNs are 40 years of age or younger, while 36% are 55 years or older (<a href="https://press.doximity.com/reports/ob-gyn-workload-and-potential-shortages-2018.pdf">Doximity</a>).</li></ul><p><strong><em>(3) Increased attention from payors on poor maternal outcomes</em></strong></p><ul><li>Per <a href="http://cms/">CMS</a>, “Despite spending more per capita on maternal health care than any other nation, the U.S. has disproportionately high rates of adverse pregnancy outcomes compared to other high-income nations.”</li><li>Our diligence calls with payors indicated that maternal health — particularly high C-section rates — is a top priority for them.</li><li>On December 15, 2023, CMS launched a new care delivery model called <a href="https://www.cms.gov/priorities/innovation/innovation-models/transforming-maternal-health-tmah-model">Transforming Maternal Health</a> (“TMaH”) to address maternal health outcomes and increase pregnancy and postpartum care access. The model calls for a “whole-person approach to pregnancy, childbirth, and postpartum care access…address barriers that limit access to valuable resources, such as midwives, doulas, and perinatal Community Health Workers (CHW).”</li></ul><p><strong><em>(4) Competition between hospital systems to acquire maternity patients</em></strong></p><ul><li>Hospitals compete fiercely for maternity patients because this episode of care is viewed as a critical time to acquire a family for downstream care.</li><li>Hospitals are looking for partners to introduce new midwifery care models. We spoke to various hospitals that have tried, and failed, to build this themselves.</li></ul><p><strong><em>(5) Attractive margins driven by technology and lower costs</em></strong></p><ul><li>Oula’s proprietary technology stack and lower fixed cost structure enable higher margins and profitability for both Oula and its hospital partners.</li><li>I believe the market will see strategic value in a business that owns the patient relationship and delivers hybrid care (both in-person and virtual) — and can move the needle on consumer experience, outcomes and cost. This lends itself to a tech-enabled services model like Oula’s. We’ve studied many such models at Maverick, and we think Oula’s is best in class.</li></ul><h3>Why Maverick?</h3><p>Maverick co-led this round with Revolution Ventures, alongside GV and existing investors including 8VC, Chelsea Clinton and Female Founders Fund. Maverick was an early investor in One Medical, and we are eager to apply our experiences as Oula scales its clinics into new regions and launches additional services. This investment fits into Maverick’s strategy of expanding our East Coast presence and partnering with companies that can scale their technology and services into broader geographies, including emerging markets where the need for innovative women’s health services is in high demand.</p><p>On a personal level, I am thrilled to be part of Maverick’s first investment in the women’s health space. Before joining Maverick, I was on the founding team of a venture-backed healthcare startup building software and hardware for fertility clinics and saw first-hand the massive need for better reproductive care in the U.S. and abroad. Oula is helping to fill this gap by building a patient-centered, midwife-driven maternity care model, and I am honored to be part of that journey.</p><p><em>The views expressed herein are solely the views of the author(s) and are not necessarily the views of Maverick Capital, Ltd. or any of its affiliates. This article is not intended to provide, and should not be relied upon for, investment advice.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96fd47390565" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What the History of Software Development Tells Us about the Hurdles to Enterprise Adoption of LLMs]]></title>
            <link>https://maverickventures.medium.com/what-the-history-of-software-development-tells-us-about-the-hurdles-to-enterprise-adoption-of-llms-c96bc968456d?source=rss-41af1c295e83------2</link>
            <guid isPermaLink="false">https://medium.com/p/c96bc968456d</guid>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Maverick Ventures]]></dc:creator>
            <pubDate>Tue, 06 Feb 2024 20:07:54 GMT</pubDate>
            <atom:updated>2024-02-13T05:33:04.420Z</atom:updated>
            <content:encoded><![CDATA[<p>By <a href="https://www.linkedin.com/in/joohoyeo/">JooHo Yeo</a> and <a href="https://www.linkedin.com/in/matthew-kinsella/">Matt Kinsella</a></p><p>Generative AI is predicted to add up to $4.4T of value to the global economy annually, according to a <a href="https://www.mckinsey.com/capabilities/mckinsey-digital/our-insights/the-economic-potential-of-generative-ai-the-next-productivity-frontier#introduction">McKinsey study</a>. However, enterprise adoption of the technology has been slow and mostly in the experimentation stage. A <a href="https://menlovc.com/2023-the-state-of-generative-ai-in-the-enterprise-report/">recent survey by Menlo Ventures</a> showed that enterprise investment in generative AI is still surprisingly small ($2.5 billion) compared to those of traditional AI ($70 billion) and cloud software ($400 billion).</p><p><strong>So, what is preventing enterprises from mass adoption of LLMs?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/1*pCsa-dS61gah76_siaMBOA.png" /><figcaption>Generated by DALL-E</figcaption></figure><p>To answer this question, we looked to history as well as 35 ML engineers working on LLM applications in enterprises.</p><p>We believe a shortage of quality options for engineers to test and evaluate models and applications throughout the LLM App Development Life Cycle is a significant blocker to adoption and thus, a great opportunity for startups to address.</p><h3>Traditional Software Development Life Cycle (SDLC)</h3><p>The advent of modern software development processes can be traced back to the 1970s when Dr. Winston Royce made the first reference to the Waterfall model in his paper titled “Managing The Development of Large Software Systems.” This method defined software development into a series of sequential phases where each phase depended on the outputs of the previous one, similar to manufacturing processes of the time. The model quickly gained support from managers because of the structure it brought to software development, ensuring there was a clear spec to follow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*2gkXlKSLLO-cveDdkH7bdQ.png" /><figcaption>Source: “Managing the Development of Large Software Systems” by Dr. Winston Royce</figcaption></figure><p>However, the model had its limitations such as the inability to take feedback from users and risking the entire process by delaying testing. Software development also started to change with the rise of the Internet. As expectations grew to bring software to market faster (along with the development tools and platforms to enable the increased pace), the rigidity and overhead of the Waterfall model became a hindrance to developer velocity. In 2001, a group of 17 developers published the “Manifesto for Agile Software Development,” a document that outlined the 12 principles of Agile development. This model revolutionized software development by emphasizing iterative releases that adapt to evolving requirements. This shift discarded rigid structures in favor of enhanced collaboration between development teams and business stakeholders. Shipping Minimum Viable Product (MVP) and iterating became a core part of the Agile methodology.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*z6wZB0Y_4aD0mklofneQfA.png" /><figcaption>Source: TrustRadius</figcaption></figure><p>As the Agile methodology took off, a new challenge arose: operations had to be able to deploy and deliver software to users at a rapid pace. Based on his observations of the need for more synergy between development and operations teams, Patrick Debois coined the term “DevOps” in 2009. The objective of DevOps was an extension of the Agile methodology: to facilitate continuous delivery of features, bug fixes, and updates. DevOps necessitated a harmonious collaboration between traditionally separated units — development, quality assurance, operations, security, and more. In order to bring these functions together and deliver continuously to users, new steps were required.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/651/1*IvCTuzI--O8VCFaH3r8Y4w.png" /><figcaption>Source: Wikipedia</figcaption></figure><p><strong>The rise of DevOps tools contributed to </strong><a href="https://a16z.com/why-software-is-eating-the-world/"><strong>software eating the world</strong></a><strong> (as Marc Andreesen phrased it) </strong>in the 2010s. By the end of the 2010s, “every enterprise became a software company.” The global spend on enterprise software grew 129% over the decade, from $269B in 2011 to $615B in 2021. DevOps tools fueled the developer productivity needed to support the cloud and mobile platform shifts. With client expectations for reliability rising, tools that addressed testing and monitoring needs were especially critical for growing enterprise adoption.</p><p>As the software development life cycle matured, each step gave way to successful startups. Here are just a few:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*hpU0pdLtr5JHOB4lNvas2g.png" /></figure><h3>Machine Learning Development Life Cycle (MLDC)</h3><p>As Machine Learning came into mainstream software development, it had different requirements from traditional software development, making it challenging to apply and scale ML solutions to business applications. In 2015, the term “MLOps” was coined in “Hidden Technical Debts in the Machine Learning System,” a research paper that brought the aforementioned problems into the spotlight.</p><ul><li>Data collection, annotation, and curation were introduced as critical steps. Labeled data became crucial to training and evaluating models.</li><li>Model training and evaluation across various model types required iteration and collaboration among stakeholders. Models need to be assessed for their accuracy and compared against each other in an objective manner.</li><li>Post-deployment, models need to be continuously monitored for reliability and accuracy, whereas for traditional software, the need for maintenance was solely on reliability. Various changes in the distribution of the training, validation, and production data can happen overtime degrading the performance of the predictions.</li></ul><p>A fresh set of companies arose to support these new developer needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*AdcPRvZzkMZJBsD2a0bAgg.png" /></figure><h3>LLM App Development Life Cycle (LLMADC)</h3><p>Akin to Machine Learning, LLMs have changed how applications are developed. Most notably, the data barrier has been knocked down. Developers no longer need upfront troves of labeled data to train ML models. The pre-trained LLMs are sufficient for demos and even some MVPs. However, data is still necessary for evaluating not only the outputs of the customized models, but also those of the overall LLM application.</p><p>Text generation tasks are also much harder to evaluate than classification tasks. Classification tasks are either correct or incorrect, whereas generated text has multiple dimensions to be evaluated on like factual consistency, relevance, and coherence. It is a difficult task to not only comprehensively assess output quality, but also weigh them systematically to be helpful in the comparison.</p><p>As LLMs can take unstructured data as inputs, the applications run the risk of behaving unexpectedly based on unexpected user inputs. The need to monitor and evaluate how the user interacts with the LLM application becomes even more important.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*_ZKtObbb-okYHUnb_C2hFw.png" /></figure><h3>What Actually Is Testing, Evaluation, and Monitoring?</h3><p>Development teams need methods to assess and enhance the quality of their projects. Testing tools help developers gather information about their software’s functionality, usability, and performance. Some examples of testing tools include tools for integration testing, bug tracking, automation testing, and performance testing. Beyond testing, these tools encompass broader test management solutions that feature collaborative functions that facilitate effective communication between testers and developers.</p><p>Evaluation tools help teams during the development phase to deploy different versions and choose the best variant to ship to production. Evaluation measures and compares various business metrics that are important to the company, like conversion rates or customer engagement. This allows software engineering teams to translate their technical progress into business value, creating a common language between all teams.</p><p>Monitoring and observability tools help ensure that software run reliably in production. Monitoring refers to watching a set of metrics to detect failures and performance issues. These performance metrics include network traffic, latency, and error rates. Monitoring tools provide information about what happened. Observability takes the concept of monitoring a step further and equips engineers with logs and traces to understand the root cause of problems. Observability tools offer the insights into why the issue occurred and how to fix it.</p><h3>Testing, Evaluation, and Monitoring Have Historically Been Large Markets</h3><p>Testing, evaluation, and monitoring have always been critical for companies that offer software products. While all steps of the software development process are important, ultimately the objective of developing software is to deliver value to users reliably. As a reflection of its significance, testing has been a large portion (<a href="https://www.statista.com/statistics/500641/worldwide-qa-budget-allocation-as-percent-it-spend/">20–30%</a>) of enterprise software development budgets.</p><p>Especially with the shift of software testing from manual to automated, software unicorns like Browserstack, Postman, and SmartBear have been able to capitalize on increasingly faster and comprehensive testing needs. The automated testing market is $15B and is projected to grow at a CAGR of 18.8%.</p><p>Beyond testing during the software development process, it is essential for businesses to monitor their software post-deployment to make sure it is operating as expected. As cloud infrastructures have gotten more complex, observability solutions have become essential in debugging reliability and latency issues as well as removing them proactively. Observability products can also help businesses better analyze and thus, reduce their infrastructure spending. According to <a href="https://www.honeycomb.io/blog/how-much-should-my-observability-stack-cost">Honeycomb</a>, companies spend up to 30% of their infrastructure costs on observability. Several multi-billion dollar companies have emerged in this space including Datadog, New Relic, and Dynatrace.</p><p>Companies are looking further than just ensuring functionality and now measuring the value their products deliver and quantifying the ROI of their software projects. The market for online evaluation software, including AB testing software, is over $1B with companies like LaunchDarkly, Optimizely, Split.io, and Statsig.</p><p>For LLM app development, evaluation and monitoring will continue to be critical and new companies will spring up in this space. To understand what the future looks like for the LLM evaluation, we surveyed 25 ML practitioners working at large companies<a href="#_msocom_1">[JO1]</a> <a href="#_msocom_2">[JY2]</a> (with over 20 developers) that are developing genAI-powered products. In addition to the survey, we also conducted interviews with 10 engineers in this space.</p><h3>Survey Results</h3><p>We asked experts to rank their top barriers to deploying LLMs to production from one to six (one being highest and six being lowest).</p><p>More than a third of the experts cited <strong>“evaluating model quality”</strong> as the top barrier for deploying LLMs to production. When looking at the average rank of each barrier, we see the same sentiment of model quality evaluation being the number one concern.</p><p>These quantitative results were in line with the learnings from our expert interviews where engineers emphasized the importance of mitigating hallucination, especially for developing software in regulated industries like finance and healthcare which have low tolerance for error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*FDanRJ1UpQZXUc5AFiq6Fw.png" /></figure><p>Barriers of Deploying LLMs to Production (average rank)</p><p>1) Evaluating model quality (2.04)<br>2) Data privacy and security (3.08)<br>3) Testing and QA (3.48)<br>4) Costs of training and serving (3.72)<br>5) Model reliability (3.84)<br>6) Data pre-processing for ingestion (4.92)</p><p>LLM evaluation is currently a <strong>time-consuming process</strong>. When asked what percentage of their time they spend on testing and evaluation, respondents said that they spend 35%. In addition, the top issues with their current LLM evaluation process were that it takes “too much engineering resources” (76%) and “too much time” (68%).</p><p>There are two reasons why evaluation is currently a time sink.</p><p>1) The process is manual. At 64%, offline manual evaluation (done by ML engineers or team members) was by far the most common form of evaluating LLMs noted by practitioners. This is consistent with the many times we heard of “eyeballing” as a “good enough” solution from our interviews. The lack of tooling and standard process also leads to repetitive work that makes the experience even more time consuming.</p><p>2) Stitching together tools ad-hoc is time consuming and unscalable. 40% of respondents said their organization uses an internally developed solution for testing and evaluation, while only 4% said they are evaluating external solutions. The lack of available external options have driven companies to build their own solutions. For example, a large tech company spent nine months of precious time of eight engineers to develop an internal evaluation platform. To build and maintain LLM evaluation tooling can be a significant engineering lift.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*YqtcUBF4xY1T14Avfp3ZsQ.png" /></figure><p>From selecting LLMs to monitoring the application in production, <strong>evaluation affects each stage of the LLM App Development Life Cycle</strong>. 80% of respondents said they perform evaluation at more than one stage of the process, with 68% replying that they evaluate during at least three out of the six major steps listed below. In LLM app development, evaluation and testing is no longer a single step, but continuously executed.</p><p>When asked why LLM evaluation is important, experts responded with user trust and continuous improvement as the top two responses (84% and 76% respectively). While utilizing the power of evaluation to enable iteration ranked highly, only 40% use evaluation to make shipping decisions, pointing to a gap of processes and tools today. In fact, the highest response to the question of the effect of the lack of testing and evaluation was lack of confidence in the product (86%). While LLM-based app development is still in its early innings, we believe that evaluation will bring the confidence to supercharge innovation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*sEKWsgQrGG62yXZYnjDVIA.png" /></figure><p>As mentioned before, very few companies are currently using external tools for evaluation.</p><p>Through the survey and interviews, we tried to get an understanding of <strong>what their criteria was for off-the-shelf solutions</strong>.</p><p>What we heard consistently was that engineers are searching for an all-in-one solution that covers everything from testing in development to monitoring in production. “Evaluation and observability” ranked the highest (80%) when asked what qualities are considered when considering external options. Buyers are not looking for point solutions, but rather an end-to-end solution that can support all parts of the workflow. We also believe that the practitioners prioritize observability because it is harder to predict how users will interact with LLM applications due to the unconstrained input space. Therefore, while offline evaluation to inform engineers during the development process are table stakes, online evaluation capabilities will also be necessary.</p><p>The second highest desired quality was customizability (72%). While default test sets and metrics are helpful, practitioners are looking for solutions where they can also add testing and evaluation methods specific to their business needs. This result is unsurprising as many engineers have voiced frustrations about the unreliability of generic benchmarks to assess the potential of LLMs in their specific use cases. For external solutions, buyers will be looking for industry and application (i.e., summarization, Q&amp;A, etc.) specific data sets, test cases, and evaluation metrics. Generating test cases based on subject matter expert inputs and actual user inputs will also help meet these customizability needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*5mfkeJcQ8lUmCWf7S0MHkQ.png" /></figure><h3>LLM Evaluation Market Size Estimation</h3><p>While we are still at the beginning of LLM app development, we believe that there is a multibillion dollar market opportunity in evaluating LLMs based on our study of software development history and practitioner input.</p><p>Based on our survey, 96% of respondents indicated willingness to pay for an external LLM evaluation product. Approximately half of the respondents were in the $10–50K and $50–100K buckets, which was consistent with the contract sizes that LLM evaluation startups we spoke with were closing with their smaller-sized clients. LLM evaluation software providers are observing six-figure contracts as they start to engage with commercial and enterprise clients, which is also reflected in the survey responses as over a third of respondents indicated that they are willing to pay six to seven figures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*kLfESuvEebWKzYBZyZBjRA.png" /></figure><p>To estimate the market size of LLM evaluation, we calculated the weighted average of the survey responses, which was approximately <strong>$326K</strong>. There are many different ways to think about how large the target customer base could be:</p><p>1) According to <a href="https://www.gartner.com/en/newsroom/press-releases/2023-10-11-gartner-says-more-than-80-percent-of-enterprises-will-have-used-generative-ai-apis-or-deployed-generative-ai-enabled-applications-by-2026">Gartner</a>, more than 80% of enterprises will have used genAI APIs or models, and/or deployed genAI-enabled applications in production environments by 2026. If we define enterprises as companies with over 1000 employees, there are around 8000 enterprises in the USA.</p><p>2) According to <a href="https://www.godfrey.com/insights/3-considerations-navigating-rapidly-evolving-ai-landscape#:~:text=According%20to%20Tracxn%2C%20there%20are,simple%20places%20businesses%20can%20begin.">Tracxn</a>, there are approximately 14,700 AI companies in the USA.</p><p>3) During <a href="https://www.microsoft.com/en-us/investor/events/fy-2024/earnings-fy-2024-q1.aspx">Microsoft’s October 2023 earnings call</a>, Satya Nadella said that more than 37,000 organizations subscribe to Github Copilot for Business.</p><p>Taking the most conservative approach of option one (80% of 8000 enterprises), we can estimate a domestic market size of approximately <strong>$2.1B.</strong></p><p>Another methodology of estimating market size is by looking at enterprise budgets. Menlo Ventures estimated that enterprises spent approximately $2.5B on genAI in 2023, while all AI spend was about $70B. Since companies have allocated 20–30% of software development budgets for quality assurance and testing in the past, we can expect the LLM evaluation budget to be in the $500M-750M range. If the enterprise LLM spend grows at a double-digit CAGR following the LLM market growth rate, we can expect the LLM evaluation market size to surpass $1B in the next few years.</p><h3>Why LLM Evaluation is Holding Up Adoption:</h3><p>Evaluation is the bottleneck throughout the development life cycle. Compared to the sexier parts like model training and deployment, testing/evaluation may seem boring, but they are critical to supporting each step of the LLM app development life cycle, from model selection to experimentation and finally, to monitoring. Companies are looking for solutions that can provide capabilities at each step over point solutions. Consistency is key for versioning and unified form of assessment.</p><ul><li><strong>Langfuse </strong>is a startup offering testing and evaluation solutions from the proof-of-concept phase to full rollout. Their model- and framework-agnostic product tracing feature makes it easier for engineers to debug throughout the development process. While some might argue that it is too early for enterprises to think about observability, Langfuse is building a data architecture to support the whole LLM app development life cycle for the needs that will inevitably come.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*8R7kIGK2vLxZRDK0Fplv3A.png" /></figure><p>Generic LLM benchmarks are insufficient to evaluate for specific business use cases. Even disregarding customizations, a LLM may perform very differently across general tasks vs specific business use cases. To borrow <a href="https://generallyintelligent.substack.com/p/is-your-language-model-any-good-nobody">David Hershey’s analogy</a>, benchmarks are like SATs: they are helpful in assessing one’s skills in a range of subjects. But what matters to enterprises are the job interviews: how the candidate performs on tasks pertinent to the specific business use case in that environment.</p><ul><li>Startups like <strong>Patronus AI </strong>can give enterprises more confidence in deploying LLMs by<strong> </strong>generating adversarial, comprehensive test cases and scoring model performance in specific, real world scenarios.</li><li>Industry-specific datasets could be an initial edge for the evaluation providers. Patronus AI<strong> </strong>released FinanceBench (a high quality, large-scale set of 10,000 question and answer pairs based on publicly available financial documents) to be a first line of evaluation for LLMs on financial questions.</li></ul><p>Among offline evaluation methods, human evaluation is the most common, but manual review is expensive, slow, and vulnerable to human error — all factors which slow down the development process. Deterministic and model-driven evaluation can be cheaper, faster, and provide unbiased alternatives. Shortening iteration cycles can accelerate software development.</p><ul><li>As mentioned in the survey analysis section, companies that have built internal solutions have experienced the large time and human capital investment required to build a comprehensive evaluation platform. Maintaining these internal tools in the LLM world, where models, architectures, and even evaluation methods frequently change, requires significant expertise and overhead. Buying an external solution can relieve development teams from these headaches.</li><li><strong>Confident AI </strong>provides a Pytest for unit testing LLM applications (called DeepEval), which supports various metrics such as hallucination, answer relevancy, RAGAS, etc. Confident AI is also implementing state-of-the-art (SOTA) LLM-based evaluation approaches, like <a href="https://arxiv.org/pdf/2303.16634.pdf">G-Eval</a>, and building proprietary multi-agent-based evaluation methods (JudgementalGPT) to address limitations of LLM-based evaluation like <a href="https://arxiv.org/pdf/2306.05685.pdf">position and verbosity biases</a>.</li><li>Currently, there is a block in running automated evals at scale with SOTA models due to cost and latency concerns. Evaluation startups like <strong>HoneyHive AI</strong> are investing in R&amp;D for evaluation-specific foundation models that can be used to run as offline and online evaluators, delivering performance in line with GPT-4 at a fraction of the price and speed.</li></ul><p>The unpredictability of how users will engage with LLM products is a challenge for enterprise software, where there is little room for error. Online evaluation becomes especially important to understand how users engage with the product and get ahead of quality issues.</p><ul><li>Startups like <strong>Athina AI</strong> are now running topic classification on user queries to help clients understand how models perform live on different types of queries.</li><li>Ultimately, what matters to companies is how much value the LLM products are delivering to the users. To measure the impact of the LLM products on the business-wide KPIs, A/B testing platforms like <strong>Statsig</strong> and <strong>Eppo</strong> can be utilized.</li></ul><p>The evolved needs of various stakeholders within organizations require a new solution. ‍</p><ul><li>LLM applications are not only developed by traditional ML practitioners, but also increasingly by product engineering teams and generalist software engineers. Compared to traditional software development, non-technical stakeholders, like product managers, are also more easily able to plug themselves into the development process steps like prompt engineering and monitoring. Inputs of subject matter experts are also vital perpetually across the LLM app development cycle. The shift in workflows is addressed today by stitching together various disconnected tools such as spreadsheets, internal dashboards, and product analytics software, which causes inefficiencies and risks. Companies like HoneyHive AI are building a unified workflow addressing each of these stakeholders’ new workflow needs that enable seamless collaboration among all parties.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/1*G9uFZZ6_2o_8sCtFwyCHPA.png" /><figcaption>Source: HoneyHive</figcaption></figure><h3>Why Evaluation Will Become More Important Over Time:</h3><p>Based on our interviews, we have seen that engineers working at enterprises are currently focused on shipping MVPs that are driven by top-down influence.</p><ul><li>In order to iterate (to reduce costs, improve accuracy, and provide additional value to users), companies will need the ability to consistently track performance and experiment in parallel.</li><li>Articulation of ROI will be paramount for further investments.</li></ul><p>Rising complexity created by an increasing number of stakeholders and their diverse backgrounds (both technical and non-technical roles) will make eyeballing and tracking with spreadsheets impossible to scale.</p><p>LLM systems are “fragile” because of the quickness at which new models and architectures are introduced as well as other dependencies. Without rigorous tests at the unit and system level, things are bound to break and take up already scarce ML engineering time.</p><ul><li>Reliance on models that are pre-trained by other companies creates risks that enterprises are unlikely to take on. Enterprises will not want to rely on communication from model providers to understand these risks like model unavailability and updates to models.</li><li>To support evolving complexities of LLM application architectures, startups including <strong>UpTrain </strong>are going beyond offering basic evaluation of the quality and appropriateness of response and retrieved-context. They are building capabilities to judge the quality of AI-driven conversations and the correctness of AI agent actions.</li></ul><p>While proprietary models like OpenAI and Anthropic dominate, the trend is moving towards companies using multiple models (even startups that route prompts to the best model based on use case, like <strong>Martian,</strong> are starting to pop up). The <a href="https://www.theinformation.com/articles/openais-customers-consider-defecting-to-anthropic-google-cohere">OpenAI scare</a> last November led to hundreds of OpenAI customers reaching out to its rivals over just that weekend.</p><ul><li>With multiple models used, the need for choosing, experimenting, and monitoring will increase. The increasing complexity of model choice and architecture lends to more need for evaluation.</li></ul><h3>Market Map of LLM Evaluation Startups</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7ot7nNqNWIu7Tw8X8yyGsw.png" /></figure><p>This is a non-comprehensive visualization of startups that are working in the LLM Evaluation space bucketed by problems they address. The line denotes a distinction between development and production stages. Security and Routing were also included as needs adjacent to evaluation engineers face in the production stage.</p><h3>Final thoughts</h3><p>Thought leaders are predicting that <strong>2024 will be the year when we will see companies move from prototype to production</strong> for genAI adoption. Matt McIlwain @ Madrona put nicely that the shift will be from “‘<a href="https://medium.com/@mattmcilwain/genai-in-2024-another-decade-in-one-year-1c7d858ac0d6">Let’s Try’ to ROI</a>.” Evaluation will be the first step required to assess ROI for past projects and forecast investments needed for future projects.</p><p>While genAI MVP projects were driven by top-down decisions, the <strong>LLM app development process will mature similarly to the systematic traditional software development process.</strong> In order for mass enterprise adoption to happen, the process will not only require quantitative data for objective comparisons, but also extensive collaborative capabilities for stakeholder alignment. These additional needs will make the already time-intensive evaluation process even lengthier unless there is tooling to support them. The ability to explain metrics will become critical to bringing diverse stakeholders to agreement.</p><p><strong>Evaluation needs will become increasingly more industry- and application-specific,</strong> while general evaluation benchmarks will be suggestive at best. As companies continue to customize LLM applications for their evolving business use cases, the ability to tailor tests and evaluation metrics will be imperative.</p><p>We are <strong>optimistic about startups in this space because enterprises have expressed positive initial signals for buying rather than building</strong>. Based on our survey and interviews, we have validated how much of a pain point evaluation is, how troublesome it is to build an internal solution, and how high the willingness is to pay for an external solution. Additionally, startups offering evaluation tools are saying that they are experiencing a wave of inbounds this quarter. We are excited to continue to follow this space closely.</p><p><em>We’re looking to learn from pioneers who are leading the future of LLM evaluations. If you are building a LLMOps startup or contributing to the evolution of the LLM App Development Life Cycle in any way, we would love to chat! Please reach out to Matt Kinsella (</em><a href="/service/mailto:Matthew.Kinsella@maverickcap.com"><em>Matthew.Kinsella@maverickcap.com</em></a><em>), JooHo Yeo (</em><a href="/service/mailto:joohoyeo@stanford.edu"><em>joohoyeo@stanford.edu</em></a><em>), or both of us!</em></p><h3><em>The views expressed herein are solely the views of the author(s) and are not necessarily the views of Maverick Capital, Ltd. or any of its affiliates. This article is not intended to provide, and should not be relied upon for, investment advice.</em></h3><h3>Appendix</h3><h4>Survey Results</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*7qM6KRcFwyo1KhGlqpw8cA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*-vzC7CeP9-NNGmO93MDO5w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*sSn2LMkibEvID4UGg3YS3w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*IT1EroNpHdLDqIzM8uysIA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c96bc968456d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Welcome to our newest Principal, Lexi Henkel!]]></title>
            <link>https://maverickventures.medium.com/welcome-to-our-newest-principal-lexi-henkel-d38d9320f1d3?source=rss-41af1c295e83------2</link>
            <guid isPermaLink="false">https://medium.com/p/d38d9320f1d3</guid>
            <dc:creator><![CDATA[Maverick Ventures]]></dc:creator>
            <pubDate>Mon, 11 Sep 2023 18:29:02 GMT</pubDate>
            <atom:updated>2023-09-11T23:48:39.020Z</atom:updated>
            <content:encoded><![CDATA[<p>We’d like to extend a warm welcome to <a href="https://www.linkedin.com/in/lexi-henkel-168709a4">Lexi Henkel</a>, the newest investor at Maverick Ventures.</p><p>As a Principal, Lexi will focus on early-stage healthcare and technology investments. Lexi is officially planting the Maverick Ventures flag on the East Coast — she will be based in New York City and will help expand our community of entrepreneurs and strategic partners from coast to coast.</p><p>Lexi’s passion lies at the intersection of early-stage company building, healthcare and technology. She was previously on the founding team of <a href="https://tmrwlifesciences.com/">TMRW Life Sciences</a>, a fertility technology startup where she ran finance and strategy. Lexi worked closely with the co-CEOs to grow the company from pre-revenue stage through commercial launch and from less than 10 employees to 150. She also helped lead three rounds of fundraising as TMRW scaled from Series A through Series C. Lexi started her career in investment banking at Morgan Stanley.</p><p>Lexi earned her BA from Yale, where she graduated summa cum laude and Phi Beta Kappa with a dual degree in Economics and Spanish. A condensed version of her Economics thesis, “The Effect of Acquisitions on Pharmaceutical Drug Prices,” was published in a peer-reviewed medical journal. She also holds an MBA from Stanford University, where she was President and CFO of the Healthcare Club.</p><p>We are thrilled for Lexi to expand our coverage in the many areas of healthcare and technology we are tracking closely — from generative AI to tech-enabled services and women’s health. Lexi is eager to partner with entrepreneurs solving the most complex challenges in healthcare and technology.</p><p>Welcome, Lexi!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I-zZJvIpy3EJjyxe4-nv1Q.jpeg" /></figure><p><em>The views expressed herein are solely the views of the author(s) and are not necessarily the views of Maverick Capital, Ltd. or any of its affiliates. They are not intended to provide, and should not be relied upon for, investment advice.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d38d9320f1d3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI’s Inflection Point: How MosaicML is making AI models more accessible]]></title>
            <link>https://maverickventures.medium.com/ais-inflection-point-how-mosaicml-is-making-ai-models-more-accessible-9a5c107809bf?source=rss-41af1c295e83------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a5c107809bf</guid>
            <dc:creator><![CDATA[Maverick Ventures]]></dc:creator>
            <pubDate>Fri, 17 Feb 2023 16:34:27 GMT</pubDate>
            <atom:updated>2023-02-17T16:39:41.954Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By Jéssica Leão and Ambar Bhattacharyya</em></p><p>We’re excited to announce Maverick Ventures’ investment in MosaicML, a company at the forefront of the generative AI space and our ninth investment in an AI-driven innovator.</p><p><strong>What does MosaicML do?</strong></p><p>MosaicML’s products make compute useful for large-scale ML training, enabling enterprises to access state-of-the-art AI/ML tools, methods, and infrastructure, and putting the power of large models in the customer’s hands.</p><p>Think: any enterprise — whether a bank, media company, or SaaS vendor, can integrate a fully functioning AI model into their products, built on their proprietary data, and done so cheaply and efficiently.</p><p>MosaicML’s secret sauce is an optimization of systems and algorithms, orchestration for improved resource utilization and resilience, as well as packaging for ease of use, that makes every dollar spent on computing when training large ML models many multiples more efficient and effective. And this isn’t just talk: Mosaic <a href="https://www.mosaicml.com/blog/gpt-3-quality-for-500k">published the costs to train compute-optimal LLMs</a>, all measured on the MosaicML Cloud, and the profiling shows that a compute-optimal model, with the recipe to reach the same quality as the original GPT-3, can be trained for less than $500K.</p><p>MosaicML also recently partnered with Stanford to create <a href="https://www.mosaicml.com/blog/introducing-pubmed-gpt">BioMedLM</a>, a purpose-built AI model trained to interpret biomedical language, and the cost of training this quality model from scratch was only ~$38,000! Mosaic also put itself to the test with MLPerf, and submitted results with their algorithmic stack to both ResNet50 and BERT benchmarks, and won them both (see: <a href="https://protect-us.mimecast.com/s/77UPCW6KYycD6q3zILz3jY?domain=mosaicml.com">ResNet results</a> and <a href="https://protect-us.mimecast.com/s/HCAoCXDXZzTO4KABsrsFHw?domain=mosaicml.com">BERT results</a>).</p><p><strong>The MosaicML Team</strong></p><p>The caliber of people building Mosaic is first-class. Naveen, co-founder and CEO of MosaicML, is a second-time founder, formerly the founder and CEO of Nervana Systems, a deep learning ML platform. Nervana designed machine learning chips and software architecture, and was early to the deep learning space by launching a hosted cloud platforms for “deep learning on demand.” <a href="https://www.vox.com/2016/8/9/12413600/intel-buys-nervana--350-million">Nervana was later acquired by Intel</a>, where Naveen spent close to four years leading the AI vision, most recently as the Corp VP and General Manager of the Artificial Intelligence Products Group.</p><p>Building MosaicML alongside Naveen are his co-founders Hanlin Tang, Mosaic’s Chief Technology Officer and former Director of Intel AI Lab; and Jonathan Frankle, Chief Science Officer and author of seminal paper on efficient deep learning: <a href="https://arxiv.org/abs/1803.03635">The Lottery Ticket Hypothesis</a>.</p><p>We are thrilled to back this exceptional team. Their efforts show: user feedback is glowing. Customers highlight how they’ve experienced astonishing results with MosaicML, which allows them develop models that they thought were impossible to build. And for those who were already on their AI journey, they report that with MosaicML, what would have taken months now happens in a matter of 1–2 days.</p><p><strong>Our Partnership</strong></p><p>For all these reasons, and many more, we are incredibly proud to announce our investment in MosaicML. We know this will be a partnership for years to come, and we cannot wait to them continue to build towards their vision of making AI models more accessible for everyone.</p><p><em>The views expressed herein are solely the views of the author(s) and are not necessarily the views of Maverick Capital, Ltd. or any of its affiliates. They are not intended to provide, and should not be relied upon for, investment advice.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a5c107809bf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The AI Boom: Why We’re Investing in the Future of Intelligence]]></title>
            <link>https://maverickventures.medium.com/the-ai-boom-why-im-investing-in-the-future-of-intelligence-f471e5485b85?source=rss-41af1c295e83------2</link>
            <guid isPermaLink="false">https://medium.com/p/f471e5485b85</guid>
            <dc:creator><![CDATA[Maverick Ventures]]></dc:creator>
            <pubDate>Tue, 06 Dec 2022 18:54:52 GMT</pubDate>
            <atom:updated>2022-12-06T18:58:13.118Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By Jéssica Leão</em></p><p>I — like many others — have been spellbound by OpenAI’s latest release, ChatGPT, and have spent the past days since its launch ferociously contributing towards the occasional “system is far too busy” message. It is simply addictive. There are numerous examples shared by the community around it acting as a code debugger, haiku writer, and inspirational eulogy deliverer. The potential applications are truly limitless and I’ve already seen many try to tinker and build apps on top of the release — in less than 2 days!</p><p>For anyone following AI — whether new to the space or steadily tracking its progress — the past couple of months have been more than exciting. Think: GPT-3’s upgrade with Davinci 003 &amp; ChatGPT showing what creative AI looks like, BLOOM’s massive contribution to open source, PaLM’s breakthrough performance, DALL-E 2 &amp; Stability Diffusion’s jaw-dropping image generation capabilities, and really so much more. We are seeing transformational changes to the field and are starting to glimpse at how industries as we know them will be reoriented by AI.</p><p>As a venture investor, I look forward to seeing AI adopted by my portfolio companies, and anxiously await the companies that will be created in an AI-native world. Among the many potential areas where AI can come in handy (such as data analysis and insights across sectors, automation of tasks and processes, personalized CX), perhaps the one I am most excited about is around innovation in product development. We’re seeing a glimpse of that power today with ChatGPT’s capabilities with code. I can see AI being leveraged to create new and optimize existing algorithms… or even to develop new applications using itself:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*xtaWeqOh5o2K9z6Q0Pvl1g.png" /></figure><p>Admittedly the above example does not alone create a chatbot — but you get the idea. I look forward to seeing what talented entrepreneurs will come up with as adoption of these models continue to proliferate. I am similarly enthusiastic about where AI can add leverage to legacy industries. Some of the top of mind examples for me include:</p><ul><li>Legal: My very first job out of college was at a law firm, and as I remember my painful hours spent combing through Bluebook to make sure my citations were accurate, or as I recall drafting the 342th version of the same boiler plate filing document, I know how easily AI algorithms can be leveraged to automatically analyze, extract, and write key information from briefs, contracts, clauses, and more.</li><li>Healthcare: How often do we consult Dr. Google only to leave feeling hypochondriac? Something better can be built that is just as easy to access by consumers. I think a great company will be created in the AI healthcare space — assist doctors in making diagnoses, predicting patient outcomes, and identifying potential treatments. AI algorithms can and will be able to analyze large amounts of medical data to identify patterns and potential risk factors for diseases, and can even assist in the development of new drugs.</li><li>Finance: This one almost goes without saying but AI’s magic is the ability to process so much data at once. We will see a lot of impact in the financial services world — improve the efficiency and accuracy of financial processes, such as fraud detection and risk assessment. Algorithms can analyze large amounts of data to identify potential fraudulent transactions, and can also help banks and other financial institutions to make more informed decisions about lending and investing.</li><li>Retail: Fresh off Black Friday shopping, I know I could use better CX. AI can dramatically improve the customer experience in the retail industry by analyzing customer data and purchase history to make ever-more-personalized product recommendations. AI can also be used on the backend to improve supply chain management and optimize inventory levels.</li><li>Manufacturing: We all remember what it was like to wait 7+ months for a coffee table to ship because of supply chain issues. AI can help there: optimize production schedules, identify inefficiencies in the production process, and even assist with tasks such as quality control and maintenance.</li></ul><p>Lastly — and this will merit its own blog post at a later time — one has to always be thinking of “picks and shovels.” If I believe in the future of an AI world — and I do! — then what are the necessary conditions for this world to flourish? I predict a lot will happen in the AI stack. We’ll see an explosion of companies around compute optimization, MLOps (more ML models being used means more training, observability, and deployment loops), cybersecurity, and much, much more.</p><p>I had a hard time coming up with a sign off here, so I went back to ChatGPT to help. Here goes: “Thanks for reading my blog post about AI! I hope you learned something new and exciting, and that you’re now eager to start building your own AI applications. Remember: with AI, the sky’s the limit (or at least the ceiling of your data center). Until next time, keep learning and keep AI-ing!”</p><p><em>The views expressed herein are solely the views of the author(s) and are not necessarily the views of Maverick Capital, Ltd. or any of its affiliates. They are not intended to provide, and should not be relied upon for, investment advice.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f471e5485b85" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing, Volta Labs!]]></title>
            <link>https://maverickventures.medium.com/introducing-volta-labs-efab260557d3?source=rss-41af1c295e83------2</link>
            <guid isPermaLink="false">https://medium.com/p/efab260557d3</guid>
            <dc:creator><![CDATA[Maverick Ventures]]></dc:creator>
            <pubDate>Thu, 17 Mar 2022 13:16:06 GMT</pubDate>
            <atom:updated>2022-03-17T13:16:06.507Z</atom:updated>
            <content:encoded><![CDATA[<p>By David Singer and Prateesh Maheshwari</p><p>Thursday, March 17</p><p>We are proud to announce our continued investment in <a href="https://www.voltalabs.com/">Volta Labs</a>, leading its $20M Series A funding round, with participation from Illumina co-founder John Stuelpnagel, Khosla Ventures, Casdin Capital, Anne Wojcicki (CEO of 23andme), Paul McEwan (co-founder of KAPA Biosystems &amp; Agencourt Biosciences), E14 Funds, and Blindspot Ventures.</p><p>Over the last two decades, the cost of <a href="https://www.genome.gov/about-genomics/fact-sheets/DNA-Sequencing-Costs-Data">DNA sequencing</a> has plummeted but there haven’t been nearly enough advances in the cost of sample preparation (or “library prep”), a necessary precursor to being able to sequence DNA. Therefore, budgets at laboratories growing DNA sequencing volumes are increasingly consumed by the costs of sample preparation. While robots have been deployed in high volume sequencing facilities, developments for lower-throughput operations have lagged behind.</p><p>Volta Labs has built a suite of components aimed at automating the sequencing sample preparation process. Volta Labs is developing a novel platform using electrowetting technology to manipulate fluids on a chip, automating a wide range of laboratory processes in a more highly controlled, effective and efficient modality, e.g., sample prep for DNA sequencing; DNA synthesis; and, eventually many of the emerging “omics”, including proteomics. With regards to sample prep, tens of thousands of labs are preparing samples manually. Automation can dramatically increase the quality and the repetition of sample prep and aid labs in their efforts to be more agile, reliable, and scale operations quicker. Overall, Volta will enable further adoption of sequencing by reducing cost and simplifying the upstream processes.</p><p>Since first leading the seed round in 2019, we have been blown away by Volta Labs co-founder and CEO, <a href="https://www.linkedin.com/in/udayanu/">Udayan Umapathi</a>. He is a thoughtful, driven entrepreneur who is a first principled thinker. He first came up with the idea in 2018 during his tenure at MIT’s Media Lab, where he was exploring microfluidic technologies to make biology look like engineering. <a href="https://www.linkedin.com/in/willlangford/">Will Langford</a>, co-founder and Head of Engineering, comes from the MIT Center for Bits &amp; Atoms and helps make engineering applicable to biology. The team has done an exceptional job hiring including <a href="https://www.linkedin.com/in/abdul-m-mohammed-b53a83101/">Abdul Mohammed</a>, Associate Director of Genomic Applications, and recently bringing on a Chief Commercial Officer, <a href="https://www.linkedin.com/in/kfarnsworth/">Keith Farnsworth</a>.</p><p>With this additional round of funding, Volta Labs will accelerate the development of its sequencer-agnostic front-end platform for commercial release in 2023, with a limited release program by the end of this year. To support these efforts, the company is actively hiring across engineering, applications, research and development, and sales — see the roles <a href="https://jobs.lever.co/voltalabs">here</a>.</p><p>We are so proud of the work Udayan, Will, Abdul, Keith and the entire Volta Labs team are doing — it’s been an honor to be a part of the journey thus far, and we are looking forward to what’s next.</p><p><em>The views expressed herein are solely the views of the author(s) and are not necessarily the views of Maverick Capital, Ltd. or any of its affiliates. They are not intended to provide, and should not be relied upon for, investment advice.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=efab260557d3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Congratulations, Centivo!]]></title>
            <link>https://maverickventures.medium.com/congratulations-centivo-9372a92a3e9c?source=rss-41af1c295e83------2</link>
            <guid isPermaLink="false">https://medium.com/p/9372a92a3e9c</guid>
            <dc:creator><![CDATA[Maverick Ventures]]></dc:creator>
            <pubDate>Wed, 22 Sep 2021 21:03:46 GMT</pubDate>
            <atom:updated>2021-09-23T18:20:02.867Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By Ambar Bhattacharyya</em></p><p>We are proud to announce our continued investment in Centivo, and we are pleased to be one of the lead investors in their most recent $51M funding round, co-leading with B Capital Group.</p><p>Centivo has experienced rapid growth and expansion since implementing its first client in 2019. During the first nine months of 2021, Centivo significantly expanded its market presence to key MSAs in New York, New Jersey, Connecticut, Iowa, Pennsylvania and Wisconsin (joining central Florida, North Carolina, Southern California and Nebraska.) It also recently developed new strategic distribution arrangements with BHCG in eastern Wisconsin.</p><p>Self-funded employers and their employees are facing a true healthcare affordability crisis, while rising premiums, surprise bills, and the lack of transparency in our complex healthcare system is leading to low-quality primary care. By providing data-driven tools to recommend the most appropriate and affordable care, Centivo lowers healthcare spend for employers, and offers employees and their families the high-quality healthcare they need. We believe Centivo is primed to be a leading driver of change for the better.</p><p>Since first investing in the company back in April 2018, we’ve seen firsthand the tremendous growth they’ve experienced, as well as their profound mission to make healthcare effective and affordable. Congratulations again to Ashok and the entire Centivo team and we look forward to the continued success ahead.</p><p><em>The views expressed herein are solely the views of the author(s) and are not necessarily the views of Maverick Capital, Ltd. or any of its affiliates. They are not intended to provide, and should not be relied upon for, investment advice.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9372a92a3e9c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Congratulations, Chapter!]]></title>
            <link>https://maverickventures.medium.com/congratulations-chapter-6bb8b58aa999?source=rss-41af1c295e83------2</link>
            <guid isPermaLink="false">https://medium.com/p/6bb8b58aa999</guid>
            <dc:creator><![CDATA[Maverick Ventures]]></dc:creator>
            <pubDate>Tue, 21 Sep 2021 18:35:58 GMT</pubDate>
            <atom:updated>2021-09-21T18:35:58.755Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By Prateesh Maheshwari and Ambar Bhattacharyya</em></p><p>We are thrilled to announce our investment in <a href="https://www.getchapter.com/">Chapter</a>. We co-led the seed in February 2020 and continued to invest as part of the $17m Series A announced today.</p><p>Chapter is personalizing the retirement journey. Chapter’s first product is providing personalized, end-to-end Medicare guidance through its platform and team of experts. Chapter has built and maintains a proprietary technology platform used by its advisors to search every Medicare plan nationwide, including Medicare Advantage, Medicare Supplement, and standalone Part D prescription plans. The company gives unbiased guidance on how to select into Medicare. As a result of its unique model, Chapter not only supports Americans directly, but also the types of organizations they trust most, including non-profits, religious institutions, community groups, financial advisors, employers, and employee benefits organizations.</p><p>We’re excited to back entrepreneurs like Cobi, Corey, and Vivek who are tackling important problems in healthcare with novel approaches. Congratulations again to Chapter, this investment is just the beginning.</p><p><em>The views expressed herein are solely the views of the author(s) and are not necessarily the views of Maverick Capital, Ltd. or any of its affiliates. They are not intended to provide, and should not be relied upon for, investment advice.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6bb8b58aa999" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>