<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Divya Jain on Medium]]></title>
        <description><![CDATA[Stories by Divya Jain on Medium]]></description>
        <link>https://medium.com/@divyajain1?source=rss-70995a473b4d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*dRvrcpujA9ph1E--t2Agkw.jpeg</url>
            <title>Stories by Divya Jain on Medium</title>
            <link>https://medium.com/@divyajain1?source=rss-70995a473b4d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 09:03:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@divyajain1/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Open Models vs. Frontier AI: History Says It’s a Marathon, Not a Sprint]]></title>
            <link>https://medium.com/@divyajain1/open-models-vs-frontier-ai-history-says-its-a-marathon-not-a-sprint-12d8f9900288?source=rss-70995a473b4d------2</link>
            <guid isPermaLink="false">https://medium.com/p/12d8f9900288</guid>
            <dc:creator><![CDATA[Divya Jain]]></dc:creator>
            <pubDate>Mon, 31 Aug 2026 22:15:19 GMT</pubDate>
            <atom:updated>2026-08-31T22:15:19.497Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PWYvHFzvPLKlSkbstUBRsw.jpeg" /></figure><p>When reports surfaced over the weekend that NVIDIA is in talks to buy Hugging Face for $13 billion, my social feeds filled up with a familiar take: <em>open source is eating AI, and proprietary frontier models are on borrowed time.</em></p><p>I saw that and couldn’t help thinking back. I’ve seen this movie before, from the inside.</p><p>Early in my career, I worked at Sun Microsystems. For anyone who didn’t live through that era, Sun wasn’t just another tech company. We built Solaris, and at the time, we were convinced we were doing the best engineering on the planet. Honestly, we probably were. We had ZFS before self-healing filesystems were a thing, DTrace before real-time kernel observability existed, and Solaris Zones a decade before anyone called them containers. <strong>By almost every technical metric, Solaris was the finest operating system around.</strong></p><p>And Linux still crushed us.</p><p>Linux didn’t win because the code was better. It won because it was free, open, and ran on cheap x86 boxes. A global community could hack on it, build tools for it, and deploy it without asking anyone for a license. The ecosystem snowballed, and Sun’s expensive, proprietary SPARC hardware got eaten from the bottom up.</p><p>It is tempting to look back at that and assume open source always wipes out closed systems. But that’s only half the story. Other closed operating systems survived. Microsoft Windows didn’t die. Apple didn’t die.</p><p>Staying alive took an incredible amount of painful, ugly work, but the ecosystems they built were what saved them. Apple narrowed its focus to an integrated user experience that people would gladly pay a premium for. Microsoft survived brutal platform shifts by embedding itself deep into enterprise software and the cloud. Neither company survived because their kernel was better than Solaris. They survived because of a rule people tend to forget: <strong>open platforms win volume, but proprietary ecosystems win the margins.</strong></p><p>We saw the exact same playbook in mobile. Android open-sourced its way to more than 70% of global phones, an unbelievable distribution win. Yet Apple still captures most of the profit because its hardware, software, and services are locked together in a way users don’t want to leave.</p><p>The companies that got squeezed out (Sun, BlackBerry, Symbian) were stuck in no-man’s-land. They had great tech, but they had neither the massive volume of an open platform nor the defensive moat of an integrated ecosystem.</p><p>Sun learned this the hardest way with Java. We built a language on the promise of “write once, run anywhere.” And it worked. Java became the backbone of enterprise software and eventually powered billions of devices, including Android. But Sun never had the application layer or consumer ecosystem to make money from it. We gave the world the plumbing, and everyone else built trillion-dollar businesses on top of it. Inventing an open standard is great, but capturing the value from it is an entirely different game.</p><p>That brings us to the AI model race today.</p><p>Right now, the consensus is that open-weight models (like Llama, Mistral, Gemma, and DeepSeek) are hitting benchmark parity fast and will commoditize frontier labs out of business. That’s why NVIDIA making a play for Hugging Face makes so much sense. Just like IBM pouring a billion dollars into Linux in 2000, or Google backing Android, NVIDIA wants to own the rails of the open volume layer so its GPUs remain the default engine for the world.</p><p>Make no mistake: open models <em>will</em> win the volume game. High-frequency, cost-sensitive, specialized enterprise workloads are going to run on them.</p><p><strong>Where the consensus goes wrong is assuming this means the death of frontier models.</strong></p><p>Every major platform race leaves room for high-margin proprietary ecosystems, as long as they do the tedious, unglamorous work of building workflow lock-in. A model by itself isn’t a product; it’s an engine. And engines only make money when they’re bolted into a car people actually drive: the tools where daily work happens, the enterprise security IT departments already trust, and the devices sitting in everyone’s pockets.</p><p>I work at Google today, and I hear a very specific critique all the time: Google published “Attention Is All You Need,” invented the Transformer architecture behind modern AI, and then watched startups steal the spotlight. <em>“It’s Sun and Java all over again.”</em></p><p>It’s an easy comparison to make. But it misses the biggest piece of the puzzle: <strong>the ecosystem. </strong>Sun couldn’t capture Java because we had no consumer platform, no everyday office software, and no mass hardware surface. Google is in a completely different spot. When a model architecture is tied directly into two billion Android phones, the world’s default search box, the workspace apps millions of businesses run on, and deep enterprise cloud infrastructure, the math changes completely.</p><p>Anyone declaring a winner based on <strong>this month’s benchmark leaderboards might be celebrating a little too soon.</strong> Platform marathons are never decided in the first few miles. That doesn’t make anyone invincible. Sun taught me firsthand that engineering arrogance is fatal, and size without speed is just inertia.</p><p><strong>The real lesson from history is simple:</strong> If your entire moat is scoring three percent higher on an evaluation benchmark this week, remember the Solaris story. Even the most brilliant technology gets left behind if it stands alone.</p><p>Open models are going to take the volume. But the frontier players who survive won’t just be the ones publishing celebrated research papers or being the darling of the developer world. They’ll be the ones who build the stickiest, most essential ecosystems around actual human work.</p><p>The marathon has barely begun.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12d8f9900288" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The 0.3% Obsession: Time to Give LLMs a Proper Report Card]]></title>
            <link>https://medium.com/@divyajain1/the-0-3-obsession-time-to-give-llms-a-proper-report-card-c33b5e08a5e3?source=rss-70995a473b4d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c33b5e08a5e3</guid>
            <category><![CDATA[llm-evaluation]]></category>
            <category><![CDATA[ai-evaluation-metrics]]></category>
            <category><![CDATA[ai-sovereignty]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[llm-benchmarks]]></category>
            <dc:creator><![CDATA[Divya Jain]]></dc:creator>
            <pubDate>Mon, 08 Jun 2026 16:45:49 GMT</pubDate>
            <atom:updated>2026-06-08T16:45:49.884Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IPgZ_oTAjK58LHUqvcMNtQ.jpeg" /></figure><p>If you want to see what seventh-grade panic looks like in a room full of brilliant adults, just tell an AI engineering lead that a competitor’s new model beat theirs on a legacy benchmark by 0.3%.</p><p>Suddenly, years of advanced training, PhDs from elite institutions, and millions of dollars in compute evaporate. You are no longer looking at world-class computer scientists; you are looking at an awkward teenager standing at a teacher’s desk, palms sweating, desperately pleading for a quarter-point on a life science quiz because their classmate got a 96.8% and their 96.5% feels like an existential crisis.</p><p>Today, some of the most brilliant minds in our field are engaged in these exact, high-stakes playground fights. We panic over decimal points on models that are already phenomenally capable. Teams will spend forty-eight straight hours debugging, re-allocating compute, and losing sleep over a 0.3% regression on a legacy evaluation harness.</p><p>Whenever I witness these decimal-point skirmishes over models that can already draft complex legal briefs and write highly functional code, I find myself asking a simple, uncomfortable question: <em>Does this actually matter?</em></p><p><strong>To an optimization loop or a TPU cluster, perhaps it does. But in the real world, this is completely imperceptible to human users.</strong> It doesn’t tell a doctor if a model can safely summarize a patient’s medical history, nor does it tell a software developer if it can refactor a legacy codebase without introducing a massive security vulnerability. We have built an evaluation system that is incredibly precise for machines, yet utterly useless for humans. We desperately need a translation layer, a way to turn raw, abstract math into human-readable capability.</p><p>Maybe it is time to stop treating AI like a statistical anomaly and start evaluating it the way we evaluate human intelligence: with something similar to a school grading system.</p><h3>Goodhart’s Law: Teaching to the Test</h3><p>The problem with raw percentage ranking systems — whether in a middle school classroom or an AI evaluation lab — is that they encourage the wrong behaviors. When my childhood school ranked students by raw decimals, we became obsessed with a few points, spending hours arguing over quarter-points just to protect our decimal-point rank. <em>Yes, I did that too, not proud of it.</em></p><p>When we eventually switched to a standard letter grade system (A, B, C), the entire atmosphere changed. The new system recognized a fundamental truth about human capability: there is no meaningful difference in intelligence between a 97% student and a 96% student. Both have masterfully grasped the material.</p><p>In the AI world, we are currently suffering from the exact same behavior. AI labs have fallen victim to Goodhart’s Law: when a measure becomes a target, it ceases to be a good measure. Models are being trained, fine-tuned, and sometimes overfit specifically to “pass” narrow, multiple-choice benchmarks like MMLU. We are building test-taking savants that can recite bizarre trivia but fail to follow simple, multi-step instructions in a real-world application.</p><p>The public, and our enterprise customers, are understandably baffled. They read headlines claiming a new model has achieved “98% human-level accuracy,” but when they actually try to deploy it to plan a family vacation or debug a script, it hallucinates, loses the thread, or fails entirely.</p><h3>Human-Readable AI Report</h3><p><em>To be absolutely clear: I am not suggesting we throw away raw decimal metrics.</em> If you are an ML engineer running a regression test in your CI/CD pipeline, you absolutely need granular metrics to ensure a code change didn’t break a critical feature. But we must stop using internal engineering telemetry as a public product standard. Instead, we can translate these metrics into a standardized, intuitive Academic Level &amp; Letter Grade framework that separates task complexity from execution reliability.</p><p>Under this framework, we organize capabilities into Education Levels reflecting task complexity. These range from Elementary School (K-5) for basic instructions, to Middle School (6–8) for simple synthesis, and High School (9–12) for standard writing and debugging. Higher tiers include the Undergraduate level for complex engineering and professional exams, culminating in the Graduate (Masters/PhD) level for original reasoning and frontier mathematical proofs.</p><p>We then pair these tiers with Letter Grades for execution reliability. A Grade A represents highly reliable performance (a 95% success rate) with strict adherence to safety and formatting constraints. A Grade B denotes a model that is generally reliable but still requires human oversight due to occasional minor logic drifts. A Grade C marks a functional but flawed tool with high hallucination rates and a frequent need for course correction, while a Grade F is reserved for models that completely fail to execute tasks at that complexity tier. By structuring capabilities this way, we make AI performance instantly understandable to anyone.</p><h3>Math Genius Who Can’t Write a Poem</h3><p>The beauty of this system is that it allows us to evaluate models across different subjects, just like a report card. We often talk about LLMs as if they are monolithic minds. But in reality, a model might be a brilliant programmer but a terrible creative writer.</p><p>Under an Academic Grading Framework, a model’s report card might look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oJ_YdQo0bBKrh-ZF8g8GCg.png" /></figure><p>Suddenly, a non-technical product manager looking to integrate AI into their workflow knows <em>exactly</em> what they are getting. They don’t need to understand what “HELM” or “MMLU” stands for. They can look at the report card and see that the model has an Undergraduate A- in coding, which fits their software team perfectly.</p><h3>The Industry Has Started Grading</h3><p>We are already starting to see the industry naturally shift toward this kind of tiered, qualitative thinking, even if the execution is still highly fragmented. For instance, OpenAI recently introduced their five-level scale for tracking progress toward Artificial General Intelligence, moving from simple conversational chatbots to autonomous “agents.” It is a positive step toward establishing clear benchmarks of utility, though it remains a proprietary marketing framework rather than an open, industry-wide standard.</p><p>At the same time, we have seen the rise of crowd-sourced evaluations like the LMSYS Chatbot Arena. By relying on blind, head-to-head “vibe checks” judged by real humans, the Arena bypasses static test sheets entirely. It is currently one of the most honest metrics we have, but because it relies on an Elo chess rating system, it only tells us who is <em>better</em> than whom. It still cannot tell us the absolute capability level of a model — is a 1300 Elo model performing at a high school level or a master’s level? We simply don’t know.</p><p>Even in academia, researchers are beginning to divide tests by academic tiers, classifying math benchmarks strictly into “Undergraduate” and “PhD” levels. These specialized tests are excellent, but their results remain siloed inside academic papers, rarely translating into a unified, consumer-friendly standard that the public can use to make deployment decisions.</p><h3>Even Governments and Global Leaders are Demanding a Rubric</h3><p>This debate has quickly outgrown Silicon Valley boardroom tables. As AI transitions from a software tool to a strategic element of national defense — a concept known as <strong>AI Sovereignty</strong> — governments are stepping in to audit these systems. But a 92% score on a Western-centric benchmark like MMLU means absolutely nothing to a policymaker in Paris trying to evaluate if a model respects French civil law, or a regulator in Ottawa worried about preserving indigenous languages.</p><p>Because of this gap, governments are starting to intervene directly. For example, the US government recently established a voluntary framework asking developers of advanced “covered frontier models” to grant the federal government secure access for a 30-day cybersecurity review before releasing them to trusted partners. But without a standardized, qualitative grading rubric to assess real-world risk, a 30-day review window is just an administrative waiting room. Regulators cannot safely clear a model based on decimal-point improvements; they need a rigorous, multi-subject report card that assesses tangible utility and safety boundaries.</p><p>The calls for a more human-centric evaluation system are coming from the highest moral offices as well. When Pope Francis addressed G7 leaders regarding the “cognitive-industrial revolution,” he called for <strong>“algor-ethics”</strong> — a framework ensuring technology serves human dignity rather than reducing human lives to statistical computations. Whether we are discussing geopolitical security or ethical guardrails, the conclusion is identical: a model’s value cannot be measured in raw parameters or compute efficiency. We need a standardized grading system that doesn’t just evaluate if an AI is fast or smart, but whether it is safe, unbiased, and aligned with human values.</p><h3>Time to Hand Out the Report Cards</h3><p>As builders, researchers, and technology leaders, we have to ask ourselves who we are ultimately building this technology for. If we are optimizing solely to beat our competitors by 0.3% on static, contaminated benchmarks, we are wasting precious compute and brilliant engineering hours tuning models to pass yesterday’s tests. Transitioning to an open, standardized, academic grading system would allow us to demystify AI for the public by replacing sensationalized fear with clear, intuitive expectations of utility.</p><p><strong><em>Yes, implementing this is incredibly difficult, and we cannot do it in silos. </em></strong>We will have to wrestle with massive technical and structural hurdles: How do we scale subjective rubrics without falling into the trap of biased “LLM-as-a-judge” setups? How do we map a single letter grade to vastly different risk profiles — where a B-minus in creative writing is a slightly boring blog post, but a B-minus in medical synthesis is a catastrophic lawsuit? And crucially, who gets to decide where the grade boundaries lie?</p><p>Just as we did not build our human education system overnight, we cannot expect a single company or lab to build this AI syllabus alone. We need to start working on putting this structure together collectively. It will require academic boards, industry coalitions, independent auditing bodies, and public stakeholders to establish a trusted, unified accreditation framework. It is time to start building the shared, institutional infrastructure that lets the world truly understand what these incredible machines can do.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c33b5e08a5e3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Myth of the Surplus Employee in the AI Era]]></title>
            <link>https://medium.com/@divyajain1/the-myth-of-the-surplus-employee-in-the-ai-era-16257243b415?source=rss-70995a473b4d------2</link>
            <guid isPermaLink="false">https://medium.com/p/16257243b415</guid>
            <category><![CDATA[agenticworld]]></category>
            <category><![CDATA[big-tech-layoffs]]></category>
            <dc:creator><![CDATA[Divya Jain]]></dc:creator>
            <pubDate>Mon, 02 Feb 2026 22:54:12 GMT</pubDate>
            <atom:updated>2026-02-02T22:54:12.237Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hYFrEDNVNUe5VVtOY2UG7g.jpeg" /></figure><p>Lately, my coffee chats have taken on a somber tone. Dozens of talented engineers have reached out with the same fear: <em>Will I still have a job in twelve months?</em> They see the “right-sizing” headlines and feel the ground shifting. I tell them all the same thing: it is a massive failure of strategic imagination at the top.</p><p>In the sterile language of boardroom slides, layoffs are called “rebalancing.” But as someone who has spent twenty years in tech — fifteen of those at the edge of AI development — I see these moves for what they really are: shortsighted. We are standing at the threshold of the most significant transition in the history of labor, moving from manual logic to agentic intelligence. Yet, instead of preparing the fleet for a new continent, leaders are throwing the crew overboard to lighten the load before the ship even reaches the shore.</p><h4>The Efficiency Trap</h4><p>The justification for these layoffs is usually that AI can now automate the tasks of junior and mid-level staff. This assumes “the work” is a finite pile of bricks that just needs to be moved.</p><p>In twenty years, <strong>I’ve never seen a high-impact project with “surplus” people.</strong> I’ve only seen projects with a surplus of ambition and a deficit of time. Certainly, “zombie” projects require pruning, but indiscriminate layoffs fail to distinguish between bloat and untapped potential.</p><p>When an AI agent handles boilerplate code or unit tests, the engineer is not redundant. They are <strong>liberated</strong>. They are free to tackle the decade of architectural debt we’ve ignored and move from “maintaining” to “disrupting.” When the calculator was invented, we didn’t fire the mathematicians; we asked them to solve harder problems.</p><h4>Judgment: The Ultimate Human Multiplier</h4><p>In a world where the cost of generating code and logic is approaching zero, the only thing that retains a premium is <strong>judgment</strong>.</p><p>AI is a powerful tool, but its output is only as reliable as the person directing it. Domain experts understand the idiosyncrasies of a codebase and the edge cases an LLM will never predict. Relying on a smaller, less experienced workforce to manage AI systems is a dangerous gamble. It leads to a “model collapse” of institutional knowledge. Once the experts are gone, the AI has no one left to learn the truth from. Without deep context, the multiplier is wasted.</p><p>This isn’t just a tech problem. If AI handles diagnostic triage or administrative grading, we shouldn’t be reducing staff. We should be celebrating that a physician can finally spend time on holistic patient care, or a teacher can provide deep, one-on-one mentorship. The value isn’t the paperwork; it’s the human outcome.</p><h4>Reinvesting in the “Next Level”</h4><p>The winners of the next decade won’t be the companies that used AI to cut payroll by 20%. Those companies will be stuck in a race to the bottom. The winners will be the organizations that <strong>reinvested</strong> their human capital into higher-order work:</p><ul><li><strong>In Tech:</strong> Shifting from fixing bugs to building systemic, self-healing architectures.</li><li><strong>In Healthcare:</strong> Moving from reactive treatment to high-touch preventative coaching and genetic counseling.</li><li><strong>In Education:</strong> Transitioning from the transmission of facts to the facilitation of ethical reasoning and critical thinking.</li><li>….</li></ul><h4>The Friction of the Pivot</h4><p>Admittedly, this pivot is hard. It requires a “strategic architecture” mindset rather than a “bookkeeping” one. It requires a tolerance for the friction of re-skilling.</p><p>While reducing headcount provides a short-term boost to the balance sheet, the long-term cost is the loss of the institutional memory needed to find the North Star. <strong>We shouldn’t be cutting the future to subsidize the present.</strong> We should be empowering our experts to lead us into the agentic era to build lasting value.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=16257243b415" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Infrastructure for Intelligence: Human Blueprint for Agents]]></title>
            <link>https://medium.com/@divyajain1/infrastructure-for-intelligence-human-blueprint-for-agents-293893930145?source=rss-70995a473b4d------2</link>
            <guid isPermaLink="false">https://medium.com/p/293893930145</guid>
            <dc:creator><![CDATA[Divya Jain]]></dc:creator>
            <pubDate>Mon, 11 Aug 2025 17:56:23 GMT</pubDate>
            <atom:updated>2025-08-11T17:56:23.257Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*okGo-wIY0vdyfKN7hieOQQ.png" /></figure><p>The rise of agentic AI promises a future where autonomous programs work alongside us, completing complex tasks and solving problems. This is an exciting prospect, but as we stand on the cusp of this new world, a critical question arises: what will it take for these agents to be truly effective? <strong>The answer lies not just in the agents themselves, but in the infrastructure that supports them</strong>. Much like human professionals, agents require a sophisticated ecosystem of tools, memory, and guidance to evolve from simple assistants into reliable, high-performing collaborators.</p><p>Consider the journey of a new software engineer. They may possess a deep understanding of programming languages and algorithms, capable of writing a functional application on their own. However, for that application to be deployed, maintained, and scaled for billions of users, that engineer needs a vast support system. This includes version control systems like Git to manage code changes, continuous integration/continuous deployment (CI/CD) pipelines to automate testing and deployment, monitoring tools to track performance and catch errors, and collaboration platforms to document decisions and communicate with the team. <strong>This infrastructure is what transforms a skilled individual into a professional who can deliver a reliable, enterprise-grade product.</strong></p><p>Agents will face a similar, if not greater, challenge. An agent designed to create code, for instance, might be able to generate a solution in a single turn. But how does it ensure that code is secure, scalable, and compatible with existing systems? How does it learn from past mistakes or apply best practices from a hundred previous projects? This is where a dedicated agentic infrastructure becomes essential. Agents need their own equivalents of a human’s <strong>memory, context, databases, and evaluation systems.</strong></p><p>A new software engineer learning a company’s codebase doesn’t just read the code; they consult the internal wiki, read design documents, and talk to experienced colleagues to understand the “why” behind the “what.” This is the same for agents. To move beyond single-turn tasks, agents need a robust <strong><em>Persistent Memory and Context Store combined with a Structured Knowledge &amp; Information Layer.</em></strong> The persistent memory acts as the agent’s long-term recall, a knowledge graph or vector database that remembers past project details, successful strategies, and failures to avoid. The structured knowledge layer is its company wiki, providing access to a canonical source of information — API documentation, architectural patterns, and brand guidelines. Together, these systems provide the agent with the crucial context it needs to make informed decisions and produce work that is not only functional but also aligned with a company’s standards. This memory and knowledge infrastructure is what allows an agent to build on its experience and truly grow into a valuable collaborator.</p><p>But most importantly, we need a <strong>Performance Management and Evaluation Framework</strong>. Just as a team lead works with an intern or a new graduate, guiding them through complex projects and providing actionable feedback, a similar supervisory role will be critical for agents. <strong>This is the agent’s manager, team lead, and peer review process all in one.</strong> This framework would automatically evaluate an agent’s output based on objective metrics. For a software-writing agent, this might mean running unit tests, checking performance numbers, and even conducting a static code analysis to ensure quality. It would provide agents with specific, measurable feedback, allowing them to iteratively improve their performance. This system would also track an agent’s reliability and efficiency, providing valuable data for future optimization and training.</p><p>Before this framework becomes fully autonomous, a critical phase — let’s call it <strong>The Agent Nurturing Phase</strong> — will involve human software engineers taking on this role directly. This new agentic world requires a thoughtful, phased plan. We can’t simply flip a switch. They will need specialized tools to efficiently perform these management tasks. Having access to an agent’s historical data, performance metrics, and evaluation results in a consumable form will be paramount. These tools will enable humans to provide clear feedback and make precise tweaks to an agent’s behavior, making this period of co-development a crucial step before we can confidently transition to a fully agentic world.</p><p>This means defining the right set of KPIs and metrics for our agents and setting a baseline to measure agent improvements. We need to go beyond pass/fail and establish clear, measurable criteria for code quality, security vulnerabilities, performance benchmarks, and adherence to style guides. Next, we need to build intuitive UIs that allow us to quickly review an agent’s work, understand its decision-making process, provide precise feedback, and make tweaks without having to dive into complex configurations. Ultimately, this is about building a system that fosters agent growth while ensuring the team maintains control and accountability.</p><p>In the fully agentic world, this framework could also facilitate what we might call <strong>“multi-agent supervision,”</strong> where one agent acts as a manager, overseeing the work of others, ensuring consistency, and providing high-level guidance based on the project’s goals. However, before we can go into this powerful, self-organizing system where agents not only perform tasks but also improve the collective intelligence of the team, we need to go through a nurture phase.</p><p>The transition to an agentic world is not just about building smarter models, but about building the entire ecosystem that will allow them to flourish. Just as world-changing technologies like the printing press, automobile, or the internet required an entire new infrastructure of systems and processes to reach their full potential, agents will require a similar foundation. By creating this infrastructure, we are not just enabling agents; we are laying the groundwork for a new, more powerful form of human-AI collaboration.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=293893930145" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Glimpse Under the Hood of Adobe’s AI and ML Innovations]]></title>
            <link>https://medium.com/adobetech/a-glimpse-under-the-hood-of-adobes-ai-and-ml-innovations-54c8155801a8?source=rss-70995a473b4d------2</link>
            <guid isPermaLink="false">https://medium.com/p/54c8155801a8</guid>
            <category><![CDATA[mls]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[adobe]]></category>
            <category><![CDATA[adobe-sensei]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Divya Jain]]></dc:creator>
            <pubDate>Fri, 01 Nov 2019 16:48:15 GMT</pubDate>
            <atom:updated>2019-11-01T16:48:35.477Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Glimpse Under the Hood of Adobe’s AI and ML Innovations: Adobe Sensei ML Framework</h3><p>Adobe Sensei is our AI and machine learning technology, powering intelligent features across all of the Adobe ecosystem. From Stock search to content aware fill in Photoshop to Auto Reframe in Premiere Pro, users across Adobe’s different suites are experiencing the benefit of features powered by Adobe Sensei. Let’s take a closer look at the framework that is powering this amazing technology.</p><p>Adobe Sensei ML Framework is built with one goal in mind: democratizing AI at Adobe. In other words, it means growing an ecosystem that makes it easier to build production worthy machine learning features and take them to market quickly. It is tackling the challenge of ML at scale. In August, we looked at the internals of the framework focused on delivering personalized experience through <a href="https://medium.com/adobetech/accelerate-intelligent-insights-with-adobe-experience-platform-data-science-workspace-89538bacbbea">Data Science Workspace</a>. The DSW is Adobe’s tool for quickly bringing content understanding and intelligence to users, helping them access and unlock insights from data more effectively.</p><p>Before we jump into the details of the framework, and how it’s being used in other exciting ways at Adobe, let us take a quick look at one of features “Smart Crop,” powered by this framework.</p><figure><img alt="Adobe Sensei Smart Crop uses artificial intelligence to determine the main subjects in a photo of a woman playing frisbee." src="https://cdn-images-1.medium.com/max/1024/0*sRE3XCJI0s00wFb-" /><figcaption>Adobe Sensei Smart Crop uses artificial intelligence to determine the main subjects in a photo of a woman playing frisbee with her dog in a park and automatically crops them into focus.</figcaption></figure><h3>Adobe Sensei ML Framework</h3><p>We are seeing trends in the rise of internal AI platforms in many different companies: Uber has Michelangelo, Facebook has FBLearn, and Adobe has Adobe Sensei. The rise of these platforms is necessary because of companies’ specific needs; these needs cannot be fulfilled by general purpose platforms, and the rigor and workflow needed to sustain the scale for each company calls for an in-house machine learning framework.</p><p>Adobe has very unique scale requirements. Along with the sheer volume of users and assets that we have, scale expands in a number of different dimensions:</p><ul><li>The range of content types is not limited to images and videos, but rather spans a complex portfolio of content types like PDF, PSD, AE, etc.</li><li>Adobe’s product suite is supported on a wide range of user devices and supporting ML on different kinds of hardware with the same expectations poses its own challenges. How do you run an algorithm effectively on a CPU when it was trained for a high compute GPU?</li><li>Adobe products have a diverse set of users. Making features that scale from novice users to experts in tools like Photoshop is not an easy task. This requires you to look at machine learning from a different perspective. Your AI grows and changes as your user changes over a period of time.</li><li>Given our commitment to innovation, we are always pushing the envelope of research and need to make sure that we are always working with the latest infrastructure. We never want to box ourselves into one technology. Working with multiple different technologies at the same time and bringing them together as one, has its own challenges.</li></ul><h3>The three pillars of the Adobe Sensei ML Framework</h3><p>Given the specific needs of our company around ML, we took up the big task of creating a ML platform that supports and solves at scale. We created a three-part framework to do this, with each pillar coming together to solve the problem end-to-end. The three pillars of this framework are:</p><ul><li>Sensei Training Framework</li><li>Sensei Inference and Content Processing Framework</li><li>Sensei On-Device Framework</li></ul><h3><strong>A deeper look at the Adobe Sensei Training Framework</strong></h3><p>The goal of our training framework is to get researchers, working in AI and ML, started in the minimum amount of time while at the same time giving them flexibility to use the tools they are used to. This framework provides researchers with SDKs and templates for all popular ML frameworks like Tensorflow, Pytorch etc. so they can start working immediately without worrying about setting up the environments. These researchers start by choosing the framework and the type of compute they would need and the Training Framework has it provisioned for them in an instant.</p><p>Once the environment is ready, the first big step to solving any machine learning problem is data. Most of the time is spent in getting data in a form that the model can be trained with. The framework provides common datasets to researchers in forms that they don’t have to spend time creating from scratch. It also provides workflows that bring in new datasets and runs distributed spark jobs for data processing and cleaning.</p><p>The other aspect of the modeling framework is around running experiments to train a model. Supporting Adobe’s full research organization, which runs hundreds of training experiments at the same time, is no easy task. Along with the cost of GPU machines, we needed to develop a training framework that gave the researchers an ability to track and reproduce their experiments, without worrying about the changes that are done between the experiment runs. We created workflows that save the information around experiment runs and helps researchers compare results from various experiments. This is accomplished by encapsulating all the information around an experiment in an entity called “engine.”</p><p>Looking to the future, we are developing more standardized evaluation and visualization tools to make it easier for researchers to find common ways of evaluating the models. We are also looking at emerging technologies like AutoML for hyperparameter tuning and Neural Network architecture selection.</p><figure><img alt="A flow chart highlighting the Adobe Sensei Training framework and how it interacts with the Adobe Experience Platform." src="https://cdn-images-1.medium.com/max/1024/1*9SMlXnYdAjz0cgwNjWoIYA.jpeg" /><figcaption>Inference and Content Processing Framework: Dealing with unique scaling challenges.</figcaption></figure><h3>Inference and Content Processing Framework: Dealing with unique scaling challenges</h3><p>Once a model is trained, it is ready to be deployed; however, deploying a model for a product integration has lot of challenges — it needs to be scaled properly, all the environment details need to be replicated in the production, and the right monitoring and alerting needs to be in place. All of these challenges lead to a long tech transfer time between when the model is ready and when it becomes part of the product.</p><p>For Adobe Sensei ML Framework, we created an infrastructure that is cloud agnostic and can scale up and down as needed. It also provides capabilities to configure right access controls, rate limiting policies, and more. This is done by extending the ‘engine’ to have a service specification file, generated by the framework and based on your choices. This helps in deploying the model in online and offline modes with just a few clicks.</p><p>Along with this, the ‘engines’ are configured in a way that they can be stacked together to build more complicated workflows and services. This required creating a special language to do the stitching. Soon, we realized that ML models don’t work in isolation; what we needed was a lot of pure content processing in the same infrastructure that could be stitched together. This led to developing more capabilities in the framework to support streaming and complex fan-in and fanout scenarios.</p><p>One of the other key requirements that emerges when a new model is ready, is the ability to re-index existing assets. There are billions of assets that need to be re-indexed every time a new analyzer comes in; we needed a batch inference pipeline to optimize on the resources without affecting the real-time pipeline. To solve for this, we optimized at different levels, starting with the downloading and caching of assets to memory and RAM optimizations to successfully re-indexing 100 million assets in less than a day. The following figure shows the architecture of the current pipeline.</p><figure><img alt="A flow chart highlighting Adobe’s Inference and Content Processing Framework." src="https://cdn-images-1.medium.com/max/1024/1*9UFYFvBAfa0CSte3G-lqWw.jpeg" /><figcaption>Adobe’s Inference and Content Processing Framework.</figcaption></figure><h3>Creating an on on-device framework for machine learning</h3><p>The third pivotal pillar of the framework is the On-Device Framework. One of the biggest challenges for Adobe and its suite of products is to support many different kinds of hardware that users are using across the globe. Given the data privacy concerns and latency requirements, a need for <a href="https://medium.com/adobetech/addressing-the-challenges-of-on-device-machine-learning-1f71ebcedd69">ML on devices</a> is a very important aspect of scaling ML for Adobe.</p><p>Since there are already hundreds of models deployed in cloud, starting from scratch and building new on-device models is not possible. We needed a solution that make these features available on device quickly. We are seeing innovations across the industry in this field and we wanted a solution that uses the best possible stack, while also eliminating the stress that app developers face in having to understand the specific details of the stack. To accomplish this, we developed an abstraction of the SDK that runs on-device, taking care to encapsulate the best framework for the device while keeping the same API across devices. Based on the device, it automatically chooses the optimized framework.</p><p>We are also working towards improving conversion and compression pipelines — this would automate the conversions of models for device-specific runtimes with the best performance and accuracy. Federated learning is also an important research topic, and we are already seeing some great results that will become part of the framework soon.</p><figure><img alt="SDK that uses a specific h/w layer optimization and ML framework combo based on the device type." src="https://cdn-images-1.medium.com/max/1024/1*dJPwzZEBjOKXAsPOpDURdA.jpeg" /><figcaption>SDK that uses a specific h/w layer optimization and ML framework combo based on the device type.</figcaption></figure><h3>Adobe Sensei Content Framework: Accelerating AI innovation</h3><p>Adobe Sensei ML Framework is a complex large-scale distributed system. Along with data scientists, machine learning engineers, distributed system experts, device engineers, and experts in software engineering, great product managers and program managers were also needed to make it a reality. Our investment in building Sensei ML Framework shows how Adobe is accelerating AI innovations throughout the company. The framework addresses the unique needs of our efforts to achieve AI innovation at scale; it makes it easier than ever for internal Adobe teams to build and produce AI and ML features and deliver the best experience to our customers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=54c8155801a8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/adobetech/a-glimpse-under-the-hood-of-adobes-ai-and-ml-innovations-54c8155801a8">A Glimpse Under the Hood of Adobe’s AI and ML Innovations</a> was originally published in <a href="https://medium.com/adobetech">Adobe Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Addressing the Challenges of On-Device Machine Learning]]></title>
            <link>https://medium.com/adobetech/addressing-the-challenges-of-on-device-machine-learning-1f71ebcedd69?source=rss-70995a473b4d------2</link>
            <guid isPermaLink="false">https://medium.com/p/1f71ebcedd69</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[adobe-sensei]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Divya Jain]]></dc:creator>
            <pubDate>Tue, 29 Oct 2019 20:20:16 GMT</pubDate>
            <atom:updated>2019-10-30T16:00:51.777Z</atom:updated>
            <content:encoded><![CDATA[<p>On-device machine learning is a recent breakthrough that enables machine intelligent features to run on users’ personal devices. A few years back, deep learning changed the paradigm for machine learning (ML), and with that we saw many new products emerge in various verticals including health care, education, industrialization, and finance. Most of these are possible because of the unlimited compute of the cloud, along with the intersection of a vast amount of data. However, this has also led to thoughts and questions on the challenges around privacy and data governance, and it’s opened a greenfield for various companies looking to bring the intelligence closer to the data, on users’ devices. On-device ML also brings other benefits like real-time performance and personalization, and therefore is an important and natural space for growth in the ML world, despite its associated challenges.</p><figure><img alt="Some of the popular libraries and interfaces used for on-device machine learning." src="https://cdn-images-1.medium.com/max/737/0*wKsK1Dt6OHScK7Mo" /><figcaption>Some of the popular libraries and interfaces used for on-device machine learning.</figcaption></figure><p>The industry is witnessing a lot of innovations across the stack around on-device ML, from new hardware to applications. However, for an enterprise starting fresh in this space, it is a struggle to architect a solution that can scale across millions of devices with different versions and different usages. These are similar to the challenges we saw pre-cloud when we were distributing software to customers; it became very difficult to upgrade and maintain various versions. Once again, we find ourselves facing this problem.</p><p>Here at Adobe, given our vast number of applications and the supported devices, this challenge has been even more daunting. Let’s start by walking through some of the challenges we’ve faced when implementing on-device ML in our offerings. The figure above shows some of the popular libraries and interfaces used for on-device ML.</p><h3>The challenge of framework choices</h3><p>One of the key challenges we faced was to come up with a solution that spans several frameworks, with each one providing its own set of features and benefits. There is no common intermediate layer, and to add to that, the ecosystem is still evolving with inference libraries getting more mature and technologies like <a href="https://tvm.ai/">TVM</a> and <a href="https://developer.android.com/ndk/guides/neuralnetworks">NNAPI</a> still emerging.</p><p>To address these challenges, our approach was to build a solution that wouldn’t lose agility but would support cross-platform work. This solution is what we call the Adobe Sensei On-Device Framework. We created an abstraction layer to allow experimenting with evolving technology and quick iterations. Cross-platform portability, platform-specific optimization, and performance were all key requirements — so we built a common C++ SDK that used a specific h/w layer optimization and ML framework combo for a given device. For example, for devices using Windows operating systems, we bundled WinML, whereas for devices on iOS we used Core ML. The top applications use this SDK to enable ML features, and the SDK takes care of encapsulating all the details of the different frameworks.</p><p>This has helped us build a solution that uses state-of-the-art ML while at the same time giving us the flexibility to change the underlying framework without needing to push new releases for applications. It’s definitely future-proofing us in an ever-changing landscape.</p><figure><img alt="SDK that uses a specific h/w layer optimization and ML framework combo based on the device type." src="https://cdn-images-1.medium.com/max/1024/0*Wlyq4i4cqoM2unok" /><figcaption>SDK that uses a specific h/w layer optimization and ML framework combo based on the device type.</figcaption></figure><h3>The challenge of model conversion</h3><p>Most of the training and research for ML models is still done using Pytorch, Caffe, and Tensorflow frameworks, and these models can reach up to hundreds of megabytes in size. Downloading and using such big models is not feasible for devices. Therefore, the models need to be converted into a device friendly format like <a href="https://developer.apple.com/documentation/coreml">CoreML</a>, <a href="https://www.tensorflow.org/lite">TFLite</a>, or <a href="https://docs.microsoft.com/en-us/windows/ai/">WinML</a>, which also bring the added bonus of better inference performance. However, these device frameworks only support basic operators, and the moment you try to convert Pytorch or Tensorflow models, they start to fail at various levels — from operators not being found, to compilation mistakes, to big sizes, and more.</p><p>We had to develop a pipeline for conversion that automates parts of the problem. However, many parts of the pipeline are still manual and need intervention at various points to do the conversion and compression. The open-source conversion tools out there don’t work for all frameworks and have limited support around compression.</p><h3>The challenge of device configuration</h3><p>Another large challenge we faced had to do with device versioning. Users have different device preferences, ranging from the iPhone 6 to iPhone 11 to the Samsung Galaxy 7 to Pixel 3. Many of these users never upgrade their devices and are running older versions of their OS.</p><p>Most of the frameworks worked on the latest versions of various operating systems, but since our customers use all various versions we need to support all of them. We had to either compile them on older versions or find alternate solutions for implementing compatibility between systems.</p><p>Similarly, not all Android vendors implement the NNAPI in the same way as Google, and TFLite cannot be used as per this reference. Therefore, we must create a pipeline for solving this problem at scale — a project that is still underway.</p><h3>The challenge of project execution</h3><p>We are seeing lots of interest from our product teams in integrating this SDK to deliver ML on- device, but a big win for our project was the execution of this project. We created an internal open-source project with different teams coming together to contribute. It was very important for people from different domains, like those that work with images, videos, or documents across the company, to come together to contribute and build something that can be reused and scaled.</p><p>This resulted in many features that are reusable across teams. Getting different teams on the same page and dealing with tradeoffs between features was a big win for all of us.</p><h3>This is just a start</h3><p>Still, we’ve only solved part of the equation. We are still in the middle of building a full-fledged conversion pipeline and working on various areas in the research around this. As we embark upon this journey further, our goal is to make it easier for AI/ML features to be available across all surface types at scale. Stay tuned to see what we accomplish next!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f71ebcedd69" width="1" height="1" alt=""><hr><p><a href="https://medium.com/adobetech/addressing-the-challenges-of-on-device-machine-learning-1f71ebcedd69">Addressing the Challenges of On-Device Machine Learning</a> was originally published in <a href="https://medium.com/adobetech">Adobe Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Leveraging AI and Deep Learning for Video Summarization]]></title>
            <link>https://medium.com/adobetech/leveraging-ai-and-deep-learning-for-video-summarization-b29e5e1c164b?source=rss-70995a473b4d------2</link>
            <guid isPermaLink="false">https://medium.com/p/b29e5e1c164b</guid>
            <category><![CDATA[summarization]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[adobe-sensei]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[videos]]></category>
            <dc:creator><![CDATA[Divya Jain]]></dc:creator>
            <pubDate>Thu, 16 May 2019 12:06:00 GMT</pubDate>
            <atom:updated>2019-06-05T16:34:49.233Z</atom:updated>
            <content:encoded><![CDATA[<p>The global video market is taking center stage — According to <a href="https://www.forbes.com/sites/tjmccue/2018/06/22/video-marketing-2018-trends-continues-to-explode-as-the-way-to-reach-customers/#7c5b00b3598d">Forbes</a>, more than 500 million hours of video are watched on YouTube every day. <a href="https://think.storage.googleapis.com/docs/shopping-micro-moments-mobile-trends.pdf">Google</a> adds that almost 50 percent of internet users look for videos related to a product or service before visiting a store.</p><p>Many such statistics show how video content is growing and will remain the mainstream as a means of sharing information. We are already seeing a shift from copy and text to snapshot stories and visual posts (e.g. Instagram) for sharing content. Artificial intelligence (AI) is also playing a large role in this shift to video. We can use AI to improve video quality by stabilization, to understand and classify content for editing purposes, or to better deliver and target.</p><p>AI is also playing a key role with video summarization, a process of shortening a video by selecting keyframes or parts of videos that captures the main points in the video. Summarization has many use cases, with one of the most significant being the ability to gauge interest in content. A flashcard summary can determine how many people will actually watch an entire video. Even a single thumbnail plays a crucial role in determining how many people will click on a video to play it. Along with determining video clicks, video summarization is also necessary for efficient viewing of the material and for video length adaptation for different mediums, like Instagram, Facebook, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/859/0*YTjPho9r92-ENDWs" /><figcaption><em>Global Video Market is increasing and taking the center stage.</em></figcaption></figure><p>Recently, there have been many advances in using deep learning to increase the processing of images; the ability for AI to understand an image’s context has rapidly improved in accuracy. Similar techniques can be used to understand videos too, but this is a much more complex process. Video is not just a collection of a large number of frames or images, but videos are multi-dimensional — including audio, motion, and a time-series dimension. Each of these dimensions are key in understanding a video, and depending on what the summarization is targeting, different dimensions can be crucial.</p><h3>The anatomy of AI video summarization</h3><p>Video summarization can be categorized into two broad areas of machine learning: supervised and unsupervised. Supervised summarization entails learning patterns from previously annotated videos and examples. This works very well in case of videos where a pattern exists, like sporting events. For these videos, we can annotate some sequences and learn from them. However, the biggest challenge with supervised learning is the labeled data. It is costly to create these well-defined datasets. Labeling of data requires domain knowledge and does not work well when it comes to the wide variety of content that is present on the web.</p><p>The other machine learning form of summarization is unsupervised, where a smaller number of frames are selected from the original video through change detection in the video. Low-level features such as color, motion, and texture have been commonly used to create histograms and clusters to determine the similar frames within a video; a few frames are then selected that are deemed useful for the summary based on the information that they are conveying from the original video. These techniques work best when the video have distinct visual content, for example a video taken throughout the different days of a vacation. However, these summaries often lack the context and come out as disjointed images.</p><p>Recent forms of deep learning look very promising in addressing the above mentioned challenges; they lend themselves to much more effective creation of video summaries. While supervised deep learning techniques popularized the process, unsupervised techniques such as generative adversarial networks (GANs) and reinforcement learning are showing great promise, offering excellent advantages that are making them a forerunner in video summarization.</p><h3>The power of emerging unsupervised deep learning techniques in video summarization</h3><p>For videos that don’t adhere to any pattern and are completely different from each other, GANs work very nicely. GANs have two neural nets: an encoder that tries to mimic the real data and a decoder that is trying to learn if the generated data is fake or not. This helps GANs learn the data distribution very effectively and create data that is very difficult to distinguish from the original dataset. In this case, each video can be described as a dataset, with GANs creating a subset of frames that are most representative of the given videos. This generates unique summaries for videos while preserving the context and meaning of the videos themselves. This technique can be used by marketers for creating smaller versions of full length ads or campaigns based on the devices and target the right audience. This can also be used by creative artists to give a preview of their upcoming releases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pWWksM5BXDF7Gsa_" /><figcaption><em>Unsupervised Learning using GANs.</em></figcaption></figure><p>For videos that have a common structure, like sporting events, reinforcement learning is more effective than supervised learning because it does not require labeled data. Here, the neural nets can learn which frames to choose based on a reward function. They learn from previous summaries to determine whether certain frames were watched or skipped. Different kinds of rewards functions can also be defined in ways where previous information is not required, such as frame diversity and representativeness or frame category classification. Such techniques can be employed by campaign managers to create more watchable and memorable summaries from past experience and engage with their customers effectively.</p><p>At Adobe MAX 2018, we previewed a proof-of-concept demo of unsupervised deep learning technology in action, through our Video Ad AI presentation during Adobe Sneaks. We took the example of a 60-second TV spot that needed to be rolled out across social media platforms. The video is first auto-tagged for benchmarking against similar content, and then <a href="https://www.adobe.com/sensei.html">Adobe Sensei</a>, our AI and machine learning technology, assesses past performance of similar content. This leads to a set of recommendations to improve the ad’s effectiveness score (e.g. to change the length and flow of the video), and those are sent to video advertisers in Adobe Premiere Pro, along with the original ad, for editing. Advertisers can adjust the ad to retain creative elements, while aligning with marketing objectives to deliver high-performing content. Check it out in the video below:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FZ8zqOpChQnU%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DZ8zqOpChQnU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FZ8zqOpChQnU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/0f4be7120b7fe07c3d4f939ec4d4a11b/href">https://medium.com/media/0f4be7120b7fe07c3d4f939ec4d4a11b/href</a></iframe><p>These new unsupervised techniques are just the start of a new era in deep learning technology when it comes to video summarization. Many advances will be made in the near future to create and optimize the best summaries based on the audience, delivery medium, and intent of summarization. Our Adobe Sensei team is working with Adobe Research to bring these techniques to the Sensei framework. Together with efforts across the industry, we’ll make video summarization highly scalable, reliable, and incredibly efficient.</p><p><em>Portions of this article were originally published on </em><a href="https://www.clickz.com/ai-deep-learning-video-summarizing/236598/?utm_source=hootsuite&amp;utm_medium=&amp;utm_term=&amp;utm_content=&amp;utm_campaign="><em>ClickZ</em></a><em>.</em></p><p><em>Follow the </em><a href="http://medium.com/adobetech"><em>Adobe Tech Blog</em></a><em> for more developer stories and resources, and check out </em><a href="http://twitter.com/adobedevs"><em>Adobe Developers on Twitter</em></a><em> for the latest news and developer products.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b29e5e1c164b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/adobetech/leveraging-ai-and-deep-learning-for-video-summarization-b29e5e1c164b">Leveraging AI and Deep Learning for Video Summarization</a> was originally published in <a href="https://medium.com/adobetech">Adobe Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>