<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by John Duprey on Medium]]></title>
        <description><![CDATA[Stories by John Duprey on Medium]]></description>
        <link>https://medium.com/@jduprey?source=rss-f2ce837c4d4c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*xoyKfLL6TIVSoDWcugOe6A.png</url>
            <title>Stories by John Duprey on Medium</title>
            <link>https://medium.com/@jduprey?source=rss-f2ce837c4d4c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 09:37:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jduprey/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A First Look at CLI aNd Editor (CLINE)]]></title>
            <link>https://medium.com/tr-labs-ml-engineering-blog/a-first-look-at-cli-and-editor-cline-c96dbc7a6331?source=rss-f2ce837c4d4c------2</link>
            <guid isPermaLink="false">https://medium.com/p/c96dbc7a6331</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[ai-assisted-development]]></category>
            <dc:creator><![CDATA[John Duprey]]></dc:creator>
            <pubDate>Wed, 16 Apr 2025 20:26:52 GMT</pubDate>
            <atom:updated>2025-04-25T14:19:49.392Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>For Enterprise Developers</em></h4><h3>The Rise of AI-Assisted Development</h3><p>Imagine the leap from <a href="https://en.wikipedia.org/wiki/Teleprinter">teletype</a> and <a href="https://youtu.be/q456Rw3sq-M?si=u8-bIJNglUpP_QTE">punch cards</a> to interactive computer terminals that responded to your commands on screen. While MANY advancements have followed, few compare to the impact of today’s emerging AI tools.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VAMf_VkoEp1__DM6E25FTg.jpeg" /><figcaption>This punch card was given to us in our ES 111 Intro to Programming class — as a novelty. We laughed at how antiquated punch cards were.. what aspects of programming today will seem antiquated in a few year’s time?</figcaption></figure><p><strong>AI-assisted development</strong> is rapidly transforming how we write code. This transformation is powered by improved large language models (LLMs) and tools that support <a href="https://www.ibm.com/think/topics/agentic-workflows">agentic workflows</a>, where AI can take more autonomous actions to complete coding tasks. The emerging <a href="https://modelcontextprotocol.io/introduction">Model Context Protocol (MCP) standard</a> further enhances these tools by enabling external tool execution.</p><p>This article is a follow-up to my previous piece “<a href="https://medium.com/tr-labs-ml-engineering-blog/ai-assisted-development-redefining-how-and-when-we-code-098a16688e86">AI-Assisted Development: Redefining How (and When) We Code</a>”. While that article explored the broader implications, here I’ll focus specifically on <strong>CLINE</strong> (CLI aNd Editor), an open-source AI coding assistant that has garnered interest at Thomson Reuters.</p><p>The landscape of AI coding tools is expanding rapidly. Commercial options like <a href="http://cursor.com/">Cursor</a>and <a href="https://replit.com/">Replit</a> demonstrate significant potential in automating coding tasks. Meanwhile, free and open-source software (FOSS) alternatives like <a href="https://github.com/cline/cline">CLINE</a> and <a href="https://openalternative.co/alternatives/cursor">others</a> offer similar capabilities with greater transparency and the flexibility to use enterprise-licensed models.</p><p>I’ll be examining CLINE through an enterprise software development lens, grounding our exploration in business values rather than unconstrained possibilities. While tools like CLINE can accelerate development, they may also introduce risks to an organization’s security and intellectual property (IP). Businesses must understand and mitigate these risks to leverage the benefits responsibly.</p><blockquote><strong><em>TL;DR:</em></strong><em> CLINE works as advertised and offers powerful AI-assisted development capabilities. For enterprise use, suggested risk mitigation includes: limiting usage to enterprise-controlled LLMs, using development containers to minimize tool execution impact, and implementing auditing mechanisms.</em></blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FKFQkhd9gQpY%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DKFQkhd9gQpY&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FKFQkhd9gQpY%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a513ecb001a5f789439a553b90f6d265/href">https://medium.com/media/a513ecb001a5f789439a553b90f6d265/href</a></iframe><h3>What is CLINE? Core Capabilities and Features</h3><p>CLINE is a <a href="https://code.visualstudio.com/">Visual Studio Code (VS Code)</a> extension that functions as a “collaborative AI coder.” At its core, CLINE offers two distinct operational modes:</p><ol><li><strong>Plan Mode</strong>: Users describe their tasks, and CLINE recommends a step-by-step path to completion, providing reasoning for each step. This mode allows for discussion and refinement before any code is written or modified.</li><li><strong>Act Mode</strong>: The assistant executes tools and code to complete the tasks, with options to verify each command/tool before execution.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*96F_qCaEVB5rp-Jcuu48_Q.png" /><figcaption><em>⤴ CLINE in “Plan” mode, showing how it breaks down tasks into logical steps</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BQ9tkwIGCuX2dBPZxumSoQ.png" /><figcaption><em>⤴ CLINE in “Act” mode, requesting confirmation before executing commands</em></figcaption></figure><p>It’s common to start in “Plan” mode and interactively outline what you’d like to accomplish. CLINE will outline a course of action. When you are finally satisfied with the course of action, you switch to “Act” mode. It will then start to make changes to your project. Your level of interaction with it in “Act” mode depends on your settings.</p><p>CLINE provides significant control over its behavior through configurable settings. One of the most important settings is “Auto-approve: …” which determines whether CLINE will automatically execute commands or request confirmation first.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*7tGTsnDkMrz1KM2gchi7nw.png" /><figcaption><em>⤴ CLINE configuration options for command approval settings</em></figcaption></figure><blockquote><strong><em>Recommendation:</em></strong><em> I recommend keeping “Auto-approve Commands, MCP” </em><strong><em>off </em></strong><em>or limited to only “safe commands” until you’re comfortable with the tool. While reviewing commands can slow down the </em><a href="https://en.wikipedia.org/wiki/Vibe_coding"><em>vibe coding</em></a><em> experience, it provides an important safety layer, especially in enterprise environments. You can back this off based on your situation, perceived risks, and the environment you are working in. To truly “vibe code,” you will have to enable most auto approval because the volume of code produced can be too much to review all up front.</em></blockquote><h3>Benefits for Enterprise Developers</h3><p>After extensive testing, I can confirm that <strong>CLINE works as advertised</strong>. It delivers on its promise of interactive and agent-based development, offering several key benefits for enterprise developers:</p><ol><li><strong>Accelerated Development</strong>: CLINE can significantly speed up coding tasks, from boilerplate generation to complex implementations.</li><li><strong>Reduced Context Switching</strong>: The integrated nature of CLINE within VS Code means less jumping between documentation, Stack Overflow, and your IDE.</li><li><strong>Improved Code Quality</strong>: With the right LLM, CLINE can suggest best practices and identify potential issues in your code.</li><li><strong>Learning Opportunities</strong>: Working with CLINE can expose developers to new patterns and approaches they might not have considered.</li></ol><p>My testing of CLINE was ad-hoc, involving tasks such as creating Python UIs, parsing CSV and markdown files, and converting markdown to a static site with <a href="https://astro.build/blog/astro-5/">Astro 5</a>.</p><p><strong>Testing Results:</strong></p><ul><li><strong>Initial Testing</strong>: With basic models, results were mixed. Some code required manual correction, and in some cases, writing from scratch would have been more efficient.</li><li><strong>Advanced Models</strong>: Switching to Claude Sonnet 3.7 drastically improved output quality. Most code worked with minimal intervention, typically only requiring fixes for simple logic issues in unit tests.</li></ul><blockquote><strong><em>Key Finding</em></strong><em>: The choice of LLM makes a significant difference in output quality. Claude models performed best in my testing, while Google Gemini models excelled specifically at refactoring and cleaning up code created by other models.</em></blockquote><p><a href="https://docs.cline.bot/exploring-clines-tools/checkpoints">Cline Checkpointing</a> is your friend. Use it to review changes after the fact and, if needed, restore to a previous checkpoint. Also use your project’s version control to save functional checkpoints that you are happy with. I’ve had scenarios where CLINE has created a perfectly working solution, but upon follow up to build out more unit tests or add additional features has “gone off the rails” — deviating from its previous good design, mixing disparate ways to do the same thing, sometimes introducing more errors.</p><p>Other details that may be of use to enterprise users can be found in CLINE’s <a href="https://docs.cline.bot/enterprise-solutions/security-concerns">Security Concerns</a> and <a href="https://docs.cline.bot/enterprise-solutions/mcp-servers">MCP Servers</a> documentation.</p><h3>Model Selection and Integration</h3><p>For enterprise use, model selection is a critical consideration. CLINE supports various model providers, including those with which enterprises may have licensing agreements.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/126/1*q3mjabXYTjUqMNVLkL8jGw.png" /><figcaption><em>⤴ CLINE’s supported model providers</em></figcaption></figure><blockquote><strong>Important Enterprise Consideration: M<em>odels not licensed by your organization should NOT be used.</em></strong><em> Your business likely considers its proprietary code and related data found in source code projects its intellectual property. As such, it will want to review and control the terms and conditions of how that data is used and where it is sent.</em></blockquote><h3>GitHub Copilot Integration</h3><p>If your organization has already licensed GitHub Copilot, CLINE offers a “VS Code LM API” integration, which uses LLMs available through VS Code extensions like GitHub Copilot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5hCfXpovirnorSRWmnigfA.png" /><figcaption><em>⤴ VS Code LM API integration showing available models from GitHub Copilot</em></figcaption></figure><blockquote><strong><em>Note:</em></strong><em> It has not been confirmed if this integration pathway is compatible with GitHub Copilot terms of service. Seek guidance from your organization’s procurement team before using this integration.</em></blockquote><blockquote><em>⚠️ At the time of testing, </em><em>copilot-claude-3.7-sonnet* did not work with this integration, though this may change in future updates.</em></blockquote><h3>MCP Integration: Extending CLINE’s Capabilities</h3><p>CLINE’s <a href="https://docs.cline.bot/mcp-servers/mcp">Model Context Protocol (MCP) integration</a> significantly extends its capabilities beyond basic code generation. This feature enables more powerful app creation through CLINE’s direct interaction with external tools like web browsers, databases, and more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*_zj8M7DXnQQWQ_NjHcyRfQ.png" /><figcaption><em>⤴ The growing marketplace of CLINE-compatible MCP tools</em></figcaption></figure><p>The MCP marketplace continues to grow, adding new tool capabilities regularly. However, it’s important to remember that each MCP tool is essentially its own application with its own behaviors and security posture. Cline <a href="https://docs.cline.bot/enterprise-solutions/mcp-servers">reports to handle MCP server security at multiple levels</a>, but at the end of the day these tools are their own application.</p><blockquote><em>⚠️ </em><strong><em>Recommendation:</em></strong><em> Carefully evaluate each MCP tool before use in an enterprise environment. Consider the tool’s data handling practices, security implications, and compatibility with your organization’s policies.</em></blockquote><p>Consider how company-specific MCP servers might <strong>help</strong> mitigate risks or further accelerate development. You could add code scanners or other tools to help vet code generated by the LLM. You could make company-specific data resources available to Cline that would allow it to better generate code for company-specific APIs. Thomson Reuters is exploring such options. Stay tuned for more developments in this area!</p><h3>Risks and Responsible Use</h3><p>At Thomson Reuters, we enjoy considerable freedom in choosing OSS tools, but with that freedom comes responsibility. <strong>Our engineers are expected to act responsibly</strong> when adopting new technologies.</p><p><strong>When considering tools like CLINE, developers should ask themselves</strong>:</p><blockquote><em>What does this tool actually do? How does it do it? What third parties might it interact with, and what will it do with my code, data, etc.?</em></blockquote><h3>Primary Enterprise Risks</h3><p>There are two main risks when using AI-assisted development tools in an enterprise environment:</p><ol><li><strong>IP Loss</strong>: Sending proprietary code and data to third-party LLMs (and other tools used in agentic workflows) could potentially expose intellectual property.</li><li><strong>System Security</strong>: Execution of commands and tools in the development environment poses risks through operator error or potentially malicious outputs.</li></ol><h3>Comprehensive Mitigation Strategies</h3><p>To address these risks while still benefiting from CLINE’s capabilities:</p><blockquote>The biggest risk in any system are.. people!</blockquote><ul><li><strong>Educate Developers</strong>: <em>The biggest risk in any system are.. people</em>! Implement regular training on safe and responsible tool use for all developers.</li><li><strong>Use Approved LLM Providers Only</strong>: Restrict usage to enterprise-approved and licensed LLM providers that have appropriate data handling agreements in place.</li><li><strong>Limit MCP Tool Usage</strong>: Only use MCP tools that have been evaluated and deemed safe for your organization.</li><li><strong>Implement Development Containers</strong>: Run CLINE in a sandboxed environment to limit the impact of errant tool execution. Configure containers to only map the specific project path/folder into the sandbox.</li><li><strong>Establish Auditing Mechanisms</strong>: At the organization level, implement processes to check and correct workstations for compliance with security policies.</li><li><strong>Contribute to Security Improvements</strong>: Because CLINE is open-source, organizations can propose and submit changes to enhance its security features.</li></ul><h3>Comparison with Other AI Coding Tools</h3><p>As AI-assisted development tools proliferate, it’s valuable to compare CLINE with other popular options. Tools like Cursor, Claude Code, GitHub Copilot (with Agent Mode), and others offer similar capabilities with varying approaches.</p><h3>Quality Assessment: It’s About the LLM</h3><p>In my testing across Python CLI development, data analysis tasks, and Astro static site creation, I discovered an important insight:</p><blockquote><strong>The quality of results was remarkably similar across tools when using the same underlying LLM</strong>. At least in my experience, the biggest differentiator was not the tool itself, but which LLM you select.</blockquote><p>Claude Sonnet 3.7 consistently produced high-quality results across different tools, while Google’s Gemini 2.0 models showed particular strength in refactoring tasks. (Gemini 2.5 is supposed to be even better.)</p><blockquote>This finding suggests that as organizations evaluate these tools, they should focus as much on which LLMs they have licensed.</blockquote><h3>Feature Comparison Table</h3><p>Here’s how CLINE compares to other popular AI coding assistants:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/11be4929dde9e56a76e910ebb69b61fd/href">https://medium.com/media/11be4929dde9e56a76e910ebb69b61fd/href</a></iframe><blockquote>⚠️<em> * The CLINE models list are actually various provider/integration pathways. Needless to say, the list is comprehensive and can support any model these providers offer.</em></blockquote><blockquote>CLINE’s key differentiators are its open-source nature, extensive MCP integration capabilities, and flexibility to use virtually any LLM including AWS Bedrock. However, <strong>all these tools are rapidly evolving, and the distinctions between them may blur over time</strong>.</blockquote><h3>Recommendations and Best Practices</h3><p>Based on my testing and analysis, here are key recommendations for enterprise adoption of CLINE:</p><p><strong>1. Start with Restricted Auto-Approve Settings</strong></p><ul><li>Keep “Auto-approve Commands, MCP” off initially</li><li>Gradually enable for safe commands as you become comfortable with the tool</li><li>Consider your development environment when determining what’s “safe”</li><li>To fly with “vibe coding” you will need to “let it cook” which means more permissive autonomous actions — balance use case with risk mitigation</li></ul><p><strong>2. Choose LLMs Strategically</strong></p><ul><li>Use only enterprise-licensed models</li><li>Test different models for your specific use cases</li><li>Consider Claude models for general coding tasks</li></ul><p><strong>3. Implement Proper Sandboxing</strong></p><ul><li>Use development containers to isolate CLINE’s operations</li><li>Configure containers to only map necessary project paths</li><li>Consider read-only access for sensitive directories</li></ul><p><strong>4. Establish Coding Norms with CLINE</strong></p><ul><li>Explore using <a href="https://docs.cline.bot/improving-your-prompting-skills/prompting">.clinerules</a> and <a href="https://docs.cline.bot/improving-your-prompting-skills/cline-memory-bank">memory bank</a> to help CLINE keep key details about your project and standards in context. (I’ve yet to explore this, myself, but seems like an obvious way to help it remember how to add dependencies, run tests, etc.)</li></ul><p><strong>5. Establish Clear Usage Policies</strong></p><ul><li>Define when and how CLINE should be used in your development workflow</li><li>Create guidelines for reviewing AI-generated code</li><li>Establish protocols for handling potential IP concerns</li></ul><p><strong>6. Monitor and Adapt</strong></p><ul><li>Keep track of how CLINE affects development productivity and code quality</li><li>Stay informed about updates to both CLINE and competing tools</li><li>Be prepared to adjust your approach as the technology evolves</li><li>Monitor spend closely, set spend limits, and adjust as you learn</li></ul><h3>Return on Investment Considerations</h3><p><strong>Cost factors</strong> for CLINE vary depending on your approach. While CLINE offers its own credit-based purchasing system suitable for individuals or small teams, most enterprises will leverage their existing LLM licensing agreements. CLINE’s transparent token usage metrics help manage costs by clearly displaying context window, token consumption, and resulting API cost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/447/1*qgnZlZhmPWlKw0M5IDBiyw.png" /><figcaption>⤴ CLINE’s transparency features help monitor and control costs</figcaption></figure><p><strong>Measuring value</strong> should focus on tangible outcomes. Look for improvements in:</p><ul><li>Development velocity (e.g. faster story completion)</li><li>Code quality metrics (e.g. test coverage, defect rates)</li></ul><p>At Thomson Reuters, we track impact through developer surveys attached to pull requests and have implemented deeper telemetry (e.g. suggestion acceptance rate) in tools like GitHub Copilot and intend to do the same for CLINE.</p><blockquote>This technology is too promising to ignore — invest strategically, but track impact against your organization’s established productivity and quality metrics to weigh the ROI.</blockquote><h3>Conclusion</h3><p><strong>CLINE works as advertised and offers capabilities beyond current commercial GitHub Copilot</strong>. It represents one of many next-generation AI-assisted tools that promise to revolutionize software development. For enterprise organizations, denying this class of tool could potentially harm competitive advantage, though we are still in the early days of this technology.</p><blockquote><em>The ecosystem of competing tools is rapidly evolving. Determining the best AI-assisted tools for enterprise developers remains an open question. Until clear winners emerge, we need to monitor this space, make informed decisions, and use these tools responsibly. </em><strong><em>Think before you act!</em></strong></blockquote><p>By implementing the recommended mitigation strategies, organizations can leverage these tools’ benefits while minimizing potential security and IP threats. Continuous education, careful selection of LLM providers, prudent use of MCP tools, and robust auditing practices are essential to maintaining a secure and efficient development environment.</p><p>The recommendations provided here are generally applicable to OSS tools like CLINE and largely fall under the category of “think before you act!” As these tools continue to evolve, so too should our approaches to using them responsibly in enterprise environments.</p><h3>💬 Join the Conversation</h3><p>What is your experience with AI-assisted tools? Have you been using tools beyond GitHub Copilot? Any thoughts on responsible use — techniques to minimize risk? How is your org measuring impact to developer productivity? If we were to do a deeper feature comparison, what features do you think are the most important to consider?</p><p>Share your experiences in the comments, or start a discussion in our <a href="https://www.linkedin.com/groups/8168422/">LinkedIn Group</a>!</p><h3>Further Reading/References</h3><ul><li><a href="https://cline.bot/">Cline — AI Autonomous Coding Agent for VS Code</a></li><li><a href="https://simonwillison.net/2025/Mar/11/using-llms-for-code/">Here’s how I use LLMs to help me write code</a> — A practical blog post on using AI-assisted tools for development</li><li><a href="https://medium.com/tr-labs-ml-engineering-blog/ai-assisted-development-redefining-how-and-when-we-code-098a16688e86">AI-Assisted Development: Redefining How (and When) We Code</a> — My own blog post on the implications of AI-assisted development</li><li><a href="https://youtu.be/KFQkhd9gQpY?si=83O6IqGODGgezJE8">YouTube Video on how One Developer uses CLINE</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c96dbc7a6331" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tr-labs-ml-engineering-blog/a-first-look-at-cli-and-editor-cline-c96dbc7a6331">A First Look at CLI aNd Editor (CLINE)</a> was originally published in <a href="https://medium.com/tr-labs-ml-engineering-blog">Thomson Reuters Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reflecting on Labs’ Medium Publication in 2024]]></title>
            <link>https://medium.com/tr-labs-ml-engineering-blog/reflecting-on-labs-medium-publication-in-2024-58a0d95b4a71?source=rss-f2ce837c4d4c------2</link>
            <guid isPermaLink="false">https://medium.com/p/58a0d95b4a71</guid>
            <category><![CDATA[blogging-for-business]]></category>
            <category><![CDATA[research-and-development]]></category>
            <category><![CDATA[innovation]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[business]]></category>
            <dc:creator><![CDATA[John Duprey]]></dc:creator>
            <pubDate>Tue, 07 Jan 2025 21:17:20 GMT</pubDate>
            <atom:updated>2025-01-07T22:15:25.583Z</atom:updated>
            <content:encoded><![CDATA[<h4>Contemplating 2025…</h4><p>This year, we published over 30 blog posts on Medium, broadening our scope beyond machine learning engineering and quadrupling our content from previous year.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lEYOFBb6s-5fhnWR" /><figcaption>Photo by <a href="https://unsplash.com/@boliviainteligente?utm_source=medium&amp;utm_medium=referral">BoliviaInteligente</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In May, <a href="https://medium.com/tr-labs-ml-engineering-blog/welcome-to-the-thomson-reuters-labs-blog-1beb27998f40">we re-introduced our blog and its intent</a>. With <a href="https://www.thomsonreuters.com/en-us/posts/our-purpose/thomson-reuters-labs-at-30-celebrating-three-decades-of-innovation/">over 30 years of applied AI research at Thomson Reuters</a> our motivations were articulated:</p><blockquote><em>Thomson Reuters is a technology company, but we often fly under the radar — we don’t talk enough about the great work we do and the hard problems we solve. We truly are a world class AI company in the enterprise space, where it is traditionally </em>extremely<em> challenging to deliver and drive AI adoption. We have a great story to tell, and we could benefit from joining in the global dialog about developing and applying technology to solve real world problems. </em><strong><em>We hope this will generate some healthy discussion in science, engineering, and even business areas. Perhaps we can learn from each other’s experiences. And, lets be honest, it will help our brand as a technology company — to continue to attract the world class people needed to further progress our mission</em></strong><em>. — </em><a href="https://medium.com/tr-labs-ml-engineering-blog/welcome-to-the-thomson-reuters-labs-blog-1beb27998f40"><em>Welcome to TR Labs Medium Blog</em></a></blockquote><p>With the increased volume, we learned a lot about the process and the content we can publish. In this short article, I reflect on this and contemplate what it could mean for this publication in <strong>2025</strong>.</p><h3>Return on Investment and the Medium Platform</h3><p>While Medium is free to use, publishing quality content comes with a cost. Our very talented staff spent many hours proposing, drafting, and reviewing the content.</p><p>We are willing to invest in the endeavor to reinforce Thomson Reuters’ brand as a technology company — driving conversation and interest in what we do from both business and technology sectors. Enticing skilled, creative, workers to our doors would be one potential side effect! (<a href="https://careers.thomsonreuters.com/us/en/search-results?keywords=Labs">Labs continues to grow</a> its team into <strong>2025</strong>, and we are always looking for a healthy pipeline of talent.)</p><h3>About our Authors</h3><p>Every published article was authored by a scientist, engineer, or other Labs technologist who has a “day job”. The content was often directly related to their work, and they had to carve out additional hours to draft, revise, and ultimately publish to Medium.</p><p>24 Labs individuals published content this year, with many others helping draft and review that content. If we are to continue this effort, <strong>we should strive for even more participation in 2025 to drive a wider breadth of topics AND perspectives</strong>.</p><p>We largely let the authors propose the topics, and use their own voice and writing style. We did encourage some best practices and, of course, ensured that the content followed Thomson Reuters’ communication guidelines and policies.</p><h3>Increased Engagement and 2025 Outlook</h3><blockquote><em>Key achievements this year included: 30+ articles by 24 authors, with over 1,000 views per month — an order of magnitude larger than previous years. Several of our stories received 50 claps or more. </em>Humble beginnings.. but a good start! <em>See our </em><a href="https://medium.com/tr-labs-ml-engineering-blog/trending"><em>trending</em></a><em> and </em><a href="https://medium.com/tr-labs-ml-engineering-blog/archive/2024"><em>most read stories of 2024</em></a><em>.</em></blockquote><p>Our most read story was “<a href="https://medium.com/tr-labs-ml-engineering-blog/evaluating-vector-databases-101-5f87a2366bb1">Evaluating Vector Databases 101</a>” by <a href="https://medium.com/@adrianomitsuo.yoshino">Adriano Mitsuo Yoshino</a> which tracks because it was also the first published story of the year. Our most clapped story is the recent “<a href="https://medium.com/tr-labs-ml-engineering-blog/evaluating-quality-in-large-language-models-a-comprehensive-approach-using-the-legal-industry-as-a-a014459a9454">Evaluating Quality in Large Language Models: A Comprehensive Approach using the legal industry as a use case</a>” by <a href="https://medium.com/@afolabimkay">Adeshola Afolabi</a>.</p><p>I enjoyed personally reviewing many of the stories published this year. I particularly enjoyed reading <a href="https://medium.com/@tzielund">Tom Zielund</a>’s unique perspectives in “<a href="https://medium.com/tr-labs-ml-engineering-blog/the-parlor-game-that-learned-to-parlez-b0aa50380c4c">The Parlor Game that Learned to Parlez</a>” (third most clapped of the year, too) and “<a href="https://medium.com/tr-labs-ml-engineering-blog/prompt-engineering-as-a-new-kind-of-user-interface-44fa76dd09e9">Prompt engineering as a new kind of user interface</a>”; <a href="https://medium.com/@scott.arnottTR">Scott Arnott</a>’s “<a href="https://medium.com/tr-labs-ml-engineering-blog/performance-engineering-in-document-understanding-161b59afc9f0">Performance Engineering in Document Understanding — Towards Better Data Formats</a>”; and <a href="https://medium.com/@charles.godfrey">Charle’s Godfrey</a>’s “<a href="https://medium.com/tr-labs-ml-engineering-blog/rag-in-the-era-of-long-context-llms-b8ecda2d5693">RAG in the era of long-context LLMs</a>”.</p><p>And I learned a lot from many more of the work we published! Hopefully you did too.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*YOtemmoaGAbTF3yh77ypOA.jpeg" /><figcaption>Steady Growth in Medium Views and Follows Month over Month</figcaption></figure><p>We have seen a marked increase in engagement with <a href="https://medium.com/tr-labs-ml-engineering-blog">this publication</a>, but I think we will need to see an order magnitude more views and engagement before an ROI is realized.</p><blockquote>I’d love to hear your thoughts on how we can increase engagement while retaining our engineering and science-based content.</blockquote><p><strong>💬 Let the conversation begin</strong>! Did you have a favorite story of <strong>2024</strong>? How can we increase engagement on Medium? Let us know in the comments, or come talk with us on <a href="https://www.linkedin.com/groups/8168422/">our LinkedIn group</a>. Please be sure to follow us to see what happens in <strong>2025</strong>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58a0d95b4a71" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tr-labs-ml-engineering-blog/reflecting-on-labs-medium-publication-in-2024-58a0d95b4a71">Reflecting on Labs’ Medium Publication in 2024</a> was originally published in <a href="https://medium.com/tr-labs-ml-engineering-blog">Thomson Reuters Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI-Assisted Development: Redefining How (and When) We Code]]></title>
            <link>https://medium.com/tr-labs-ml-engineering-blog/ai-assisted-development-redefining-how-and-when-we-code-098a16688e86?source=rss-f2ce837c4d4c------2</link>
            <guid isPermaLink="false">https://medium.com/p/098a16688e86</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[careers]]></category>
            <category><![CDATA[ai-assisted-writing]]></category>
            <category><![CDATA[llm-applications]]></category>
            <dc:creator><![CDATA[John Duprey]]></dc:creator>
            <pubDate>Tue, 26 Nov 2024 18:02:52 GMT</pubDate>
            <atom:updated>2024-12-03T14:11:04.697Z</atom:updated>
            <content:encoded><![CDATA[<p>An open-minded look at the current state of the art, pros/cons, and how Thomson Reuters is accelerating software development with it today.</p><p>AI coding assistants, like GitHub Copilot, are now commonplace — sparking debates from “software engineers are obsolete” to “AI creates more problems than it solves.” The truth lies somewhere in between. This article explores the current state of AI coding assistants, their pros and cons, and Thomson Reuters’ approach to integrating these tools.</p><blockquote><strong>Spoiler Alert</strong>: Coding assistants won’t replace most software engineers, but they will require adaptation. Engineers will shift from routine tasks to areas where human intelligence excels — learning, adapting, and creatively solving problems with code.</blockquote><h3>The AI Coding Assistant Landscape</h3><h4>Back in MY Day …</h4><p>In 1995 when I started my career at Lawyers Cooperative Publishing (LCP[1]), the <a href="https://en.wikipedia.org/wiki/History_of_the_World_Wide_Web">World Wide Web (WWW)</a> was young, the business was run on IBM mainframes and <a href="https://en.wikipedia.org/wiki/Oracle_Solaris">Sun Solaris Unix</a>. Like the other C/C++ developers at LCP, I coded in text editors like <a href="https://en.wikipedia.org/wiki/Vi_(text_editor)">vi</a> or <a href="https://en.wikipedia.org/wiki/Emacs">emacs</a>[2] in text-based terminals. In my earliest days, I learned from the electronic <a href="https://en.wikipedia.org/wiki/Man_page">manual pages</a> ,[3] that there were <em>graphical</em> developer tools on <a href="https://en.wikipedia.org/wiki/Oracle_Solaris">Solaris</a>. So I pleaded for help from the “sysadmin gods”. A few days later, a very crusty lead sysadmin[4] came by my cubicle[5] and slammed down a 12-inch stack of printed manuals. He, somewhat sarcastically, wished me luck — humble beginnings..</p><p>Hard to find manuals and “<a href="https://amazingalgorithms.com/definitions/grey-beard/">grey beard</a>” colleagues were our best resources in figuring out the craft of software engineering when I was starting out as an engineer. While computer science had existed since the 1950’s, the whole discipline of developing complex and robust software was still quite new. A nascent discipline at universities, you mostly had to learn as you went along, and the web lacked the content and search capabilities of today.</p><p>Fast forward 30 years past the operating system wars,[6] the <a href="https://en.wikipedia.org/wiki/Browser_wars">browser wars</a>, <a href="https://en.wikipedia.org/wiki/Dot-com_bubble">dot-com bubbles</a> and before us lies a very exciting chapter in the field of software engineering. Just a few short years ago, well documented APIs, Google Search, <a href="https://en.wikipedia.org/wiki/Stack_Overflow">Stack Overflow</a>, and (sometimes) <a href="https://en.wikipedia.org/wiki/Reddit">Reddit</a> made it pretty easy for a good developer to learn just what they needed, when they needed it.</p><p><strong>But today, we are <em>literally</em> talking to our computers to get the job done!</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*5wRr_p_6j46v5Ge43jvpOw.gif" /><figcaption>Scotty: “Hello Computer…”</figcaption></figure><h4>LLMs Can Write Code</h4><p><a href="https://en.wikipedia.org/wiki/Transformer_(deep_learning_architecture)">Transformer-based</a>, <a href="https://en.wikipedia.org/wiki/Large_language_model">large language models (LLMs)</a>, can encode large amounts of information and can respond, often times intelligibly, to natural language prompts. They are not only masterful at modeling human language, but in modeling computer languages.</p><p>With the launch of <a href="https://openai.com/index/openai-codex/">Open AI’s Codex model in 2021</a>, LLMs have been purpose-built to assist coders writing code. Of course, computer-assisted coding has existed for years — starting with intelligent, context-aware, <a href="https://en.wikipedia.org/wiki/Code_completion">code completion</a> (like <a href="https://code.visualstudio.com/docs/editor/intellisense">Intellisense</a>). Later offerings, such as those from <a href="https://www.tabnine.com/">Tabnine</a> used machine learning models for intelligent completions. But today’s LLM-based assistants, including Tabnine’s, go far beyond simple completions.</p><h4>What Can Today’s Coding Assistants Do?</h4><p>An incomplete list of how AI coding assistants can help developers today include:</p><ol><li>Generating code</li><li>Completing code as you type</li><li>Providing code examples and explaining what it does</li><li>Debugging/analyzing existing code</li><li>Refactoring or suggesting improvements to existing code</li><li>Testing code by generating tests and test inputs</li></ol><p>Let me try to illustrate some these using GitHub Copilot…</p><h4>AI Coding Assistance, An Example</h4><p>Recently I was looking at providing a simple user interface on top of Snowflake (RDBMs / SQL) data. I started by communicating the basic idea to GitHub Copilot in Microsoft VisualCode.[7]</p><blockquote>Me: I wish to create a streamlit app that displays a table from snowflake and allows the user to edit the table data.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/1*hABlhBmLt3BzLgfz8Ukb3g.png" /><figcaption>Initial Prompt and GHC Plan</figcaption></figure><p>Copilot breaks down the solution step-by-step, then provides code that you can apply in your editor to implement it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/1*rAyBnuxBbVc-ORoL612doQ.png" /><figcaption>Generated Code Snippet</figcaption></figure><p>The user interface allows me to rapidly apply this code into existing or new code files:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/227/1*6IuD3N_--TOcZQqXMrfmuw.png" /><figcaption>GitHub Copilot Interface</figcaption></figure><p>I follow up with additional refinement to what I want:</p><blockquote>ME: the user will need to pass in the passphrase for the snowflake PEM</blockquote><p>Copilot adjusts the code to accommodate and gives me the option to apply the changes to the code. Pretty amazing, right!!?! Hold on. It’s not all roses..</p><p>When I try running the code, I get an error about a experimental_data_editor not existing. When I report this to Copilot:</p><blockquote>ME: how do I add experimental_data_editor</blockquote><p>It responds with:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*3i0DuZTktJvFZFX1T796Hg.png" /><figcaption>GitHub Copilot doesn’t know which features exist in which versions of Streamlit</figcaption></figure><p>Nope. I know, even before I execute the command, that is <em>not</em> the problem. I let Copilot know, that didn’t work:</p><blockquote>ME: AttributeError: module ‘streamlit’ has no attribute ‘experimental_data_editor’</blockquote><p>Copilot’s response is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/1*0PBJ5i5iF2tcZreALDohbw.png" /><figcaption>Some suggestions may not be optimal — its up to you to decide!</figcaption></figure><p>And this does indeed work, but I knew there was a better way. Falling back to good old Google search, I found the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*_2xuB6zqCFWlUbUfeAy49g.png" /><figcaption>Google finds the docs about component deprecation — which tells me what to use instead.</figcaption></figure><p>And so I manually update my code to use the new data_editor component.</p><blockquote>APIs are forever changing and any one application will have many different APIs that it codes to. Models are only as up-to-date as their last training, and approaches like <a href="https://en.wikipedia.org/wiki/Retrieval-augmented_generation">retrieval-augmented generation (RAG)</a>, used to synthesize new information, are not foolproof, as the above example illustrates.</blockquote><h4>Can You Explain This?</h4><p>AI Assistants can explain what code does in plain human language. e.g. I can highlight some code and ask Copilot to explain what it does:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*_TvS9mxWBmOVzK8eVF3QkQ.png" /><figcaption>You can select a block of code, then ask Copilot to explain what it does.</figcaption></figure><p>In addition to breaking it down line-by-line, it summarizes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*aSo-kC2JRXKkXQFkER305Q.png" /><figcaption>Code summary</figcaption></figure><p>For very bespoke code, the assistant may only be able to explain what is mechanically happening and not the <strong>intent</strong> of the code itself, like it did in this example. So, your mileage may vary.</p><blockquote>A human will always have more context (outside of the code itself) than the LLM does.</blockquote><p>In a similar vein, you can ask Copilot to fix your code. Highlight the error and ask it to fix it. If it can’t fix it, it can usually help explain the error in more natural language.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/1*eDuIQR-5hBcKmru1eTQ0GQ.png" /><figcaption>You can ask Copilot to fix your code.</figcaption></figure><blockquote>AI assistants seem to be pretty good at explaining cryptic runtime or compile-time errors in context of the current code. They may NOT be able to correct an error for the same reasons mentioned above — insufficient context .</blockquote><h3>Writing Tests</h3><p>AI Coding assistants are pretty good at creating unit test cases — especially with coming up with valid input values. I found that it was not as good at also determining the expected output of the test[8], but it was able to remove a lot of the boilerplate coding I’d have to do and allows me to focus on the intent of the test.</p><p>Suggestions are shown in gray, you just have to hit tab to accept them, then edit as needed.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F8hTkVA1TKHs%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D8hTkVA1TKHs&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F8hTkVA1TKHs%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/27bb3f1218a50bb989790ea3f3a4c270/href">https://medium.com/media/27bb3f1218a50bb989790ea3f3a4c270/href</a></iframe><h4>Is that it… Really? When Does AI Replace Software Engineers?</h4><p>Are you underwhelmed? I’ve barely scratched the surface (e.g. it can help with <a href="https://en.wikipedia.org/wiki/Regular_expression">regular expressions</a>, executing CLI tools, etc.) but that is largely “more of the same”. You may be asking, where do we get to the part where AI obviates the need for software engineers altogether?</p><p>Enter <a href="https://devin.ai/">Devin</a>! OK, JK, <a href="https://www.reddit.com/r/LocalLLaMA/comments/1c60xha/pretty_comprehensive_overview_on_how_devin_is_a/">NOT Devin</a>. Enter <a href="https://www.cursor.com/">Cursor</a>! Cursor is a much more refined LLM-IDE solution. But maybe Cursor, barely out of the crib, is <a href="https://www.youtube.com/watch?v=Aj9yUs95obg">already surpassed by… Replit</a>? Let’s go with that!</p><p><a href="https://replit.com/">Replit</a> combines a lot of the emerging techniques used in other generative AI domains such as <a href="https://en.wikipedia.org/wiki/Multi-agent_system">multi-agent</a> processing and <a href="https://www.ibm.com/topics/chain-of-thoughts">chain of thoughts (CoT)</a> to take a more comprehensive and wholistic view of the developer’s task. It removes a LOT of the complexities in a development environment by managing the development environment for you — drastically lowering the barrier of entry. <strong>Or so this is the promise</strong>. Unlike Devin, it seems like it is more than marketing, smoke and mirrors.</p><p>I’m sure there are, or soon will be, dozens more competing solutions[9]. AI Coding Assistants (replacements?) will become more sophisticated in their approaches as we are already seeing in the likes of Replit.</p><blockquote>And I think that certain types of applications may be possible without software engineers. There. I said it.</blockquote><p>This is not that dissimilar to how it’s now possible to create web sites without knowing how to code HTML, CSS, or Javascript. But we still have PLENTY of engineers who code HTML, CSS, and Javascript. Why?</p><ul><li>Because these tools are limited in the scope of capabilities — supporting only the most common use cases.</li><li>Because these tools require people who DO know how to code to create and maintain them!</li></ul><p>Like those web tools that allow non-technical people to create web pages, AI is going to help lower the barrier for non-technical users to create complex solutions without writing code. These tools work well for common, well understood, problems that have cookie-cutter solutions. <strong>The minute the use case deviates too far from these parameters, someone is going to have to write some clever code.</strong></p><blockquote><strong>Want to ensure you can enter or maintain a career in software engineering?</strong> Then deepen your understanding of technology and how they interconnect. Focus less on the syntax and grammar of coding and more on the fundamental concepts and how these can solve problems optimally.</blockquote><p>Specialize in emerging areas. (I hear there’s a lot happening in AI these days…). Embrace emerging tools like AI coding assistants so you can focus more on where YOU add value — with that beautiful, clever, creative, mind of yours.</p><blockquote><strong>Beware of missing concepts by blindly letting AI assistants do the work. (See “skill atrophy” below.)</strong></blockquote><h3>The Dark Side of AI Generated Code</h3><p>AI coding tools are transforming software development, but is this change entirely positive? Key concerns include:</p><ul><li><strong>Code Quality and Understanding</strong>: Can we trust AI to produce optimal code? What if the AI fails to meet our needs? Do we understand the code well enough to intervene?</li><li><strong>Security and Vulnerability</strong>: Who ensures AI-generated code is secure?</li><li><strong>Intellectual Property</strong>: AI tools might reproduce copyrighted material, raising legal issues. GitHub Copilot, for example, stops generating code if it detects OSS code, and is currently facing a <a href="https://www.courtlistener.com/docket/65669506/doe-1-v-github-inc/">DMCA lawsuit</a> (<a href="https://www.legal.io/articles/5516216/Judge-Throws-Out-Majority-of-Claims-in-GitHub-Copilot-Lawsuit">though most of the claims have been dismissed</a>).</li><li><strong>Skill Atrophy</strong>: Will reliance on AI erode our coding skills, especially for beginners?</li></ul><p>An <a href="https://www.axios.com/2024/06/13/genai-code-mistakes-copilot-gemini-chatgpt">Axios article</a> highlights that while AI coding tools boost productivity, they can also introduce up to 30% incorrect code — security flaws and errors, requiring significant post-commit fixes. Thus, initial productivity gains may be offset by later corrections.</p><p>A robust software development lifecycle (SDLC) can mitigate these issues. At Thomson Reuters, we peer review code and use multiple scans to:</p><ul><li>Detect security vulnerabilities</li><li>Identify copyrighted or open-source code</li><li>Check license compatibility</li></ul><p>Despite these safeguards, it’s best to catch issues early or avoid them altogether.</p><blockquote>While AI coding assistants improve, human oversight remains crucial for complex applications.</blockquote><h3>Humans Are Still the Grown Up in the Room…</h3><p>Earlier, we saw how AI coding assistants can lose sight of the full context when it couldn’t figure out which component to use in Streamlit. It had no idea about which version of Streamlit I was using. Software has complex inter-dependencies and myriad versions of libraries. Even with RAG, in an open-ended application space, the assistant can lose track of critical information.</p><p>It may also generate code that is not optimal given the nature of (yet unseen) input data or suggest code that is not as secure as required. What priorities or trade offs do you think the LLM is considering when generating code? Even if you tell it the priorities, it may not connect these concepts to the code.</p><p>Newer approaches like <a href="https://www.ibm.com/think/insights/agentic-ai">agentic AI</a> and <a href="https://www.ibm.com/topics/chain-of-thoughts">chain of thoughts</a> processing <strong>will</strong> get better at this. How much better is still to be determined.</p><blockquote>Until then <strong>you, the coder, are the “grown up” in the room</strong>. You must take ownership of all the code regardless of whether it was written by you or suggested by AI.</blockquote><p>Your AI assistant can reduce time spent reading lengthy manuals, or parsing through (sometimes toxic) Stackoverflow/Reddit threads.</p><h3>AI Assisted Development in the Industry</h3><p>Google’s CEO Sundar Pichai reported that <a href="https://www.forbes.com/sites/jackkelly/2024/11/01/ai-code-and-the-future-of-software-engineers/">25% of Google’s new code is AI-generated</a>, with human review boosting productivity. A GitHub survey found that <a href="https://github.blog/news-insights/research/survey-reveals-ais-impact-on-the-developer-experience/">92% of U.S. developers use AI coding tools</a>. This widespread adoption suggests the benefits of AI assistants outweigh the downsides.[10]</p><p>If this is truly indicative of the adoption of AI coding assistants, then the tech industry is clearly not seeing the downsides of AI assistants outweighing the positive.</p><h4>AI-Assisted Development at Thomson Reuters</h4><p>Thomson Reuters aimed for 100% adoption of AI coding assistants this year, starting with GitHub Copilot and later adding Amazon Q. Adoption was swift, reaching near 100% in Thomson Reuters Labs within months.</p><p>While opinions on productivity gains varied, most agreed Copilot was beneficial for boilerplate code and tests but less effective for complex logic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YC_w0WjnEtHYDkff6Rg70g.png" /><figcaption><em>TR Labs Self-Reported Productivity</em></figcaption></figure><p>See some <em>generalized</em> feedback mapped over productivity ratings below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/506/1*sqBhmlba26h3eBR1o2z5NQ.png" /><figcaption><em>TR Labs Comments on Self-Reported Productivity — Sentiment by Rating</em></figcaption></figure><p>As developers learn to use these tools more effectively and as the tools themselves improve, productivity is expected to increase. Recent enhancements to GitHub Copilot include better IDE integration, reduced latency, larger context windows, and multi-model support (e.g., GPT-4, Claude, Gemini).</p><h3>The Verdict…</h3><blockquote>AI coding assistants are here to stay.</blockquote><p>Can someone with little tech knowledge create an app with AI and no engineering help? Yes, but mostly for well-defined, problem spaces — like basic web apps or simple mobile apps. These could be “no code” or “low code” solutions with minimal developer tweaks.</p><p>While LLMs often impress with their connections, they can also disappoint with errors. This is most detrimental when these errors are subtle and detected late in the SDLC.</p><p>Currently, coding assistants learn from human-oriented materials. It’s possible, if training material is optimized for AI first, they could reliably handle more “undifferentiated work” over time.</p><blockquote>It’s never been more important to be “good engineers” — those who understand technology and how to apply it creatively, who understand the nuances and complexities and can make those connections.</blockquote><blockquote>The days of getting paid to simply “wire up” APIs are numbered.</blockquote><p>Cost-conscious companies should not assume AI will significantly cut their engineering workforce. It’s a flawed assumption that 25% AI-generated code means you can cut 25% of your engineering workforce. This is a clear indicator that you don’t understand what value your engineers are providing. It would be akin to <a href="https://www.researchgate.net/publication/267761547_A_statistical_study_of_the_relevance_of_lines_of_code_measures_in_software_projects">falsely assuming Lines of Code (LoC) is an indication of software complexity or developer effort</a>.</p><p>An AI assistant increases the velocity of engineers allowing for faster or larger deliveries. These advantages are available to our engineers as well as to other companies, if we would cut our workforce by 25% we would lose relative velocity and possibly market share compared to our competitors.</p><p>This pivots how we should look at AI assisted coding.</p><blockquote>It is not about if we adopt it, we gain 25% speed. It is more likely that if we do not adopt it, we will be 25% slower than others.</blockquote><p><strong>What do you think about the current and future state of AI code assistants?</strong></p><p><strong>💬 Let the conversation begin.</strong> Let’s <a href="https://trten-my.sharepoint.com/personal/john_duprey_thomsonreuters_com/Documents/Shared/Labs%20Authoring/LINK-To-Post%20comments%20if%20applicable">talk about it here</a> or on our <a href="https://www.linkedin.com/groups/8168422/">LinkedIn Group</a>!</p><p>[1] LCP was acquired by Thomson Corporation in 1989</p><p>[2] Emacs. I have the gnarled pinky fingers to prove I was a power user.</p><p>[3] “man pages” were all-text documents you could access from the command line. See them <a href="https://www.man7.org/linux/man-pages/dir_all_alphabetic.html">online now</a>.</p><p>[4] “Al” was his name. He may have been a Hells Angel biker. Legend. 💪</p><p>[5] Cubicles were… never mind..</p><p>[6] The operating system wars of the 1990s largely revolved around Microsoft Windows and GNU-Linux. However other’s like <a href="https://en.wikipedia.org/wiki/OS/2">IBM’s OS/2</a> and <a href="https://en.wikipedia.org/wiki/BeOS">BeOS</a> vied for a position as well. In the early 2000’s <a href="https://en.wikipedia.org/wiki/MacOS">Mac OS X</a> arrived lending more credibility to Unix operating systems like <a href="https://en.wikipedia.org/wiki/FreeBSD">FreeBSD</a>. Learn more: “<a href="https://en.wikipedia.org/wiki/Halloween_documents">The Halloween Documents</a>”<a href="https://en.wikipedia.org/wiki/Halloween_documents,">,</a> <a href="https://en.wikipedia.org/wiki/Linux#Commercial_and_popular_uptake">Linux’s Rise</a><a href="https://en.wikipedia.org/wiki/Linux#Commercial_and_popular_uptake,">,</a> <a href="https://www.forbes.com/sites/rajsabhlok/2013/09/20/microsoft-kill-shots-how-ibmlinux-deals-the-latest-blow-to-redmond-empire/">IBM Throws Weight behind Linux</a><a href="https://www.forbes.com/sites/rajsabhlok/2013/09/20/microsoft-kill-shots-how-ibmlinux-deals-the-latest-blow-to-redmond-empire/,">,</a> <a href="https://www.theverge.com/2020/5/18/21262103/microsoft-open-source-linux-history-wrong-statement">Microsoft Backpedals on OSS and Linux</a></p><p>[7] I could have done the same with any number of other IDEs or text editors — e.g. JetBrains IDEs, neovim, etc.</p><p>[8] Perhaps future agentic-AI solutions will execute the test code to determine the expected output?</p><p>[9] <a href="https://codeium.com/">Codeium / Windsurf</a>, the <a href="https://code.visualstudio.com/blogs/2024/11/12/introducing-copilot-edits">New GitHub Copilot “Edits” feature</a>, <a href="https://bolt.new/">Bolt.new</a> / or <a href="https://github.com/coleam00/bolt.new-any-llm">Forks of</a>, …</p><p>[10] This was based on a Wakefield Research survey of 500 U.S.-based developers in enterprise companies: “This report draws on a survey conducted online by Wakefield Research on behalf of GitHub from March 14, 2023 through March 29, 2023 among 500 non-student, U.S.-based developers who are not managers and work at companies with 1,000-plus employees. For a complete survey methodology, please contact <a href="/service/mailto:press@github.com"><strong>press@github.com</strong></a>.” — <a href="https://github.blog/news-insights/research/survey-reveals-ais-impact-on-the-developer-experience/">https://github.blog/news-insights/research/survey-reveals-ais-impact-on-the-developer-experience/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=098a16688e86" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tr-labs-ml-engineering-blog/ai-assisted-development-redefining-how-and-when-we-code-098a16688e86">AI-Assisted Development: Redefining How (and When) We Code</a> was originally published in <a href="https://medium.com/tr-labs-ml-engineering-blog">Thomson Reuters Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Goodbye Twitter, Hello Bluesky]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@jduprey/goodbye-twitter-hello-bluesky-b2a6973fae6c?source=rss-f2ce837c4d4c------2"><img src="https://cdn-images-1.medium.com/max/1209/1*sHw2P3ttgO3DL968R_8HBQ.png" width="1209"></a></p><p class="medium-feed-snippet">Originally published at https://www.iduprey.com on November 17, 2024.</p><p class="medium-feed-link"><a href="https://medium.com/@jduprey/goodbye-twitter-hello-bluesky-b2a6973fae6c?source=rss-f2ce837c4d4c------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@jduprey/goodbye-twitter-hello-bluesky-b2a6973fae6c?source=rss-f2ce837c4d4c------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2a6973fae6c</guid>
            <category><![CDATA[bluesky-social]]></category>
            <category><![CDATA[social-media]]></category>
            <category><![CDATA[twitter]]></category>
            <dc:creator><![CDATA[John Duprey]]></dc:creator>
            <pubDate>Sun, 17 Nov 2024 00:00:42 GMT</pubDate>
            <atom:updated>2024-11-19T03:18:32.342Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Part 2/2 — Scaling Thomson Reuters’ Language Model Research]]></title>
            <link>https://medium.com/tr-labs-ml-engineering-blog/part-2-2-scaling-thomson-reuters-language-model-research-e392d5aefe7f?source=rss-f2ce837c4d4c------2</link>
            <guid isPermaLink="false">https://medium.com/p/e392d5aefe7f</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[genai]]></category>
            <dc:creator><![CDATA[John Duprey]]></dc:creator>
            <pubDate>Mon, 29 Jul 2024 13:47:57 GMT</pubDate>
            <atom:updated>2024-07-29T21:17:52.935Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part 2/2 — Scaling Thomson Reuters’ Language Model Research</h3><h4>with Amazon SageMaker HyperPod</h4><p>In <a href="https://medium.com/tr-labs-ml-engineering-blog/part-1-2-scaling-thomson-reuters-language-model-research-52dfb0becedb">Part 1</a> of this series we covered the business drivers and research objectives behind our custom LLM training research. In this final part, we’ll tell you about how this compute-intensive research required specialized hardware, posing resource scale challenges. Through close partnership with Amazon Web Services (AWS), we were able to use <a href="https://aws.amazon.com/sagemaker/hyperpod/">Amazon SageMaker HyperPod</a> to address those challenges. (<strong>Note</strong>, I’ll use “Hyperpod” for brevity/readability throughout this article.) HyperPod allowed Thomson Reuters to scale a large cluster of NVIDIA A100 GPUs in a reliable and predictable manner. This gave us the scale we needed to explore various approaches to training domain-adapted Large Language Models (LLMs).</p><p>Check out the joint presentation <a href="https://www.linkedin.com/in/simone-zucchet">Simone Zucchet</a> and I made on this topic at the AWS Summit in London back in April, 2024:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FPDG8VLW4ts8%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DPDG8VLW4ts8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FPDG8VLW4ts8%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube-nocookie" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/b4db2050b5d885e13c7e5f0510d3fbd4/href">https://medium.com/media/b4db2050b5d885e13c7e5f0510d3fbd4/href</a></iframe><h3>Scaling Language Model Training</h3><p>We knew training LLMs would require significant computing power. Training a large language model of even 7b parameters is a compute intensive operation — requiring multi-node distributed computing capabilities. Further, these compute nodes need large GPUs or similar hardware. In our case, we focused on NVIDIA’s high performance A100 family of GPUs.</p><p>To estimate just how much, we used the <a href="https://medium.com/@zaiinn440/training-at-scale-chinchilla-scaling-laws-for-compute-optimal-training-of-llms-eca49f58c358">Chinchilla scaling law</a> to determine how much training data (in tokens) we would need to retain quality at a given model size. The scaling law is based on <a href="https://arxiv.org/abs/2203.15556">published research</a> that found the model size to training tokens scale proportionally. From there, we used other publicly available information to estimate how much time (in days) would be required to complete training at a given number of GPUs.</p><p>By our calculations, the compute scale to number of training days is as shown in this table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*PncI_UTt33deb7Vz6HQ4QA.png" /><figcaption>Our scale calculations based, in part, on chinchilla point.</figcaption></figure><p>So, for Example, a 7b parameter model, would require 132b input tokens and take about 7 days to finish training with 64 A100 GPUs (or 8 p4ds). We used <a href="https://aws.amazon.com/ec2/instance-types/p4/">p4d ec2 instance types</a> which offers 8x100 NVIDIA GPU tensor cores and 40GB of RAM.</p><h4>Scaling Challenges</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-AhxoecJKwz7KXvo3mX9dQ.png" /><figcaption>Desert Generated with AI (Microsoft Copilot Designer, DALL-E 3)</figcaption></figure><p>We quickly discovered that high performance GPUs are in very high demand. It was virtually impossible to get just 1 p4d let alone 8 or 16! We know because we tried for over a month to get just 1 on demand. We failed. We also tried alternate regions where there was supposedly more availability with no luck as well. Capacity reservations were equally slow to acquire p4ds. After a month of trial and error, it became clear that we had a very serious problem — available GPU capacity.</p><p>We also looked at a number of third parties that promised to help provide the compute scale we required. However, these solutions would require significant portions of Thomson Reuters data leaving the boundaries of our cloud storage and posing potential intellectual property and data security risk.</p><h3>Solution: Amazon SageMaker HyperPod</h3><p>AWS is a trusted Thomson Reuters partner, and we asked them how we could scale our training when we couldn’t even get 1 p4d in any of our regions! It turns out, this was a pain point many customers were having and AWS was actively working on ways to manage large scale capacity needs that would ebb and flow over time.</p><p>They introduced us to <a href="https://aws.amazon.com/sagemaker/hyperpod/">Amazon SageMaker HyperPod (HyperPod)</a>. With HyperPod, the customer communicates the GPU capacity they need over time and HyperPod provides it, managing health and resiliency of the worker nodes.</p><p>The following table is the capacity plan we used over the course of 5–6 months.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/1*0xx-RjBhlvN5BBXE_FhDvQ.png" /><figcaption>Sample Capacity Plan</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B7tsm6JDb9pI65mXeU_UnQ.jpeg" /><figcaption>Oasis Generated with AI (Microsoft Copilot Designer, DALL-E 3)</figcaption></figure><h3>Outcomes and Next Steps for Us</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*047jXP5lEJ5hnLVPNBjVzg.png" /><figcaption>Photo by <a href="https://unsplash.com/@nasa?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">NASA</a> on <a href="https://unsplash.com/photos/space-shuttle-challenger-launches-from-kennedy-space-center-dCgbRAQmTQA?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p>Finally! With our HyperPod cluster setup, our capacity plan needs shared with the HyperPod team, and a Labs custom command line interface (CLI) to ease training job management, we were ready to experiment with training LLMs. This was a journey!</p><blockquote><strong>By the numbers:</strong> Over the course of <strong>~5 months</strong>, we successfully ran <strong>~20 training jobs</strong>. We scaled our cluster <strong>up to 16 p4ds</strong> and our largest job utilized the entire cluster. We trained a <strong>70b parameter model on 400b input tokens and it took 36 days to complete</strong>.</blockquote><p>The most amazing aspect of this was that we had zero hardware failures! This is perhaps a testament to HyperPod’s pre-flight health checks performed before they are made available in the cluster.</p><h4>Report on initial trainings done, findings</h4><p>While our experimentation is far from complete, we do have some positive preliminary findings. <strong>What I’m sharing here is an informal summary.</strong> More detailed analysis and results will be published in the future by the Labs Foundational Research team.</p><p><strong>Continuous Pre-Training (CPT)</strong></p><p>In continuous pre-training (CPT), you train from an existing open source LLM checkpoint. More than a time-saver; it is a strategic decision that allows for the nuanced growth of the model’s capabilities over time.</p><p>The preliminary results of our experimentation showed that we were able to train models on the legal domain without losing general knowledge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*MGHuo2LExZCCN3Togkd3hw.png" /><figcaption>CPT Legal vs. General Perplexity</figcaption></figure><p>We used a measure called perplexity. It quantifies how well the model predicts a sample of text. In essence, perplexity measures the confidence a model has in its predictions. Lower perplexity indicates that the model is more certain about its predictions. From the graphs above you can see that as we increased our batches of training, legal perplexity decreased while general perplexity increased somewhat, it quickly leveled off.</p><p>Part of our experimentation was determining the right split of domain specific (legal) and general data to train with.</p><p><strong>Instruct fine-tuning (IFT)</strong></p><p>Instruct fine-tuned LLMs are tuned to respond to specific instructions, enabling tasks such as question answering, summarization, and brainstorming. For instance, human-written instruction datasets include prompts like “summarize this article” or “list fun weekend activities.” Our hypothesis is that Legal LLMs can benefit from diverse legal instructions.</p><p>We have discovered that our Legal LLM greatly benefits from a vast array of diverse instructions. By compiling legal instructions, such as drafting legal headnotes, and combining them with publicly available instructions, our MPT-TR-7b model, derived from MPT-7b, has showcased improvements correlated with an increased number of instruction data sets provided.</p><p>We used an automatic measure called rouge to determine how well our domain adapted models performed compared to GPT-4. This automatic measure, based on term overlap, is not the same as human preference judgment, but gives us some degree of confidence we are headed in the right direction.</p><p><strong>Legal Summarization</strong></p><p>Our MPT-TR-7b model has demonstrated proficiency in legal summarization tasks, rivaling GPT-4’s performance when evaluated with automatic metrics assessing word overlap with reference summaries. While a human-based evaluation would offer deeper insights, the initial results are compelling evidence of the model’s capabilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*nDeiRo3gs9Ix_kH8mp-wpg.png" /><figcaption>AS IFT collections were added, rouge evaluations were on par or better than GPT4</figcaption></figure><p><strong>Legal Classification</strong></p><p>In other legal tasks, such as classification that was measured in accuracy and precision/recall, there’s still room to improve when compared to GPT-4. Nonetheless, the performance uptick is evident with the expansion of instruction datasets. Even more exciting is the leap in performance observed with larger base models like MPT-30b.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*6yZzkh3q73Sfm2OCpzPLbA.png" /><figcaption>Even for larger models, rouge scores did not reach GPT4 levels in this task</figcaption></figure><blockquote><strong>NOTE</strong>: Results for the third task, legal question answering, are not available at this time.</blockquote><h4>Next Steps</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8_SUfp4hwQjB39gaYi0A-Q.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@alpifree?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Sylvain Mauroux</a> on <a href="https://unsplash.com/photos/2-men-hiking-on-mountain-during-daytime-WI8phMvAEMI?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p>With the advent of even more capable models like Mistral-7b, which matches MPT-30b’s performance, we are eager to explore the potential of more recently released models like Mixtral8x7b and LLaMa-3–70b. As next step, we have been training on the Mixtral8x7b and the LLaMa-3–70b models that seem to give us even better performance than the smaller models we have been training.</p><p>Looking ahead, the integration of new alignment methods, such as DPO (Direct Preference Optimization), could further narrow the performance gap, paving the way for the next generation of specialized LLMs that could revolutionize the legal tech landscape. Its impact on training scale requirements is positive, as it simplifies the process and reduces computational overhead.</p><h3>HyperPod and LLM Training, Is it Right for You?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oQfxf92KXv3hiiIx" /><figcaption>Photo by <a href="https://unsplash.com/@marknealphoto?utm_source=medium&amp;utm_medium=referral">Mark Neal</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>HyperPod works</strong>. We were able to scale up capacity of HyperPod with a ~2-week notice to AWS. And scaling down could often happen faster. We did find it somewhat challenging to plan for capacity since the results of current experiments would often determine the capacity requirements of the next experiment. We often retained more capacity than we needed for weeks at a time to ensure we would have it when we needed it. The cost of this was somewhat offset by our savings plans with AWS.</p><h4>Is HyperPod Right for Your Business?</h4><p>Is HyperPod right for you? The first question to answer is does it make sense to train and maintain your own models. Think back on some of the benefits I previously outlined in <a href="https://medium.com/tr-labs-ml-engineering-blog/part-1-2-scaling-thomson-reuters-language-model-research-52dfb0becedb">Part 1</a>: Quality, Agency/Control, Operational Efficiency. If your business operates in specialized or deep verticals with knowledge not generally available on the web and you have domain specific tasks, it may make sense. At the same time, you’ll need to weigh the costs associated with training and inference as well as keeping up with rapidly advancing LLM technology.</p><p>Like Thomson Reuters, you might want to start with retrieval-augmented generation (RAG) solutions with off-the-shelf LLMs as a first step, then consider customization options from there. Besides quality, think about the amount of control, speed and cost your use cases may need.</p><p>If you DO decide training LLMs makes sense, then you’ll need considerable computational power. Depending on your model parameters, you’ll likely need at least 64 A100 or better GPUs and HyperPod can ensure you have the necessary capacity and resilience. <a href="https://aws.amazon.com/ec2/capacityblocks/">EC2 Capacity Blocks for ML</a> might work for smaller, shorter running training jobs where resiliency is less of a concern. The<a href="https://aws.amazon.com/machine-learning/trainium/"> Trainium instance type</a> may also be something to consider because it seems to be a more plentiful resource, currently. However, in our experience, Trainium <a href="https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/index.html#pytorch-neuron-training-tutorials">only supports certain model architectures</a>, versions and frameworks.</p><h4>From Thomson Reuters Labs’ Viewpoint</h4><p><strong>Our research is promising</strong>. The possibility of being able to run smaller, tuned models can make good business sense — giving us more autonomy, improved task-specific quality, and reduced operational cost in the long run. We have a long way to go in terms of vetting an end to end solution. Research on quality continues and as things firm up we will need to do further analysis on operational concerns — including designing for enterprise-scale inference/hosting.</p><blockquote>While we do all of this, very large commercial LLMs continue to advance. With ever-increasing context windows and increasing availability/capacity, whether smaller, tuned, LLMs benefits will outweigh the cost is still to-be-determined for us.</blockquote><p><strong>💬 Let the conversation begin</strong> Let’s <a href="https://medium.com/LINK-To-Post%20comments%20if%20applicable">talk about it here</a>!</p><h4>Further Reading/References</h4><ul><li><a href="https://aws.amazon.com/sagemaker/customers/#thomson-reuters">Amazon SageMaker HyperPod Testimondials</a> — including TR Labs’ very own <a href="https://www.linkedin.com/in/joel-hron-90a3421a/">Joel Hron</a></li><li><a href="https://aws.amazon.com/events/summits/emea/london/">2024 AWS London Summit</a></li><li><a href="https://youtu.be/PDG8VLW4ts8">AWS — Labs Joint Presentation on HyperPod at 2024 AWS London Summit</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e392d5aefe7f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tr-labs-ml-engineering-blog/part-2-2-scaling-thomson-reuters-language-model-research-e392d5aefe7f">Part 2/2 — Scaling Thomson Reuters’ Language Model Research</a> was originally published in <a href="https://medium.com/tr-labs-ml-engineering-blog">Thomson Reuters Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 1/2 — Scaling Thomson Reuters’ Language Model Research]]></title>
            <link>https://medium.com/tr-labs-ml-engineering-blog/part-1-2-scaling-thomson-reuters-language-model-research-52dfb0becedb?source=rss-f2ce837c4d4c------2</link>
            <guid isPermaLink="false">https://medium.com/p/52dfb0becedb</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[business]]></category>
            <dc:creator><![CDATA[John Duprey]]></dc:creator>
            <pubDate>Tue, 16 Jul 2024 14:55:59 GMT</pubDate>
            <atom:updated>2024-08-01T20:56:12.200Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part 1/2 — Scaling Thomson Reuters’ Language Model Research</h3><h4>Business Drivers and Research Objectives</h4><p>2023 proved to be an inflection point for generative AI, prompting Thomson Reuters to consider how their high-value, curated, data could improve on language models for customer-specific tasks.</p><p>In this two-part series, I explore the journey Thomson Reuters took to enable our cutting-edge research in training domain-adapted LLMs with Amazon SageMaker HyperPod. In this first part, I’ll go over some of the developments in the AI LLM landscape in 2023, the business drivers, and the research we were looking to conduct.</p><p><strong>Update</strong>: <a href="https://medium.com/tr-labs-ml-engineering-blog/part-2-2-scaling-thomson-reuters-language-model-research-e392d5aefe7f">Part 2</a> is now available.</p><h3>Large Language Models Disrupt The Industry, Businesses Move to Adopt</h3><p>In 2023, leaders in the generative AI space, such as OpenAI, released ground breaking large language models (LLMs) that drastically improved over previous model capabilities. Many others followed suit. The resulting technology opened new doors to enhancing customer experiences — tailoring content, recommendations, and responses to individual customers in natural chat-like interfaces.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E5pgr3_0mBUp-yjLDHpvVg.png" /><figcaption>Generated with AI (Microsoft Copilot Designer, DALL-E 3)</figcaption></figure><p>In retrospect generative AI’s impact is obvious, but the speed and impact at which it hit in 2023 was undeniably disruptive. For many businesses, the race was on to bring this technology into their products to maintain or gain competitive advantage. Thomson Reuters was no exception and keenly felt the need to help its customers be successful in this burgeoning, AI-Augmented, world.</p><p>However, as with any technology, proper application and understanding of its limitations is critical. How effectively we address these limitations will make or break a business. Consider the following:</p><ul><li><strong>Large Language Models Make Things Up</strong>: Large language models have a tendency to hallucinate or confabulate. Simply put, they may generate text that is not true! LLMs are quite remarkable in their ability to respond to natural language, and clearly encode significant amounts of knowledge. However, the stochastic nature of the technology means that its responses are based on probability of word occurrences. It doesn’t model “facts” so much as it models language. The model has no idea if the words (tokens) it’s generating are factually correct, though it may have successfully modeled the correct sequence of words to represent facts.</li><li><strong>Public Language Models May Not Be of Sufficient Quality</strong>: While the general knowledge encoded in the latest LLMs is remarkably good, it may not be enough for your business or customer domains. Public and commercial LLMs are based on the knowledge of the Internet — not what’s behind your business’ closed doors. Adding to the problem, bias and factually incorrect information exists on the Internet and there often isn’t enough transparency in what data is used and how commercial models are trained with it. Further, LLMs will only have encoded knowledge since their last training. They may not be up to date and businesses don’t control the frequency of model retraining.</li><li><strong>Large Language Models Can be Slow, Expensive, or Lack Sufficient Capacity</strong>: Depending on your use cases, you may find existing commercial LLMs are either too slow or too expensive or be in such high demand that you cannot purchase enough capacity to meet your requirements. (This may only be a temporary challenge, as we’ve observed increased capacity and reduced cost as hardware, optimizations, and economies of scale continue to improve.)</li></ul><p>Every business must take into consideration these limitations and devise strategies to address them. Thomson Reuters is no exception.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/1*JbU6XOhDDxYwM503iQQk9g.png" /></figure><p>Thomson Reuters’ customers have no tolerance for inaccuracies. They are professionals with discerning information needs in legal, corporate, tax, risk, fraud, compliance and news domains.</p><p>Take, for example, legal. US law is based on legal precedent — the outcomes of past trial cases are used to determine decisions in new cases. Not only does Thomson Reuters curate and enhance the publicly available content published by the courts, it has centuries of editorial content that analyzes and reflects on all aspects of the law. Legal research is a critical area for Thomson Reuters customers — it needs to be as complete as possible and there is no tolerance for made up facts or citations. Thomson Reuters products should not misrepresent the facts — they should not hallucinate or confabulate information.</p><blockquote><strong>The reality is that LLM hallucinations are a phenomenon that exists even when techniques, such as retrieval augmented generation (RAG), are used to minimize it. Solutions should be designed in such a way to allow for user verification. We should also set clear expectations with our users on effective use of our generative AI products. (See this recent Legal Current article “</strong><a href="https://www.legalcurrent.com/our-commitment-to-our-customers/"><strong>Our Commitment to our Customers</strong></a><strong>” by Mike Dahn, head of Westlaw Product Management, Thomson Reuters.)</strong></blockquote><h3>Thomson Reuters Gen AI Strategy — Buy, Build, Partner</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FI1VAe2xn8ynCklA9jI4tA.png" /><figcaption>Pillars Generated with AI (Microsoft Copilot Designer, DALL-E 3)</figcaption></figure><p>Thomson Reuters <a href="https://www.thomsonreuters.com/en/press-releases/2023/november/thomson-reuters-unveils-generative-ai-strategy-designed-to-transform-the-future-of-professionals.html">generative AI Strategy</a> includes investing an additional $100M per year on AI over the next several years with a <strong>buy</strong>, <strong>build</strong> and <strong>partner</strong> approach. In early 2023, we launched into this aggressively. While we worked to <strong>acquire</strong> <a href="https://www.thomsonreuters.com/en/press-releases/2023/august/thomson-reuters-completes-acquisition-of-casetext-inc.html">Casetext’s CoCounsel for $650M</a>, we also launched into <strong>building</strong> our first round of generative AI products. By November, we had our first beta of <a href="https://legal.thomsonreuters.com/blog/legal-research-meets-generative-ai/">Westlaw’s AI-Assisted Research</a>. It went to production in December. This was closely followed by <a href="https://legal.thomsonreuters.com/blog/legal-how-to-meets-generative-ai/">Ask Practical Law AI</a> in January and others are on their way. Lastly, <a href="https://www.thomsonreuters.com/en/ventures/.html">Thomson Reuters Ventures</a> is actively seeking investment and partnership opportunities with emerging AI startups.</p><p>These first round of solutions largely centered around <a href="https://www.datastax.com/guides/what-is-retrieval-augmented-generation">retrieval-augmented generation (RAG)</a> patterns and commercially available LLMs — such as <a href="https://openai.com/index/gpt-4/">OpenAI’s GPT-4</a> and <a href="https://www.anthropic.com/claude">Anthropic’s Claude</a>. RAG allowed us to address some of the limitations around missing domain knowledge and helped mitigate risks of hallucination. Rather than exposing a bare LLM to our customers, the LLM is a central component of a larger, intelligent, system.</p><p>We also utilized multiple LLM vendors where quality was comparable for specific tasks to address capacity and single point of failure.</p><p>And we are now launching an enterprise-wide <a href="https://www.thomsonreuters.com/en/artificial-intelligence.html">generative AI platform for a unified customer experience that spans products</a>. But that’s a story for another time!</p><p>While we were working on those near-term RAG solutions, the Labs foundational research team started work on how we might build our own LLMs. This research work, particularly around our journey to scale training with Amazon SageMaker HyperPod, will be the focus for the rest of this story.</p><h3>Our Research</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A2jDq2hzsUFr1mqPIrNrXA.png" /><figcaption>Research Workbench Generated with AI (Microsoft Copilot Designer, DALL-E 3)</figcaption></figure><p>Thinking back on the limitations of publicly available, commercial models (hallucinations, lack of domain knowledge in customer verticals, lack of transparency in the training data and processes, can be slow or costly to operate, etc.), we asked ourselves the following questions:</p><ul><li>Can Thomson Reuters’ editorially created, curated or enhanced data be used to improve LLM knowledge for specific business tasks?</li><li>Would smaller LLMs (e.g. 12–30b parameters) trained with Thomson Reuters data perform on par with very large LLMs upwards of a trillion parameters?</li><li>What methods could we employ to train the Thomson Reuters domain specific models to get the best results?</li></ul><p>The potential benefits as we see it fell along three areas: <strong>quality</strong>, <strong>agency</strong>, and <strong>operational efficiency</strong>.</p><p>Having full access to model training, it’s possible we could tune LLM generation to our domain and allow for tighter RAG integration. This would directly impact <strong>quality</strong>. And if we own the models, we <strong>control</strong> how and when they are trained and updated. We would have full audit and control over what went into the model training. Lastly, if smaller tuned models could perform sufficiently, it could be a more cost effective and scalable solution — improving <strong>overall operational efficiency</strong>.</p><h4>LLM Training Experimentation</h4><p>Our research focused around answering these specific questions:</p><ul><li>How well do foundational models (in the 7–30B parameters range) perform on specific tasks, unmodified? (This would be our baseline.)</li><li>Does performance <strong>improve</strong> for specific tasks when <strong>augmented</strong> with Thomson Reuters domain-specific data using various training techniques?</li></ul><p>To frame this research, and give us concrete evaluation targets, we focused on a number of real world <strong>tasks</strong>: legal summarization, classification, and question answering. We used publicly available general textual <strong>data</strong>, and domain specific textual data from our comprehensive stores of primary and secondary US law material. Primary law would include content published by the courts, and enhanced by TR. Secondary law, would include TR subject matter expert analysis and annotation of the law.</p><p><strong>Finding the Right Training Recipe Requires Experimentation</strong></p><p>There’s a lot of public research on training and adapting LLMs. We knew we would need to run a series of experiments — training LLMs from 7b to 30+b parameters, starting with a foundational model and continuous pre-training (using various techniques) with a mix of TR and general data. We would then finetune the model and evaluate how much better it performed on our specific legal tasks, while at the same time evaluating for any loss in general knowledge or language understanding.</p><p>1. <strong>Continuous Pre-Training:</strong> By further pre-training an existing foundational model, we wished to enrich its understanding of legalese without compromising its general language abilities. This was largely an experiment in finding the right mix of domain and general training data to retain <em>general</em> knowledge while increasing <em>domain-specific</em> knowledge. We used perplexity to measure impact of domain specific training on general knowledge capabilities of the model.</p><p>2. <strong>Instruction-Finetuning:</strong> This would be an exercise in generating impactful instruction datasets, including legal and general tasks. We experimented with pre-training open source foundational models, such as MPT, Flan-T5, Mistral, etc. and compared against industry standard commercial models, like OpenAI’s GPT-4. We used rouge to measure how well models performed on tasks.</p><blockquote><strong>In short, our experimentation would involve lots of LLM training!</strong></blockquote><h4>To be continued…</h4><p>In the first of this two-part series, we talked about how generative AI and Large Language Model (LLM) technology exploded in 2023. This created new business drivers that influenced Thomson Reuters’ AI strategy. One area involved exploring LLM customization.</p><p>This customization would require a computational scale we had not tackled before. That is where our engineering and technical challenges begin! Come back for <a href="https://medium.com/tr-labs-ml-engineering-blog/part-2-2-scaling-thomson-reuters-language-model-research-e392d5aefe7f">part 2</a> where we talk about these scale challenges and how we addressed them.</p><p><strong>💬 Let the conversation begin</strong> in the comments here or <a href="https://www.linkedin.com/feed/update/urn:li:activity:7219008152137084929/">on our LinkedIn Group</a>!</p><h4>Further Reading/References</h4><ul><li>Thomson Reuters <a href="https://www.thomsonreuters.com/en/press-releases/2023/november/thomson-reuters-unveils-generative-ai-strategy-designed-to-transform-the-future-of-professionals.html">generative AI Strategy</a></li><li>Thomson Reuters <a href="https://www.legalcurrent.com/our-commitment-to-our-customers/">commitment to our Customers</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52dfb0becedb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tr-labs-ml-engineering-blog/part-1-2-scaling-thomson-reuters-language-model-research-52dfb0becedb">Part 1/2 — Scaling Thomson Reuters’ Language Model Research</a> was originally published in <a href="https://medium.com/tr-labs-ml-engineering-blog">Thomson Reuters Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Welcome to the Thomson Reuters Labs Blog]]></title>
            <link>https://medium.com/tr-labs-ml-engineering-blog/welcome-to-the-thomson-reuters-labs-blog-1beb27998f40?source=rss-f2ce837c4d4c------2</link>
            <guid isPermaLink="false">https://medium.com/p/1beb27998f40</guid>
            <category><![CDATA[enterprise]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[innovation]]></category>
            <dc:creator><![CDATA[John Duprey]]></dc:creator>
            <pubDate>Tue, 28 May 2024 10:02:46 GMT</pubDate>
            <atom:updated>2025-01-07T21:19:13.757Z</atom:updated>
            <content:encoded><![CDATA[<h4>Our Medium Home Rebrand, AKA Hello from Labs!</h4><h4>Driving Innovation with AI and Emerging Technology</h4><p>Hello readers. Welcome to the Thomson Reuters Labs blog! Thomson Reuters Labs aims to be the leading force in AI innovation across all industries served by TR, creating a world where AI not only informs the way forward but drives it, transforming the way people access and utilize information, and ultimately, fostering a more informed, efficient, and just society. This blog has been around for a while, but we’re refreshing our focus expanding the range of topics we cover from machine learning, and other computer science topics to very specific applied innovation topics that Labs is spearheading on behalf of our business and customers. We truly are at the forefront — not just of cutting-edge AI research, but also in the application of it to real world, high value enterprise problems. There are few companies in the world who have released as much AI powered products to the market as we have, and we’re here to tell our story to the world.</p><figure><img alt="Rebranded Blog: Thomson Reuters Labs" src="https://cdn-images-1.medium.com/max/1024/1*gPoyUsV8iAOXqM5lQK5Ndg.png" /><figcaption>Rebranded Blog: Thomson Reuters Labs</figcaption></figure><h4>There’s Plenty to Talk About</h4><p>We are particularly interested in talking more about our how we are creating AI-based capabilities in our products. The explosion in generative AI technology has drastically accelerated our ability to solve our customers’ most challenging problems. And indeed AI has accelerated development for many software companies; but as a trust business, we have a particular interest in not just developing at pace, but doing so in a way that earns and maintains the trust and truthfulness that our brand has been known for so many decades. We think there are a lot of good stories to share in this space and Labs has a unique perspective being at the heart of the AI in these capabilities. We are limited only by the amount of time we have in a day.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*80k6-KN0QoDR2tbDcwjE3w.png" /><figcaption>Westlaw AI-Assisted Research</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-0hNJo1OmjgJpMnLuGh6xA.png" /><figcaption>Ask Practical Law AI</figcaption></figure><p>For example, we released the retrieval-augmented generation (RAG) based question answering system in Thomson Reuters’ first generative AI assistant — <a href="https://legal.thomsonreuters.com/blog/legal-research-meets-generative-ai/">Westlaw’s AI-Assisted Research, released last November</a>. This was followed up with <a href="https://legal.thomsonreuters.com/blog/legal-how-to-meets-generative-ai/">Ask Practical Law AI in January of this year</a> and <a href="https://www.thomsonreuters.com/en/press-releases/2024/april/thomson-reuters-unveils-checkpoint-edge-with-cocounsel-featuring-its-generative-ai-assistant-for-tax-research.html">Checkpoint Edge with CoCounsel just last month</a>. And many more solutions are on the way!</p><p>While our product innovation continues to accelerate, we are actively exploring how our unique competitive advantage of 150 years of world class content and subject matter expertise can be incorporated into generative AI — through domain adaption of language models, large and small, and other techniques. We recently shared <a href="https://youtu.be/PDG8VLW4ts8?si=wcPvz1fplRdPjvLb&amp;t=1353">our journey of scaling LLM customization through continuous pretraining and instruction finetuning of open source LLMs at the 2024 AWS London Summit</a>. (Follow this blog for a post about that soon!) And more results of that research will be shared over the coming months.</p><h3>Why are we Doing This?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*NIjffypOKx20-jrzG_-9Nw.png" /><figcaption>Labs Workshop</figcaption></figure><p>Labs has been at the heart of much of Thomson Reuters innovation <a href="https://www.thomsonreuters.com/en-us/posts/our-purpose/thomson-reuters-labs-at-30-celebrating-three-decades-of-innovation/">over the past 30 years</a>. What started out as a <em>handful</em> of people in the early 1990s focused on applied AI in the legal domain, has expanded into a team of ~170 applied scientists and research engineers focused on all segments of Thomson Reuters’ business.</p><p>So we have plenty to talk about, but why put in the time and effort to share here? Thomson Reuters is a technology company, but we often fly under the radar — we don’t talk enough about the great work we do and the hard problems we solve. We truly are a world class AI company in the enterprise space, where it is traditionally <em>extremely</em> challenging to deliver and drive AI adoption. We have a great story to tell, and we could benefit from joining in the global dialog about developing and applying technology to solve real world problems. We hope this will generate some healthy discussion in science, engineering, and even business areas. Perhaps we can learn from each other’s experiences. And, lets be honest, it will help our brand as a technology company — to continue to attract the world class people needed to further progress our mission.</p><blockquote><strong>Shameless Plug</strong>: <a href="https://careers.thomsonreuters.com/us/en/search-results?keywords=TR%20Labs">Labs <em>is</em> actively hiring so check out our careers page</a>. If you like the content you see here, you might like to come work for us!</blockquote><h3>Follow Us!</h3><p>Where to go from here? Please consider following us here and on our <a href="https://www.linkedin.com/groups/8168422/?highlightedUpdateUrn=urn:li:groupPost:8168422-7196929350141648899&amp;q=highlightedFeedForGroups">LinkedIn Group</a> and join in on the conversation! Look for us to increase our volume of content over the coming months. Curious about specific areas of our work, what its like to work in applied R&amp;D, or something else? Let us know in the comments. We’re building a backlog of story ideas.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/345/1*k7xn1Tq24Vp6M1cCEFWbCg.png" /><figcaption>Please follow us!</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1beb27998f40" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tr-labs-ml-engineering-blog/welcome-to-the-thomson-reuters-labs-blog-1beb27998f40">Welcome to the Thomson Reuters Labs Blog</a> was originally published in <a href="https://medium.com/tr-labs-ml-engineering-blog">Thomson Reuters Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>