<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Toby Fotherby on Medium]]></title>
        <description><![CDATA[Stories by Toby Fotherby on Medium]]></description>
        <link>https://medium.com/@fotherby?source=rss-a17f98955c2d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*qAT-I8VxCDLEKIlYszhxRQ.jpeg</url>
            <title>Stories by Toby Fotherby on Medium</title>
            <link>https://medium.com/@fotherby?source=rss-a17f98955c2d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 17:15:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@fotherby/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Efficient Generative AI model evaluation using Amazon Bedrock Model Evaluation LLM-as-a-Judge]]></title>
            <link>https://medium.com/ai-ml-genai-solution-architecture/efficient-generative-ai-model-evaluation-using-amazon-bedrock-model-evaluation-llm-as-a-judge-0038a3aa11dd?source=rss-a17f98955c2d------2</link>
            <guid isPermaLink="false">https://medium.com/p/0038a3aa11dd</guid>
            <category><![CDATA[generative-ai-solution]]></category>
            <category><![CDATA[llm-evaluation]]></category>
            <dc:creator><![CDATA[Toby Fotherby]]></dc:creator>
            <pubDate>Sun, 29 Dec 2024 02:19:08 GMT</pubDate>
            <atom:updated>2025-02-12T17:52:45.741Z</atom:updated>
            <content:encoded><![CDATA[<p>Authors: Toby Fotherby and Atul Varshneya</p><p>A common challenge when creating and upgrading Generative AI LLM solutions is determining which model to use. For green-field solutions, the starting point tends to be either models which have demonstrated success in the past for similar use cases, or the most powerful model available, to get a quick start on understanding the nuances and challenges of the use cases in focus. This will then evolve into the more standard scenario, which also occurs periodically for more mature solutions, which may be summarized as: <em>Now that we have a reasonable understanding of the use case and good test data (LLM inputs and target outputs), is there a better LLM that we could be using? </em>Where <em>better</em> may mean lower-latency, low-cost, more accurate generative output, etc., depending on business goals and current state.</p><p>With faster, cheaper and more capable LLMs becoming available at high frequency, and that prompt compatibility across models is less of obstacle that it used to be, there is significant opportunity for improvement for those who can quickly take advantage of these advances.</p><p>This is where <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation-judge.html">Amazon Bedrock Model Evaluation LLM-as-a-Judge</a>, in Public Preview on AWS at the time of writing, comes in useful. In a nutshell<em>, </em>LLM-as-a-Judge enables you to rapidly create a model evaluation job that uses an LLM judge model to score your target model’s response (which is generated as part of the job), and then provide an explanation of why a prompt and response pair received the score. Scores and explanations are available in a report card, and the full responses are available in an Amazon S3 bucket that you specify when you create the model evaluation job.</p><p>To help show its potential value for your use cases, the main body of this article shows a full example of its use. An LLM as a Judge evaluation job is configured, run and results are considered, along with the overall utility of the tooling .</p><p>The starting point is the use case that you want to evaluate. For this example our use case is a natural language to SQL (NL2SQL) scenario. Many LLMs have been trained on NL2SQL tasks, and the <a href="https://github.com/defog-ai/sqlcoder/blob/main/prompt.md">structure of prompt data for NL2SQL</a> is fairly consistent across difference LLMs. This allows us to consider that the NL2SQL prompts that we have created for our base model, should be largely transferable to other LLMs without specific tuning.</p><p>An evaluation test data for this use case is needed to create the evaluation job. The objective is to see how accurate an alternative model is given the same prompt as the base generative model. To create this evaluation test set, the current (golden) test set for the base generative model is leveraged. The test set has the natural language query input, and the (golden) ground truth SQL output. To create the evaluation test set, the generative prompt for each test case is generated and saved. This set of prompts is then paired with the (golden) ground truth SQL output. Together, the generative prompt and the golden SQL output, form the evaluation test set. The specific format that this needs to be in for LLM as a Judge is described later in this article.</p><p>With the test data ready for use, configuring an LLM as a Judge evaluation run is straightforward and intuitive. The starting point is the Amazon Bedrock Console. From there select the Evaluations option in the side-navigation. This will bring you to the home-screen for evaluations.</p><figure><img alt="The home page for Amazon Bedrock Evaluations" src="https://cdn-images-1.medium.com/max/1024/1*Ah2-EYkRjahGASKoCZ5Dpg.png" /><figcaption>Starting Screen for Amazon Bedrock Model Evaluations</figcaption></figure><p>From here select the button to create an Automatic evaluation using Model as a judge.</p><figure><img alt="Selecting the option for Model as a judge" src="https://cdn-images-1.medium.com/max/1024/1*M4AWMxVbfTwX2B_XqKzS5A.png" /><figcaption>Selecting the Option for Model as a Judge</figcaption></figure><p>This will take you to the screen where you can configure the parameters for the model evaluation.</p><p>The first section of the configuration captures some metadata about the evaluation, the name you want to give it and a description. These are useful as reference for comparison of evaluation results across different jobs. Following this is a key decision point, choosing which model to use as the judge, evaluator of the results. At the time of writing this there are several model options available, the set of models you will see will vary by region. For this example task, I have chosen Claude 3.5 Sonnet, based on the complexity of the evaluation task and its ability to process evaluation tasks such as this with a good degree of accuracy. Last in this section, is the configuration of the Generator model, i.e., the model to be evaluated. This model will be invoked with the test prompts for the evaluation run. The set of model options will again vary by region. At the time of writing, models from AI21 Labs, Amazon, Anthropic, Cohere, Meta, and Mistral were available in the us-east-1 region. I have chosen to Amazon Nova Micro model, as I want to compare its behavior to its sister model Amazon Nova Lite.</p><figure><img alt="Initial Configuration Definition" src="https://cdn-images-1.medium.com/max/1024/1*RxBQmLOou_lJBZ43TYol8Q.png" /><figcaption>Initial Configuration Definition</figcaption></figure><p>The next part of the configuration is to select the metrics that you want the evaluation to use to assess the output of the generator model. For this task, which is essentially a code generation task, I have chosen two quality metrics: Correctness, and Completeness. I based this decision on my objective (to compare the output quality of Nova Micro vs. Nova Lite for this use case), and on review of the <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-type-judge-prompt.html">evaluator prompts</a> that will be used for the evaluation process.</p><figure><img alt="Configuration of Metric for Evaluation" src="https://cdn-images-1.medium.com/max/1024/1*bb1Blh_IaVBQc3p4yypudQ.png" /><figcaption>Configuration of Metric for Evaluation</figcaption></figure><p>As shown above, there are a broad set of metrics available, making this LLM as a Judge process applicable to a wide range of model comparison and evaluation scenarios: A key strength of this solution.</p><p>The penultimate configuration section is specifying the evaluation datasets. The input, the prompt dataset, needs to be a single file on Amazon S3 in the same region as the evaluation job that is being configured. The <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-prompt-datasets-judge.html">input file</a> needs to be in a JSON lines format, use the .jsonl file extension, and can include up to 1000 JSON documents, one per test case. Briefly, the structure of the <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-prompt-datasets-judge.html">input file </a>is as follows:</p><pre>[<br>  {<br>    &quot;prompt&quot;: &quot;...&quot;, - The complete prompt for the generative model <br>    &quot;referenceResponse&quot;: &quot;...&quot;, - The ground truth response (optional)<br>    &quot;category&quot;: &quot;...&quot; - Evaluation class or sub-class (optional)<br>  },<br>  {<br>    &quot;prompt&quot;: &quot;...&quot;,<br>    &quot;referenceResponse&quot;: &quot;...&quot;,<br>    &quot;category&quot;: &quot;...&quot;<br>  },<br>...<br>]</pre><p>In addition to specifying the input file location, you need specify an output folder location on Amazon S3 where the results will be saved.</p><figure><img alt="Specifying the Input and Output S3 Locations" src="https://cdn-images-1.medium.com/max/1024/1*nus0SsZieJSLWPe7gYd1Lg.png" /><figcaption>Specifying the Input and Output S3 Locations</figcaption></figure><p>Note: For jobs created using the console you must update the Cross Origin Resource Sharing (CORS) configuration on the S3 bucket. To learn more about the required CORS permissions, see <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-security-cors.html">Required Cross Origin Resource Sharing (CORS) permissions on S3 buckets</a>.</p><p>Once you have done this, there’s the final section of specifying the IAM role to use to run the job. The user interface guides you through this, including creating a new role if you desire.</p><figure><img alt="Specifying the IAM Role" src="https://cdn-images-1.medium.com/max/1024/1*bjGcDBTAixeZCdXxuOk2og.png" /><figcaption>Specifying the IAM Role</figcaption></figure><p>With these elements defined you can push the button at the bottom of the configuration to run the evaluation job. Evaluation jobs are shown on the Amazon Bedrock Evaluations screen.</p><figure><img alt="Evaluation Jobs Summary" src="https://cdn-images-1.medium.com/max/1024/1*ZwmKUxhybOLQ-99dmKY4gA.png" /><figcaption>Evaluation Jobs Summary</figcaption></figure><p>The results for each completed job can be viewed in the console as an easy to read model evaluations report. A subset of a report is shown below. The results from different job runs can also be compared through the console, or programmatically, by comparing the results of the job outputs which are saved to Amazon S3 with each run.</p><figure><img alt="Model Evaluation Report" src="https://cdn-images-1.medium.com/max/1024/1*EMRPvfc02bFV4G1JYmLW8Q.png" /><figcaption>Model Evaluation Report for Nova Micro</figcaption></figure><p>In this particular report, there are some instances where the results are not positive. The Bedrock Console provides a user interface to see the detail of each test case instance, the prompt input, the generated output, and the ground truth, enabling further exploration.</p><p><strong>Closing thoughts</strong></p><p>A key aspect of this tooling is enabling rapid evaluation of LLMs across a wide range of metrics focused on output quality and responsible AI. These evaluations can be used to spot check Bedrock based LLM solutions across these metrics and/or to assess if other models may be more suitable for the use cases that we develop. With LLMs getting smarter, faster and cheaper all the time, and the ease of use of this tooling, this can help us improve our applications.</p><p>API vs. Console? The console tooling provided by Amazon Bedrock, shown in this article, is really well put together. It provides excellent ease of use and is a great way to get familiar with the capabilities offered. For systematic use, there are <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-jobs-management-create-judge.html">APIs</a> that use can use to drive the evaluation. With these <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-jobs-management-create-judge.html">APIs</a>, programatic evaluation of quality and responsible AI metrics, as part of the solution release process, is now more readily achieveable.</p><p>Pricing of this is very reasonable. The cost is simply the standard <a href="https://aws.amazon.com/bedrock/pricing/">Amazon Bedrock pricing</a> for model inference. There will be model inference for the generative model (being called with the evaluation test prompt), and there will be model inference for the LLM as a Judge prompt also (which includes the generative prompt input, the results of the generative output, and the judge prompt). There are no additional charges for evaluation jobs themselves or console use. The models used are billed according to their normal on-demand pricing. So all in all its a great deal.</p><h4>About the Authors</h4><p><strong>Toby Fotherby </strong>is a Senior AI and ML Specialist Architect at Amazon Web Services (AWS), helping customers use cloud-based AI/ML and GenAI services to rapidly scale their innovations. He has over a decade of cross-industry expertise leading strategic initiatives. Toby also leads a program training the next generation of AI Solutions Architects.</p><p><strong>Atul Varshneya </strong>is formerly a Principal AI and ML Specialist at Amazon Web Services (AWS). He currently focuses on developing solutions in the areas of AI/ML, and particularly generative AI. In his career of four decades, Atul has worked as the technology R&amp;D leader in multiple large companies and startups. Outside of work he enjoys performing Hindustani classical music.</p><p><em>This article represents the opinions of the authors, not the opinions of their employer.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0038a3aa11dd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-ml-genai-solution-architecture/efficient-generative-ai-model-evaluation-using-amazon-bedrock-model-evaluation-llm-as-a-judge-0038a3aa11dd">Efficient Generative AI model evaluation using Amazon Bedrock Model Evaluation LLM-as-a-Judge</a> was originally published in <a href="https://medium.com/ai-ml-genai-solution-architecture">AI/ML &amp; GenAI Solutions Architecture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is the role of an AI Solutions Architect in 2025?]]></title>
            <link>https://medium.com/ai-ml-genai-solution-architecture/what-is-the-role-of-an-ai-solutions-architect-in-2025-a417446a3d8f?source=rss-a17f98955c2d------2</link>
            <guid isPermaLink="false">https://medium.com/p/a417446a3d8f</guid>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[solution-architect]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Toby Fotherby]]></dc:creator>
            <pubDate>Thu, 19 Dec 2024 23:55:06 GMT</pubDate>
            <atom:updated>2025-01-02T19:57:09.077Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="An AI generated image of a slightly sureal tiger" src="https://cdn-images-1.medium.com/max/1024/1*_4GPMNxpCiMyPToW68mIlg.jpeg" /></figure><p>This is a question that I am asking myself as I kick-off my next online training program called <em>AI Solution Architect; </em>a live sixteen-part program to help IT and software developers transition into AI solutions roles. I also happen to be an AI/ML Specialist Solution Architect with AWS, collaborating with customers on their journey, and considering how their roles are evolving.</p><p>The last couple of years have been pivotal for AI practitioners. The advances in tooling — LLMs, multi-modal models, no-code/low-code AI/ML, and AI as a service — has changed what is possible, viable and expected. Along with this, the role of the AI Solution Architect is changing. For those transitioning to this role and those consolidating their skills, considering the expectations, opportunities and nuances will help you move in the right direction.</p><p>The foundation of AI Solutions Architect (AI SA) role is very much that of the standard Software Solutions Architect role. For instance, looking at some of the core behaviors, there is much similarity:</p><ul><li>Designing and developing comprehensive solutions that align with business requirements, best practices and technical constraints</li><li>Evaluating and selecting appropriate technologies, tools, and frameworks to build robust and scalable solutions</li><li>Collaborating with product stakeholders and providing technical leadership to ensure effective communication and alignment</li><li>Staying up-to-date with emerging technologies and industry trends to continuously improve and optimize solutions</li></ul><p>Looking for differences helps kick-start a more specific definition the AI Solution Architect role. As an AI SA, central concerns include:</p><ul><li>Designing and developing AI solutions and patterns that align with business objectives and requirements</li><li>Assessing and selecting appropriate AI technologies, frameworks, and tools for projects in focus</li><li>Overseeing the integration of AI systems and patterns with existing infrastructure and applications</li><li>Ensuring the scalability, performance, security, and maintainability of AI solutions</li></ul><p>Attending to these concerns can be particularly challenging:</p><ul><li>AI technologies are evolving extremely rapidly</li><li>Key solution aspects, such as acceptable quality, latency, and cost, often lack clarity during scoping and design</li><li>Assessing and communicating what is possible, design options, challenges, and risks, etc.</li></ul><p>This suggests a particular requirement for soft-skills, along with specific AI skills, to bridge the gap for stakeholders and to facilitate decision making.</p><p>It may seem as if the AI SA role needs so many skills that it must be a Unicorn: It’s impossible! More reasonably, perhaps it’s more like a juggler who is able to focus and deep-dive on the specifics of their domain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tkmM0GjLrnDVHGenogTW1w.jpeg" /><figcaption>AI Solution Architect: Juggler (generated by Stable Diffusion)</figcaption></figure><p><strong>How is AI Solution Architecture in 2025 different?</strong></p><p>Changes in state of the art:</p><ul><li>Smarter high-end models</li><li>Smarter light-weight model</li><li>Agent based reasoning and application frameworks</li><li>RAG 2.0</li><li>Multi-modal models and solution patterns</li></ul><p>Change in the application space:</p><ul><li>New domains adopting Generative AI</li><li>Evolution of stakeholder understanding and expectations</li><li>Application and domain specific challenges</li></ul><p>With the scope of the AI SA practice, the scope of tools, and the scope of the business domains, all expanding, there is an increasing need for specialization with the AI SA role.</p><p><strong>Opportunities for AI SA Specialism</strong></p><p><strong>Closing Thoughts</strong></p><p>The industry needs more AI Solution Architects. Skilled software professionals and product developers are in an excellent position to transition into AI and build on their domain and industry knowledge. With AI being more powerful and easier to apply than ever before, 2025 is a great time start. And for those who are already in the ranks as an AI SA, 2025 is excellent time for further specialization.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a417446a3d8f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-ml-genai-solution-architecture/what-is-the-role-of-an-ai-solutions-architect-in-2025-a417446a3d8f">What is the role of an AI Solutions Architect in 2025?</a> was originally published in <a href="https://medium.com/ai-ml-genai-solution-architecture">AI/ML &amp; GenAI Solutions Architecture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>