<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dylan Pieper on Medium]]></title>
        <description><![CDATA[Stories by Dylan Pieper on Medium]]></description>
        <link>https://medium.com/@dylanpieper?source=rss-139626b3bf16------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*5k2im6v6dXOAAOv2mD5B6A.jpeg</url>
            <title>Stories by Dylan Pieper on Medium</title>
            <link>https://medium.com/@dylanpieper?source=rss-139626b3bf16------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 08:04:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dylanpieper/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Batch and Compare the Similarity of LLM Responses in R]]></title>
            <link>https://medium.com/@dylanpieper/batching-and-evaluating-the-similarity-of-llm-responses-in-r-dd9f347f2a11?source=rss-139626b3bf16------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd9f347f2a11</guid>
            <category><![CDATA[r]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[batch]]></category>
            <category><![CDATA[similarity]]></category>
            <dc:creator><![CDATA[Dylan Pieper]]></dc:creator>
            <pubDate>Mon, 03 Mar 2025 19:56:14 GMT</pubDate>
            <atom:updated>2025-03-21T16:55:05.210Z</atom:updated>
            <content:encoded><![CDATA[<p>R is leading the way for data scientists to make the most of large language models. Posit’s official package <a href="https://ellmer.tidyverse.org/">ellmer</a> provides a powerful functional interface for chatting, streaming responses, extracting data, and calling functions — built on the modern, safe, and fast HTTP client <a href="https://httr2.r-lib.org/">httr2</a>.</p><figure><img alt="R package hex logo featuring a cartoon turquoise elephant with colorful patchwork ears against a vibrant background of multicolored patterned squares, with the name “ellmer” prominently displayed." src="https://cdn-images-1.medium.com/max/240/0*4gatFWACsRA_aWkj.png" /></figure><p>Data scientists often need to provide multiple prompts to LLMs to complete many actions at time — a process referred to as <strong>batch processing</strong>. Batch processing can be achieved at an API level for some providers (<a href="https://platform.openai.com/docs/guides/batch">OpenAI</a>, <a href="https://docs.anthropic.com/en/docs/build-with-claude/batch-processing">Anthropic</a>, <a href="https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/batch-prediction-gemini#generative-ai-batch-text-drest">Gemini</a>, and <a href="https://mistral.ai/news/batch-api">Mistral</a>). This option is around 50% cheaper than requesting responses in real-time and delivers responses within 24 hours. See <a href="https://edubruell.github.io/tidyllm/">tidyllm</a> for a CRAN package allows you to implement batch APIs.</p><p>Real-time batching costs more, but it delivers immediate responses and supports any LLM provider. While the CRAN release of ellmer (0.1.1) lacks batch processing capabilities, package creator Hadley Wickham is enhancing ellmer and httr2 with both real-time and API batching in their development GitHub repositories. In the meantime, I created a robust wrapper package called <a href="https://dylanpieper.github.io/hellmer/">hellmer</a> to fill this gap with a user-friendly interface with rich features such as <a href="https://ellmer.tidyverse.org/articles/tool-calling.html">tooling</a> and <a href="https://ellmer.tidyverse.org/articles/structured-data.html">structured data extraction</a>, progress tracking and recovery, automatic retry with backoff, and more.</p><h3>Sequential vs. Parallel Processing</h3><p>There are two methods you can use for batch processing: <strong>sequential</strong> or <strong>parallel</strong> processing. Sequential processing always uses one process and makes one API request at a time.</p><p>Sequential processing is slow, which could be good or bad depending on your goal, and generally safer because you can save your progress at each step. Parallel processing is fast but makes it tricky to keep track of all of the responses. In ellmer, parallel processing uses one R process using httr2, makes multiple requests at a time, and if interrupted, returns the partial responses; however, they won’t be saved to the disk. In hellmer, parallel processing uses multiple R processes using <a href="https://future.futureverse.org/">future</a> and saves the responses in chunks of prompts to the disk. If interrupted, the last chunk that was saved can be resumed with no additional steps. If you care about speed, the processing times between ellmer and hellmer are very similar because most of the time is waiting for the API to respond. One limitation of hellmer is that you have to set the chunk size to equal the number of prompts to achieve the speeds seen below, which sacrifices safety as it will not return partial responses if interrupted like ellmer will.</p><figure><img alt="Violin plot comparing ellmer vs hellmer library performance. In parallel mode, hellmer (15–17s) outperforms ellmer (17–18s). In sequential mode, ellmer (180–200s) outperforms hellmer (220–240s). Based on OpenAI tests with 200 prompts x 5 runs." src="https://cdn-images-1.medium.com/max/750/1*wXEJSOVnAXjUQZeR0znhIg.png" /></figure><h3>Batch with hellmer</h3><p>Implementing real-time batch processing in R is fairly easy. I will focus on using hellmer, but if you are curious how to batch using ellmer alone, check out the <a href="https://gist.github.com/dylanpieper/bb8d0b41463db5c2b808b3c6df061c60">gist for the real-time batch performance benchmark</a>.</p><p>I recommend setting your API keys for your LLM providers in my user or project environment using usethis::edit_r_environ(scope = c(&quot;user&quot;, &quot;project&quot;), which will open a .Renviron file where you can add any API key, such as OPENAI_API_KEY=your-key or ANTHROPIC_API_KEY=your-key.</p><p>Install the package from CRAN: install.packages(&quot;hellmer&quot;).</p><p>The two primary functions chat_sequential and chat_future create a sequential or parallel processor around an ellmer chat function. For example:</p><pre>library(hellmer)<br><br># Option 1: Sequential processing<br>chat &lt;- chat_sequential(chat_openai, system_prompt = &quot;Reply concisely&quot;)<br><br># Option 2: Parallel processing via future<br>chat &lt;- chat_future(chat_openai, system_prompt = &quot;Reply concisely&quot;)<br><br>result &lt;- chat$batch(list(&quot;What is R?&quot;,<br>                         &quot;What is Python?&quot;,<br>                         &quot;What is Julia?&quot;,<br>                         &quot;What is Rust?&quot;))<br># Methods<br>result$progress() # Return batch progress (if interuppted)<br>result$texts()    # Return list of responses<br>result$chats()    # Return ellmer chat objects</pre><h3>Compare the similarity of LLM responses</h3><p>Imagine I have a batch of LLM responses. I ask for the sentiment of texts (positive, neutral, or negative). Like a good scientist, I should manually review the responses to assess their validity. In true psychometric fashion, I may even ask research assistants or colleagues to independently review the responses to determine the <a href="https://en.wikipedia.org/wiki/Inter-rater_reliability">inter-rater reliability</a>. I would then analyze the ratings and resolve any conflicts.</p><p>But, what if, I could simulate this process using different LLM models or providers. I could ask OpenAI and Claude to complete the same task, compare their responses, and resolve any conflicts. This strategy is often referred to as LLM-as-a-judge<strong>; </strong>although, in this case, a more accurate term might be LLM-as-an-independent-rater.</p><p>That’s how I got the idea for <a href="https://dylanpieper.github.io/samesies/">samesies</a> — a package I developed to compare lists of texts, factors, or numerical values to measure their similarity.</p><p>Install the package from CRAN: install.packages(&quot;samesies&quot;).</p><p>The three primary functions (same_text, same_factor, and same_number) accept two or more lists as inputs, including nested listed, but are inherently typed and will not work with mixed types. For example:</p><pre># Text<br>r1 &lt;- list(&quot;R is a statistical computing software&quot;, <br>           &quot;R enables grammar of graphics using ggplot2&quot;, <br>           &quot;R supports advanced statistical models&quot;)<br>r2 &lt;- list(&quot;R is a full-stack programming language&quot;,<br>           &quot;R enables advanced data visualizations&quot;, <br>           &quot;R supports machine learning algorithms&quot;)<br>tex &lt;- same_text(r1, r2)<br><br># Factors<br>cats1 &lt;- list(&quot;R&quot;, &quot;R&quot;, &quot;Python&quot;)<br>cats2 &lt;- list(&quot;R&quot;, &quot;Python&quot;, &quot;R&quot;)<br>fct &lt;- same_factor(cats1, cats2, <br>                   levels = c(&quot;R&quot;, &quot;Python&quot;))<br><br># Numbers<br>n1 &lt;- list(1, 2, 3)<br>n2 &lt;- list(1, 2.1, 3.2)<br>num &lt;- same_number(n1, n2)</pre><figure><img alt="R pacakge hex logo image of three Spiderman characters — from different movies — pointing at each other. This is based on a meme that was created from a 1967 episode of the Spider-Man cartoon series, featuring two Spider-Men accusing each other of being impostors. The original image is fan art created by the Reddit user WistlerR15." src="https://cdn-images-1.medium.com/max/1024/1*HY8IbaEQOFBel2t8aQqXmw.png" /></figure><h3>Complete example</h3><p>I’ll use a minimal example to demonstrate how to batch and evaluate the similarity of LLM responses in R using hellmer and samesies.</p><p>I will use hellmer to chat with OpenAI (gpt-4o) and Claude (claude-3-5-sonnet-latest). I’ll ask for the sentiment of our prompts as a:</p><ul><li>text (emotional tone),</li><li>factor (positive, neutral, or negative), and</li><li>number (0.0 to 1.0).</li></ul><p>Below are my prompts and code to batch the LLM responses:</p><pre>library(hellmer)<br><br>prompts &lt;- list(<br>  # Positive<br>  &quot;R makes data visualization incredibly easy with ggplot2.&quot;,<br>  &quot;I love how R integrates statistics and data science seamlessly.&quot;,<br>  &quot;The R community is really supportive and welcoming.&quot;,<br>  # Neutral<br>  &quot;R is commonly used in academic research.&quot;,<br>  &quot;R has both base functions and tidyverse functions for data manipulation.&quot;,<br>  &quot;RStudio is one of the most popular IDEs for R development.&quot;,<br>  # Negative<br>  &quot;R is painfully slow for large datasets compared to Python.&quot;,<br>  &quot;R&#39;s object-oriented system is confusing and inconsistent.&quot;,<br>  &quot;Installing packages in R can be frustrating due to dependency errors.&quot;,<br>  # Ambiguous<br>  &quot;I use R every day, but I&#39;m not sure if I love it or hate it.&quot;,<br>  &quot;Tidyverse makes R more accessible, but it adds another layer of abstraction.&quot;,<br>  &quot;R has a steep learning curve, but once you get it, it&#39;s great.&quot;<br>)<br><br>openai &lt;- chat_future(ellmer::chat_openai)<br>claude &lt;- chat_future(ellmer::chat_claude)<br><br>type_sentiment &lt;- type_object(<br>  &quot;Extract sentiments&quot;,<br>  sentiment_str = type_string(&quot;Describe the emotional tone in one word&quot;),<br>  sentiment_fct = type_enum(&quot;The sentiment type&quot;, c(&quot;positive&quot;, &quot;neutral&quot;, &quot;negative&quot;)),<br>  sentiment_num = type_number(&quot;Negative to positive sentiment score, 0.00 to 1.00&quot;),<br>)<br><br>openai_dat &lt;- openai$batch(prompts, type_spec = type_sentiment)<br>claude_dat &lt;- claude$batch(prompts, type_spec = type_sentiment)<br><br>openai_dat &lt;- openai_dat$structured_data()<br>openai_dat &lt;- list(<br>  sentiment_str = purrr::map_chr(openai_dat, &quot;sentiment_str&quot;),<br>  sentiment_fct = purrr::map_chr(openai_dat, &quot;sentiment_fct&quot;),<br>  sentiment_num = purrr::map_dbl(openai_dat, &quot;sentiment_num&quot;)<br>)<br><br>claude_dat &lt;- claude_dat$structured_data()<br>claude_dat &lt;- list(<br>  sentiment_str = purrr::map_chr(claude_dat, &quot;sentiment_str&quot;),<br>  sentiment_fct = purrr::map_chr(claude_dat, &quot;sentiment_fct&quot;),<br>  sentiment_num = purrr::map_dbl(claude_dat, &quot;sentiment_num&quot;)<br>)</pre><p>I wrangle the nested list that ellmer returns to a list of lists, because I will extract them based on type to run my samsies functions.</p><p>For those who are curious, here’s what the wrangled data looks like:</p><pre><br>openai_dat<br># $sentiment_str<br>#  [1] &quot;appreciative&quot;             &quot;appreciative&quot;             &quot;Positive and encouraging&quot; &quot;neutral&quot;                 <br>#  [5] &quot;informative&quot;              &quot;Informative&quot;              &quot;Critical&quot;                 &quot;frustrated&quot;              <br>#  [9] &quot;frustration&quot;              &quot;ambivalent&quot;               &quot;Balanced&quot;                 &quot;Encouraging&quot;             <br><br># $sentiment_fct<br>#  [1] &quot;positive&quot; &quot;positive&quot; &quot;positive&quot; &quot;neutral&quot;  &quot;neutral&quot;  &quot;neutral&quot;  &quot;negative&quot; &quot;negative&quot; &quot;negative&quot; &quot;neutral&quot;  &quot;neutral&quot; <br>#  [12] &quot;positive&quot;<br><br># $sentiment_num<br>#  [1] 0.85 0.95 0.88 0.50 0.50 0.75 0.35 0.15 0.25 0.50 0.50 0.75<br><br>claude_dat<br># $sentiment_str<br>#  [1] &quot;enthusiastic&quot; &quot;enthusiastic&quot; &quot;enthusiastic&quot; &quot;favorable&quot;    &quot;neutral&quot;      &quot;appreciative&quot; &quot;frustrated&quot;   &quot;frustrated&quot;  <br>#  [9] &quot;frustrated&quot;   &quot;ambivalent&quot;   &quot;ambivalent&quot;   &quot;optimistic&quot;  <br><br># $sentiment_fct<br>#  [1] &quot;positive&quot; &quot;positive&quot; &quot;positive&quot; &quot;positive&quot; &quot;neutral&quot;  &quot;positive&quot; &quot;negative&quot; &quot;negative&quot; &quot;negative&quot; &quot;neutral&quot;  &quot;neutral&quot; <br># [12] &quot;positive&quot;<br><br># $sentiment_num<br>#  [1] 0.85 0.85 0.90 0.75 0.50 0.75 0.20 0.20 0.20 0.50 0.50 0.70</pre><p>I’m ready to compare the responses:</p><pre>library(samesies)<br><br>check_str &lt;- same_text(<br>    &quot;openai&quot; = openai_dat$sentiment_str |&gt; as.list(),<br>    &quot;claude&quot; = claude_dat$sentiment_str |&gt; as.list()<br>  )<br>average_similarity(check_str)<br>#     osa      lv      dl hamming     lcs   qgram  cosine jaccard      jw soundex<br>#   0.390   0.390   0.390   0.194   0.219   0.584   0.643   0.495   0.666   0.250<br><br>check_fct &lt;- same_factor(<br>  &quot;openai&quot; = openai_dat$sentiment_fct |&gt; as.list(),<br>  &quot;claude&quot; = claude_dat$sentiment_fct |&gt; as.list(),<br>  levels = c(&quot;positive&quot;, &quot;neutral&quot;, &quot;negative&quot;)<br>)<br>average_similarity(check_fct)<br># exact<br># 0.833<br><br>check_num &lt;- same_number(<br>  &quot;openai&quot; = openai_dat$sentiment_num |&gt; as.list(),<br>  &quot;claude&quot; = claude_dat$sentiment_num |&gt; as.list()<br>)<br>average_similarity(check_num)<br>#      exact        raw        exp    percent normalized      fuzzy<br>#      0.417      0.056      0.948      0.883      0.930      0.950</pre><p>From this analysis, I learned that the two models have good agreement on the factor-based sentiment classification (83% exact match), excellent agreement on the numerical sentiment scores (93% normalized similarity), and moderate agreement on the text descriptions (64% cosine similarity). The lower agreement on text descriptions reflects the more subjective nature of choosing a single word to describe emotional tone.</p><h3>Conclusion</h3><p>Working with LLMs in R has evolved rapidly, and the combination of hellmer for efficient batching and samesies for response comparison creates a powerful toolkit for data scientists. These packages fill important gaps in the R ecosystem and enable more sophisticated LLM workflows.</p><p>The ability to validate LLM outputs against each other provides a level of quality assurance that’s essential for production systems. The example demonstrated that models tend to agree strongly on structured outputs, with more variation in free-text responses — a pattern that can guide how we design extraction prompts.</p><p>Looking ahead, these tools will become increasingly important as organizations integrate LLMs into their data pipelines. The R ecosystem continues to lead in providing pragmatic solutions for working with AI, allowing data scientists to leverage these technologies without sacrificing reliability or interpretability.</p><p>I hope that these packages can become part of the standard toolkit for anyone working with LLMs in R. The combination of efficient batching and systematic comparison provides the foundation for reliable, production-ready LLM implementations.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd9f347f2a11" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>