<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Andrew Docherty on Medium]]></title>
        <description><![CDATA[Stories by Andrew Docherty on Medium]]></description>
        <link>https://medium.com/@docherty?source=rss-dba307069819------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*xY4oj30Y_oZ8UWXJFb-ERQ.png</url>
            <title>Stories by Andrew Docherty on Medium</title>
            <link>https://medium.com/@docherty?source=rss-dba307069819------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 13:13:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@docherty/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Controlling your LLM: Deep dive into Constrained Generation]]></title>
            <link>https://medium.com/@docherty/controlling-your-llm-deep-dive-into-constrained-generation-1e561c736a20?source=rss-dba307069819------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e561c736a20</guid>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[structured-data]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Andrew Docherty]]></dc:creator>
            <pubDate>Tue, 06 May 2025 05:52:44 GMT</pubDate>
            <atom:updated>2025-05-08T23:17:58.817Z</atom:updated>
            <content:encoded><![CDATA[<h4>Mastering Structured Output in LLMs 5: How do the techniques to ensure LLMs give output in JSON or conform to a RegEx work?</h4><p>Constrained generation is a fundamental tool for AI practitioners seeking to ensure Large Language Models (LLMs) produce outputs conforming to specific structures or rules — whether that’s valid JSON, correctly formatted URLs, or custom data schemas. This technique underpins features like OpenAI’s <a href="https://openai.com/index/introducing-structured-outputs-in-the-api/">structured output API</a> strict mode, <a href="https://ollama.com/blog/structured-outputs">Ollama’s structured outputs</a>, LLama.cpp’s <a href="https://medium.com/better-programming/testing-out-llama-cpp-grammar-constraint-based-sampling-f154e48e6028">constraint-based sampling</a>, and JSON mode in OpenAI and <a href="https://docs.fireworks.ai/structured-responses/structured-response-formatting">other</a> <a href="https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/json-mode?tabs=python">cloud</a> <a href="https://docs.mistral.ai/capabilities/structured-output/json_mode/">providers</a>.</p><p>In this post, we’ll walk through how constrained generation in large language models (LLMs) works by manipulating the raw logits in the output layer of an LLM. We’ll build some examples using HuggingFace Transformers progressing to writing a class capable of constraining the output to an arbitrary regular expression.</p><h4>Other articles in the series</h4><ol><li><a href="https://medium.com/@docherty/mastering-structured-output-in-llms-choosing-the-right-model-for-json-output-with-langchain-be29fb6f6675">Choosing the Right Model for JSON Output with LangChain</a></li><li><a href="https://medium.com/@docherty/mastering-structured-output-in-llms-revisiting-langchain-and-json-structured-outputs-d95dfc286045">Revisiting LangChain and JSON structured outputs</a></li><li><a href="https://medium.com/@docherty/mastering-structured-output-in-llms-3-langchain-and-xml-8bad9e1f43ef?source=your_stories_page--------------------------------------------">Structured Output in XML using LangChain</a></li><li><a href="https://medium.com/@docherty/python-libraries-for-llm-structured-outputs-beyond-langchain-621225e48399?source=your_stories_page--------------------------------------------">Python libraries for LLM structured outputs beyond LangChain</a></li></ol><h3>Takeaways</h3><ul><li>The core mechanism for influencing what an LLM generates lies in the underlying numerical probabilities (logits) associated with each possible token. By understanding logits and how to manipulate them, you can gain a level of direct control over LLM behavior.</li><li>The fundamental technique of constrained generation boils down to “masking”, or reducing the logits of, unwanted tokens.</li><li>Constrained generation is conceptually straightforward — we implement a class to apply RegEx constraints in under 40 lines of Python!</li><li>Try constrained generation yourself in the Colab notebook: <a href="https://colab.research.google.com/drive/1YH4i86uiqMbftBPejO_HEEuVycUbgmrA?usp=sharing">https://colab.research.google.com/drive/1YH4i86uiqMbftBPejO_HEEuVycUbgmrA</a></li></ul><h3>How Language Models Generate Text</h3><p>A typical language model generates output by choosing one item from it’s vocabulary iteratively, producing an output sequence. This is technically termed a causal language model which outputs each symbol given past symbols and without knowledge of future symbols, and includes the majority of LLMs currently in use. If you’d like to read more try “<a href="https://jalammar.github.io/illustrated-transformer/">The Illustrated Transformer</a>”, a great intro to transformers.</p><p>For example, consider a simplified language model with a vocabulary of the digits 0–9. In unconstrained mode, the model can emit any digit at every step of the generation, in the figure below the model outputs the digits “12379”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*ZoisMKJDe68kF10IOqBLuw.png" /><figcaption>Unconstrained output of a sequence of numbers from a language model.</figcaption></figure><p>Constrained generation, however, restricts this freedom, allowing only a subset of tokens that satisfy specific rules. This approach is commonly used when the model output needs to follow a specific format, such as JSON or XML, but arbitrary rules can be implemented to constrain the output.</p><h3>Constrained generation starts with rules</h3><p>Constrained generation begins with rules (the constraints) defining valid and invalid outputs. These rules must be evaluable on incomplete sequences — essentially, a rule is a boolean function that returns True for a valid sequence and False otherwise.</p><p>For example, a rule might <strong>allow odd numbers only at the start of a sequence or after an even number</strong>. For this rule, valid sequences include:</p><pre>22222<br>1234567<br>24142167</pre><p>And invalid sequences include:</p><pre>1235<br>11<br>148233</pre><p>During generation, this rule is used to mask tokens that would lead to an invalid sequence. For instance, given the sequence “124”, only even numbers can follow to maintain validity. Simply speaking, at each step constrained generation applies the rules to determine which next items in the vocabulary (tokens) maintain a valid output and which do not. Then, the model is prevented from outputting the vocabulary items that do not lead to a valid output.</p><p>For example, for the sequence above let’s go through at each step:</p><ol><li>At step 1 all digits are valid by the rule.</li><li>After outputting a 1, only even digits are now valid.</li><li>We output an even number at the last step, so all digits are valid.</li><li>After outputting an off number at the last step, only even digits are now valid.</li></ol><p>This masking is visualised in the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*-HEhn0PNGAFgmtsHLKh94g.png" /><figcaption>Constrained output of a sequence of numbers from a language model. The constraint rule is “allow odd numbers only at the start of a sequence or after an even number”.</figcaption></figure><h3>Understanding model logits</h3><p>Constrained generation techniques operate directly on the logits — the raw, unnormalized log-likelihoods output by the language model at each step, representing the model’s prediction for the next token.</p><p>The logits are the log-probabilities of the estimated distribution over vocabulary items. Writing the logit for vocab item vocab[<em>i</em>] as <em>Li</em>, the model probability of the token is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/312/1*uemp2DqRJ25VKHS-rfJwYA.png" /></figure><p>Greedy sampling selects the token with the highest likelihood, while multinomial sampling samples tokens proportionally to their probabilities. A temperature parameter <em>t</em> modulates this sampling process, transitioning between standard multinomial sampling (<em>t</em>=1) and greedy sampling as <em>t</em> approaches 0. If you want to learn more about how sampling logits gives LLM outputs, I recommend this <a href="https://huggingface.co/blog/how-to-generate">HuggingFace’s blog post</a>.</p><p>As an example, consider a model with a vocabulary of digits 0–9. The model’s output for a given input is a probability distribution over these digits. For example, a logit distribution at a given step might look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*TD_U7aKvpfv6LQaFblUVKg.png" /><figcaption>Example Logit distribution for the digits 0–9.</figcaption></figure><p>With greedy sampling, ‘7’ would be selected as it has the highest logit value.</p><p>To constrain token generation, we modify these logits, reducing the probability of unwanted tokens to zero. This is achieved by setting their logits to a large negative value, typically -∞, as exp(-∞️) = 0.</p><p>For example, referring to the example rule above, suppose the sequence is “123” and we want to ensure only even numbers follow. We can set the logits for odd numbers (1, 3, 5, 7, 9) to a large negative value. The resulting logits would appear as in the following figure. After this masking operation the token “4” is now the highest, and only the even-numbers have non-zero probabilities of being output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*U9XGQvWLs-C-jTjG7SJxJA.png" /><figcaption>Example masked Logit distribution for the digits 0–9.</figcaption></figure><h3>Practical demo with HuggingFace Transformers</h3><p>Let’s implement this process practically using HuggingFace Transformers. To get a primer on how to use this library <a href="https://huggingface.co/transformers/">check out the documentation</a>.</p><p>Run all this code in the COLAB notebook here:<br><a href="https://colab.research.google.com/drive/1YH4i86uiqMbftBPejO_HEEuVycUbgmrA?usp=sharing">https://colab.research.google.com/drive/1YH4i86uiqMbftBPejO_HEEuVycUbgmrA</a></p><p>First, let’s use the <a href="https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct">HuggingFace SmolLM2 model</a>, which has only 360M parameters and can be run directly on a Colab free instance (CPU or GPU).</p><pre>from transformers import pipeline<br>from transformers import AutoTokenizer, AutoModelForCausalLM<br><br>model_name=&quot;HuggingFaceTB/SmolLM2-360M-Instruct&quot;<br>model = AutoModelForCausalLM.from_pretrained(model_name, device_map=&quot;auto&quot;)<br>tokenizer = AutoTokenizer.from_pretrained(model_name)</pre><p>The tokenizer is a vital part of the system to understand when considering structured generation, as often our rules that we want (e.g. a regex) operate at the level of the text whereas the LLM operates on tokens. When we have discussed the vocabulary above, this is controlled by the tokenizer, which is responsible for converting between text and vocabulary items, or tokens.</p><p>For example, let’s get the tokenizer to convert the following text to tokens. It outputs a list of integers which are the token IDs. These are the numbers the LLM works with directly.</p><pre>model_inputs = tokenizer.encode(&quot;A list of colors: red, blue&quot;)<br>model inputs.token_ids<br><br>&gt;&gt; [  49, 1398,  282, 4683,   42, 2382,   28, 4461]</pre><p>We can see the string representation of each token with the .decode method of the tokenizer. Notice that the token string often contains a word prefixed with a space, rather than the space being a separate token. This can become very important for constrained generation, as we will touch on below.</p><pre>[tokenizer.decode(token_id) for token_id in model_inputs]<br><br>&gt;&gt; [&#39;A&#39;, &#39; list&#39;, &#39; of&#39;, &#39; colors&#39;, &#39;:&#39;, &#39; red&#39;, &#39;,&#39;, &#39; blue&#39;]</pre><h4>Logits and their value</h4><p>Now we have a model we can use it to query the model to continue the prompt. We can easily ask Transforms to return raw logits at each step in the generation by suppling the options output_logits=True and return_dict_in_generate=True; note you need both options.</p><pre>prompt = [&quot;A sequence of numbers: 2, 4, 8, 14&quot;]<br>model_inputs = tokenizer(prompt, return_tensors=&quot;pt&quot;).to(&quot;cuda&quot;)<br>model_output = model.generate(**model_inputs, max_new_tokens=20, do_sample=False, output_logits=True, return_dict_in_generate=True)</pre><p>The model_output now contains the output sequence in sequences and logits contains the length $L$ tuple of $B\times N$ logits each, where L is the output length and N is the vocab size and B the batch dimension, which will be 1 for all experiments here.</p><p>The output can be reconstructed from the token IDs using tokenizer functions, note here the output is truncated artificially as we use max_new_tokens=20.</p><pre>tokenizer.decode(model_output.sequences[0])<br><br>&gt;&gt; Continue the sequence of numbers: 2, 4, 8, 16, 32, 64, 128, 2</pre><p>We can get the same from a greedy decoding of the outputs by converting the logits to a numpy array. This gives output_logits as a L⨉N array where L is the output length and N is the vocab size.</p><pre>output_len = len(model_output.logits)<br>output_logits = np.array([model_output.logits[k][0].cpu() for k in range(output_len)])</pre><p>Let’s check that we can reconstruct the outputs sequence from the logits (note the logits only contain the continuation characters, whereas model_outputs.sequences contains all tokens including the prompt). For greedy sampling, the output sequence are the token IDs of the maximum token for each step in the decoding process.</p><pre>output_token_ids = output_logits.argmax(axis=1)<br>np.allclose(output_token_ids, model_output.sequences[0, -output_len:])<br><br>&gt;&gt; True</pre><h4>Visualising the logits</h4><p>So the code above generated the sequence , 16, 32, 64, 128, 2 to continue our prompt, the model correctly guessing that these were powers of two.</p><p>Let’s have a look at the raw logits themselves for this sequence. Below we plot the value of the logit for the highest scoring tokens. The verticle axis represents the tokens the model is generating at each step, with the string representation of the token on the left axis. The horizontal axis are the generation steps for the 20 tokens we requested. On the top we show the step (index) and the string representation of the token.</p><p>The numbers in each square represent the logit value, and the colours match them going from the lowest logit value in blue to the highest in orange. The maximum logit in each column is highlighted in white text, and this matches the (greedy) generated token at each step.</p><p>Note that on the vertical axis we include the top 15 tokens (over the entire sequence) as well as all tokens contained in the output sequence from the LLM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bixly86k-qrcOrUyi0Fi_w.png" /></figure><p>Note that the tokenizer used by SmolLM2 uses single tokens for every numeric value. This makes numeric processing very straightforward as we don’t need to consider tokens that contain multiple digits, such as ‘32’.</p><p>What can we see in this diagram? Firstly, we see the difference between the highest logit value in the columns and the other high logits differs between the first number in each pair and the second number. This means the model is very “sure” of the first number in each sequence and less sure of the second number.</p><h4>Masking the logits with LogitsProcessor</h4><blockquote>I based some of the code below on one of the original Python libraries created to perform constrained generation using RegEx, <a href="https://github.com/r2d4/rellm">https://github.com/r2d4/rellm</a>.</blockquote><p>Check it out as it contains straightforward code and is great to learn more about how constrained generation with RegEx works.</p><p>Logits are not only there for us to look at after the fact, transformers allows us to modify them during duration to guide the model as it produces the output sequence. The transformers class to do this is a LogitsProcessor . Consider the code below, it subtracts 10 from the logits of all tokens that are not in the token IDs we specify.</p><pre>from transformers import LogitsProcessor<br><br>class LogitsMask(LogitsProcessor):<br>    def __init__(self, allowed_token_ids):<br>        self.allowed_token_ids = set(allowed_token_ids)<br>    def __call__(self, input_ids, scores):<br>        mask = torch.ones_like(scores) * -10<br>        for token_id in self.allowed_token_ids:<br>            mask[:, token_id] = 0<br>        scores = scores + mask<br>        return scores</pre><p>Let’s consider the output above and say that we don’t want the model to output commas, only spaces. We can do this with the following code, restricting the allowed symbols to only numbers and a space. We also include the end-of-sequence special token, because it’s good to get used to letting the model end when it want to.</p><pre>allowed_tokens = list(&quot;0123456789 &quot;)<br>allowed_token_ids = [tokenizer.encode(t)[0] for t in allowed_tokens] + [tokenizer.eos_token_id]<br>lm_test = LogitsMask(allowed_token_ids)</pre><p>And we use the logits processor by specifying it in the arguments to generate as follows:</p><pre>model_output = model.generate(<br>	**model_inputs,<br>	max_new_tokens=20,<br>	logits_processor=[lm_test]<br>)</pre><p>The model now outputs &#39; 16 32 64 128 256 51&#39; with no commas as we expected. Looking at the logit values we can see that all symbols apart from those we’ve allowed are suppressed (more blue than before).</p><p>In a real application you’d likely want to subtract more than 10, in fact subtracting torch.inf drives the probability to zero. However, here we subtract a small number so that we can visualize the raw logits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EUq-eS3wreZLsPdVCwEErw.png" /></figure><h3>Constraining output to a RegEx</h3><p>For the previous example we constrained the model to only output tokens that we have chosen at all steps. For many structured generation tasks the tokens that are valid for the model to output will change at each step. We could for every structured output pattern code a new LogitsProcessor that changes the allowed_token_ids at each step according to custom code. A better way to do this is to build a general LogitsProcessor that takes a pattern such as a RegEx and generates valid tokens that match this pattern at each step.</p><p>We want to test at generation step if the generated string up to now appended by each token string in the vocabulary would be valid. This requires a <em>partial match</em>, where the regex processor returns valid if the supplied string is consistent with the RegEx pattern so far. This is provided in the python regex library (but not the inbuilt re library).</p><p>For example, let’s consider a simple regex that matches the base URL of a web address, and enforces the “http” prefix. This simplicity is intentional for our examples, if you want a RegEx that actually matches general valid URLs see <a href="https://stackoverflow.com/questions/3809401/what-is-a-good-regular-expression-to-match-a-url#3809435">this thread</a> for a good discussion of the options.</p><pre>import regex<br>regex_http = regex.compile(r&quot; ?https?:\/\/(www\.)?\w{1,256}\.\w{1,6}/?$&quot;)</pre><p>This regex will not match an incomplete address by default:</p><pre>regex_http.match(&quot;&lt;http://test&gt;.&quot;)<br>&gt;&gt; None</pre><p>However, with partial=True it will.</p><pre>regex_http.match(&quot;&lt;http://test&gt;.&quot;, partial=True)<br>&gt;&gt; &lt;regex.Match object; span=(0, 12), match=&#39;&lt;http://test&gt;.&#39;, partial=True&gt;</pre><p>This is just what we need to implement our new LogitsProcessor that handles an arbitrary RegEx, as shown below.</p><pre>from transformers import LogitsProcessor<br><br>class RegexMask(LogitsProcessor):<br>    def __init__(self, regex, tokenizer):<br>        self.regex = regex<br>        self.tokenizer = tokenizer<br>        self.vocab_map = {token_id: tokenizer.decode(token_id) for _, token_id in tokenizer.get_vocab().items()}<br>        self.start_index = 0<br><br>    def __call__(self, input_ids, scores):<br>        partial_output_str = self.tokenizer.decode(<br>          input_ids[0, self.start_index:]<br>      )<br>        # Always allow the model to finish (even if not valid)<br>        allowed_token_ids = [self.tokenizer.eos_token_id]<br><br>        # Search through all tokens<br>        for token_id, token_str in self.vocab_map.items():<br>            if self.regex.match(partial_output_str + token_str, partial=True):<br>                allowed_token_ids.append(token_id)<br><br>        # Mask scores for tokens that don&#39;t match regex<br>        mask = -self.subtract_on_mask * torch.ones_like(scores)<br>        for token_id in allowed_token_ids:<br>            mask[:, token_id] = 0<br>        scores = scores + mask<br>        return scores</pre><p>Some notes on the code:</p><ol><li>As the input_ids contain the prompt as well as the output, if we want the regex to only match the output we need to set the start_index to the end of the prompt;</li><li>As we always include the end-of-sequence (EOS) token, the generation can stop before the output matches the RegEx;</li><li>The code queries the regex for every possible token for each step of the generation, and as there are commonly tens of thousands of tokens this makes the code very inefficient! There are multiple techniques that could be used to improve this see <a href="http://arxiv.org/abs/2307.09702">these</a> <a href="http://arxiv.org/abs/2411.15100">papers</a> for details.</li><li>Note that the regex is done over the token strings, which will often have a space at the start. This needs to be accounted for in the regex.</li></ol><h3>An example: outputting a valid URL</h3><p>First let’s obtain a continuation of the prompt &quot;One of the most common web addresses is:&quot; without constraining the generation. We get a sensible answer, although one without the http prefix, an unnecessary new line, and extra text after describing it:</p><pre>One of the most common web addresses is:<br>www.google.com<br>This is a well-known and</pre><p>Now let’s try the constrained generation using the class above, using the RegEx&quot; ?https?:\/\/(www\.)?\w{1,64}\.\w{1,6}/?&quot;</p><pre>One of the most common web addresses is: http://www.google.com</pre><p>Great! The model output the web address as we wanted it.</p><p>It’s important to note there is an optional space at the start of the regex above. This is needed to match the tokens as they are typically prefixed by spaces, for example the token for http is &quot; http&quot; not &quot;http&quot;.</p><p>The logits map shows some interesting things, first after masking the top candidates were &quot; &quot;, &quot; http&quot;, and &quot; https&quot; . In this case the model chose a space then the &quot;http&quot;token. However, it is good practice to not mask the token&quot; http&quot;.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*npwhxVxIDf3NSL3AVlfs6Q.png" /></figure><p>Take a look at the colab notebook to see some more examples of constrained generation and token boundaries.</p><p>With constrained generation we are not restricted to greedy sampling, we can use multinomial sampling of the masked logits to generate different outputs. For example, for the same prompt and constraints above we can sample repeatedly to get multiple valid websites:</p><pre>One of the most common web addresses is: http://www.example.com/<br>One of the most common web addresses is: http://www.google.com<br>One of the most common web addresses is: http://www.somewebsite.com/path/to/file.php?fileName=fileName.txt<br>One of the most common web addresses is: http://www.google.com/search<br>One of the most common web addresses is: http://www.google.com/.<br>One of the most common web addresses is: http://www.google.com/search?hl=en<br>One of the most common web addresses is: https://www.google.com/.<br>One of the most common web addresses is: http://localhost.com<br>One of the most common web addresses is: https://www.google.com/maps/search/.<br>One of the most common web addresses is: http://www.google.com</pre><h3>Conclusions</h3><p>Whether it is ensuring JSON output, correctly formatted URLs, or custom structured outputs for various domains, constrained generation is a fundamental tool in the toolbox for AI practitioners.</p><p>In this article, we’ve looked at the core principles of constrained generation; from how logits are used to generate outputs in language models to how constrained generation is implemented by masking invalid tokens. Finally we developed a practical implementation using Hugging Face Transformers that allows us to constrain outputs to an arbitrary RegEx.</p><p>Hopefully this helps you understand how structured output works and avoid some common pitfalls when applying these techniques. In particular, the interplay between tokenization and constraints demands careful consideration, as the LLM operates on tokens, not raw text.</p><p>The code in this blog is purely to understand how constrained generation works. The code is inefficient and cannot guarantee that the generated output will actually give the output you want. For implementations to use in your code look at <a href="https://github.com/dottxt-ai/outlines">Outlines</a>, <a href="https://github.com/guidance-ai/guidance">Guidance</a> or <a href="https://github.com/mlc-ai/xgrammar">XGrammar</a>.</p><h3>References</h3><p>OpenAI’s structured output API<br><a href="https://openai.com/index/introducing-structured-outputs-in-the-api/">https://openai.com/index/introducing-structured-outputs-in-the-api/</a></p><p>Ollama’s structured output support<br><a href="https://ollama.com/blog/structured-outputs">https://ollama.com/blog/structured-outputs</a></p><p>Blog article on Lamma.cpp constraint based sampling<br><a href="https://medium.com/better-programming/testing-out-llama-cpp-grammar-constraint-based-sampling-f154e48e6028">https://medium.com/better-programming/testing-out-llama-cpp-grammar-constraint-based-sampling-f154e48e6028</a></p><p>A great introduction to how a transformer language model works<br><a href="https://jalammar.github.io/illustrated-transformer/">https://jalammar.github.io/illustrated-transformer/</a></p><p>Hugging Face Transformers Overview<br><a href="https://huggingface.co/transformers/">https://huggingface.co/transformers/</a></p><p>Sampling from language models on HuggingFace blog<br><a href="https://huggingface.co/blog/how-to-generate">https://huggingface.co/blog/how-to-generate</a></p><p>Understanding Logits, Sigmoid, Softmax, and Cross-Entropy Loss in Deep Learning<br><a href="https://wandb.ai/amanarora/Written-Reports/reports/Understanding-Logits-Sigmoid-Softmax-and-Cross-Entropy-Loss-in-Deep-Learning--Vmlldzo0NDMzNTU3">https://wandb.ai/amanarora/Written-Reports/reports/Understanding-Logits-Sigmoid-Softmax-and-Cross-Entropy-Loss-in-Deep-Learning--Vmlldzo0NDMzNTU3</a></p><p>One of the first RegEx constrained generation libraries, ReLLM<br><a href="https://github.com/r2d4/rellm">https://github.com/r2d4/rellm</a></p><p>Logits processors Documentation<br><a href="https://huggingface.co/transformers/main_classes/logits_processors.html">https://huggingface.co/transformers/main_classes/logits_processors.html</a></p><p>Deeper dive into how to implement efficient constrained generation with a FSM<br><a href="https://blog.dottxt.co/coalescence.html">https://blog.dottxt.co/coalescence.html</a></p><p>NVIDIA has a range of logit processors that manipulate the sequence as it’s being generated<br><a href="https://github.com/NVIDIA/logits-processor-zoo">https://github.com/NVIDIA/logits-processor-zoo</a></p><p>Outlines’ FSM-based method for efficient constrained masking<br><a href="http://arxiv.org/abs/2307.09702">http://arxiv.org/abs/2307.09702</a></p><p>XGrammar’s efficient binary masking approach for constrained masking<br><a href="http://arxiv.org/abs/2411.15100">http://arxiv.org/abs/2411.15100</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e561c736a20" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python libraries for LLM structured outputs beyond LangChain]]></title>
            <link>https://medium.com/@docherty/python-libraries-for-llm-structured-outputs-beyond-langchain-621225e48399?source=rss-dba307069819------2</link>
            <guid isPermaLink="false">https://medium.com/p/621225e48399</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[structured-data]]></category>
            <dc:creator><![CDATA[Andrew Docherty]]></dc:creator>
            <pubDate>Tue, 29 Apr 2025 01:43:03 GMT</pubDate>
            <atom:updated>2025-04-29T01:43:03.340Z</atom:updated>
            <content:encoded><![CDATA[<h4>Mastering Structured Outputs in LLMs 4: Many libraries offer to structure the output from LLMs. What are they and how do they differ?</h4><blockquote>“This state of wondering paralysis cries out for criticism, which promises to sort through the glut, to assist in the formation of choices, to act as gatekeeper to our besieged sensoria.” — A. O. Scott , <a href="https://www.goodreads.com/book/show/25614595-better-living-through-criticism">Better Living Through Criticism</a></blockquote><p>In previous blog posts I’ve looked at how to get output structured in both <a href="https://medium.com/@docherty/mastering-structured-output-in-llms-choosing-the-right-model-for-json-output-with-langchain-be29fb6f6675">JSON</a> and <a href="https://medium.com/@docherty/mastering-structured-output-in-llms-3-langchain-and-xml-8bad9e1f43ef">XML</a> using LangChain. In this blog post I take a look at the options for Python libraries beyond LangChain that offer different approached to structure output from LLMs. I use Google’s NotebookLM to compare multiple options including Guidance, Outlines, DSPy, LMQL, Instructor, and more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lt1jbSxvj-TbGY4BWXdebQ.jpeg" /><figcaption>Image created by <a href="https://gemini.google.com/">Google Gemini</a></figcaption></figure><p>Other articles in the series:</p><ol><li><a href="https://medium.com/@docherty/mastering-structured-output-in-llms-choosing-the-right-model-for-json-output-with-langchain-be29fb6f6675">Choosing the Right Model for JSON Output with LangChain</a></li><li><a href="https://medium.com/@docherty/mastering-structured-output-in-llms-revisiting-langchain-and-json-structured-outputs-d95dfc286045">Revisiting LangChain and JSON structured outputs</a></li><li><a href="https://medium.com/@docherty/mastering-structured-output-in-llms-3-langchain-and-xml-8bad9e1f43ef">LangChain and XML</a></li></ol><h3>Takeaways</h3><ul><li>There are a lot of libraries that support structured outputs in different capacities, some like LangChain and Instructor focus on specifying structure with Pydantic and use JSON under the hood; others like Outlines and Guidance give you tools to define your own arbitrary structure.</li><li>The Python packages that I looked at can be classified into three broad types, servers that run models and offer an inference API, python libraries that run models directly and provide low-level constrained generation, and packages that harness existing inference APIs.</li><li>Comparing many different options using Google’s NotebookLM gave a table that was good enough for me to categorise packages and choose one that will likely fill my use case with minimal effort. However, it’s important to remember that some details may not be correct.</li></ul><p>See below for an embedded table summarizing the options. Are the any I’ve missed, let me know! Read the rest of the article for the methodology I went through to create this table, and an explanation of the different categories.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fairtable.com%2Fembed%2FappHoRywVjyPA5B1R%2Fshr1nBp1vvgDe2TKS%2FtblDGgNdrnptptphj&amp;display_name=Airtable&amp;url=https%3A%2F%2Fairtable.com%2FappHoRywVjyPA5B1R%2Fshr1nBp1vvgDe2TKS%2FtblDGgNdrnptptphj&amp;image=https%3A%2F%2Fstatic.airtable.com%2Fimages%2Foembed%2Fairtable_brand.png&amp;type=text%2Fhtml&amp;schema=airtable" width="800" height="533" frameborder="0" scrolling="no"><a href="https://medium.com/media/661c77d54346c3a58a4d4a5d6901584a/href">https://medium.com/media/661c77d54346c3a58a4d4a5d6901584a/href</a></iframe><h3>Methodology</h3><p>I find it a lot of work to figure out what different libraries actually offer, and their differences from one another. To make this process easier I’ve (of course) turned to AI to help me. <a href="https://notebooklm.google.com/">Google’s NotebookLM</a> offers an easy solution to add multiple sources and ask questions from the collated data (and a cool podcast functionality!)</p><p>To make the table above I went through the following process. First, I manually identified a number of Python libraries that offer structured outputs in some form. Then, I performed the following steps for each Python library</p><ol><li>Add the GitHub home page to NotebookLM’s sources</li><li>Add the documentation page on structured outputs, or the closest to this, to the sources</li></ol><p>Next, I prompted NotebookLM to summarize each library, given the documents above, and create a comparison table, using the following prompt:</p><pre>You are tasked with evaluating Python libraries that enable structured output<br>from LLMs. You are given information about each library and it&#39;s features.<br><br>Compare the libraries and produce a table that lists the options in rows<br>and the features in columns, include the following features:<br><br>* Number of GitHub stars<br>* Main purpose of library<br>* Is there custom constrained generation or does it use the function calling API?<br>* List ALL ways that the constraint generation rules can be constructed (regex, EBNF etc)?<br>* Does it support custom models (bring your own pretrained models)?<br>* Other significant features<br><br>Display your table in Markdown with the libraries as rows and the features<br>as columns of the table.</pre><p>Finally, I copied the resulting table and manually spot checked the information in the table, and edited it for brevity. I also adding some Python libraries I found after the fact manually. See if you can spot the rows that I added manually and those created by NotebookLLM!</p><p>This was an experiment in how to go through the many structured output options, understand what they actually provided, and make a decision on which ones are useful for my needs. I found that NotebookLM gave a great start to the search and made it easy to create these comparison tables with little effort.</p><p>An important note is that I explicitly approached this process as creating a table that’s good enough to make a decision on which library to try rather than expecting that all information in the table is 100% correct. Therefore, there may be errors in the table that I haven’t spotted, if you see any let me know!</p><h3>Categorisation of Libraries</h3><p>In the table there are <em>three different approaches to structured outputs</em> that I have categorised the packages (excluding DSPy). These are specified in the type column and reinforced by the colours.</p><h4>Servers</h4><ul><li>Servers that run an LLM inference engine and are interacted with through an API (typically OpenAI API compatible) and often use another package to structure outputs under the hood.</li><li>Typically they restrict structured outputs to JSON, although there are exceptions.</li><li>These packages are essentially offering a roll-your-own replacement for an LLM cloud provider.</li></ul><h4>Low-level constraints</h4><ul><li>These packages implement constrained generation at the level of the model token logits. This guarantees the LLM output will be in a specific, and custom format (Classification, Markdown, XML, (almost) anything we want).</li><li>These typically run models directly using HuggingFace transformers or other LLM inference engines (as opposed to accessing them through an API).</li><li>They are not able to use hosted inference services such as OpenAI API or Anthropic API to run inference, as these APIs don’t give enough low-level control.</li></ul><h4>High-level wrapper for LLM APIs</h4><ul><li>Packages that structure outputs at a high level using Pydantic (or similar) and use LLM API calls to structure them as JSON formats under the hood.</li><li>The general approach for these packages is to check if the LLM output conforms to a specific test, and if not to retry the call with guidance on how to conform.</li><li>These typically interact with models through an API (compatible with cloud services such as OpenAI).</li></ul><p>I’ve organised DSPy in the group of high-level wrapper for LLM APIs, although it takes a unique approach and positions itself as way to program, rather than prompt, language models. I am curious to learn more; however, for now I’m going to focus on libraries with low-level constraints.</p><p>In the next blog article I’ll dive into libraries for low-level constraints, investigate how they work, and how to use them to get only the outputs we want — with no surprises.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=621225e48399" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Structured Output in XML using LangChain]]></title>
            <link>https://medium.com/@docherty/mastering-structured-output-in-llms-3-langchain-and-xml-8bad9e1f43ef?source=rss-dba307069819------2</link>
            <guid isPermaLink="false">https://medium.com/p/8bad9e1f43ef</guid>
            <category><![CDATA[llm-evaluation]]></category>
            <category><![CDATA[structured-data]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Andrew Docherty]]></dc:creator>
            <pubDate>Thu, 20 Mar 2025 02:00:38 GMT</pubDate>
            <atom:updated>2025-04-28T06:31:47.291Z</atom:updated>
            <content:encoded><![CDATA[<h4>Mastering Structured Output 3: Structured output for LLM doesn’t only mean JSON, this post shows how to get consistent XML output and the best models to use.</h4><blockquote>“Everyone who tells a story tells it differently, just to remind us that everybody sees it differently.”<br>― <strong>Jeanette Winterson, </strong><a href="https://www.goodreads.com/work/quotes/1411520"><strong>Oranges Are Not the Only Fruit</strong></a></blockquote><p>Structured output for LLM doesn’t only mean JSON. In this post I take you through how to create prompts to get consistent XML output and the best models to use.</p><p>Structured output in LLMs instructs the returned answer to be in a defined and specific format. This is great to integrate the output in code that expects specific data to be returned. Using JSON to structure outputs has become the preferred format, with OpenAI and other LLM providers having specialised functionality to ensure the outputs of the LLM are in JSON format.</p><p>All articles in the series</p><ol><li><a href="https://medium.com/@docherty/mastering-structured-output-in-llms-choosing-the-right-model-for-json-output-with-langchain-be29fb6f6675">Choosing the Right Model for JSON Output with LangChain</a></li><li><a href="https://medium.com/@docherty/mastering-structured-output-in-llms-revisiting-langchain-and-json-structured-outputs-d95dfc286045">Revisiting LangChain and JSON structured outputs</a></li><li>LangChain and XML [this article]</li></ol><h3>Takeaways</h3><ul><li>Large and newer LLMs reliably output XML formats conforming to a given schema; however, <em>unlike JSON there are no guarantees that they will output valid XML or a valid schema</em>.</li><li>The best LLMs tested were <em>Anthropic Sonnet 3.5 and Haiku 3.5, Meta Llama 3.3, Microsoft Phi4, and Alibaba Qwen 2.5. </em>They all performed excellently at handling XML outputs. Phi 4 is the smallest model tested that gives almost 100% conformance to the XML schema (note Gemma 3 wasn’t tested).</li><li><em>Currently, using LangChain output parsers to give XML output works poorly, </em>in particular the prompting instructions given to the LLM do not specify the schema explicitly. I provide an alternative method with the goodness of Pydantic XML and some custom code that performs better especially with complex schemas.</li><li>If you are struggling with obtaining conformant XML output for a LLM: <em>choose a different LLM with good XML performance </em>or if changing is not possible <em>give the LLM a clear prompt (Approach 2 below) in the user prompt, and use a temperature of zero</em>.</li></ul><p>For each of three providers, one commercial provider (Anthropic), the locally hosted Ollama, and Cloud-based Fireworks I’ve tested the support for XML structured outputs. Four levels of structure complexity are tested with the <em>results below showing the percentage of LLM responses that are valid XML and conform to the requested schema</em>.</p><p>See all code used to generate the results in this article in <a href="https://github.com/adocherty/mastering-structured-output/tree/publish">this GitHub repo</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/408/1*Eb9_-jxvWdoBwGOS_AZ8TA.png" /><figcaption>The overall proportion of outputs that conform strictly to the requested XML schema. These results insert XML formatting instructions in the system prompt, the performance is improved with user prompting see below. These results are aggregated over multiple prompts, temperatures, and levels of schema complexity as described in the article.</figcaption></figure><h3>JSON is not the only format</h3><p>JSON is not the only format used to get structured outputs from LLMs — and sometimes it may not be the best choice depending on your application. Some evidence that JSON may not always be best:</p><ul><li>Anthropic has <a href="https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/use-xml-tags">recommended using XML to structure prompts</a> and to <a href="https://docs.anthropic.com/en/docs/test-and-evaluate/strengthen-guardrails/increase-consistency">increase consistency in outputs</a> (along with JSON).</li><li><a href="https://david-gilbertson.medium.com/llm-output-formats-why-json-costs-more-than-tsv-ebaf590bd541">JSON takes a lot of tokens</a>, other formats use less, although XML is not specifically compared to.</li><li>Aider has shown that <a href="https://aider.chat/2024/08/14/code-in-json.html">LLMs are not good at returning code in JSON</a></li><li>Constraining the LLM to JSON (or other constraints) can impact the reasoning abilities of models and stricter format constraints give worse performance. See: <a href="https://arxiv.org/abs/2408.02442">Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models”</a></li></ul><p>LLMs are trained on a lot of data in different formats. A hypothesis I have is the kind of data that are most represented in each format in the training data will bias the LLM and influence its output in that format. Therefore, which format to use for a particular application could depend on the (unknown) distributions of data represented in different formats.</p><p>We can make an educated guess at the likely data represented in different formats based on what we know the formats are commonly used for. For example:</p><ul><li><strong>XML</strong> — Favoured by Anthropic for structuring prompts and a common format for structuring text and web data;</li><li><strong>CSV, TSV</strong> — The simplest and most portable way to share and represent tabular data such as spreadsheets;</li><li><strong>TOML</strong> — Simple structure often used for parameters;</li><li><strong>Code</strong> — Code is a structured data format!;</li><li><strong>Markdown</strong> — The web’s default rich text format, and often output (even when not asked) by LLMs. Is Markdown structured?</li><li><strong>Diff</strong> — good for code changes, <a href="https://aider.chat/2024/08/14/code-in-json.html">as demonstrated by Aider</a></li></ul><p><em>In this article I’ll take a look at XML</em>. My hypothesis is that using an XML format for larger blocks of text such as those for an article will perform better than JSON. How can we define better? More to come in a later blog!</p><h3>How to get XML output using LangChain</h3><figure><a href="https://mybinder.org/v2/gh/adocherty/mastering-structured-output/publish?urlpath=%2Fdoc%2Ftree%2F5-langchain-xml-output-examples.ipynb"><img alt="" src="https://cdn-images-1.medium.com/max/109/1*Ueo_jqDkWcnQDutYZMPW1g.png" /></a><figcaption><em>Follow through with full code on Binder by clicking above</em></figcaption></figure><p>To output XML in LangChain there area two main approaches, firstly <em>LangChain’s inbuilt approach using output parsers</em>. This approach has two important limitations:</p><ul><li>As the format instructions are not given a schema, they do not prompt the LLM for a particular structure and only provide a list of tags that the LLM should use.</li><li>The XML parsing of the output doesn’t check the schema to ensure that it conforms. Therefore custom code to ensure the output contains the expected fields is required.</li></ul><p>As a second approach I <em>propose a custom workflow that uses Pydantic XML to defined the structure and parse the XML</em>. This let’s us define the structure programattically as a Pydantic XML object and ensures the returned XML conforms to the schema. Currently this requires custom code to provide XML format instructions that give the LLM clear directions and an example of the output schema.</p><p>I detail both these approaches below using the following example prompt directing the LLM to provide the answer to the user question in the format defined by format_instructions :</p><pre>prompt_system_format = ChatPromptTemplate.from_messages(<br>    [<br>        (<br>            &quot;system&quot;,<br>            &quot;Write an article in the provided format that answers the user &quot;<br>            &quot;question.\\n{format_instructions}&quot;,<br>        ),<br>        (&quot;human&quot;,  &quot;...&quot;)<br>    ]<br>)</pre><h3>Approach 1: Output Parsers</h3><p>LangChain uses the concept of output parsers<strong> </strong>to both parse the output from the LLM and also provide instructions to the LLM for the required format. To create an output parser for XML we <a href="https://python.langchain.com/api_reference/core/output_parsers/langchain_core.output_parsers.xml.XMLOutputParser.html">use the class </a><a href="https://python.langchain.com/api_reference/core/output_parsers/langchain_core.output_parsers.xml.XMLOutputParser.html">XMLOutputParser</a> and provide it with both the name of the root XML tag as well as the tags to be used. Note that this doesn’t give much ability to define the structure of complex schemas, and the <em>LLM is required to infer the hierarchy and order of these tags</em>.</p><pre>xml_parser = XMLOutputParser(<br>    name=&quot;article&quot;, tags=[&quot;article&quot;, &quot;title&quot;, &quot;problem&quot;, &quot;answer&quot;]<br>)</pre><p>Then to instruct the LLM to produce the required XML structure above we use the .get_format_instructions method combined to add the format instructions.</p><pre>prompt_with_format = prompt_system_format.partial(<br>	format_instructions=xml_parser.get_format_instructions()<br>)</pre><p>Next we can combine the prompt with a LLM and finally use the XML parser to parse the XML output from the model to a Python dict object</p><pre>xml_chain = (<br>  prompt_with_format<br>    | llm_model<br>    | xml_parser<br>)<br><br>output = xml_chain.invoke(...)</pre><p>Considering that the LLM is only given the XML tags and no heirarchical or contextual information, the fact that it can still infer the correct structure and content is impressive!</p><p>However, this can break for more complex schemas as there is no explicit structure information to guide the LLM. Also, the output parser does not perform any validation of the schema. The next section shows how to use Pydantic XML to improve both of these problems.</p><h3>Approach 2: Pydantic XML</h3><p>Here I propose what I think is a better approach using the power of Pydantic, which is standard for JSON structured output. In particular, <a href="https://pydantic-xml.readthedocs.io/en/latest/">Pydantic XML</a> is a library that provides similar functionality to Pydantic for XML, specifically providing XML parsing and validation.</p><p>As an example, let’s specify the above article using Pydantic XML. Compared to regular Pydantic instead of specifying Field classes the XML element is used.</p><pre>from pydantic_xml import BaseXmlModel, element<br><br>class ArticleXML(BaseXmlModel, tag=&quot;article&quot;):<br>    title: str = element(description=&quot;Title of the article&quot;)<br>    problem: str = element(<br>        default=&quot;Summary of the writer&#39;s question, write concisely&quot;<br>    )<br>    answer: str = element(default=&quot;Answer the writer&#39;s question&quot;)</pre><p>However, currently there is no support to create prompt instructions from Pydantic XML in LangChain (or other LLM packges as far as I know); therefore, I have written a simple helper function called that does this. To convert the above schema to prompt instructions, let’s consider how we could prompt it manually with the following prompt:</p><pre>Your answers must strictly adhere to the following XML format:</pre><pre>&lt;article&gt;<br>&lt;title&gt;<br>[Title of the article.]<br>&lt;/title&gt;</pre><pre>&lt;problem&gt;<br>[Summary of the writer&#39;s question, write concisely.]<br>&lt;/problem&gt;</pre><pre>&lt;answer&gt;<br>[Answer the writer&#39;s question.]<br>&lt;/answer&gt;<br>&lt;/article&gt;</pre><p>To generalise this format, I use a function pydantic_to_xml_instructions that writes format instructions for an arbitrary Pydantic XML, see Appendix 1 for the function code (although there are limitations to the schemas it can handle currently). See also <a href="https://github.com/adocherty/mastering-structured-output/blob/publish/5-langchain-xml-output-examples.ipynb">the XML examples Jupyter Notebook</a> for full code of all examples from this article.</p><p>Using this function we can prompt the LLM like follows:</p><pre>xml_chain = (<br>    prompt_format.partial(<br>        format_instructions=pydantic_to_xml_instructions(ArticleXML)<br>    )<br>    | llm_model<br>)<br><br>output = xml_chain.invoke(...)</pre><p>And finally we can run the chain and create a Pydantic object from the Pydantic XML object we created above using the .from_xml method. Note that <em>we can’t use either </em>PydanticOutputParser, as this only works with JSON formatted output, or XMLOutputParser which doesn’t provide validation.</p><pre>ArticleXML.from_xml(output.content)</pre><p>After all this we get an XML output parsed to a dictionary in the expected format. Great!</p><pre>ArticleXML(<br>  title=&#39;The Oldest Recorded Fossil: A Window into the Past&#39;,<br>  problem=&#39;What is the oldest recorded fossil and how was it discovered?&#39;,<br>  answer=&quot;Astronauts on board NASA&#39;s Apollo 11 mission in 1969 discovered a fossilized piece of stromatolite, a layered structure created by ancient microorganisms.&quot;<br>)</pre><h3>Evaluating XML output for different models</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/109/1*Ueo_jqDkWcnQDutYZMPW1g.png" /><figcaption><em>Follow through with the experiment code on Binder by clicking above</em></figcaption></figure><p>In order to test how well different models can output XML using both approaches, I evaluate and compare three providers and multiple models both open source and a commercial (Anthropic only). The three providers are Anthropic (commercial), Ollama (local open models), and <a href="http://fireworks.ai/">Fireworks.ai</a> (cloud served open source models).</p><p>The models evaluated, their sizes, and their model key is provided below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*pPBLhu6SFySC0U8EOFjrKA.png" /><figcaption>All models tested with provider key and parameter size.</figcaption></figure><p>I set up a single task given in the prompt below. Notice the length hint of less than 200 words, this is to constrain the outputs when the models ignore the formatting instructions (quite often) so they don’t have too long an output.</p><blockquote>You are a professional science writer tasked with responding to members of the general public who write in asking questions about science. Write an article responding to a writer’s question for publication in a science magazine intended for a general readership with a high-school education. You should write clearly and compellingly, include all relavent context, and provide motivating stories where applicable.</blockquote><blockquote>Your response must be less than 200 words.</blockquote><blockquote>The question given to you is the following: {question}</blockquote><p>To introduce some variety the {question} is replaced with one of 20 different science related questions, and we run through all 20 for each test of model and schema level.</p><p>There are 4 different schema levels tested as shown in the table below. These roughly increase in complexity and match the JSON schema levels used in the previous JSON blog post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*kTlkQT--UB0etT-gnthrOQ.png" /></figure><p>The full definition of these levels in Pydantic XML used in these experiments is available in <a href="https://github.com/adocherty/mastering-structured-output/blob/publish/6-langchain-xml-output-evaluation.ipynb">this Jupyter notebook</a>.</p><h3>Experiments</h3><p>Using the above setup I set out to answer the following research questions:</p><ol><li>Which models produce the most compliant XML output?</li><li>Does the custom approach with Pydantic XML work better than LangChain output parsers for XML?</li><li>Does a change in temperature change the model conformance to XML?</li><li>Which prompts are best at getting conformant XML output: System prompt vs user prompt?</li></ol><h3>Results</h3><h4>Which models produce the most compliant XML output?</h4><p>The results below show the percent of the runs that produced outputs that conformed with our requested schema for each method. This is repeated for each of the providers and models above and each of the four schemas, from least complex Level 1 to most complex Level 4. System prompting was used, and temperature was set to either 0 or 0.8.</p><p>We can see that newer models including <em>Anthropic Haiku 3.5, Sonnet 3.5, Phi4, Qwen 2.5 and Llama3.3 are excellent</em> <em>at outputting XML</em>. They not only consistently produce valid XML but also conform to the schema requested.</p><p>Smaller and older models are not as capable of producing valid XML structured outputs. In particular, conforming to typing hints (such as returning an integer) are hard for many models. This can be observed by looking at the difference between Level 1 and Level 1 (No int) performance where the only difference is if the number field is required to be an integer or not.</p><p>Note that performance for the weaker models is improved significantly by putting format instructions in the user prompt, as we detail in a following section. However, our default setup is to follow LangChain’s advice and put the format instructions in the system prompt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/1*B4_4qn_1QXNVqp2Z5XmWhg.png" /><figcaption>The results show the percent of the outputs from the LLM that conformed with our requested XML schema for each level of structure complexity. The confidence intervals are estimated usign a binomial model and a 90% confidence interval. The interval is stated as the largest variation in the two-sided bounds.</figcaption></figure><h4>Does the custom approach with Pydantic XML work better than LangChain output parsers for XML?</h4><p>This experiment compares the Approach 1 using LangChain’s default output parsers with Approach 2 using Pydantic XML and custom format instructions. The results speak for themselves.</p><p>It’s clear that LangChain XML output parser format instructions do not give the LLM enough information about the format to specify more complex XML formats. That being said, the latest models are suprisingly good at infering the structure from even this small amount of information, specifically Qwen 2.5, Sonnet 3.5, Llama 3.3 and Phi 4.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/1*HJ0MlbNv4j-y-ZEnjRuo0g.png" /><figcaption>The results show the percent of the outputs from the LLM that conformed with our requested XML schema for our custom Pydantic XML method versus LangChain output parsers. The confidence intervals are estimated usign a binomial model and a 90% confidence interval. The interval is stated as the largest variation in the two-sided bounds.</figcaption></figure><p>Note also that the Pydantic XML can be given a search_order parameter that specifies how the XML is parsed. The default “strict” value requires all elements in the XML to be in the same order as the schema and no extra elements are accepted. If we change this to “unordered” we can relax this condition and elements are accepted in any order. This morderately improves performance, but not enough to beat the custom Pydantic XML prompting approach.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*gV-lYuH-43f-NL9oOM-jyA.png" /></figure><p>For all other results in this article, I use the “strict” parsing mode to assure the results show outputs that conform exactly to the provided schema without changes in element order or hallucinated elements.</p><h4>Does a change in temperature change the model conformance to XML?</h4><p>When generating an output LLMs have a configurable parameter that enables them to control the sampling of the next token. With a temperature of 0 the token with the highest model prediction is returned, with higher remperatures the next token is chosen randomly with likelihood proportional to the model’s predictions.</p><p>It seems a likely hypothesis that as the temperature is increased schema conformance will reduce; as with more randomness there is higher chance to choose less probable tokens that do not fit the requested schema.</p><p>Performing an experiment with equal setups apart from a change in temperature shows that this hypothesis is true of weaker models, with <em>Llama 3.2, Nemotron mini,</em> and <em>Phi 3</em> particularly benefiting from lower temperatures .</p><p>However, if you want reliable XML output it’s best to choose a model that works well with XML — and with those models there is little decrease in performance between the temperatures of t=0 and 0.8.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/523/1*2_M4KILSpLMKS_hI81LnHA.png" /><figcaption>The results show the percent of the runs that produced outputs that conformed with our requested schema for two temperatures t=0 and t=0.8. The confidence intervals are estimated usign a binomial model and a 90% confidence interval. The interval is stated as the largest variation in the two-sided bounds.</figcaption></figure><h4>Which prompts are best at getting conformant XML output: System prompt vs user prompt?</h4><p>The format instructions can be included in the system prompt or the user prompt. In the examples on Approach 1 and 2 above, the format instructions were included in the System prompt, like so:</p><pre>ChatPromptTemplate.from_messages(<br>    [<br>        (<br>            &quot;system&quot;,<br>            &quot;Return a publishable article in the requested format.\n&quot;<br>            &quot;{format_instructions}&quot;,<br>        ),<br>        (&quot;human&quot;, user_prompt_txt),<br>    ]<br>)</pre><p>For the User prompt format I include the format instructions at the end of the user prompt and did not include a system prompt at all, for example:</p><pre>prompt_user_format = ChatPromptTemplate.from_template(<br>    user_prompt_txt + &quot;\n\n{format_instructions}&quot;<br>)</pre><p>In general including the format instructions in the user prompt <strong>increased conformance </strong>to XML significantly for the weaker models. For the stronger models there was no significant difference.</p><p>This is of interest if you are planning to use small models with structured output — they seem to have a habit of forgetting instructions if the are included in the system prompt! Including the formatting instructions in the user prompt for these models results in better performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/1*zvxXw6-_iqbI3MSgP1t06Q.png" /><figcaption>The results show the percent of the runs that produced outputs that conformed with our requested schema for format in the system and user prompts. The confidence intervals are estimated usign a binomial model and a 90% confidence interval. The interval is stated as the largest variation in the two-sided bounds.</figcaption></figure><h4>What’s going on with DeepSeek R1?</h4><p>This is a small (7b parameter) distilled model, therefore it does not hold that it’s performance represents larger models. This distilled model is based on Qwen2 and is of similar performance as Meta Llama 3.2 even though this is a smaller model.</p><p>This distilled model is just not great with XML, with the format instructions in the system prompt it often forgets to output XML at all. With the format in the user prompt it will often output corrupt XML, even at temperature 0. This is interesting as the model is trained to output it’s thinking in between XML tags &lt;think&gt;, so naively I was thinking it would perform well with XML structure in general.</p><p>For example:</p><pre>&lt;event Johannes Kepler formulated laws of planetary motion based on <br>Tycho Brahe’s data.&lt;/event&gt;</pre><p>I have not tested larger DeepSeek R1 models, as they are expensive on Fireworks.ai. I will try this with a future post on reasoning models and structured outputs.</p><h4>What’s going on with Llama 3.2?</h4><p>Meta Llama 3.2 is a small model (only 3B parameters) so it’s not expected it would perform as well as the larger Llama 3.1 and 3.3 models. However, I’m seeing consistently that the performance of Llama 3.2 running on Ollama is not as good as the same model running on Fireworks.ai. What’s going on here?</p><p>This is most likely due to differences in quantization of the models. While Ollama’s Llama 3.2 3b model is quantized at Q4_K_M. <a href="https://fireworks.ai/blog/fireworks-quantization">Fireworks.ai takes a careful approach to quantization</a>; however, they don’t specify if their <a href="https://fireworks.ai/models/fireworks/llama-v3p2-3b-instruct">Llama 3.2 3b serverless model</a> is quantized or not. My best guess is that the Fireworks.ai base models are not quantized, and this explains the performance difference.</p><h3><strong>Conclusions</strong></h3><p>This article has shown how you can get excellent and reliable XML output that conforms to a schema that is specified in Pydantic XML, and parse and validate it.</p><p>There is a clear divide among models. Newer and larger models like Anthropic’s Haiku/Sonnet 3.5, Phi4, Qwen 2.5, and Llama3.3 consistently produce valid XML that conforms to provided schemas. In contrast, older and smaller models struggle significantly with structured XML outputs, with Microsoft Phi 3 and Nvidia Nemotron demonstrating particularly poor performance. This capability gap underscores the rapid advancement in structured output generation among frontier models.</p><p>Interestingly, implementation details matter significantly for less capable models. When working with models like Nemotron, Phi3, or Llama3.2, placing XML format instructions in the user prompt rather than the system prompt yields substantially better results. Temperature settings also affect these models, with higher temperatures degrading XML conformance. By contrast, newer models maintain consistent XML quality across user and system prompts and temperature ranges, allowing more creative variation without sacrificing structural integrity.</p><p>One notable challenge is type conformance in XML, with many models struggling to consistently output correct data types (in particular, integers) as specified in schemas. Additionally, unlike JSON — which benefits from widespread provider support through constrained generation methods — XML lacks equivalent specialized tooling in most LLM platforms.</p><p>As LLMs continue to evolve, we expect structured output capabilities to become increasingly standardized across model families. As od today, the clear recommendation is to utilize newer, more capable models when XML output is required. However, when circumstances necessitate using smaller or older models, choosing the right model for the job and implementing format instructions in user prompts and maintaining lower temperatures can significantly improve results.</p><h3>Appendix 1: Pydantic XML to Format Instructions</h3><p>The following is a simple function that given the Pydantic XML object outputs format instructions to prompt the LLM to conform to the requested schema.</p><pre>def pydantic_to_xml_instructions(<br>    model,<br>    root_name=None,<br>    root_description=None,<br>    add_instructions=True,<br>    n_list_examples: int = 2,<br>):<br>    # Get the JSON schema representation of the model<br>    schema_json = model.model_json_schema() or {}<br><br>    xml = (<br>        &quot;You must respond only in XML using the following schema.\n&quot;<br>        &quot;Do not provide any output outside the first and last XML tags.\n\n&quot;<br>        if add_instructions<br>        else &quot;&quot;<br>    )<br><br>    # Start with root element named after the model<br>    _root_name = root_name or model.__xml_tag__ or schema_json.get(&quot;title&quot;, &quot;&quot;)<br>    _root_desc = root_description or schema_json.get(&quot;description&quot;, &quot;&quot;)<br>    xml += f&quot;&lt;{_root_name}&gt;\n  &lt;!--{_root_desc}--&gt;\n&quot;<br><br>    # Process each property<br>    for field_name, field in model.model_fields.items():<br>        field_type = field.annotation<br>        description = field.description<br>        tag = field.path if field.path else field_name<br><br>        # Handle nested classes<br>        if isinstance(field_type, type) and issubclass(field_type, BaseXmlModel):<br>            xml += pydantic_to_xml_instructions(<br>                field_type, root_name=tag, add_instructions=False<br>            )<br>            xml += &quot;\n&quot;<br><br>        # Handle lists<br>        # TODO: lists of lists are not currently handled<br>        elif get_origin(field_type) is list:<br>            subtype = get_args(field_type)[0]<br>            if isinstance(subtype, type) and issubclass(subtype, BaseXmlModel):<br>                list_xml = pydantic_to_xml_instructions(<br>                    subtype,<br>                    root_name=tag,<br>                    root_description=description,<br>                    add_instructions=False,<br>                )<br>            else:<br>                list_xml = f&quot;  &lt;{tag}&gt;\n&quot;<br>                list_xml += f&quot;    {{{description} - must be type {subtype.__name__}}}\n&quot;<br>                list_xml += f&quot;  &lt;/{tag}&gt;&quot;<br><br>            # Insert list XML multiple times to prompt a list<br>            for ii in range(n_list_examples):<br>                if ii == 1:<br>                    xml += &quot;&lt;!-- First list element --&gt;\n&quot;<br>                else:<br>                    xml += &quot;&lt;!-- Next list element --&gt;\n&quot;<br>                xml += list_xml + &quot;\n&quot;<br>            xml += &quot;&lt;!-- Etc --&gt;\n&quot;<br>            xml += f&quot;  &lt;{tag}&gt;\n  ...\n  &lt;/{tag}&gt;\n&quot;<br><br>        else:<br>            # Add field as XML element with type comment and description<br>            xml += f&quot;  &lt;{tag}&gt;\n&quot;<br>            xml += f&quot;    {{{description} - must be type {field_type.__name__}}}\n&quot;<br>            xml += f&quot;  &lt;/{tag}&gt;\n&quot;<br><br>    xml += f&quot;&lt;/{_root_name}&gt;&quot;<br><br>    return xml</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8bad9e1f43ef" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Structured Output in LLMs 2: Revisiting LangChain and JSON]]></title>
            <link>https://medium.com/@docherty/mastering-structured-output-in-llms-revisiting-langchain-and-json-structured-outputs-d95dfc286045?source=rss-dba307069819------2</link>
            <guid isPermaLink="false">https://medium.com/p/d95dfc286045</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[llm-evaluation]]></category>
            <category><![CDATA[langchain]]></category>
            <dc:creator><![CDATA[Andrew Docherty]]></dc:creator>
            <pubDate>Wed, 12 Feb 2025 04:57:32 GMT</pubDate>
            <atom:updated>2025-03-19T23:49:32.476Z</atom:updated>
            <content:encoded><![CDATA[<h4>Structured outputs for LLMs are the bedrock of how to integrate them into other software systems, workflows, and applications.</h4><p>Structured output in LLMs (Large Language Models) instruct the returned answer to be in a defined and specific format. This enables us to use the output directly in code that expects specific fields to be returned.</p><p>While we can simply request in the prompt for the LLM to return output in JSON format and specify the desired keys, it’s more effective to use specific functionality to output JSON. Simply asking the LLM for JSON output, while it may work most of the time, will not guarantee that what is returned is valid JSON or make any guarantee that it will conform to the schema that you’ve provided.</p><p>A couple of months ago <a href="https://medium.com/@docherty/mastering-structured-output-in-llms-choosing-the-right-model-for-json-output-with-langchain-be29fb6f6675"><strong>I wrote a blog post on exactly this</strong></a>, but now in Feburary 2025 most of what I wrote has now changed! Since I wrote the previous article new options have been introduced into the with_structured_output methods that change how to best use LangChain to get structured outputs. At the same time, the <a href="https://ollama.com/blog/structured-outputs">Ollama has introduced a powerful new way to get structured outputs</a> that upends the comparisons that I did and makes getting valid structured outputs from almost any model robust and reliable.</p><p>My goal for this blog is to update the previous post to match the recent new features in LangChain, and clarify the best ways to get structured output from multiple models and providers.</p><h3>Takeaways</h3><ul><li>LangChain has added support for choosing which method is used to give structured outputs, including <em>function-calling</em>, <em>JSON mode</em>, and <em>JSON schema.</em></li><li><em>Ollama with the JSON schema method gives a huge improvement</em> in the reliability of structured outputs, and now most models calmost always return outputs matching the requested schema. This is competative with commercial services including OpenAI and Anthropic.</li><li><em>Some providers have not yet fully supported these new methods</em>; therefore, what is supported and how it functions depends on both the provider and the model.</li><li>To run larger models on a cloud provider, <a href="http://fireworks.ai/"><em>Fireworks.ai</em></a><em> supports structured outputs in LangChain </em>but only for some models and does not yet support the “JSON schema” method.</li><li>Full example code and evaluation code is available here: <a href="https://github.com/adocherty/mastering-structured-output">https://github.com/adocherty/mastering-structured-output</a></li></ul><p>For each of three providers, one commercial provider (Anthropic), the locally hosted Ollama, and Cloud-based Fireworks I’ve tested the support for structured outputs using a methodology simlilar to <a href="https://medium.com/@docherty/mastering-structured-output-in-llms-choosing-the-right-model-for-json-output-with-langchain-be29fb6f6675">my previous blog post</a>. The take-away results for select models are below, showing the best methods and the percentage of 20 calls to the LLM requesting a structured output that return a valid object conforming to the schema.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*lHifaKK-q_XdN5kyxNFVfA.png" /><figcaption>The results for the percentage of outputs which were both valid JSON and matched the given schema. for the best performing method. The method that gave these results is shown and if there is a tie all methods that gave the same performance are listed. Note the approximate spread in multiple experiments is ±10%.</figcaption></figure><h3>New LangChain structured output methods</h3><p>LangChain has recently introduced three potential methods to have models return structured JSON output. The interface is consistent accross all providers (such as OpenAI, Anthropic, Ollama, and more). However, not all providers or chat models implement all of these methods and furthermore they may implement the same option in different ways, or currently ignore the method as they have not yet been updated. This leads to behaviour that sometimes the same model on different providers can behave very differently. We will get to that below.</p><p>LangChain has done a great job of standardising the interface that we use to request structured outputs.</p><p>In the examples below we pass a schema. This can be a Pydantic model, a TypedDict model, or a JSON schema dict specifying the schema. Generally a Pydantic modelis recommended as it provides support for validation (so you’ll get a error message if the returned data doesn’t fit the schema) and it’s becoming standardised accross structured output libraries including OpenAI.</p><p>For example, we might want to ask the LLM to return a article with structured fields containing a title, and answer to a reader’s question and I’ve included a number of the day. After defining this structure as a Pydantic model we pass it to the LangChain LLM model and specify the method like so:</p><pre>from pydantic import BaseModel<br><br>class ArticleFromQuestion(BaseModel):<br>    &quot;&quot;&quot;Structured article for publication answering a reader&#39;s question.&quot;&quot;&quot;<br>    title: str = Field(description=&quot;Title of the article&quot;)<br>        answer: str = Field(<br>            description=&quot;Provide a detailed description of historical events to answer the question.&quot;<br>        )<br>        number: int = Field(<br>            description=&quot;An number relevant to the question.&quot;<br>        )<br><br>llm_model.with_structured_output(ArticleFromQuestion, method=&quot;...&quot;)</pre><p>Note the use of the “description” parameter to give the LLM the details of what to place in this field. This is an extention of the prompt, and how to get the prompt and the schema to best work together is an art.</p><p>For full coded examples with all methods check out this <a href="https://github.com/adocherty/mastering-structured-output/blob/main/3-langchain-structured-output-examples-updated.ipynb">Jupyter Notebook on GitHub</a>.</p><h4>Method 1: Function calling</h4><pre>llm_model.with_structured_output(<br>      ArticleFromQuestion,<br>      method=&quot;function_calling&quot;<br>  )<br>structured_llm.invoke(prompt)</pre><p>Firstly, if the model has function calling (tool-calling) support then LangChain supports a structured output method which uses the function calling API under the hood to return JSON formatted output. Models that support tool-calling have been fine-tuned to “understand” the specific tool-calling JSON specification and provide good performance for it.</p><p>This essentially means that the LLM thinks it’s calling a function with arguments as specified by the schema you specify, and under the hood LangChain returns the structure you requested converted from these arguments.</p><p>Also, not all open models support tool-calling currently. When using Ollama as a backend, the models with tool-calling support are listed here: <a href="https://ollama.com/search?c=tools">https://ollama.com/search?c=tools</a></p><p>This is the default method, and until recently the only supported method.</p><p><strong>Failures: </strong>When this method fails, I find it’s most likely a function calling failure and the output is Null.</p><h4>Method 2: JSON Mode</h4><p>Recent techniques have been developed to limit the output of models to valid JSON were introduced by OpenAI and the open-source model community. This uses constrained generation to essentially only allow the model to choose outputs that are valid JSON. This essentially replaces the output parsers method I mention in the <a href="https://medium.com/@docherty/mastering-structured-output-in-llms-choosing-the-right-model-for-json-output-with-langchain-be29fb6f6675">previous (now outdated) blog post</a>.</p><p>An important caveat is that in addition to specifying the schema in the with_structured_output you also need to include formatting instructions that specify the JSON schema you want returned in the prompt itself. LangChain output parsers PydanticOutputParser and JSONOutputParser can constructs prompt instructions for you that you can include in the prompt.</p><p>This is because with this method the language model doesn’t see the schema explicitly, unlike the other methods.</p><pre>output_parser = PydanticOutputParser(pydantic_object=ArticleFromQuestion)<br>format_instructions = output_parser.get_format_instructions()<br><br>structured_llm = llm_model.with_structured_output(<br>        ArticleFromQuestion, <br>        method=&quot;json_mode&quot;<br>    )<br>structured_llm.invoke(f&quot;{prompt} {format_instructions}&quot;)</pre><p><strong>Failures: </strong>When this method fails, it is typically that the model has hallucinated an incorrect schema.</p><h4>Method 3: JSON Schema</h4><pre>llm_model.with_structured_output(<br>      ArticleFromQuestion,<br>      method=&quot;json_schema&quot;<br>  )<br>structured_llm.invoke(prompt)</pre><p>This is the newest method on the block. It uses special API calls to backends for the explicit purpose of returning structured outputs. Different providers implment this in different ways, and it’s important to investigate what your chosen provider supports.</p><p>As far as I understand it, this method constrains the model output not just to JSON but to the subset of JSON that is consistent with the given schema. This seemst to be true for <a href="https://openai.com/index/introducing-structured-outputs-in-the-api/">OpenAI’s implementation</a> as well as <a href="https://blog.danielclayton.co.uk/posts/ollama-structured-outputs/">Ollama’s recent feature</a>; however, I don’t fully understand how different providers implement this, so please correct me if I’m wrong!</p><p>Note that for OpenAI models there’s an extra strict argument that you can supply to <a href="https://platform.openai.com/docs/guides/structured-outputs">enable strict schema conformance</a>.</p><p><strong>Failures: </strong>This method rarely fails, when it is implemented! When it does fail I’ve found it’s due to the LLM not closing the structure before it runs out of output tokens. In my case, I think it can be improved with some prompt engineering.</p><h3>Provider and model support</h3><p>In the previous post I used only local models through Ollama and commercial models from Anthropic. Now there are cloud services that enable running large LLMs too big for my local machine for costs far less than the commercial services. Of these I’ve found <a href="http://openrouter.ai/">OpenRouter.ai</a> and <a href="http://fireworks.ai/">Fireworks.ai</a> support structured outputs. However, OpenRouter requires a parameter to be passed in the REST query to only select providers that support structured outputs (only Fireworks and OpenAI currently) and currently this is not supported by LangChain.</p><p>Now a warning, not all providers or models support all modes. Even worse, some providers fail to implement the method= parameter at all and don’t check what value it is — so you can pass in a method but it will be ignored without error (currently Anthropic, although this may change quickly). So, be careful using these different options and always check the docstrings to see what methods are supported (they could change anytime!).</p><p>As of Feb 2025, the supported methods for a few select provicers are shown in the table below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*Xw7kUosfSaxYnS7rUT6dTg.png" /><figcaption>Structured output method support by provider</figcaption></figure><p>So, while LangChain is doing a great job to make the programming interface consistent accross different providers and models, there can be differences in how a provider implements or not the different methods.</p><h4>Using <a href="http://fireworks.ai/">Fireworks.ai</a> for structured outputs</h4><p>Fireworks directly supports structured outputs in most models. Although there are caveats with how well they will perform! Currently Fireworks only supports <em>function calling</em> and <em>JSON mode</em> for structured outputs; however, the <em>JSON schema</em> method is not supported.</p><p>Other than setup, the Fireworks.ai models work with structured outputs like any other mode. For full coded examples with all methods check out this Jupyter Notebook on GitHub.</p><h3>Schema Validation with Pydantic</h3><p>In the above examples you can have the returned JSON validated by Pydantic, returned as a Python dict object without validation, or a raw JSON string.</p><p>To automatically have the returned JSON converted to a Pydantic object and validated, supply a pydantic model as the schema</p><pre>llm.with_structured_output(schema=pydantic_model)</pre><p>To return a dictionay without any valdiation, use structured output with JSONSchema or TypeDict:</p><pre>llm.with_structured_output(schema=json_schema)</pre><h3>Handling Errors</h3><p>As we know, LLMs are statistical (read random) models, and therefore there is no guarantee that they will return valid JSON or if the do that the schema will conform to what we have requested.</p><p>When LangChain is given a Pydantic schema then Pydantic will be used to validate that returned data conforms to the expected schema.</p><p>For example, Llama3.2 fails often by returning the number in ArticleResponse1 as a string</p><pre>ValidationError: 1 validation error for ArticleResponse1<br>number<br>  Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value=&#39;3,500,000&#39;, input_type=str]<br>    For further information visit &lt;https://errors.pydantic.dev/2.9/v/int_parsing&gt;</pre><p>To get more information from a model, including the raw output before any validation errors, the include_raw=True argument can be passed to the .with_structured_output method. The parsed output is then contained within the &#39;parsed&#39; element of the returned dictionary.</p><p>To determine the different types of errors that can occur I’ve found the following error catching code useful:</p><pre>try:<br>    test_chain = prompt | llm_model.with_structured_output(<br>        pydantic_obj, method=method, include_raw=True<br>    )<br>    output = test_chain.invoke(input)<br>    # Typically Pydantic validation failure<br>    if output[&quot;parsing_error&quot;] is not None:<br>        raise RuntimeError(&quot;Parsing error&quot;)<br>    # Typically function-calling failure<br>    elif output[&quot;parsed&quot;] is None:<br>        raise RuntimeError(&quot;No output error&quot;)<br>    # No error - Yay!<br>    else:<br>     ...<br>   <br># Other failures (typically function-calling not supported)<br>except Exception as e:<br>    print(f&quot;Error in structured output: {type(e).__name__}&quot;)</pre><p>LangChain offers functionality to fix errors with the OutputFixingParser which passes the problematic output back to the LLM to attempt to fix the issue. However, currently this doesn’t work with the .with_structured_output as this method applies an output parser that converts the output to a Pydantic object already. Therefore we can’t apply the OutputFixingParser to this — the failure will have already occurred by the time we get there and the OutputFixingParser expects the raw LLM output.</p><h3>Evaluating JSON output for different models</h3><p>Here I’m interested in returning text descriptive outputs for a program management task in a structured way. Therefore, I’m curious to create a test case that returns <em>longer text</em> and<em> more complicated structures </em>and test how different models handle the outputs.</p><p>OpenAI gives supported structure <a href="https://platform.openai.com/docs/guides/structured-outputs#supported-schemas">guidelines for ChatGPT</a>, including:</p><ul><li>A schema may have up to 100 object properties total, with up to 5 levels of nesting.</li><li>The total string length of all property names, definition names, enum values, and const values cannot exceed 15,000 characters</li><li>All fields or function parameters must be specified as required.</li></ul><p>All the schemas I test so far have much less complexity that this with a maximum of 2 levels of nesting and less than 5 properties. Now the models are performing so well, in the future I can look at making the schemas more complicated.</p><p>I’m going to evaluate and compare three providers and multiple models both open source and a commercial (Anthropic only). The three providers are Anthropic (commercial), Ollama (local open models), and <a href="http://fireworks.ai/">Fireworks.ai</a> (cloud served open source models).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/1*svad_qUi_qql7FAiWVuQ9w.png" /><figcaption>Models and providers considered in this study and model keys used in the API.</figcaption></figure><p>For more details about Ollama models see: <a href="https://ollama.com/library">https://ollama.com/library</a><br>For more details about Fireworks models see: <a href="https://fireworks.ai/models">https://fireworks.ai/models</a></p><p>To answer the question of how these models and output methods differ with different complexities of schema I’m defining four example schema in increasing order of complexity. I summarize these in the table below. For the full definitions, see the appendix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/1*2TnIChIVVFDalbz3J_hbUA.png" /></figure><p>The example prompt follows. Notice the length hint of less than 200 words, this is to constrain the outputs when the models ignore the formatting instructions.</p><blockquote>You are a professional science writer tasked with responding to members of the general public who write in asking questions about science. Write an article responding to a writer’s question for publication in a science magazine intended for a general readership with a high-school education. You should write clearly and compellingly, include all relavent context, and provide motivating stories where applicable.</blockquote><blockquote>Your response must be less than 200 words.</blockquote><blockquote>The question given to you is the following: {question}</blockquote><p>To introduce some variety the {question} is replaced with one of 20 different science related questions, and we run through all 20 for each test of model and schema level.</p><p>The code is given in full in <a href="https://github.com/adocherty/mastering-structured-output/blob/main/4-langchain-structured-output-evaluation-updated.ipynb">this evaluation Jupyter notebook on GitHub</a>.</p><h3>Results</h3><p>The results below show the percent of the runs that produced outputs that conformed with our requested schema for each method. This is repeated for each of the providers and models above and each of the four schemas, from least complex Level 1 to most complex Level 4.</p><p>I don’t include any indication of the variance of the results, but from doing a few runs I’m guestimating that these results can vary by ±10%. Therefore, I wound’t put any importance on differences less than this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*tjWO87FkeKZAk_Y-UnJ_YA.png" /><figcaption>The percentage of outputs which were both valid JSON and matched the given schema. Note the approximate spread in multiple experiments is ±10%. NA = Method not available</figcaption></figure><p>Note that these results all have an implicit (and untested) error margin. From previous experiments I’d estimate this at around 10%. All models are run at a temperature of 0 to get the best structured outputs. However, this could come at a compromise of reduced variation in responses. I intend to address this in the future.</p><p>The results for Ollama models are a stricking difference from the previous time I tested these. Using the JSON schema method most models give a perfect performance in structured outputs accross the board.</p><p>For Fireworks, Llama 3.1 and Llama 3.3 models give excellent performance whereas Llama 3.2 falls down and does not support function calling as a method.</p><p>We can see the improvement in Antrhopics models with Haiku 3.5 providing a big improvement over Haiku 3, and how being on par with Sonnet 3.5 for structured outputs.</p><p>Phi4 has improved schema adherance from Phi3 to Phi4 using JSON mode, but using JSON schema mode in Ollama gives both models exellent performance.</p><p>There are some oddities here, Ollama’s function calling method doesn’t seem to work (it produces null outputs). It is most likely a bug that may be fixed by the time you read this!</p><h3>Conclusions</h3><p>Every couple of months LangChain’s structured output support and performance increases remarkably. A lot has changed recently and now tool use and structured output mode is supported for a wide range of models.</p><p>Just a few months ago to get excellent structured outputs you needed to use commercial models such as Anthropic’s Claude and OpenAI’s ChatGPT. Now open models support comparable performance in structured output throug Ollama and cloud services such as Fireworks.</p><h3>Further reading</h3><p>OpenAI structured output implementation: <a href="https://platform.openai.com/docs/guides/structured-outputs">https://platform.openai.com/docs/guides/structured-outputs</a></p><p>Blog post introducing Structured Outputs for OpenAI: <a href="https://openai.com/index/introducing-structured-outputs-in-the-api/">https://openai.com/index/introducing-structured-outputs-in-the-api/</a></p><p>Some examples using structured outputs in OpenAI: <a href="https://cookbook.openai.com/examples/structured_outputs_intro">https://cookbook.openai.com/examples/structured_outputs_intro</a></p><p>A blog post going through using the new OpenAI structured outputs:: <a href="https://towardsdatascience.com/diving-deeper-with-structured-outputs-b4a5d280c208">https://towardsdatascience.com/diving-deeper-with-structured-outputs-b4a5d280c208</a></p><p>Ollama structured output implementation: <a href="https://ollama.com/blog/structured-outputs">https://ollama.com/blog/structured-outputs</a></p><p>Fireworks JSON mode implementation: <a href="https://docs.fireworks.ai/structured-responses/structured-response-formatting">https://docs.fireworks.ai/structured-responses/structured-response-formatting</a></p><p>Fireworks article on the importance of structured outputs <br><a href="https://fireworks.ai/blog/why-do-all-LLMs-need-structured-output-modes">https://fireworks.ai/blog/why-do-all-LLMs-need-structured-output-modes</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d95dfc286045" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Structured Output in LLMs 1: JSON output with LangChain]]></title>
            <link>https://medium.com/@docherty/mastering-structured-output-in-llms-choosing-the-right-model-for-json-output-with-langchain-be29fb6f6675?source=rss-dba307069819------2</link>
            <guid isPermaLink="false">https://medium.com/p/be29fb6f6675</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[structured-output]]></category>
            <category><![CDATA[json]]></category>
            <dc:creator><![CDATA[Andrew Docherty]]></dc:creator>
            <pubDate>Thu, 19 Dec 2024 04:34:19 GMT</pubDate>
            <atom:updated>2025-05-30T05:43:17.538Z</atom:updated>
            <content:encoded><![CDATA[<h4>Structured outputs for LLMs are the bedrock of how to integrate them into other software systems, workflows, and applications.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xvA5nufx8480AXsnlqQVqQ.png" /><figcaption>Created with ChatGPT</figcaption></figure><blockquote>Since this article was published a lot has changed! Please <a href="https://medium.com/@docherty/mastering-structured-output-in-llms-revisiting-langchain-and-json-structured-outputs-d95dfc286045">see the updated blog post here</a> for an update to structured outputs in LangChain.</blockquote><p>Large language models (LLMs) have taken the world by storm and are powerful bridges between the human world of unstructured language and the computer world of structured data.</p><p>The problem is, these models don’t easily produce outputs that are easy to integrate them into system with other software; and when asked to output in specific format, the results can be inconsistent and furthermore can change the quality of the answer in unexpected ways.</p><p>In this article, we’ll explore some options for structuring LLM outputs and evaluate which models work best in different situations with both commercial and open source models.</p><p>In future articles we’ll look at XML and markdown and compare the different formats. My goal over this series of articles is to build an evaluation framework that can evaluate prompt output quality and schema adherence when using different structured outputs so I can choose the best format and model for the job.</p><h3>Other articles in the series</h3><ol><li>JSON output with LangChain (this article)</li><li><a href="https://medium.com/@docherty/mastering-structured-output-in-llms-revisiting-langchain-and-json-structured-outputs-d95dfc286045">Revisiting LangChain and JSON structured outputs</a></li><li><a href="https://medium.com/@docherty/mastering-structured-output-in-llms-3-langchain-and-xml-8bad9e1f43ef">Structured Output in XML using LangChain</a></li><li><a href="https://medium.com/@docherty/python-libraries-for-llm-structured-outputs-beyond-langchain-621225e48399">Python libraries for LLM structured outputs beyond LangChain</a></li><li><a href="https://medium.com/@docherty/controlling-your-llm-deep-dive-into-constrained-generation-1e561c736a20">Controlling your LLM: Deep dive into Constrained Generation</a></li></ol><h3>Takeaways</h3><p>Below I evaluate the accuracy of JSON object generation matching a defined schema for different models and at four levels of (approximate) schema complexity. The results are in the table below, see the full article below for an in-depth explanation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/1*uiGAH6KckEy7JBfLR7GuVg.png" /><figcaption>The results for the percentage of outputs which were both valid JSON and matched the given schema. Note the approximate spread in multiple experiments is ±10% except for the red numbers where the spread over 3 experiments is given explicitly.</figcaption></figure><ul><li>The function-calling API is now widely available in LangChain accross different models and can be used to produce good quality JSON outputs.</li><li>Anthropic’s latest Claude 3.5 Sonnet handles even the most complex structured excellently.</li><li>If you want to output JSON with a small model, choose Nemotron-mini. With only 4B parameters excels at producing valid JSON structure.</li><li>Other small open-weight models, including Llama3.2 3B, perform poorly for all but the simplest schema.</li><li>Gemma2 doesn’t have tool use but it quite capable of outputting structured JSON although it struggles with custom types.</li></ul><h3>Overview of structured output in LLMs</h3><p>Structured output in LLMs instructs the LLM to return the output in a specific format, often JSON or XML. This is great to integrate the output in code that expects specific fields to be returned. If we return a machine-readable format such as JSON we can easily extract structured objects from the LLM.</p><blockquote>How many days are there until Christmas? Today is December 19th. Return answer as a JSON object.</blockquote><blockquote>{“days_until_christmas”: 12}</blockquote><p>The simplest approach is to prompt the LLM for the return output to be a JSON object, and specify the desired keys. However, while this may work most of the time, there is no guarantee that what is returned is valid JSON or will conform to the schema that you’ve provided.</p><p>Recently models have vastly improved their abilities to output structured formats, particularly JSON, and OpenAI’s ChatGPT API can now guarantee to return valid JSON and conform to the provided schema. This functionality in <a href="https://openai.com/index/introducing-structured-outputs-in-the-api/">structured outputs was introduced in August 2024</a> and OpenAI reports that it improves the compliance to a JSON schema from 35% with prompting alone to 100% with this “strict mode” enabled on their evaluation dataset.</p><p>The <a href="https://github.com/dottxt-ai/outlines">outlines</a> library by <a href="https://blog.dottxt.co/">dottxt</a> aims to make structured outputs easier and more reliable. However, for this blog post I’ll focus on structured outputs using the LangChain library. In the future it’ll be interesting to see how these outlines and LangChain compare when using the same LLM model.</p><p>In the rest of this blog, I’m going to investigate how we can return JSON using LangChain and evaluate the quality of different models on some test schemas of increasing complexity.</p><h3>Using LangChain to get JSON outputs</h3><p>Firstly, let’s walk through how to get a model to output JSON in LangChain. We will focus requesting structured outputs using the LangChain API and not compare to requesting a JSON object in the prompt.</p><p>LangChain has two main methods to output JSON, and they are quite different in both usage and performance. I’ll call them Method 1 and Method 2.</p><h4>Method 1: Structured output with the tool-calling API</h4><p>Firstly, if the model has tool calling (function calling) support then <a href="https://python.langchain.com/docs/how_to/structured_output">LangChain supports a structured output method</a> which uses the tool-calling API under the hood to return JSON formatted output.</p><p>Models that support tool-calling have been fine-tuned to “understand” the specific tool-calling JSON specification and provide good performance for it. The structured output method essentially creates a dummy tool with the arguments corresponding to the schema of the provided object.</p><p>However, not all open models support tool-calling currently. OpenAI’s ChatGPT and Anthropic’s Claude both support tool-calling. When using Ollama as a backend, the models with tool-calling support are listed here: <a href="https://ollama.com/search?c=tools">https://ollama.com/search?c=tools</a></p><h4>Method 2: Prompt instructions using Output Parsers</h4><p>Alternatively, LangChain output parsers <a href="https://api.python.langchain.com/en/latest/output_parsers/langchain_core.output_parsers.pydantic.PydanticOutputParser.html">PydanticOutputParser</a> and <a href="https://api.python.langchain.com/en/latest/output_parsers/langchain_core.output_parsers.json.JsonOutputParser.html">JSONOutputParser</a> construct prompt instructions that are fed to the model to define the output format without the tool-use API required. However, as the models may not be not fine-tuned to conform to this schema, typically this method does not perform as well as the structured output method.</p><h3>How to use LangChain to get structured outputs</h3><p>I’ve provided a Jupyter notebook for the code in this section, <a href="https://github.com/adocherty/mastering-structured-output/blob/main/1-langchain-structured-output-examples.ipynb">available on GitHub</a>.</p><p>Now let’s see how we can use the structured output mode to get a model to return structured data. In essence we provide a Pydantic object that is converted to a <a href="https://json-schema.org/">JSON schema</a> in the background and fed to the LLM to define the output format. This will also parse the returned data to a Pydantic object and validate it to the schema<em>.</em></p><p>Note that with either method you can have the returned JSON validated by Pydantic, returned as a Python dict object without validation, or a raw JSON string. See the table below for the brief overview:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*1xfgHuNiue1HYIbxgKKbRg.png" /><figcaption>Overview of getting validated and unvalidated outputs with both JSON output methods</figcaption></figure><h4>How to get structured output with Method 1: Tool-Calling API</h4><p>An example <a href="https://python.langchain.com/docs/how_to/structured_output/">Pydantic object from the LangChain docs</a> to give a joke as a structured output with Method 1:</p><pre>from typing import Optional<br>from pydantic import BaseModel, Field<br><br>class Joke(BaseModel):<br>    &quot;&quot;&quot;Joke to tell user.&quot;&quot;&quot;<br>    setup: str = Field(description=&quot;The setup of the joke&quot;)<br>    punchline: str = Field(description=&quot;The punchline to the joke&quot;)<br>    rating: int = Field(description=&quot;How funny the joke is, from 1 to 10&quot;)<br><br>structured_llm = llm_model.with_structured_output(Joke)<br>structured_llm.invoke(&quot;Tell me a joke about cats&quot;)</pre><p>The code returns a Pydantic object that contains the completed fields. Using a Pydantic object automatically validates that each field requested is present and is of the expected type.</p><pre>Joke(<br>  setup=&#39;Why did the cat join a band?&#39;, <br>  punchline=&#39;Because it wanted to be the purr-cussionist!&#39;,<br>  rating=8<br>)</pre><p>If we don’t want to use Pydantic’s validation we can convert the class to a JSONschema and use this.</p><pre>structured_llm = llm.with_structured_output(schema=Joke.model_json_schema())</pre><p>A unvalidated dict is then returned, with no validation of the fields or types, which can be useful for evaluation or as input to post-hoc data fixing (see <a href="https://awstip.com/crafting-structured-json-responses-ensuring-consistent-output-from-any-llm-53213906b69f">this medium post for example</a>).</p><pre>{&#39;punchline&#39;: &#39;Why did the cat join a band?&#39;,<br> &#39;rating&#39;: &#39;8&#39;,<br> &#39;setup&#39;: &#39;Because it wanted to be the purr-cussionist!&#39;}</pre><h4>How to get structured output with Method 2: Output Parsers</h4><p>To get JSON output from models without support for the tool-use API the fallback is to specify the JSON schema directly in the prompt. Fortunately, LangChain has support to create the required prompt instructions automatically from the Pydantic schema:​</p><pre>prompt_user_format = ChatPromptTemplate.from_template(<br>    &quot;{input} \n{format_instructions}&quot;<br>).partial(format_instructions=parser.get_format_instructions())<br><br>structured_llm = prompt_user_format | llm_model | StrOutputParser()<br>structured_llm.invoke(&quot;Tell me a joke about cats&quot;)</pre><p>This is similar to the example from the LangChain documentation: <a href="https://python.langchain.com/docs/how_to/structured_output/#custom-parsing">How to return structured data from a model | 🦜️🔗 LangChain</a>; however, in the documentation they use system messages to contain the format instructions. In my experiments I find including the format specifications in the system message considerably less reliable.</p><p>Additionally, for the output parser method I enable JSON Mode (see Appendix 1) for supported backends as it improves the JSON quality, see the section below for more details.</p><h3>Errors in output formats</h3><p>As we know, LLMs are statistical (read random) models, and therefore there is no guarantee that they will return valid JSON, or if the do that the schema will conform to what we have requested. While OpenAI has features that can guarantee valid JSON, open source models currently do not have this functionaltiy and I have found often fail to return the correct schema.</p><p>When LangChain is given a Pydantic schema then Pydantic will be used to validate that returned data conforms to the expected schema. When it doesn’t conform to the schema a ValidationError will be returned. For example, Llama3.2 returns numeric fields as text with erroneous formatting, such as commas, and the validation fails as follows:</p><pre>ValidationError: 1 validation error for ArticleResponse1<br>number<br>  Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value=&#39;3,500,000&#39;, input_type=str]<br>    For further information visit &lt;https://errors.pydantic.dev/2.9/v/int_parsing&gt;</pre><p>LangChain offers functionality to fix errors with the OutputFixingParser which passes the problematic output back to the LLM to attempt to fix the issue. For example:</p><pre>try:<br>    output = parser.parse(output_raw.content)<br>except Exception as e:<br>    parser_fix = OutputFixingParser.from_llm(parser=parser, llm=llm_model)<br>    output = parser_fix.parse(output_raw.content)</pre><p>However, this involves calling the LLM again and feeding in the output, the original instructions, and the error message to try to get an improved output. This involves another LLM call with more context and may not resolve the problem. Therefore, it is best to get it right the first time.</p><h3>Experiment: Comparing JSON output using tool-use API and format instructions</h3><p>The main question I want to answer is how method 1 and 2 above compare for different complexities of schema across multiple LLMs. I’m interested in returning longer descriptive outputs to integrate with other computer software in a structured way. Therefore, I’m curious to create a test case that returns longer text and more complicated structures.</p><p>I’m going to evaluate and compare a number of different open source models to two versions of Anthropic’s Claude model. The models I’m testing are below, I also indicate if they support tool-use and also if they have a JSON mode.</p><p>I’ve also added a flag for the size of the model, both Claude models are large (<a href="https://www.reddit.com/r/ClaudeAI/comments/1c3tqlr/claude_3_model_size_speculation/">estimated size of Haiku is around 20B parameters, and Sonnet around 70B</a>) and the open source models are all small (less than 10B parameters).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*UU_YFkaORyBmv9W4QQJi6A.png" /><figcaption>List of models used in testing</figcaption></figure><h4>Problem setup</h4><p>For this test I’m going to start with a test task asking the LLM to write content for an popular scientific article for publishing and provide the response for different questions in a specific format.</p><p>The example prompt follows. Notice the length hint of less than 200 words, this is to constrain the outputs when the models ignore the formatting instructions (quite often) so they don’t have too long an output.</p><pre>You are a professional science writer tasked with responding to members<br>of the general public who write in asking questions about science. Write<br>an article responding to a writer’s question for publication in a science <br>magazine intended for a general readership with a high-school education. <br><br>You should write clearly and compellingly, include all relavent context,<br>and provide motivating stories where applicable.<br><br>Your response must be less than 200 words.<br><br>The question given to you is the following:<br>{question}</pre><p>To introduce some variety the {question} is replaced with one of 20 different science related questions, and we will have a experiment with 20 runs per configuration.</p><p>The prompt is created by adding a system prompt that specifies the output format if we are using output parsers or just the prompt above if we are using the structured output method.</p><h4>Output Formats</h4><p>To test the capabilities of the models I’ve created some increasingly complex Pydantic models to structure the output. With these I’m aiming to test longer textual structured outputs rather than short function parameters.</p><p>I’ve defined four example schema in increasing order of complexity as follows:</p><p><strong>Level 1 (Simple types)</strong></p><p>This is a simple flat structure with only string and integer fields. I’m expecting most models to handle this quite easily.</p><pre>class ArticleResponse1(BaseModel):<br>    &quot;&quot;&quot;Structured article for publication answering a reader&#39;s question.&quot;&quot;&quot;<br>    title: str = Field(description=&quot;Title of the article&quot;)<br>    answer: str = Field(<br>        description=&quot;Provide a detailed description of historical events to answer the question.&quot;<br>    )<br>    number: int = Field(<br>        description=&quot;A number relevant to the question.&quot;</pre><p><strong>Level 2 (Nested types)</strong></p><p>This gets more difficult as we are defining a custom type and using this in the actual response. A quick note that when converted to a JSON Schema the HistoricalEvent type is created then the two historical_event_x fields contain a link to this object. I expect that this is difficult for the LLM to parse.</p><pre>class HistoricalEvent(BaseModel):<br>    &quot;&quot;&quot;The year and explanation of a historical event.&quot;&quot;&quot;<br>    year: int = Field(description=&quot;The year of the historical event&quot;)<br>    description: str = Field(<br>        description=&quot;A clear description of what happened in this event&quot;<br>    )<br>class ArticleResponse2(BaseModel):<br>    &quot;&quot;&quot;Structured article for publication answering a reader&#39;s question.&quot;&quot;&quot;<br>    title: str = Field(description=&quot;Title of the article&quot;)<br>    historical_event_1: HistoricalEvent = Field(<br>        description=&quot;Provide a detailed description of one historical events to answer the question.&quot;<br>    )<br>    historical_event_2: HistoricalEvent = Field(<br>        description=&quot;Provide a detailed description of one historical events to answer the question.&quot;<br>    )</pre><p><strong>Level 3 (Lists)</strong></p><p>This is relatively straightforward, requiring a list of strings to be returned in addition to the title.</p><pre>class ArticleResponse3(BaseModel):<br>    &quot;&quot;&quot;Structured article for publication answering a reader&#39;s question.&quot;&quot;&quot;<br>    title: str = Field(description=&quot;Title of the article&quot;)<br>    further_questions: list[str] = Field(<br>        description=&quot;A list of related questions that may be of interest to the readers.&quot;<br>    )</pre><p><strong>Level 4 (List of custom types)</strong></p><p>This is the most complicated where a list of custom objects is returned. As above the custom historical event object is used is referenced in the list, creating a complicated look-up that is more difficult for the LLM.</p><pre>class ArticleResponse4(BaseModel):<br>    &quot;&quot;&quot;Structured article for publication answering a reader&#39;s question.&quot;&quot;&quot;<br>    title: str = Field(description=&quot;Title of the article&quot;)<br>    historical_timeline: list[HistoricalEvent] = Field(<br>        description=&quot;Provide a compelling account of the historical context of the question&quot;<br>    )</pre><h3>Results</h3><p>The table below shows the percent of LLM calls that gave valid JSON outputs and conformed with our requested schema for both method 1 (structured output with tool-use) and method 2 (using output parser instructions).</p><p>The open-source models are run twice for each question resulting in 40 executions per experiment. The Claude models are run once for each of 20 questions to keep the costs down.</p><p>The code to run the LLMs and calculate the results is <a href="https://github.com/adocherty/mastering-structured-output/blob/main/2-langchain-structured-output-evaluation.ipynb">available on GitHub</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/1*uiGAH6KckEy7JBfLR7GuVg.png" /><figcaption>The results for the percentage of outputs which were both valid JSON and matched the given schema. Note the approximate spread in multiple experiments is ±10% except for the red numbers where the spread over 3 experiments is given explicitly.</figcaption></figure><p>The results show that the latest Claude 3.5 Sonnet is hands down the most capable, able to output valid JSON for all schemas almost flawlessly for event the most complex schemas.</p><p>Nemotron-mini comes in next with excellent performance on JSON outputs. In fact Nemotron-mini is the best of the open source models and even beats Claude Haiku which is impressive given that Haiku is likely almost 10x bigger than Nemotron-mini.</p><p>The other open source models perform poorly, with Phi3 the worst and Llama3.2 not much better especially for complex schema. This is suprising as Llama3.2 is trained specifically for tool-use. Also interestingly the two methods are quite comparable: when a model works well to return JSON data in one format, it will return well structured JSON data using the other.</p><p>The results for Llama3.2 were not stable. I ran the experiment a couple of times and the results were within around 5–10% for all results except for Llama3.2 for Level 2 and Level 4 which showed a huge difference in results from run to run (between 10% and 60%). I’m currently unsure what the cause of this large variation was.</p><h3>Conclusions</h3><p>When I tried to get structured outputs from LLMs a few months ago I couldn’t get LangChain’s structured output to work. A lot has changed recently and now tool use and structured output mode is supported for a wide range of models.</p><p>I came into this thinking that it would be clear that using the structured output mode would improve the validity of JSON over providing the JSON schema in the prompt. However, I found that the two methods are comparable: when a model works well to return JSON data with one method, it tends to work well using the other.</p><p>Anthropic’s Claude excels at outputting JSON even with complex schemas. I haven’t included OpenAI models here; however, I expect that they will be competitive with Claude’s models and even out-compete them now they’ve released new methods to ensure the responses conform to a given schema.</p><p>Even when trained for tool-use most of the small open-weight models struggle to output complex data schemas. This makes good error-handling important when using structured outputs with these models. However, Nemotron performed suprisingly well for a model with only 4B parameters when using the tool-use API consistently returning JSON with the correct schema without errors.</p><p>In these experiments I’ve compared large commercial models from Anthropic with small open-weights models. It would be interesting to include larger open-weight models. In the future I aim to get up and running on OpenRouter to put larger open source models to the test.</p><h3>Appendix 1: JSON Mode</h3><p>Open AI strict mode as well as other JSON modes available in providers such as Llama.cpp are a form of constrained generation where the LLM is limited while decoding to produce sequences of tokens that are deemed valid. For more on the complexities of constrained generation <a href="http://arxiv.org/abs/2403.06988">this article</a> seems a good entry point, it’s not something I’ve investigated further.</p><p>LangChain supports a <a href="https://python.langchain.com/docs/concepts/structured_outputs/#json-mode">JSON mode</a> in some backends that limit the LLM to produce only valid JSON. In the table below I compare the output parsers with and without JSON mode enabled.</p><p>Enabling JSON mode showed some improvement in the validity of the JSON outputs. Where there was little or no improvement this was due to the JSON schema being incorrect. It appears that JSON mode reduces invalid JSON outputs, which helped Gemma2 and Phi3 produce valid JSON, while not affecting the adherence to the schema.</p><p>Finally, it it also notable that methods trained for tool-calling API such as Nemotron and Llama 3.2 did not improve much, but the other models have a large improvement with JSON Mode.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*JnuHyhYi-0g_2RW0WLfUKg.png" /><figcaption>Comparison of generation using Output Parsers with and without JSON mode.</figcaption></figure><p>See a list of backends with JSON mode support here: <a href="https://python.langchain.com/docs/integrations/chat/">https://python.langchain.com/docs/integrations/chat/</a>.</p><p><em>Note that when using Method 1 the </em><em>with_structured_output method </em><a href="https://python.langchain.com/docs/how_to/structured_output/#advanced-specifying-the-method-for-structuring-outputs"><em>takes a parameter that allows specifying using either tool-calling or JSON mode</em></a><em>. I don’t investigate the change in performance of this flag here, as I discovered it after I wrote the article!</em></p><h3>References</h3><p><strong>OpenAI</strong></p><p>For open models, Ollama supports JSON: <a href="https://pub.towardsai.net/openai-json-mode-vs-functions-92b15baa38d9">https://pub.towardsai.net/openai-json-mode-vs-functions-92b15baa38d9</a>.</p><p><strong>Open source options</strong></p><p><a href="https://github.com/noamgat/lm-format-enforcer">https://github.com/noamgat/lm-format-enforcer</a></p><p><a href="https://github.com/dottxt-ai/outlines">https://github.com/dottxt-ai/outlines</a> [see also the blog posts from dottxt: <a href="https://blog.dottxt.co/">https://blog.dottxt.co</a>]</p><p><strong>Anthropic’s Claude</strong></p><p>For Claude see tips on improving output consistency: <a href="https://docs.anthropic.com/en/docs/test-and-evaluate/strengthen-guardrails/increase-consistency">https://docs.anthropic.com/en/docs/test-and-evaluate/strengthen-guardrails/increase-consistency</a></p><p>And an example notebook to extract valid JSON: <a href="https://github.com/anthropics/anthropic-cookbook/blob/main/tool_use/extracting_structured_json.ipynb">https://github.com/anthropics/anthropic-cookbook/blob/main/tool_use/extracting_structured_json.ipynb</a></p><p>Also they reccomend using a tool call to return structured outputs: <a href="https://docs.anthropic.com/en/docs/build-with-claude/tool-use#json-mode">https://docs.anthropic.com/en/docs/build-with-claude/tool-use#json-mode</a></p><p><a href="https://github.com/anthropics/anthropic-cookbook/blob/main/tool_use/extracting_structured_json.ipynb">https://github.com/anthropics/anthropic-cookbook/blob/main/tool_use/extracting_structured_json.ipynb</a></p><p><strong>Miscellaneous articles on structured outputs</strong></p><p>Good overview of how to get structured outputs, and use pydantic to validate: <a href="https://medium.com/@azizbenothman76/pydantic-and-prompt-engineering-the-essentials-for-validating-language-model-outputs-e48553eb4a3b">https://medium.com/@azizbenothman76/pydantic-and-prompt-engineering-the-essentials-for-validating-language-model-outputs-e48553eb4a3b</a></p><p>Uses instructor for JSON outputsL <a href="https://medium.com/@pedro.aquino.se/how-to-get-structured-output-from-llms-applications-using-pydantic-and-instrutor-87d237c03073">https://medium.com/@pedro.aquino.se/how-to-get-structured-output-from-llms-applications-using-pydantic-and-instrutor-87d237c03073</a></p><p>How to use PydanticOutputParser (no tool calling): <a href="https://medium.com/@speaktoharisudhan/structured-outputs-from-llm-using-pydantic-1a36e6c3aa07">https://medium.com/@speaktoharisudhan/structured-outputs-from-llm-using-pydantic-1a36e6c3aa07</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be29fb6f6675" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Harnessing LLMs to manage my projects: Prompt Engineering (Part 2)]]></title>
            <link>https://medium.com/@docherty/harnessing-llms-to-manage-my-projects-prompt-engineeing-part-2-bfa5e02a31a9?source=rss-dba307069819------2</link>
            <guid isPermaLink="false">https://medium.com/p/bfa5e02a31a9</guid>
            <category><![CDATA[llm-applications]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[project-management]]></category>
            <dc:creator><![CDATA[Andrew Docherty]]></dc:creator>
            <pubDate>Thu, 10 Oct 2024 03:08:14 GMT</pubDate>
            <atom:updated>2025-06-20T00:25:49.107Z</atom:updated>
            <content:encoded><![CDATA[<p>In a <a href="https://medium.com/@docherty/can-llms-help-me-manage-my-projects-part-1-ee4342b1ca0a">previous blog post</a> I demonstrated that LLMs could take a brief project overview and create a detailed project description, and then break it down into individual tasks. Here I’m taking this idea and using some more advanced prompting techniques including prompt chaining and critique to fine-tune the process and gain greater control over the output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cdBuYwFAgQczzj6eRU8jOw.jpeg" /><figcaption>A robot project manager — created with ChatGPT4o</figcaption></figure><h4>Takeaways</h4><p>Through this project this I’ve learned the following:</p><ul><li>Getting the LLM to suggest improvements to human input and then using the suggestions in a new prompt can improve the output.</li><li>Prompting the LLM for output that is structured into specified sections works well to give results that align to our expectations.</li><li>Getting LLMs to give structured output that is machine readable is tricky: different models work better with different formats and there is no guarantee that the output will be valid. Claude and Gemma 2 work well with XML, Llama 3 not so much.</li><li>Having chained prompts not only improves the outcome noticeably but also gives you the opportunity to use and modify the intermediate outputs before feeding them to the next prompt.</li></ul><h3>Taking our Prompt to the Next Level</h3><p>Here I briefly introduce two prompt techniques that I’m going to use in the rest of the post: Prompt chaining and critique and refine. I’ll then show how they are used together to form the prompt workflow.</p><h4>Prompt chaining</h4><p>In the previous post I came up with a single prompt that took a brief project outline, created a detailed project description, and then broke it down into individual tasks. However, there’s no reason that we need to do both tasks in a single prompt.</p><p>When I think of planning a project myself, I would first write the goals, outcomes, and resources first and then break it down into tasks once that is clear. How about we get the LLM to follow this human-like approach? In LLM-land breaking a task into multiple subtasks is called prompt chaining (see details at <a href="https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/chain-prompts#example-multitenancy-strategy-review">Anthropic’s Docs</a> and <a href="https://www.promptingguide.ai/techniques/prompt_chaining">The Prompting Guide</a>).</p><p>For our system of taking a project description and breaking it down into tasks we present this visually as a graph of two prompts, one will expand the project outline into a structured description, this is then fed into the next prompt that will break the expanded description down into tasks. In later sections I’ll show that this has a number of advantages beyond improving the quality of the output. This is shown in the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/371/1*MWYF9Xt-sreVJnLcUrIsFw.png" /><figcaption>Two chained prompts going from a project outline to project description and finally the task list</figcaption></figure><h4>Critique and refine</h4><p>In the previous post another issue was when I left out critical information in the initial project outline the LLM hallucinates material to fill in this gap. As LLMs become more capable there’s capacity for them to ask for clarification, but this will only work with a chat interface. Furthermore currently they aren’t that great at asking clarification — for example, Llama3 when given a prompt to expand a project outline to a project description with the project outline entirely missing will hallucinate random projects rather than actually ask for more details.</p><p>My initial though was to critique the input outline and have the LLM give us feedback where the gaps are (where I’m assuming hallucinations may occur) and then use this feedback to fill in the critical gaps. In general I find that working with LLMs works well when I get them to critique my work, give me feedback and suggest improvements that I can take and use to refine what I’m writing.</p><p>This was my original idea for the feedback loop:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/284/1*zH-J8KszzrYr0Zow0f2VtQ.png" /><figcaption>The critique user feedback loop</figcaption></figure><p>However, I found that just feeding the suggested improvements along with the project outline into the next stage can improve the results without us having to do any actual work! However, there is a caveat: it seems to improves the results by creating examples for the content that is missing — and these may not be what we want. Nonetheless, I think it’s instructive how the LLM changes behaviour by addressing improvements that it has suggested itself.</p><p>This is very similar to the <a href="https://evjang.com/2023/03/26/self-reflection.html">Critique and Refine feedback loop</a> that is used to iteratively refine LLM outputs. The differences here are that the critique is on our input not the LLM output so there’s no iteration. We could perform this critique-and-refine on the output of Stage 2 (or any stage) but I don’t explore that here.</p><h4>New prompt workflow</h4><p>With this in mind let’s breaking down the project description and task breakdown into three stages that each have a separate call to the LLM:</p><p><strong>Stage 1:</strong> Critique outline to give suggested improvements;</p><p><strong>Stage 2:</strong> Generate detailed project description;</p><p><strong>Stage 3:</strong> Break project description down into tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/351/1*Q2OesX82i-p1SRXfDw16MQ.png" /><figcaption>Three chained prompts going from a project outline to project description and finally the task list</figcaption></figure><h4>Example Project Outline</h4><p>I expanded the outline of the project in the last section and removed the details of specific tools and added more details of the proof-of-concept nature of the project:</p><blockquote><strong>Overview:<br></strong><em>Build a simple proof-of-concept tool to use LLMs to take a brief project outline, produce a detailed project description with a specified structure, and finally break it down into tasks with enough information and clarity to be able to be useful. The project and tasks are then uploaded to a project database in Notion along with the relevant properties and a clear and comprehensive description for the project as well as each task. The project should have a simple UI where the project description is entered. The project is POC and does not need to support multiple users.</em></blockquote><blockquote><strong>Goals:<br><em>1. </em></strong><em>Design a system to generate a project description and tasks from a project outline<br>2. Build a simple POC UI where a user can enter the project outline and see the task breakdown before uploading to Notion<br>3. Evaluate the system somehow</em></blockquote><p>Next we’ll go through each of these phases, build up prompts for them, and see how they work on an example project outline.</p><h3>Stage 1: Critiquing the project outline</h3><p>When checking the feasibility of using the LLM for a the project breakdown we saw that when the project outline doesn’t specify enough then the breakdown changes significantly. I’m tasking the LLM to give critique in three areas: project objectives, evaluation methods, and research questions. Getting the LLM to give critique and suggested improvements in predefined categories is useful as it provides “guardrails” to the LLM and focus on what is important for me in the task.</p><p>Naturally, these categories would change depending on the kind of project. Potentially, we could choose different sets of categories for different projects, or they could even be generated by the LLM itself. For now I’m sticking with a fixed set of categories.</p><h4>Design the prompt</h4><p>We will design the prompt with the four category prompt design guide I explained in the first blog post.</p><p><strong>Role and context</strong></p><pre>You are a research project manager responsible developing proof of concept software for internal use.</pre><pre>Here is the project outline you will be working with:<br>&lt;project_outline&gt;<br>{{PROJECT_OUTLINE}}<br>&lt;/project_outline&gt;</pre><p><strong>Explain the task</strong></p><pre>You are tasked with giving constructive critique given the a project<br>outline above. This will help in better understanding the project scope<br>and help in planning its execution.<br><br>Return a critique of the project outline and suggestions for improvement.</pre><p><strong>Detail the task steps</strong></p><p>The task steps are as follows, this matches the structure below and also provides length guidelines for the size of the critique and the suggested improvements.</p><pre>Follow these guidelines:<br>1. Carefully read and analyze the project outline.<br>2. For each of the three areas (project objectives, evaluation methods,<br>   and research questions), do the following:<br>   a. Provide a brief comment on the strengths and/or<br>      weaknesses of this aspect of the project.<br>      Limit your response to under 100 words.<br>   b. Suggest a specific improvement that could enhance this aspect<br>      of the project.  Limit your response to under 100 words.<br>   c. Assign a score out of 10, where 1 is poor and 10 is excellent.</pre><p><strong>Specify the output format</strong></p><p>For Claude XML formatting is recommended as the model is trained to follow XML in it’s input and output. I have repeated structure for the XML sections in each category in the critique: for each category I ask the LLM to return the “critique”, “score”, and “improvement”.</p><pre>Present your output in the following XML format:</pre><pre>&lt;project_objectives&gt;<br>&lt;summary&gt;<br>[Are the research objectives specific, measurable, achievable,<br>relevant, and time-bound (SMART)?]<br>&lt;/summary&gt;<br>&lt;improvement&gt;<br>How can the project objectives be improved ?<br>&lt;/improvement&gt;<br>&lt;score&gt;<br>[A score of the clarity of the project objectives as an integer<br>between 0 and 10]<br>&lt;/score&gt;<br>&lt;/project_objectives&gt;</pre><pre>&lt;research_questions&gt;<br>&lt;summary&gt;<br>[Does the project outline clearly articulate a compelling<br> and focused research question?]<br>&lt;/summary&gt;<br>&lt;improvement&gt;<br>How can the research questions be improved?<br>&lt;/improvement&gt;<br>&lt;score&gt;<br>[A score of the quality of the research questions as an integer<br> between 0 and 10<br>&lt;/score&gt;<br>&lt;/research_questions&gt;</pre><pre>&lt;evaluation&gt;<br>&lt;summary&gt;<br>[How are the objectives of the project going to be evaluated,<br> for example for effectiveness and usability?]<br>&lt;/summary&gt;<br>&lt;improvement&gt;<br>[How can the evaluation of the project&#39;s objectives be improved?]<br>&lt;/improvement&gt;<br>&lt;score&gt;<br>[A score of the quality of the research questions as an integer<br> between 0 and 10]<br>&lt;/score&gt;<br>&lt;/evaluation&gt;</pre><p><strong>Reinforce important points</strong></p><p>Here I reinforce that I’d like constructive feedback and ask for output in valid XML. This helps the Llama 3 models with XML output (Claude doesn’t need this).</p><pre>Remember to be constructive and specific in your feedback. Your goal<br>is to help improve the research project by identifying opportunities<br>for enhancement. Your response must be in valid XML.</pre><h3>Example output</h3><p>Let’s try this prompt using <a href="http://claude.ai/">Claude.ai</a> with the project outline above. We have some harsh, but justified, critique! Note that as our output is structured as XML we can easily convert the XML to other formats, for example a markdown table (from code written by ChatGPT).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/1*cFo8QTvzZNRjRaDhtImXwg.png" /><figcaption>Critique output for the example project outline</figcaption></figure><p>We see that Claude does a pretty good job of giving a critique and some suggested improvements. Our worst category is evaluation and the LLM even pulls up our “evaluate the system somehow” throw-away line. I think it got us here!</p><p>Note that the scores aren’t calibrated at all but they should be <em>good enough</em> to use relatively and be a guide to what parts need improvement.</p><p><strong>Note on Llama 3.1 output</strong></p><p>Giving the same prompt to Llama 3.1 fails to close some XML tags and pads the XML outputs with natural language text outside the start and end tags. This is a known issue with LLMs and Claude is good at XML likely because it has been trained on XML data for this purpose (see LangChain docs on XML [REFERENCE]).</p><p>Adding extra cues to tell the model to return in XML format helps, specifically adding the following sentence to the end of the prompt: You must provide your response in valid XML. and changing Present your output in the following format:to Present your output strictly in the following XML format: helped and with them Llama 3.1 returns valid XML.</p><p>However, note there are no guarantees that most models will return valid XML. While there are methods that <a href="https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md">enforce LLM outputs to a formal grammar</a>, even <a href="https://github.com/IohannesArnold/xml_gbnf">this doesn’t guarantee valid XML</a> yet.</p><p>A better structured output format that’s supported by more models is JSON. I’ll explore this in another blog post soon, but for now we’ll stick with XML.</p><h3>Stage 2: Create a detailed project specification</h3><p>In this step we ask the LLM to create a detailed project specification from the project outline. Similar to the critique prompt, I have provided specific categories for the project description.</p><p>Guiding the LLM with a document structure improves provides some guardrails that can increase the usefulness of the output for the specific situation as well as providing a consistent format for all projects.</p><h4>Design the prompt</h4><p>A quick note is that some of the categories in this prompt structure match those in the critique stage intentionally.</p><p><strong>Role and context</strong></p><p>Again we have a similar role and the context now contains our project outline as well as the suggested improvements from the previous step.</p><p>If we don’t have the suggested improvements, we can remove the associated text from the prompt. The categories that I’ve chosen here would need to be changed for different kinds of projects such as personal projects, exploratory projects, software development projects, etc.</p><pre>You are a research project manager responsible developing proof of concept software for internal use.</pre><pre>Here is the brief project outline:<br>&lt;project_outline&gt;<br>{{PROJECT_OUTLINE}}<br>&lt;/project_outline&gt;</pre><pre>And here is the suggested improvement:<br>&lt;suggested_improvement&gt;<br>{{SUGGESTED_IMPROVEMENT}}<br>&lt;/suggested_improvement&gt;</pre><p><strong>Explain the task</strong></p><p>I’ve just explained the task as succinctly as I can. Again below if we do not have the suggested improvements I just remove that part of the text.</p><pre>Your are tasked with creating a comprehensive and well-structured project<br>description that addresses all key aspects of the research project given<br>the project outline and suggested improvements above.</pre><p><strong>Detail the task steps</strong></p><p>Here the instructions are less specific as I’ve chosen to put a lot of the details of what’s required in each section into the example format itself. I’m not sure if these descriptions are better in this section but for my own thinking I preferred to write them in the example format itself.</p><pre>Carefully analyze the provided information and expand on it to create<br>a detailed project specification. Follow these guidelines:<br>1. Carefully read and analyse the project outline<br>2. Carefully read the suggested improvements and use your expertise<br>   to fill in the gaps in the project given these suggestions<br>3. Consider the research problem, potential impact, evaluation methods,<br>   research questions, and project objectives.<br>4. Given the following format, complete each section following the<br>   instructions.</pre><p><strong>Specify the output format</strong></p><p>The output format is in XML again and we provide a lot of structure to guide the output we want. Like the critique stage, the categories we use here can change depending on the kind of project we are working with and what we find useful.</p><pre>Your expanded project specification must strictly adhere to the following XML format:</pre><pre>&lt;project_specification&gt;<br>&lt;name&gt;<br>[Provide a compelling name for the research project]<br>&lt;/name&gt;</pre><pre>&lt;summary&gt;<br>[Provide a short but comprehensive summary of the project.<br> Address the context, research questions, and potential impact.]<br>&lt;/problem&gt;</pre><pre>&lt;problem&gt;<br>[Provide a detailed description of the research problem the project<br> aims to solve. Expand on the initial description, considering the<br> context, background, and significance of the problem.]<br>&lt;/problem&gt;</pre><pre>&lt;impact&gt;<br>[Describe the potential contributions and changes the project aims<br> to have on personal processes. Consider both short-term and<br> long-term impacts, and how the project might influence efficiency,<br> productivity, or other relevant aspects.]<br>&lt;/impact&gt;</pre><pre>&lt;evaluation&gt;<br>[Provide clear and specific statements on how the project objectives<br> can be measured and the project output evaluated.<br> Include potential metrics, methodologies, or benchmarks that could be<br> used to assess the project&#39;s success.]<br>&lt;/evaluation&gt;</pre><pre>&lt;questions&gt;<br>[List at least 3-5 detailed research questions that the project should<br> answer. These questions should be specific, measurable, and directly<br> related to the project&#39;s goals.]<br>&lt;/questions&gt;</pre><pre>&lt;objectives&gt;<br>[List at least 3-5 concrete project objectives. These should be SMART<br> (Specific, Measurable, Achievable, Relevant, and Time-bound) objectives<br> that align with the research questions and overall project goals.]<br>&lt;/objectives&gt;<br>&lt;/project_specification&gt;</pre><p><strong>Reinforce important points</strong></p><p>Here I emphasize some important points that are orthogonal to the categories of the project and emphasize using the suggested improvements. Again, for Llama 3.1 the final valid XML sentence is important to steer the model to provide valid XML. I remove the penultimate sentence for the prompt without suggested improvements.</p><pre>Ensure that your expanded project specification is thorough, detailed,<br>and actionable research project. Consider the potential challenges and<br>interdependencies between different aspects of the project.<br>Ensure that you address the suggested improvements and incorporate them<br>into the overall specification.  Your response must be in valid XML.</pre><p>Let’s try feeding our project outline into this prompt, both with and without the suggested improvement from the previous step.</p><h3>Comparison with and without using the critique stage</h3><p>To test the prompts I ran them through both Claude and Llama3.1 models multiple times. I wanted to asses the diversity of responses (given the problem is under-specified and there’ll be multiple ways of filling all the details I’ve missed) and the consistency of formatting. Note for Claude I bumped the model temperature up to 0.3 to get more variety.</p><p>In the interests of space I’m not including the full output. The names proposed by the system using Claude were quite consistent, with the example project outline typically we get some variant of this boring but accurate title:</p><blockquote>LLM-Powered Project Breakdown and Task Management System (LPBTMS)</blockquote><p>The summary sections are almost identical between prompts that used the suggested improvements from the critique stage and those that didn’t. As we asked for criques only in the categories of “evaluation”, “questions”, and “objectives” these sections have the largest differences between the two prompts.</p><p>In particular the difference is clearest in the evaluation section. The output from the prompt that includes the suggested improvements are much more specific and measurable. It seems that having the critique stage suggest SMART objectives helps this stage use these specifics to come up with evaluations that are more specific too.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*1IpoG6CKuw9ynHt1pqRBiA.png" /><figcaption>Output of evaluation section from Stage 2 prompt without and with suggestions.</figcaption></figure><p>We see something similar with the objectives. They are more descriptive and contain more specific details with the suggestions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*iBRbPt5w6JuiEQvUzAw7gw.png" /><figcaption>Output of objectives section from Stage 2 prompt without and with suggestions.</figcaption></figure><p>For all the improvements, it’s likely we won’t want to keep the exact details of the objectives. What the LLM comes up with are more like placeholders — as it doesn’t know the resources we have available (I don’t have 5 project managers to ask or a database of broken-down projects — yet).</p><p>With that said, I find the objectives are better for having the LLM first critique the outline then incorporate the suggested improvements. They are specific and if we don’t keep them they are improved examples of how to write SMART objectives.</p><h3>Differences between Claude and Llama 3.1</h3><p>In general Claude gives much more consistent responses. I bumped up the temperature to 0.5 and still saw that the outputs were quite consistent between runs. Llama3.1 on the other had varies wildly between outputs both in content and XML formatting, sometimes even inventing new XML tags.</p><p>In a sense, Llama3.1 is more creative — and gives some great project names! Llama3.1 acts like the crazy cousin of the professional but boring Claude, it comes to meetings with great new ideas stuffed in colourful folders and drops half of them in it’s excitement.</p><p>A few of the names suggested by Llama 3.1, we see it’s not great with acronyms:</p><blockquote>“Automated Project Description Generation using LLMs: A Proof-of-Concept Tool”</blockquote><blockquote>“LLM-Driven Project Automation (LPAA) — A Proof-of-Concept Tool”</blockquote><blockquote>“LM-GPT: Leveraging Large-Scale Language Models for Project Management Efficiency”</blockquote><p>For example here are two different outputs of the objectives from Llama 3.1 using the same prompt:</p><p>Run 1:</p><pre>&lt;objectives&gt;<br>1. Design an LLM-based system that accurately generates detailed project descriptions from brief project outlines ( precision &gt; 80%, recall &gt; 90%).<br>2. Develop a user-friendly interface for the tool, ensuring that at least 85% of users find it easy to use and navigate.<br>...<br>&lt;/objectives&gt;</pre><p>Run 2:</p><pre>&lt;objectives&gt;<br>    &lt;objective&gt;P1.1: Design an LLM-driven system capable of accurately generating detailed project descriptions with specified structures from inputted project outlines.&lt;/objective&gt;<br>    &lt;objective&gt;P1.2: Develop a simple yet effective UI for AutoProject that allows users to easily enter their project outline and view the auto-generated project description and task breakdown before uploading it to Notion.&lt;/objective&gt;<br>..<br>  &lt;/objectives&gt;</pre><h3>Stage 3: Breaking down the project into tasks</h3><p>Now we have a detailed project description, including objectives and evaluation criteria. We want the LLM to take this project description and break down the work required into tasks.</p><p>As I was writing this prompt I was really unsure how best to specify the task directions, and what factors are important. I think this reflects my uncertainty how to do this myself! The initial suggestions I got from the Claude prompt writer were a good start, but I felt I needed something more specific to my own project breakdown. So, I asked ChatGPT to write a list of steps to break down the problem.</p><p>ChatGPT suggested the following steps (I’ve edited this list as quite a few were focussed on time estimates and dependencies, which I’m keeping out of this for the time being).</p><ol><li><strong>Define the Project Scope<br></strong>Clarify the overall goal of the project. Understand the big picture before breaking it into smaller parts. What is the project’s purpose, and what outcomes are expected?</li><li><strong>Identify Major Phases or Milestones<br></strong>Break the project down into key phases or milestones. These are large chunks of work that can be further divided into tasks. What needs to be accomplished at each stage for the project to progress?</li><li><strong>List Main Tasks for Each Phase<br></strong>For each phase, list the high-level tasks that need to be completed. Think in terms of what actions or deliverables are required to meet each milestone.</li><li><strong>Assign Resources<br></strong>Identify who or what is needed to complete each task. This could include team members, tools, or materials. Do you have the resources available when they are needed?</li><li><strong>Organize Tasks in a Sequence<br></strong>Arrange the tasks in the most logical order, considering dependencies and timelines. Which tasks should be prioritized based on the project schedule or critical dependencies?</li></ol><h3>Design the prompt</h3><p>Again, now with this list of steps and the four-part prompt template, I could write the following prompt pretty easily. I’ve taken the approach of breaking up the project into phases and then the phases into tasks, but I don’t actually surface the phases anywhere in the output. I’ll come back to this in the section “Task breakdown with explicit phases” below.</p><pre>You are a research project manager responsible developing proof of concept<br>software for internal use.</pre><pre>Here is the project specification you will be working with:</pre><pre>&lt;project_specification&gt;<br>{{PROJECT_SPECIFICATION}}<br>&lt;/project_specification&gt;</pre><pre>Your goal is to create a comprehensive task list that covers all aspects of the project while ensuring each task is clearly defined and actionable.</pre><pre>First, carefully read and analyze the project specification. Pay attention to the main objectives, deliverables, and any specific requirements or constraints mentioned.</pre><pre>Next, break down the project into a list of tasks. Follow these guidelines:<br>1. Define the Project Scope: Clarify the overall goal of the project. Consider the main objectives of the project.<br>2. Identify Major Phases**:**  Break the project down into key phases. These are large chunks of work that can be further divided into tasks. Consider what needs to be accomplished at each stage for the project to progress.<br>3. List Tasks for Each Phase: For each phase, list the high-level tasks that need to be completed. Define a clear outcome for each task: E.g., “Set up database schema with tables for X,” or “Build login screen UI.”<br>4. Organize Tasks in a Sequence: Arrange the tasks in the most logical order. Consider which tasks should be prioritized based on critical dependencies.<br>5. Consider Requirements: What programming languages, frameworks, libraries, and databases will be used? Which choices need to be made to perform each task?<br>6. Evaluation: Plan how the outcome of the task will be evaluated. How does this connect to the project&#39;s evaluation plan?</pre><pre>Output the phases first in the following XML format:<br>&lt;phases&gt;<br>&lt;phase&gt;<br>[Describe the first phase]<br>&lt;/phase&gt;</pre><pre>[Continue with additional phases...]<br>&lt;/phase&gt;</pre><pre>Then present your task list in strictly the following XML format:<br>&lt;task_list&gt;<br>&lt;name&gt;<br>[A short 3-5 word description of the task]<br>&lt;/name&gt;<br>&lt;task&gt;<br>&lt;description&gt;<br>[Describe the task clearly and in detail with all required context]<br>&lt;/description&gt;<br>&lt;outcome&gt;<br>[Clearly detail the outcome of the task]<br>&lt;/outcome&gt;<br>&lt;requirements&gt;<br>[Clearly detail the requirements of the task]<br>&lt;/requirements&gt;<br>&lt;priority&gt;<br>[The priority of the task, either low, medium, or high]<br>&lt;/priority&gt;<br>&lt;size&gt;<br>[The size of the task, either small, medium, or large]<br>&lt;/small&gt;<br>&lt;explanation&gt;<br>[Explain the rationale for giving the priority and size]<br>&lt;/explanation&gt;</pre><pre>[Continue with additional tasks...]<br>&lt;/task_list&gt;</pre><pre>Ensure that you include all the information specified in the format above for each task. If a particular field is not applicable for a task, you may indicate it as &quot;N/A&quot;. You must provide your response in valid XML.</pre><h3>Example outputs</h3><p>Now let’s test this prompt on the previously generated project specification. I’m using the project specification generated by Claude with the suggested improvements. I gave this both to Claude and to Llama 3.1 in order to do an apples-to-apples comparison of this step, rather than a comparison of the two models for the complete workflow. Again, as we are outputting structured data it’s easy to convert the XML to a table in Markdown format.</p><p>Initially Claude seemed to have missed that the intention is to build proof of concept code for internal use in a research project. It gave tasks such as “Comprehensive System Architecture Design” and “User Documentation and Knowledge Transfer” which are not things I’d expect to include for a proof of concept. On the whole the breakdown is, for lack of a better word, very corporate.</p><p><strong>Task breakdown with Claude (corporate version)</strong></p><p>For space reasons I’m only showing the name and description of the tasks</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*RNUYA2-waaGwPKJc0nSVWw.png" /><figcaption>Task breakdown from the project description produced by Stage 2.</figcaption></figure><p><strong>Task breakdown from Claude (independent researcher version)</strong></p><p>I’ve tried a few changes to the prompt to steer the LLM to produce more research-oriented tasks by changing the role that the LLM is given:</p><pre>You are an independent researcher developing proof of concept software<br>for individual use.</pre><p>However, this turned out to not change the output much at all. I also needed to add the following sentence at the end in order for the LLM to change it’s behaviour significantly. It seems the prompts to not include documentation, requirements gathering and system architecture were needed to push the LLM away from it’s happy place of corporate software development.</p><pre>Remember you are planning a project for an individual researcher,<br>therefore do not include large documentation, requirement gathering,<br>or system architecture tasks.</pre><p>With these changes, the output is more aligned with my expectations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*rZewT89OgDdnOZ_HeaNXSQ.png" /><figcaption>Task breakdown from the project description produced by Stage 2, with cues for the independent researcher.</figcaption></figure><p><strong>Task breakdown with explicit phases</strong></p><p>Notice in the prompt in the step-by-step instructions I gave included breaking the project into phases, then the phases into tasks. However, I never asked the LLM to output these phases, so it doesn’t have them in “working memory”. What happens when we ask the LLM to also output these phases, I can think of two was to do this:</p><ul><li>Output the phases in a “scratchpad” before the tasks;</li><li>Output the phase for each task as another field.</li></ul><p>I experimented with the following addition to the prompt to output the phases before the task list. This gives the LLM context for breaking down the tasks for each phase as we instructed it to. If we don’t output the phases the LLM can’t actually think about them (it has no internal short term memory at all; the outputs are only based on the input sting in total and the output tokens up to the current output position)</p><pre>Output the phases first in the following XML format:<br>&lt;phases&gt;<br>&lt;phase&gt;<br>[Describe the first phase]<br>&lt;/phase&gt;<br>[Continue with additional phases...]<br>&lt;/phase&gt;</pre><p>Even running this a few times I didn’t find a consistent enough difference between the task outputs for each prompt. I tried this both with the “corporate” prompts and the “individual researcher” prompt above and found adding the phase output did change the tasks output by the prompt but in conflicting ways. For the “corporate” prompt adding phases tended to result in multiple tasks covering the same thing in the phase-less prompt (for example, a single task UI Development became the two tasks Design UI and Implement UI). However, this wasn’t the case for the “individual researcher” prompt where having the phases in fact changed the UI to be a CLI interface (we didn’t specify that the “simple UI” in the project outline needed to be a GUI).</p><p>Therefore, I didn’t find that including the phases explicitly in the output consistently improved the task breakdown. In fact, I wonder if referencing the phases at all in the prompt helps or hinders the task breakdown. This reinforces the need for better evaluation tools if I’m going to refine the prompts any more than I have currently. I intend to explore how to evaluate the prompt outputs in the future.</p><h3>Conclusions</h3><p>Overall, I’m happy with the evolution of the prompts to get my project description. I haven’t tried it with too many projects so it’ll be important to build a system where I can adjust the prompt as I put in more projects.</p><p>I’ve found that Claude gives nice stable breakdowns and deals with XML tags really well. Llama3.1 is a little more creative and can deviate more easily from format specifications that you give it.</p><p>Chaining the prompt and getting the LLM to give suggested improvements for an input, and use these suggested improvements in the next step works surprisingly well. Potentially I could have used this technique at each step to give suggested improvements for each stage. I’m not sure how much this would help, and my intention with the critique stage was initially to improve the description myself. But why do the work when the LLM can do it for you?</p><p>As I have gone through this and read more literature out there on LLMs I’ve thought of many, many more techniques I could apply here. The difficult thing for me is to just go ahead with something that’s good enough rather than forever trying new techniques!</p><p>The prompt can definitely be improved, and I could have endless loops of adding this and that. I think that this will give limited improvements without a robust way to evaluate the system. As my intention here is to get something that works sooner rather than later, I’m not going to hold off as much as I can.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bfa5e02a31a9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Harnessing LLMs to manage my projects (Part 1)]]></title>
            <link>https://medium.com/@docherty/can-llms-help-me-manage-my-projects-part-1-ee4342b1ca0a?source=rss-dba307069819------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee4342b1ca0a</guid>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <dc:creator><![CDATA[Andrew Docherty]]></dc:creator>
            <pubDate>Tue, 24 Sep 2024 08:44:10 GMT</pubDate>
            <atom:updated>2024-10-10T01:33:19.056Z</atom:updated>
            <content:encoded><![CDATA[<h4>Exploring using LLMs for personal project management, and how to go from a simple project idea to well specified tasks</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XjynsTZkXQ0mdWhcTGDB6w.png" /><figcaption>Created with ChatGPT4o</figcaption></figure><p>I recently worked in a startup and found that while I love the excitement of a new project I find going from an exciting new project idea to a well thought out plan and actionable tasks is not my forte.</p><p>I’m now up-skilling in Large Language Models (LLMs) and the crazy speed at which they have advanced over the past few years is remarkable. While I do this I’m naturally thinking, <em>can LLMs help me overcome my biggest project management hurdles?</em></p><p>In this article, I’ll explain what I’ve done to design effective prompts to produce a structured project specification and break the project up into actionable tasks. Note that this is for short term personal research software projects and I’m not looking at the processes involved in team software development.</p><p>I have chosen to focus on my experiences with two well-known LLMs: <a href="https://claude.ai/">Anthropic’s Claude.ai</a> and <a href="https://ollama.com/library/llama3.1">Meta’s Llama 3.1</a>. I chose Claude for its <a href="https://www.anthropic.com/news/claudes-constitution">ethical AI stance</a> and cloud-based API, while Llama 3.1 offers the advantage of running locally on my own hardware (using <a href="https://ollama.com/">Ollama server</a>). I’ve also tested these prompts with other models such as ChatGPT with success, but have not focused tailored the prompts for other models.</p><h3>Feasibility of LLMs for Project Breakdown</h3><p>To gauge the feasibility of using LLMs for self-management I came up with a quick prompt to test their capabilities. My goal here is to give the LLM a simple project outline and to break down the project into actionable tasks.</p><p>I started out with this straightforward prompt:</p><pre>You are an assistant to a project manager for research software<br>development. Given the following project description, break down<br>the project into multiple tasks, arranged as a table including<br>columns &quot;Task Name&quot;, &quot;Size of Task&quot;, &quot;Priority&quot;, and &quot;Outcome.&quot;</pre><p>Next let’s add a project outline (for this very project, naturally), as follows:</p><pre>Project Outline:<br>Build a simple tool in Python that will help us in the project planning<br>process. Let&#39;s start with taking a project summary and feeding it to a<br>local Ollama server using LangChain, get structured output for a list<br>of tasks and potentially subtasks, and then upload this to Notion in <br>the correct format to fill out tasks in a project template.</pre><p>Here are the outputs from Claude and Llama3.1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/1*mvqJskSueWMK_2YIEUhMtQ.png" /><figcaption>Output of example prompt from Claude</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*WoV7r-1lzWKtg6vUmgXbHg.png" /><figcaption>Output of example prompt from Llama 3.1</figcaption></figure><p>This prompt seems pretty close to giving what I was expecting. It’s enough to convince me that it’s worth spending some more time to flesh out the concept and build prompts that can give a better project breakdown.</p><p>One key takeaway from this experiment is the importance of providing clear guidance to LLMs when generating tasks. While both LLMs generated useful breakdowns of the project outline, there are clear differences not just the steps but the outcomes of the project, for example Claude wants us to write a user guide and Llama 3.1 has no documentation task at all. This is not just a function of the different LLMs, different calls to the same LLM will provide different tasks.</p><p>This isn’t unexpected as the project is not well defined. Namely, do I want to build a utility software to be used my myself, an proof-of-concept internal tool for a large company, or a product for a startup? What software quality level are we aiming for? Who are the users of the software? What is the minimum functionality needed in this project? Without these details the models are giving us a single output sampled from a large space possible scenarios.</p><p>In the next section, I’ll improve the prompt and project outline, as well as return the data in a more machine readable form so that we can parse and upload to the task management system.</p><h3>Improving the prompt</h3><p>Now that I’ve found LLMs can breakdown projects in a useful way, let’s find out how to improve how we ask the model to do this task. This is prompt engineering: the art of crafting requests to get LLMs to do what you need effectively. There are many prompting strategies and tricks that claim to improve the output of LLMs. Before we discuss these and explore a better prompt, let’s start with a project overview that I can use to test the prompts.</p><h4>Example Project Outline</h4><p>I expanded the outline of the project in the last section and removed the details of specific tools and added more details of the proof-of-concept nature of the project:</p><blockquote><strong>Overview:<br></strong><em>Build a simple proof-of-concept tool to use LLMs to take a brief project outline, produce a detailed project description with a specified structure and finally break it down into tasks with enough information and clarity to be able to be useful. The project and tasks are then uploaded to a project database in Notion along with the relevant properties and a clear and comprehensive description for the project as well as each task. The project should have a simple UI where the project description is entered. The project does not need to support multiple users.<br></em><strong><em>Goals:<br> * </em></strong><em>Design a system to generate a project description and tasks from a project outline<br> * Build a simple POC UI where a user can enter the project outline and see the task breakdown before uploading to Notion<br> * Evaluate the project breakdowns appropriately</em></blockquote><h4>Prompt design principles</h4><p>To learn some of the basics of prompt engineering I started with the documentation from the companies providing the AI models themselves, and then read many general tutorials and blogs. I’ve collected some resources I found useful at the bottom of this post. Also I found Anthropic’s <a href="https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/prompt-generator">Prompt Generator</a> very useful to generates a prompt based on best practices from a simple prompt description.</p><p>With what I’ve learned so far, I’m starting to structure my prompts using the following four sections (inspired by the great <a href="https://towardsdatascience.com/how-i-won-singapores-gpt-4-prompt-engineering-competition-34c195a93d41">CO-STAR framework</a> that Sheila Tao used to win the Singapore’s GPT-4 Prompt Engineering competition)</p><ol><li><strong>Give the role and context:<br></strong>Give the LLM a clear role and context for the task</li><li><strong>Explain the task:<br></strong>What do you want the LLM to do? Be clear and provide important factors in the task.</li><li><strong>Break down the task into steps:<br></strong>Give a clear step-by-step breakdown of the task to be done</li><li><strong>Specify the output format:<br></strong>Give examples of the output format along with inline instructions for what goes in each section<br>For Claude, structure the input and output data using XML tags <a href="https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/use-xml-tags">[*]</a>. Other LLMs may work better with other formats.</li><li><strong>Reinforce important points:<br></strong>State or rephrase the most important aspects of the output. This I found important to get Llama 3.1 (and ChatGPT) to output valid XML for example.</li></ol><h4>Writing my next prompt</h4><p><strong>Role and context<br></strong>Let’s start by implementing a recommended role and some context for the test. Specifically, I want to implement the project ideas as a proof of concept software, without this the LLM is likely to provide objectives suited to software development of a product.</p><p>Also, we will give it the project outline itself here (the variable in double curly braces will be replaced by the actual description)</p><pre>You are a research project manager responsible for developing proof of<br>concept software for internal use.<br><br>Here is the project outline you will be working with:<br><br>&lt;project_outline&gt;<br>{{PROJECT_OUTLINE}}<br>&lt;/project_outline&gt;</pre><p><strong>Explain the task<br></strong>Next, I gave the LLM a clear description of the task to be done. Note that I have two tasks in this prompt, first writing a more detailed project specification and then breaking down the project into tasks. This may not give the best results and in the next section I’ll break this down into multiple prompts that do just one of these tasks.</p><pre>You are tasked with expanding a project outline into a detailed,<br>structured project description and breaking it down into specific<br>tasks. This will help in better understanding the project scope<br>and planning its execution. <br><br>Ensure that your expanded description is comprehensive, clear,<br>and aligned with the original project outline. The tasks should<br>be actionable.</pre><p><strong>Break down the task into steps<br></strong>How do I want the LLM to go about breaking down tasks? This step may be less clear, and I was helped by asking the LLM to provide a step-by-step breakdown of how to do the task and editing it.</p><pre>First, expand this outline into a structured project description. <br>Your goal is to create a comprehensive and well-structured project plan<br>that addresses all key aspects of the project.<br>Follow these guidelines:<br>1. Begin with an introduction that provides an overview of the project.<br>2. For each main point in the outline, create a detailed paragraph or<br>   section.<br>3. Elaborate on the ideas, providing context, rationale, and any<br>   relevant details.<br>4. Ensure logical flow and coherence between sections.<br>5. Conclude with a summary of the project&#39;s goals and expected outcomes.<br><br>Next, break down the project into specific tasks. For each task:<br>1. Provide a clear, actionable description.<br>2. Indicate which section of the project it relates to.<br>3. Estimate the time or effort required (e.g., hours, days, or complexity<br>   level).<br>4. Identify any dependencies or prerequisites.</pre><p><strong>Specify the output format<br></strong>Finally, I specify how the LLM should return the results. For Claude XML formatting is recommended as the model is trained to follow XML in it’s input and output. The XML format may not be appropriate for other LLMs, and other formatting could improve the results. I’ll explore this in another blog post but to keep things simple for now I’m going to use XML formatting exclusively.</p><pre>Present your output in the following format:<br><br>&lt;expanded_description&gt;<br>[Your expanded project description goes here]<br>&lt;/expanded_description&gt;<br><br>&lt;task_breakdown&gt;<br>1. [Task description] - [Related section] - [Time/Effort estimate] - [Dependencies, if any]<br>2. [Task description] - [Related section] - [Time/Effort estimate] - [Dependencies, if any]<br><br>[Continue for all tasks...]<br>&lt;/task_breakdown&gt;</pre><p><strong>Reinforce important points</strong></p><p>Llama3.1 often fails to return valid XML, the the XML tags are often not closed and the returned text has Markdown formatting. However, Claude <a href="https://python.langchain.com/docs/how_to/output_parser_xml/">is optimized for XML output</a>. By the following Llama 3.1 now returns valid XML.</p><pre>Your response must be in valid XML.</pre><p>However, there are no guarantees that the model will return valid XML. There are methods to ensure models return XML syntax by limiting their outputs to that specified by formal grammar, but this <a href="https://github.com/IohannesArnold/xml_gbnf">doesn’t guarantee the output is valid XML</a>.</p><h4>Results</h4><p>How do we do with this prompt? Here’s one output from Claude given the above prompt with our example project outline inserted.</p><pre>&lt;expanded_description&gt;<br>Project: LLM-Powered Project Management Tool</pre><pre>Introduction:<br>This project aims to develop a proof-of-concept (POC) tool that leverages Large Language Models (LLMs) to streamline the project planning process. The tool will take a brief project outline as input and generate a detailed project description with a specified structure, along with a comprehensive task breakdown. The generated project information will then be automatically uploaded to a Notion database, creating a centralized and organized project management system.</pre><pre>System Design:<br>The core of the system will be built around an LLM, which will be responsible for interpreting the project outline and generating the detailed description and tasks. The system will be designed to understand project management concepts and produce structured output that aligns with best practices in project planning. The LLM will be fine-tuned or prompted to generate output in a consistent format that can be easily parsed and organized.</pre><pre>User Interface:<br>A simple, user-friendly interface will be developed to allow users to input their project outlines. The interface will display the generated project description and task breakdown, allowing users to review and potentially edit the information before uploading it to Notion. The UI will be designed with simplicity in mind, focusing on essential functionality for this POC stage.</pre><pre>Notion Integration:<br>The system will integrate with Notion&#39;s API to automatically create new project entries in a designated database. Each project entry will include the detailed description, individual tasks, and relevant properties such as estimated time/effort and dependencies. The integration will ensure that all information is properly formatted and organized within the Notion workspace.</pre><pre>Evaluation:<br>To assess the effectiveness of the POC tool, an evaluation process will be implemented. This may include comparing the LLM-generated project plans with manually created ones, gathering user feedback on the accuracy and usefulness of the output, and analyzing the time saved in the project planning process. The evaluation will help identify areas for improvement and determine the potential value of further developing the tool.</pre><pre>Conclusion:<br>This proof-of-concept tool aims to demonstrate the potential of using LLMs in project management, specifically in the crucial phase of project planning and task breakdown. By automating these processes and integrating with Notion, the tool has the potential to significantly streamline project initialization and organization, saving time and improving consistency in project management practices.<br>&lt;/expanded_description&gt;</pre><pre>&lt;task_breakdown&gt;<br>1. Define LLM requirements and selection - System Design - 2 days - None<br>2. Design prompt engineering strategy - System Design - 3 days - Depends on #1<br>3. Implement LLM integration for project description generation - System Design - 5 days - Depends on #1, #2<br>4. Develop algorithm for task breakdown generation - System Design - 4 days - Depends on #3<br>5. Create simple user interface for project outline input - User Interface - 3 days - None<br>6. Implement result display functionality in UI - User Interface - 2 days - Depends on #5<br>7. Design Notion database structure for projects and tasks - Notion Integration - 1 day - None<br>8. Implement Notion API integration for data upload - Notion Integration - 3 days - Depends on #7<br>9. Develop data formatting and organization logic for Notion upload - Notion Integration - 2 days - Depends on #8<br>10. Design evaluation metrics and methodology - Evaluation - 2 days - None<br>11. Implement evaluation process - Evaluation - 3 days - Depends on #10<br>12. Conduct initial testing and debugging - All sections - 4 days - Depends on #1-#11<br>13. Gather user feedback and perform iterations - All sections - 5 days - Depends on #12<br>14. Document system architecture and usage instructions - All sections - 2 days - Depends on #13<br>15. Prepare final POC demonstration and presentation - All sections - 1 day - Depends on #14<br>&lt;/task_breakdown&gt;</pre><p>This is looking better, we have a detailed project specification and broken down tasks. However, I think we can do better. Some issues I’d like to address are:</p><ul><li>The sections in the project description change with different runs of the LLM and different project outlines.</li><li>The scope of the project is still hard to control; for example, there are still tasks for documentation even though we have stated that it’s a proof-of-concept software for a single user.</li><li>The tasks themselves have little detail on what is involved in them.</li><li>The output is not very structured, for my goal of uploading them to my project management system I want more explicit structure.</li></ul><h3>Conclusions</h3><p>LLMs can effectively generate detailed project descriptions from a simple project outline and break the project down into tasks. This will hopefully be valuable for me in the future with new projects, getting me over staring at a blank page and give me something to start with!</p><p>In the next blog post, I’ll delve deeper into prompt chaining and critique &amp; refinement techniques to further improve the quality and accuracy of LLM-generated project plans. I’ll also explore ways to better structure the output so that it can be uploaded to my project management tool.</p><h3>Useful prompt engineering resources</h3><p>A small selection of resources I’ve found useful:</p><p><strong>Documentation from AI companies</strong></p><ul><li><a href="https://platform.openai.com/docs/guides/completions/prompt-design">OpenAI API Prompting Guide</a></li><li>Anthropic’s Claude — <a href="https://github.com/anthropics/courses/tree/master/prompt_engineering_interactive_tutorial">Prompt engineering interactive tutorial</a></li><li>Anthropic’s Claude — <a href="https://github.com/anthropics/courses/tree/master/real_world_prompting">Real world prompting</a></li></ul><p><strong>General guides and courses on prompting</strong></p><ul><li><a href="https://www.promptingguide.ai/">PromptingGuide.ai</a> by Elvis Saravia</li><li><a href="https://learnprompting.org/">Learn Prompting</a></li><li><a href="https://www.deeplearning.ai/short-courses/chatgpt-prompt-engineering-for-developers/">Prompt Engineering for Developers</a> — <a href="http://deeplearning.ai/">DeepLearning.AI</a></li><li><a href="https://www.youtube.com/watch?v=lA_9GH5T7ec">Mastering Prompt Engineering Video</a></li></ul><p><strong>Other tutorials, blogs and guides</strong></p><ul><li><a href="https://github.com/promptslab/Awesome-Prompt-Engineering">Awesome Prompt Engineering GitHub</a></li><li><a href="https://www.vellum.ai/blog/prompt-engineering-tips-for-claude">https://www.vellum.ai/blog/prompt-engineering-tips-for-claude</a></li><li><a href="https://medium.com/the-modern-scientist/best-prompt-techniques-for-best-llm-responses-24d2ff4f6bca">https://medium.com/the-modern-scientist/best-prompt-techniques-for-best-llm-responses-24d2ff4f6bca</a></li><li><a href="https://github.com/dmatrix/genai-cookbook/blob/main/README.md">https://github.com/dmatrix/genai-cookbook/blob/main/README.md</a></li><li><a href="https://medium.com/p/34c195a93d41">https://medium.com/p/34c195a93d41</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee4342b1ca0a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>