<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mowafak Allaham on Medium]]></title>
        <description><![CDATA[Stories by Mowafak Allaham on Medium]]></description>
        <link>https://medium.com/@mowafak?source=rss-d4e55c30816d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*AlVjXy8m9poDudMX</url>
            <title>Stories by Mowafak Allaham on Medium</title>
            <link>https://medium.com/@mowafak?source=rss-d4e55c30816d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 00:39:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mowafak/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Query Your Data Using Natural Language with LlamaIndex?]]></title>
            <link>https://generative-ai-newsroom.com/how-to-query-your-data-using-natural-language-with-llamaindex-8988904e6d25?source=rss-d4e55c30816d------2</link>
            <guid isPermaLink="false">https://medium.com/p/8988904e6d25</guid>
            <category><![CDATA[data-journalism]]></category>
            <category><![CDATA[journalism]]></category>
            <category><![CDATA[ai-prompting]]></category>
            <category><![CDATA[llamaindex]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Mowafak Allaham]]></dc:creator>
            <pubDate>Wed, 07 Aug 2024 23:22:44 GMT</pubDate>
            <atom:updated>2024-08-08T13:17:09.191Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JM4uWWbi0z7GJi4x" /><figcaption><em>Image was generated using GPT-4o: “Generate an image for a blog post that provides an overview of an AI tool called “LLamaIndex” for data analysis in journalism.”</em></figcaption></figure><p>Many journalists can benefit by being able to extract insights and find answers to questions in the heaps of documents and data they may collect in the course of reporting. How can generative AI help facilitate this? Previous GAIN blog posts have illustrated how less technical journalists can analyze data by integrating <a href="https://generative-ai-newsroom.com/bringing-open-source-models-to-spreadsheets-c440fc4818b4">open source Large Language Models (LLMs) into Google Sheets</a> or by building an <a href="https://generative-ai-newsroom.com/can-llms-help-us-understand-data-49891c4e1771">agentic</a> solution that relies on the <a href="https://arxiv.org/abs/2210.03629">ReAct</a> (Reason + Act) prompting framework to answer questions written in natural language. But what if data journalists want to build a system for the newsroom to enable their colleagues with less technical expertise to query data from files using natural language, without the need for deep expertise in prompt engineering?</p><p>In this post, I present an approach to work with data using <a href="https://docs.llamaindex.ai/en/stable/">LlamaIndex</a> — an open source framework for connecting custom data sources to Large Language Models (LLMs). In particular, I outline a query pipeline that enables journalists to query semi-structured data, such as tabular data stored in .csv files, using natural language without using any database or query languages. Data journalists/scientists may re-configure this pipeline and potentially deploy it as a <a href="https://docs.llamaindex.ai/en/stable/understanding/putting_it_all_together/apps/fullstack_app_guide/">web application</a> in the newsroom. Accordingly, this post is oriented towards data journalists and scientists who are familiar with coding and data analysis. It aims to illustrate how LlamaIndex works and how it can be leveraged to set up a query pipeline as a tool for data analysis.</p><p>The following sections provide a brief description of the data source I chose to illustrate this post and how LlamaIndex works on this dataset. Later, I delve into the details of the Text-to-Pandas architecture, powered by a query pipeline, that enables the functionality of querying data in natural language. Finally, I conclude with a sample evaluation of LlamaIndex and its current limitations in the context of querying a .csvfile for data analysis.</p><h3><strong>Data Source</strong></h3><p>The query pipeline I will set up in the following sections can be leveraged in conjunction with LlamaIndex <a href="https://docs.llamaindex.ai/en/stable/module_guides/loading/connector/">Data Connectors</a> to query sources of unstructured data such as documents stored in Microsoft Word or PDF formats (as illustrated by these <a href="https://llamahub.ai/l/readers/llama-index-readers-file?from=readers">examples</a>), <a href="https://docs.llamaindex.ai/en/stable/examples/index_structs/struct_indices/SQLIndexDemo/">structured</a>, and semi-structured data (e.g, tabular data saved in a .csvfile).</p><p>To demonstrate the capabilities of LlamaIndex on semi-structured data, specifically data saved in.csvfiles, I chose a <a href="https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/VOQCHQ">dataset</a> describing the county-level votes for U.S presidential elections from 2000 to 2020 that was retrieved from <a href="https://electionlab.mit.edu/data#data">MIT Elections Data and Science Lab</a>. As we are approaching the U.S presidential elections in November, I reasoned that journalists might use this data for understanding previous voter turnout and patterns across states as they are covering the elections.</p><h3><strong>What is LlamaIndex?</strong></h3><p><a href="https://www.llamaindex.ai/open-source">LlamaIndex</a> is an open-source framework that can be used to <a href="https://docs.llamaindex.ai/en/stable/understanding/loading/loading/">ingest</a> (i.e., load data), <a href="https://docs.llamaindex.ai/en/stable/understanding/indexing/indexing/">index</a>, and query your data when building generative AI applications powered by LLMs such as <a href="https://docs.llamaindex.ai/en/stable/use_cases/q_and_a/">Retrieval Augmented Generation (RAG) pipelines</a> for question answering, <a href="https://docs.llamaindex.ai/en/stable/use_cases/chatbots/">chatbots</a>, and <a href="https://docs.llamaindex.ai/en/stable/use_cases/agents/">agents</a>. As mentioned earlier, one of the appealing and key features of LlamaIndex is its <a href="https://docs.llamaindex.ai/en/stable/module_guides/loading/connector/#llamahub">Data Connectors</a> that are hosted on <a href="https://llamahub.ai/?tab=readers">Llama Hub</a>. This feature offers a range of plug-and-play solutions to connect your application to various data sources such as <a href="https://docs.llamaindex.ai/en/stable/examples/data_connectors/TwitterDemo/">Twitter</a>, <a href="https://docs.llamaindex.ai/en/stable/examples/data_connectors/NotionDemo/">Notion</a>, <a href="https://docs.llamaindex.ai/en/stable/examples/data_connectors/DatabaseReaderDemo/">database</a>, or <a href="https://docs.llamaindex.ai/en/stable/examples/data_connectors/WebPageDemo/">webpage</a>s to build <a href="https://docs.llamaindex.ai/en/stable/examples/pipeline/query_pipeline/">query pipelines</a> — which I will elaborate on in the next section — that enables <em>chatting</em> with the content of these data sources in natural language. In the following section, I will demonstrate how query pipelines can be used to convert user queries from text to Pandas expressions in Python that can be executed on a dataframe containing voting data loaded from a .csv file.</p><h3><strong>Text-to-Pandas Architecture</strong></h3><p><a href="https://docs.llamaindex.ai/en/stable/examples/pipeline/query_pipeline/">Query pipelines</a> are objects used to create workflows that combine and connect different stages of a data pipeline in a sequential order that can be run end to end. This pipeline enables users to query various types of data sources using Large Language Models (LLMs) such as GPT-4o. The details of this pipeline can be broken down into three stages (as outlined below) and the corresponding prompts for some stages that GPT-4o will use to complete the task. Although LlamaIndex <a href="https://docs.llamaindex.ai/en/stable/examples/low_level/oss_ingestion_retrieval/">supports open-source models</a>, I decided to proceed with GPT-4o for its easy integration with LlamaIndex.</p><ul><li><strong>Stage 1: Convert questions to executable python expressions. </strong>This stage entails prompting GPT-4o to convert users queries to executable Python expressions that can run on the dataframe containing the loaded data from the .csv file. Depending on the type of query users are requesting, GPT-4o may return python expressions that could alter the data such as modifying values or removing rows and columns. Because the LLM has access to the eval() function in Python via LlamaIndex, this poses a <a href="https://medium.com/@murungaephantus/why-you-should-not-use-eval-and-exec-in-python-77d19345128c">security vulnerability</a> that LlamaIndex <a href="https://docs.llamaindex.ai/en/stable/examples/query_engine/pandas_query_engine/#:~:text=WARNING%3A%20This%20tool%20provides%20the%20LLM%20access%20to%20the%20eval%20function.%20Arbitrary%20code%20execution%20is%20possible%20on%20the%20machine%20running%20this%20tool.%20While%20some%20level%20of%20filtering%20is%20done%20on%20code%2C%20this%20tool%20is%20not%20recommended%20to%20be%20used%20in%20a%20production%20setting%20without%20heavy%20sandboxing%20or%20virtual%20machines.">warns</a> about due to its potential impact on both the data and the system where the pipeline is deployed. It’s recommended to use the query pipeline to query the data for insights instead of relying on it to transform data, and to run the code in a sandbox environment where there is no chance it could interfere with your primary systems. To convert user queries to Python expressions, I will use the following prompt in this stage and pass it to GPT-4o via the query pipeline setup:</li></ul><pre>pandas_prompt = (<br>    &quot;You are working with a pandas dataframe in Python.\n&quot;<br>    &quot;The name of the dataframe is `df`.\n&quot;<br>    &quot;This is the result of `print(df.head())`:\n&quot;<br>    &quot;{df_top_5_records}\n\n&quot;<br>    &quot;Follow these instructions:\n&quot;<br>    &quot;1. Convert the query to executable Python code using Pandas.\n&quot;<br>    &quot;2. The final line of code should be a Python expression that can be called with the `eval()` function.\n&quot;<br>    &quot;3. The code should represent a solution to the query.\n&quot;<br>    &quot;4. PRINT ONLY THE EXPRESSION.\n&quot;<br>    &quot;5. Do not quote the expression.\n&quot;<br>    &quot;Query: {user_query}\n\n&quot;<br>    &quot;Expression:&quot;<br>)</pre><p><em>Note: variables in the curly brackets will be replaced with actual values resulting from each step of the query pipeline. For more details, read about </em><a href="https://www.geeksforgeeks.org/formatted-string-literals-f-strings-python/"><em>f-strings</em></a><em> in Python.</em></p><p>This prompt includes four key pieces of information:</p><ul><li>df : is the name of the dataframe I would like GPT-4o to consistently use when referencing it, instead of making one up.</li><li>df_top_5_records: refers to the output of df.head() which includes the first 5 rows of a dataframe. Including this information in the prompt will inform GPT-4o about the column names, data types, and the overall formatting of the data.</li><li>Five instructions elaborating for GPT-4o the task of converting queries to python expressions as outlined in the prompt above.</li><li>user_query: refers to the submitted question by a user to query the data and receive an answer.</li></ul><p>To execute this stage and retrieve the Python expression from GPT-4o, I first had to create a prompt template object that has the aforementioned prompt along with its values. Prompt templates serve as an interface between user prompts and LlamaIndex modules. In addition, I had to instantiate an output parser to parse out the output from GPT-4o and retrieve a well formatted answer that can be used as an input to the following stage. To this end, I used the custom Pandas output parser supported by LlamaIndex which parses pandas syntax and makes it accessible for LLMs.</p><pre># Prompt template object containing pandas_prompt and the first five rows of a dataframe<br>pandas_prompt_object = PromptTemplate(pandas_prompt).partial_format(df_top_5_records=df.head(5))<br><br># Prompt template object containing response_synthesis_prompt  <br>response_synthesis_prompt = PromptTemplate(response_synthesis_prompt)<br><br># Output parser<br>pandas_output_parser = PandasInstructionParser(df)<br><br># Instantiate GPT-4<br>llm = OpenAI(model=&quot;gpt-4o&quot;,api_key=&quot;YOUR_API_KEY&quot;,max_new_tokens=300)</pre><p>Next, I created a query pipeline consisting of modules (i.e, components) and links (i.e. steps) controlling how modules are connected together.</p><pre>qp = QP(<br>modules={<br>&quot;input&quot;: InputComponent(), # parse and process inputs.<br>&quot;pandas_prompt&quot;: pandas_prompt, # prompt to convert user queries to Python expressions.<br>&quot;expression_to_llm&quot;: llm, # LLM used for processing user query.<br>&quot;pandas_output_parser&quot;: pandas_output_parser, # parser to render the output in a readable format.<br>&quot;response_synthesis_prompt&quot;: response_synthesis_prompt, # prompt to generate the final answer.<br>&quot;pandas_to_answer_llm&quot;: llm, # LLM synthesizing the findings from the dataframe into a single answer based on the data.<br>},<br>verbose=True,<br>)</pre><p>The modulesparameter of the query pipeline includes the necessary components (i.e., building blocks) for the pipeline to function. This includes defining the objects that will pass the input to the pipeline, the relevant prompts at each stage, and the LLM (GPT-4o) that will be used as part of the pipeline.</p><p>Linkis another data structure that LlamaIndex supports to connect modules together to create various stages of a query pipeline at a more granular level. A link describes the module that will be executed next, the input to the module, the prompt to be used in that module, and its output.</p><p>Returning back to the task of converting user queries into Python expressions, I created three links each performing a subtask. The first link passes a user query to query_to_python_expression prompt to populate the place holder user_query with the query entered by the user. Next, the query_to_python_expression prompt with the populated user query will be passed as an input to GPT-4o, per the second link. Lastly, the output of GPT-4o will be passed to pandas_output_parser to get a well formatted response that can be used in the subsequent stage pertaining to executing the Python expression on the dataframe.</p><pre>qp.add_links([<br>    # First chain of processing<br>    Link(&quot;input&quot;, &quot;pandas_prompt&quot;,dest_key=&quot;user_query&quot;),<br>    Link(&quot;pandas_prompt&quot;, &quot;expression_to_llm&quot;),<br>    Link(&quot;expression_to_llm&quot;, &quot;pandas_output_parser&quot;),<br>)]</pre><ul><li><strong>Stage 2: Executing python expressions on pandas dataframe. </strong>To execute the Python expression from Stage 1, I extended the set of links from the previous stage to include a few additional links that leverage the following prompt:</li></ul><pre># Response synthesis prompt (response_synthesis_prompt)<br>response_synthesis_prompt = (<br>    &quot;Given an input question, synthesize a response from the query results. Always include numbers in your answer to support your claims. \n&quot;<br>    &quot;Your response must ALWAYS include three components: 1) Response, 2) Python expression to query the dataframe, and 3) brief explanation of the Python expressoin. \n&quot;<br>    &quot;If the answer requires a full dataframe, display it in the results.&quot;<br>    &quot;Include in your answer the column names you used to filter data on \n&quot;<br>    &quot;Query: {python_expression}\n\n&quot;<br>    &quot;Pandas Instructions (optional):\n{pandas_instructions}\n\n&quot;<br>    &quot;Pandas Output: {pandas_output}\n\n&quot;<br>    &quot;Response: &quot;<br>)</pre><p>Similar to the previous stage, this prompt also has three key components: python_expressionreferring to the query obtained from the previous stage, pandas_instructions referencing the output after locally executing the Python expression on a pandas dataframe, and pandas_output referring to the final output that is synthesized by GPT-4o using the above prompt after populating all the aforementioned placeholder variables.</p><p>Using Links, the Python expression in stage 1 gets executed locally on the pandas dataframe based on the response_synthesis_prompt, and then the results are forwarded to the pandas_output field in the same prompt. The updated set of links looks like this:</p><pre>qp.add_links([<br>    # First chain of processing<br>    Link(&quot;input&quot;, &quot;pandas_prompt&quot;,dest_key=&quot;user_query&quot;),<br>    Link(&quot;pandas_prompt&quot;, &quot;expression_to_llm&quot;),<br>    Link(&quot;expression_to_llm&quot;, &quot;pandas_output_parser&quot;),<br><br>    # Links to response synthesis prompt<br>    Link(&quot;input&quot;, &quot;response_synthesis_prompt&quot;, dest_key=&quot;python_expression&quot;),<br>    Link(&quot;expression_to_llm&quot;, &quot;response_synthesis_prompt&quot;, dest_key=&quot;pandas_instructions&quot;),<br>    Link(&quot;pandas_output_parser&quot;, &quot;response_synthesis_prompt&quot;, dest_key=&quot;pandas_output&quot;)<br>])</pre><ul><li><strong>Stage 3: Synthesize a final answer. </strong>To synthesize the final answer I will pass the output from stage 2, which includes the response_synthesis_prompt with its populated values (from stage 2) and the result, to GPT-4o. To do so, I added one more link to account for this stage Link(&quot;response_synthesis_prompt&quot;,&quot;pandas_to_answer_llm&quot;) that forwards the populated response_synthesis_prompt to GPT-4o. Accordingly, the final set of links in the query pipeline is:</li></ul><pre>qp.add_links([<br>    # First chain of processing<br>    Link(&quot;input&quot;, &quot;pandas_prompt&quot;,dest_key=&quot;user_query&quot;),<br>    Link(&quot;pandas_prompt&quot;, &quot;expression_to_llm&quot;),<br>    Link(&quot;expression_to_llm&quot;, &quot;pandas_output_parser&quot;),<br><br>    # Links to response synthesis prompt<br>    Link(&quot;input&quot;, &quot;response_synthesis_prompt&quot;, dest_key=&quot;python_expression&quot;),<br>    Link(&quot;expression_to_llm&quot;, &quot;response_synthesis_prompt&quot;, dest_key=&quot;pandas_instructions&quot;),<br>    Link(&quot;pandas_output_parser&quot;, &quot;response_synthesis_prompt&quot;, dest_key=&quot;pandas_output&quot;),<br><br>    # Final link to the LLM<br>    Link(&quot;response_synthesis_prompt&quot;, &quot;pandas_to_answer_llm&quot;)<br>])</pre><p><strong>Executing the Query pipeline</strong></p><p>To put this query pipeline into action, I call the run method supported by LlamaIndex query pipelines. Under the hood, the user query is converted into a Python expression that gets executed on the pandas dataframe, LlamaIndex then parses the output, and then forwards the results to GPT-4o to synthesize a final answer.</p><pre>response = qp.run(query_str=&quot;Name the counties with the highest votes for democrats and compare them to counties that voted for republicans.&quot;)</pre><h3><strong>Evaluation</strong></h3><p>The utilization of LLMs for developing generative AI applications should always be paired with an evaluation process to confirm the validity of the generated output by the LLMs and assess the performance of the applications relying on these outputs. In the context of this blog post, it is recommended for data journalists intending to use LlamaIndex to 1) ensure the alignment of the intermediary steps between users’ queries and the generated expressions by the LLM, and 2) continuously evaluate whether the generated response by the LLM is factual with respect to the content of the file being queried.</p><p>To evaluate the generated Python expressions of the model and confirm that the query pipeline is executing the expression correctly, I instructed the LLM as part of the response_synthesis_prompt to always include in the output the query expression generated by the LLM and also provide an explanation of the generated expression so I understand the logic behind it. This is most useful when the editor has the requisite technical knowledge to understand and critique these expressions and explanations as needed to ensure the response is accurate.</p><p>On a high level, the LLM is behaving as expected by generating the relevant expression to my query that is also executable and well-explained. For instance, when querying the data using the query pipeline to “<em>Name the top counties that voted for a democratic presidential candidate in 2004</em>”, I get the following response:</p><pre><br>The top counties that voted for a Democratic president in 2004 were Los Angeles, Cook, Montgomery, Orange, and Jefferson. Los Angeles had the highest number of votes with 1,907,736.<br><br>Python expression:<br>df[(df[&#39;year&#39;] == 2004) &amp; (df[&#39;party&#39;] == &#39;DEMOCRAT&#39;)].groupby(&#39;county_name&#39;)[&#39;candidatevotes&#39;].sum().nlargest(5)<br><br>Explanation:<br>This Python expression filters the dataframe to include only data from the<br>year 2004 and where the party is Democrat. It then groups the data by county<br>name, sums the candidate votes, and returns the top 5 counties <br>with the largest total candidate votes.</pre><p>As shown in the response, the model correctly picks the right set of columns to generate an expression, LlamaIndex then executes the expression, and returns a synthesized response with the correct data. However, the model is prone to hallucinate when submitting complex queries which emphasizes the need to continuously validate the generated answers by AI. For example, when asking “<em>Who were the democratic candidates in 2004 and who were they competing with? and who won?”</em> the model generated an expression that correctly addresses the first part of the question df[(df[&#39;year&#39;]==2004)&amp;(df[&#39;party&#39;]==&#39;DEMOCRAT&#39;)][&#39;candidate].unique(). However, that same expression is not sufficient to address the remaining questions about “<em>who the democratic candidate was competing with and who won the elections”.</em> Yet, the LLM relied on its own knowledge from its training data to specify that George W. Bush was the Republican candidate and he won the 2004 election despite the absence of such information from the generated python expression by the LLM as shown below:</p><pre>In the 2004 election, the Democratic candidate was John Kerry. He was competing against the Republican candidate George W. Bush, who won the election.<br><br>Python expression:<br>df[(df[&#39;year&#39;] == 2004) &amp; (df[&#39;party&#39;] == &#39;DEMOCRAT&#39;)][&#39;candidate&#39;].unique()<br><br>Explanation: This Python expression filters the dataframe to only include rows <br>where the year is 2004 and the party is Democrat. <br>It then selects the unique values in the &#39;candidate&#39; column for these filtered rows.</pre><p>In light of these examples, it is recommended for data journalists to have an evaluation criteria that is grounded in assessing the relevance of the generated Python expression in the final answer with respect to their original queries. This may also entail some prompt engineering by refining the phrasing of the query submitted to the pipeline so the LLM can correctly parse the request. Furthermore, assuming that the generated expression by the LLM is relevant, it remains crucial to assess the factuality of the answer. This can simply be verified by running the resulting Python expression on the dataframe to confirm whether the LLM is leveraging the right sample of data to conclude the answer. Without such evaluations it becomes challenging to determine the efficacy of emerging tools, such as LlamaIndex, that are integrated with AI and their applications in the newsroom.</p><h3><strong>Limitations</strong></h3><p>By extensively testing the query pipeline across various queries that vary in complexity, I noticed that the model may occasionally make up column names from its knowledge base instead of the data file. One such case was evident when requesting to “<em>Compare the voting pattern between the republican and democratic candidate in the last election in 2020.</em>” The LLM responded with “<em>There is no data available in the dataset for the last election in 2020 to compare the voting pattern between the republican and democratic candidate</em>.” even though the data was there. This issue was due to the model hallucinating the formatting for the names of “JOE BIDEN” and “DONALD TRUMP” instead of preserving the way these names are formatted in the data file as “JOSEPH R BIDEN JR” and “DONALD J TRUMP”, respectively. Including unique values in the pandas_prompt in Stage 1 for each column in the dataset could mitigate against this issue.</p><p>Python expression:</p><pre>df_2020 = df[df[&#39;year&#39;] == 2020]<br>winner_2020 = df_2020.groupby([&#39;state&#39;, &#39;candidate&#39;])[&#39;candidatevotes&#39;].sum().idxmax()[1]<br>df_2020[(df_2020[&#39;candidate&#39;].isin([&#39;JOE BIDEN&#39;, &#39;DONALD TRUMP&#39;])) &amp; (df_2020[&#39;candidate&#39;] == winner_2020)]<br><br>Explanation:<br>The Python expression filters the dataset to only include data from the year 2020.<br>It then identifies the winning candidate in each state based on the candidate with<br>the highest total votes. Finally, it attempts to display the details of the winning candidate <br>(either JOE BIDEN or DONALD TRUMP) in the last election in 2020, but since there is no data available,<br>the output is an empty DataFrame.</pre><p>In addition, I observed that LlamaIndex doesn’t support complex queries that would require multi-indexed dataframes. For instance, re-running the previous query with the adjusted candidate names resulted in the following LlamaIndex error: “<em>can’t unstack multi-indexed data</em>” . This creates a disadvantage for adopting LlamaIndex for more complex data analysis use cases. One potential way to resolve these performance bottlenecks, but requires technical expertise, is to leverage <a href="https://platform.openai.com/docs/guides/function-calling">function calling</a> so the LLM can make explicit calls to functions that perform multi-indexing of the dataframe and use the resulting dataframe to execute user queries to generate the final answer. This solution enables more control over the implementation of the intermediate steps in comparison to LlamaIndex. This issue could also be addressed in future <a href="https://github.com/run-llama/llama_index/releases">releases</a> by LlamaIndex. Accordingly, it is recommended for data journalists/scientists to regularly check the releases page of the LlamaIndex GitHub <a href="https://github.com/run-llama/llama_index">repository</a> for bug fixes and feature enhancements.</p><h3><strong>Closing Thoughts</strong></h3><p>Open source AI tools such as LlamaIndex present an opportunity for journalists to engage with the open source community and potentially request feature <a href="https://github.com/run-llama/llama_index/issues?q=is%3Aopen+is%3Aissue+label%3Aenhancement">enhancements</a> for these tools to account for use cases in journalism. As generative AI tools continue to evolve, we invite data journalists to consider how these tools can enhance reporting and boost productivity for journalists in the newsroom. Could this approach be useful, for example, to index and query a large corpus of documents in an investigation? Or to query transcripts or reporting notes? Whether you are seeking an in-depth analysis of your data or further exploring it to unlock fresh insights, these tools could contribute new ways of unlocking the value of data for journalists.</p><p>—</p><p><em>The data and code for this blog post are available on a </em><a href="https://colab.research.google.com/drive/11eAZSklQoiXeEVPm6kxElL8SSPhyNGCT?usp=sharing"><em>Notebook</em></a><em> hosted on Google Colab.</em></p><p><em>Acknowledgements: The author would like to thank </em><a href="https://medium.com/@ndiakopoulos"><em>Dr. Nick Diakopoulos</em></a><em> for his helpful feedback.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8988904e6d25" width="1" height="1" alt=""><hr><p><a href="https://generative-ai-newsroom.com/how-to-query-your-data-using-natural-language-with-llamaindex-8988904e6d25">How to Query Your Data Using Natural Language with LlamaIndex?</a> was originally published in <a href="https://generative-ai-newsroom.com/">Generative AI in the Newsroom</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Prompting Techniques and Best Practices for Journalists]]></title>
            <link>https://generative-ai-newsroom.com/prompting-techniques-and-best-practices-for-journalists-6fa806cde8ec?source=rss-d4e55c30816d------2</link>
            <guid isPermaLink="false">https://medium.com/p/6fa806cde8ec</guid>
            <category><![CDATA[ai-prompting]]></category>
            <category><![CDATA[journalism]]></category>
            <category><![CDATA[news]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <dc:creator><![CDATA[Mowafak Allaham]]></dc:creator>
            <pubDate>Tue, 18 Jun 2024 18:52:53 GMT</pubDate>
            <atom:updated>2024-06-18T18:52:53.227Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WbIG_TfNkfdvCia7" /><figcaption><em>Image is generated by GPT-4o using the following prompt: “Generate an image in a modern vintage form of an AI guide that enables a machine to work”</em></figcaption></figure><p><em>Note:</em> This post was co-authored with: Michael Crystal, Mona Gomaa, and <a href="https://medium.com/@ndiakopoulos">Nick Diakopoulos</a></p><p>A recent <a href="https://www.researchgate.net/publication/379668724_Generative_AI_in_Journalism_The_Evolution_of_Newswork_and_Ethics_in_a_Generative_Information_Ecosystem">survey</a> from the Associated Press revealed a predominant interest in using generative AI for tasks relevant to content production such as information gathering, data analysis, multimedia content production (e.g., generating images), text summarization, translation, and transcription. Clearly, large language models (LLMs) have the potential to upend how journalists create content for news media, but in order to capture their value journalists need effective ways to adapt these models for various journalistic use cases. One way to do this is by training models for news tasks, though as one of us has recently <a href="https://generative-ai-newsroom.com/why-should-news-organizations-not-build-an-llm-c074396eec3b">argued</a>, that could be costly. Another path that is easier to get started with (though which can also require a great deal of skill and time investment) is <em>prompt engineering</em>.</p><p>In this article we describe some best practices for how LLMs can be controlled using different prompting techniques to elicit the desired outputs for tasks in journalism. We first provide some background context on prompt engineering before we elaborate on a few different prompting techniques, providing illustrative examples in news production. We close with some ideas about prompt validation and output quality evaluation.</p><h3>What is Prompt Engineering?</h3><p>Prompt engineering is an emerging communication technique between users and LLMs that is used to craft questions and instructions to elicit a desired response from LLMs. While prompt engineering appears to be straightforward on the surface, it requires domain expertise in different prompting techniques to fully reap the benefits of LLMs. For instance, <a href="https://www.promptingguide.ai/techniques">this guide</a> lists 17 different approaches to prompting, some of which are rather structured and involved. Moreover, different models may require <a href="https://huggingface.co/blog/llama2#how-to-prompt-llama-2">different prompt formats</a> or techniques to get the best performance.</p><h3>Characteristics of an Effective Prompt</h3><p>To achieve better results from LLMs, all prompts should be expressed in full sentences and communicate clearly with specific language the set of instructions a model needs to follow (and sometimes <em>not </em>follow) when completing a task. In addition, instructing the model <a href="https://platform.openai.com/docs/guides/prompt-engineering/strategy-split-complex-tasks-into-simpler-subtasks">to work out its own steps or solution</a> before it reaches a conclusion is <a href="https://proceedings.neurips.cc/paper_files/paper/2022/hash/8bb0d291acd4acf06ef112099c16f326-Abstract-Conference.html">proven to enhance the model’s capability to generate better outputs</a>, especially for analytical tasks that require reasoning. Another key characteristic of a good prompt is scoping the prompt for a specific task. Prompting for a very broad and open-ended task (e.g. “Produce an online news site”) will not be as effective as breaking down the tasks and sub-tasks of that overall goal to tackle them individually. In order to minimize the possibility of models diverging from their given instructions, OpenAI’s prompting guide <a href="https://platform.openai.com/docs/guides/prompt-engineering/split-complex-tasks-into-simpler-subtasks">recommends </a>that complex tasks are split into subtasks each having their own prompt.</p><p>In general, a good prompt is structured to include the prompt <em>text </em>and<em> context</em> (see section 3 of Joe Amditis’ <a href="https://jamditis.notion.site/Beginner-s-prompt-handbook-ChatGPT-for-local-news-publishers-15d1f07d5b194265a41fdd42955679b4">Beginner’s Prompt Handbook</a>)<em>. </em>However, depending on the task, this structure can be extended to include <em>constraints</em> (e.g., specifying the length of generated text), <em>tools</em> such as accessing external services or data sources (e.g., Yahoo Finance API), or <em>additional instructions</em> (e.g., specifying tone or writing style). For instance, for tasks that require factual information, it is better for prompts to include a trustworthy reference text in its context to ground the output.</p><p>One thing to be aware of in terms of responsible prompting is that you may need to be careful about including confidential or personally identifiable information (PII) as part of prompt context if such context is being shared with third-party AI platforms (e.g. OpenAI or Anthropic). To address the PII issue you can replace specific information with generic anonymized information (e.g. “Walter Smith” gets substituted with “&lt;NAME&gt;” in the prompt) and then after you receive a response from the service you can substitute the original name back, if needed. Another approach is to use models that are locally hosted so that prompt context is not being shared with outside parties.</p><p>Some models also support the use of <em>system prompts</em> which can set the overall stage for how a system responds in terms of tone, style, or to stay grounded in the material included in the context. For instance you might include “Your responses are always grounded in the specific document provided to you.” as a system prompt to encourage the model to only operate on the information included in a document you provide in the prompt as context. As with non-system prompts however, it’s important to remember that the model may not always follow the instructions to the letter and so checking outputs is still necessary.</p><p>Next, we discuss how these characteristics can be applied to different prompting techniques.</p><h3>An Overview of Key Prompting Techniques</h3><h4><strong>Zero-shot Prompting</strong></h4><p>Zero-shot prompting is a technique that entails instructing the model to complete a task without providing it with examples or demonstrations of the task or its output (i.e. “zero” in zero-shot refers to zero examples provided). This type of prompting can be useful for tasks that require ideation (e.g., brainstorming interview questions as illustrated in Figure 1), content generation (e.g., drafts for headlines), and translation. However, this technique requires many iterations over the base prompt to include the necessary details to reach the desired output. This could be attributed to the statistical nature of LLMs that makes it difficult to consistently predict the model’s exact responses and format when completing a task.</p><p>For use cases that require factual answers such as a summary of the latest news, zero-shot prompting has the limitation that, in the absence of other context, it relies only on the training data of the model to provide its response. Accordingly, whenever zero-shot prompting is used with minimal instructions, journalists need to be particularly wary about the model’s generated output because it may only reflect the knowledge present in the model’s training data which could be outdated. One way to ensure the text generated by a model is relevant to the task at hand is to include relevant context in the prompt such as the leading paragraph of an article when asking the model to generate engaging headlines based on that excerpt. In some cases, state of the art models such as GPT-4o can also a zero-shot prompt and then also access up-to-date sources by browsing the internet to provide context (Figures 2A &amp; 2B).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-Z8wZMQmWHukrfZx" /><figcaption>Figure 1. In this example, we ask the model to give us a question for an interview we are conducting with the Editor in Chief (EIC) of the Chicago Tribune. We provide details about the content of the interview, but not any details about the format of the question. This allows the model to decide how to structure the question.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0E1GrfRM5-GLbLn3" /><figcaption>Figure 2A. Example GPT4-o output demonstrating how the model behavior is not fully inline with the user requests. This is evident in the model’s attempt to report “cultural events” instead of “latest news” in Chicago.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zghJJWTo9n8ZfwRw" /><figcaption>Figure 2B. Example of GPT-4o output when writing an effective prompt that includes a request for the model to “provide links to sources”. In contrast to Figure 2A, the generated text from this prompt resulted in more informative output that includes a mix of events and local news from three local sources.</figcaption></figure><h4><strong>Few-shot Prompting</strong></h4><p>For use cases that require additional details, structure, and formatted response to the prompts, few-shot prompting provides an alternative to zero-shot prompting. Few-shot prompting builds on top of zero-shot prompting by including examples of the task or the formatting of the output as part of the prompt context (as illustrated in Figure 3). This enables the model to recognize the desired pattern and behavior that it should reflect in the generated text. For instance, journalists can use a few-shot prompt to classify users’ comments to news articles by providing the LLM with a few examples of negative and positive comments along with their corresponding labels. This could enable journalists to have an accessible way of understanding how the audience is responding to a news article. Similarly, few-shot prompting can be useful for information extraction (e.g., extracting named entities like people or places) or categorization (e.g. labeling content according to a schema) use cases. The main challenge for effective few-shot prompting is the development of examples that demonstrate to the model how to complete a task so it can ideally generalize across those examples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/0*UYqI3g5yOE7wMZga" /><figcaption>Figure 3. An example illustrating few-shot prompting by including four examples in the prompt context and how they helped inform the model of how we want to classify positive versus negative comments. As shown, the model accurately classifies the three comments provided to it.</figcaption></figure><h4><strong>Chain-of-Thought Prompting (CoT)</strong></h4><p><a href="https://arxiv.org/abs/2201.11903">Chain of Thought</a> prompting techniques enable models to “reason” about complex tasks through intermediary steps. However, it is not a mutually exclusive technique to the other prompting techniques we have described thus far. In its most basic form you can append “let’s think step by step” to any prompt including the zero and few shot prompts. The reason this works is that, before tackling the task, the model will output context which can further help ground the output. This prompting technique has proven to be useful for analytical tasks such as answering quantitative questions from text. It can also be combined with few-shot prompting to get better results on more complex tasks that require reasoning before responding.</p><p>As Figures 4A and 4B illustrate, instructing the model to think step-by-step using zero-shot prompting combined with CoT results in a more refined answer that includes details about the model’s reasoning steps compared to using only zero-shot prompting on the same task. The intermediate output has the additional benefit of offering some visibility for people to understand how the model arrived at the final output it did. One thing to keep in mind, though, especially when using an API to prompt a third party model, is that since CoT prompting produces more (intermediate) output tokens, the cost could be higher.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/0*XDJOyOoLhOYH-aXR" /><figcaption>Figure 4A. Example of generating a headline for an article using the leading paragraphs of the <a href="https://www.whitehouse.gov/ostp/ai-bill-of-rights/">AI Bill of Rights</a> using zero-shot prompting.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/0*jM628ovLaStrsgvF" /><figcaption>Figure 4B. Example of using CoT to generate an article headline about the AI Bill of Rights. In contrast to 4A, the model outputs the series of steps that are grounded in the excerpt of text from the bill to generate the headline. In addition, this title appears to be more specific than the one shown in 4A as it includes the Biden Administration as a named entity and it describes bias as a specific dimension the bill of rights is intended to overcome.</figcaption></figure><h3>Effective Quality Control in Prompting</h3><p>There are two basic approaches to effectively control the quality of prompts: (1) validating the prompts ahead of time to establish their robustness, and (2) assessing outputs required in task-specific use cases according to various quality criteria.</p><h4>Prompt Validation</h4><p>While writing prompts is perhaps the easiest way to get started with tapping into the power of LLMs, it can take a lot of iteration and time to refine a prompt so that it’s producing high quality outputs that you might be comfortable incorporating into your larger workflows. In this sense it can be helpful to think about the validation process for developing a prompt: <em>How do you know if it’s an effective prompt?</em></p><p>We recommend an approach requiring two datasets that you need to put together for each task you want to target: one to assess the performance of the prompt as it is developed — the <em>development</em> set — and another separate one — the <em>test</em> set — that you use to evaluate the prompt only after you’ve optimized it as much as possible on the development set. Ideally, both are random samples so there’s no underlying selection bias in either. Having two sets has the effect of helping you understand whether the prompt you developed and refined to work well with the development set also generalizes to the test set and still works well there. The larger and more varied your development and test sets, the more confident you can be that your prompt is working well across a range of conditions. Take note, however, every time you swap out a model, or a provider updates their model, you will need to re-run this validation!</p><p>As part of the prompt validation process, it is also important to account for the statistical variation of LLMs in generating the text corresponding to each iteration of the prompts. While the examples included in this blogpost were generated using the ChatGPT user interface, we encourage the use of OpenAI’s <a href="https://platform.openai.com/playground/chat?models=gpt-4o">Playground </a>when developing and validating prompts as it gives users the ability to control several parameters that are useful for this validation.</p><p>Probably the most important parameter to be aware of is <em>temperature, </em>which<em> </em>controls the level of randomness vs. determinism in the generated text<em>. </em>LLMs work by generating one token at a time, and a higher temperature gives the model more leeway to select from a larger number of next tokens. For OpenAI’s models this value defaults to 1 with a range from 0 to 2. For tasks that should be tightly grounded in the context provided as part of the prompt (e.g. ensuring a generated summary doesn’t stray from an underlying document) you should set the temperature closer to zero. Alternatively, in use cases that may require more “creative” or “verbose” text, such as brainstorming or headline ideation, setting the temperature value closer to 2 should result in a higher variation of generated texts for the same prompt. Of course, the higher the temperature, the more variance there will be in outputs and thus you will need to run more trials to assess validity.</p><p>Prompt validation is one of the under-estimated complexities of prompt engineering that must be accounted for before using the crafted prompts for a specific task at scale. At the same time, once validated in a systematic way, you can feel more confident sharing prompts within your organization and so reduce overlapping effort from everyone in your newsroom coming up with their own version. In addition, documenting the development process of prompts, by establishing benchmark development and test datasets will contribute towards more transparent, reproducible, and scalable prompt engineering and validation practices across tasks and newsrooms.</p><h4>Evaluating the Quality of AI-generated Content</h4><p>By now, many folks using LLMs are aware that they are prone to hallucinate — potentially making up information that is not grounded in reality due to the statistical nature of the way they produce outputs. If this is not accounted for in journalism, it could have ramifications for the integrity and trust in the news.</p><p>While <a href="https://jamditis.notion.site/Beginner-s-prompt-handbook-ChatGPT-for-local-news-publishers-15d1f07d5b194265a41fdd42955679b4#f1003fbd803f43f785d8f49503447200">evaluation criteria are likely to vary based on the task</a> the LLMs are used for, in general we suggest that journalists might minimally consider dimensions pertaining to information <em>accuracy</em> and <em>bias, </em>including<em> </em>both<em> </em>political and representational (over-representing or under-represented groups) bias. Additional qualitative dimensions with respect to the language of the generated text can also be accounted for in the evaluation process such as <em>coherence</em> (Does the generated text have any incomplete or unclear sentences?), <em>granularity</em> (How detailed is the language of the generated text?), and <em>relevance</em> (How relevant is the generated text to the request described in the prompt?). These are of course just a few of the many suggestions for criteria that you might evaluate, and we encourage the news industry to develop more task-specific criteria to evaluate the success of LLM outputs as well. One possible way forward might be for journalists to collaborate with technologists or social scientists to establish robust and trustworthy evaluation metrics and criteria across various news-related tasks.</p><p>But even while we recommend such evaluations as indicators of whether a prompt <em>tends</em> to produce higher quality output in general, it is nonetheless prudent from a responsible use point of view to maintain a human in the loop for many use cases. This is particularly important for dimensions like accuracy when the outputs of models might be presented directly to the audience.</p><h3>Honing Your Prompting Further</h3><p>As the capabilities of LLMs continue to advance, we recommend that you periodically check prompting guides and libraries relevant to third party models so you are up-to-date with the latest prompting techniques and their applications. Below we list a few resources you may want to bookmark as additional references for prompting:</p><ul><li><a href="https://platform.openai.com/docs/guides/prompt-engineering">Prompt Engineering — OpenAI</a></li><li><a href="https://platform.openai.com/examples">Prompt Examples — OpenAI</a></li><li><a href="https://docs.anthropic.com/en/docs/intro-to-prompting">Intro to prompting — Anthropic</a></li><li><a href="https://docs.anthropic.com/en/prompt-library/library">Prompt Library — Anthropic</a></li><li><a href="https://jamditis.notion.site/Beginner-s-prompt-handbook-ChatGPT-for-local-news-publishers-15d1f07d5b194265a41fdd42955679b4">Beginner’s prompt handbook: ChatGPT for local news publishers</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6fa806cde8ec" width="1" height="1" alt=""><hr><p><a href="https://generative-ai-newsroom.com/prompting-techniques-and-best-practices-for-journalists-6fa806cde8ec">Prompting Techniques and Best Practices for Journalists</a> was originally published in <a href="https://generative-ai-newsroom.com/">Generative AI in the Newsroom</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Should News Organizations (Not) Build an LLM?]]></title>
            <link>https://generative-ai-newsroom.com/why-should-news-organizations-not-build-an-llm-c074396eec3b?source=rss-d4e55c30816d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c074396eec3b</guid>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[journalism]]></category>
            <dc:creator><![CDATA[Mowafak Allaham]]></dc:creator>
            <pubDate>Thu, 04 Apr 2024 17:43:13 GMT</pubDate>
            <atom:updated>2024-04-04T17:43:13.689Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SJkt4On8UGAj0FkWKfuaKg.jpeg" /><figcaption><em>Image was generated using OpenAI GPT-4 chat interface using the following prompt: “Generate an image for news organization using AI. Do not focus the image on people but rather on technology. Keep the image innovative and modern with a vintage twist. Include in the image computers on a desk and make items on the desk more sophisticated”.</em></figcaption></figure><p>Integrating Large Language Models (LLMs) into the newsroom has the potential to unlock a myriad of opportunities for news organizations in tasks relevant to content creation and editing, as well as news gathering and distribution. But as newsrooms continue to explore the avenues and prospects for harnessing LLMs, a question arises around the strategic and competitive use of the technology: should news organizations strive to train their <em>own</em> LLMs?</p><p>In this post I argue that news organizations (especially those with limited resources) that use prompt engineering, fine-tuning, and retrieval augmented generation (RAG) to enhance their productivity and offerings will be strategically better off than if they train their own LLMs from scratch. I will first lay out the cost calculus for engineering and deploying your own model, and then I’ll elaborate the benefits and trade-offs of these other techniques for leveraging the value from generative AI.</p><h3><strong>Training LLMs from scratch could be a costly decision</strong></h3><p>Building and training LLMs from scratch is challenging due to the need for large datasets, extensive computing resources, and specialized talent to develop and train these models. For instance, the computing resources needed to train <a href="https://arxiv.org/abs/2303.17564">BloombergGPT</a> — an <a href="https://www.youtube.com/watch?v=byCe7-c84d4">LLM for finance</a> — is estimated to have cost approximately <a href="https://www.linkedin.com/posts/rajistics_what-it-takes-to-build-bloomberggpt-and-activity-7089609531961348096-iIAs/">$1M</a>. While the cost of serving such a model is not public information, the infrastructure required to serve even such a moderate-size model, irrespective of the number of users, is probably not <a href="https://medium.com/emalpha/the-economics-of-large-language-models-2671985b621c">cheap</a>, and is likely somewhere in the six figures. In addition, the ethical considerations around building a responsible model, such as ensuring fairness, privacy, and transparency, while sourcing ethical data, require dedicated attention and resources that could divert the focus of news organizations from their core journalistic tasks and reporting.</p><p>Optimizing an LLM’s performance with respect to the amount of compute resources required for training remains a <a href="https://arxiv.org/html/2402.04291v1">work in progress</a>. Rushing forward to train an LLM without a detailed cost-benefit analysis is likely to end up costing news organizations hefty amounts of money that may not yield a high return on investment. Since the development of BloombergGPT, in March 2023, smaller and more capable open source model architectures (such as <a href="https://mistral.ai/technology/#models">Mistral Models</a>) are becoming publicly available which present a competitive alternative to other large, costly, and proprietary models.</p><p>News organizations may instead want to consider the strategic advantages of utilizing third-party models that are accessible via API endpoints. This can reduce infrastructure costs while also ensuring access to state-of-the-art models, and even facilitate versatility in terms of quickly swapping models. For instance, news organizations could deploy the open-source and quite performant Mistral-7B model via the <a href="https://ui.endpoints.huggingface.co/">HuggingFace Inference Endpoint</a> on a single A10G GPU for $1.3 per hour for experimentation purposes. They could then decide to switch to<a href="https://huggingface.co/google/gemma-7b-it"> Gemma-7B</a> from Google at no cost while still paying the same amount for compute, allowing for rapid iteration and testing of different models.</p><p>Accordingly, News organizations exploring the potentials of prompt engineering, fine-tuning, and retrieval-augmented generation (RAG) may have a competitive cost advantage and application development agility, possibly achieving a faster return on investment through the use of readily available models (e.g., GPT-4 or Claude) via API or inference endpoints (e.g., Mistral-7B deployed via HuggingFace Inference Endpoint) for their applications.</p><h3><strong>What is prompt engineering?</strong></h3><p>Prompt engineering is an emerging communication technique between users and LLMs that is used to craft questions and instructions to elicit a desired response from LLMs. While prompt engineering appears to be straightforward on the surface, it requires domain expertise in different prompting techniques to fully reap the benefits of LLMs. For instance, <a href="https://www.promptingguide.ai/techniques">this guide</a> lists 17 different approaches to prompting, some of which are rather structured and involved. And different models may require <a href="https://huggingface.co/blog/llama2#how-to-prompt-llama-2">different prompt formats</a> or tricks to get the best performance. Yet prompt engineering is still the fastest way to get information from a general purpose LLM (at least, one that is already tuned to behave like a chat assistant similar to ChatGPT) without modifying its architecture or retraining it.</p><p>You can refer to the <a href="https://cloud.google.com/vertex-ai/docs/generative-ai/learn/introduction-prompt-design">Introduction to prompt design</a> documentation guide to learn about how to create prompts that elicit the desired output from Google’s LLMs. OpenAI offers a similar <a href="https://platform.openai.com/docs/guides/prompt-engineering">prompting guide</a> to use its models effectively. Or, <a href="https://journaliststoolbox.ai/prompt-writing/">Journalist’s ToolBox</a> offers useful prompting resources that are more oriented towards use cases in journalism.</p><h3><strong>What is Retrieval Augmented Generation (RAG)?</strong></h3><p>While prompt engineering is a very powerful and resource efficient way to generate desired content, the knowledge for many LLMs is capped by the cut-off date of their training data. For example, GPT-4 has a cut off training date of <a href="https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo">December 2023</a>. In other words, without merging GPT-4’s knowledge with the information available online, the model won’t be able to infer the latest updates in the world. News organizations can build their own cost-efficient RAG systems using externally hosted LLMs (such as GPT-4 or Claude) or internally hosted open source models (e.g., Mistral-7B) to enable journalists and users to sift through and converse with a large corpus of archival documents, knowledge bases, or reporting material similar to the <a href="https://www.theverge.com/2024/3/23/24106296/ask-ft-generative-ai-chatbot-answers-tool">Financial Times AI </a>chatbot. RAG services can also be multi-modal. Using multi-modal open source vector databases such as <a href="https://weaviate.io/">Weaviate</a>, users can query and retrieve audio, video, and text data in natural language.</p><p>Overall, RAG allows LLMs to access real-time information (via connecting and retrieving information from the internet) or domain-specific knowledge (e.g., archival data) from a specific set of sources. This capability can potentially enable journalists to generate answers to questions that are grounded in a curated set of factual and up-to-date information, enhancing the accuracy of the LLM.</p><h3><strong>When to fine-tune an LLM?</strong></h3><p>News organizations interested in specializing pre-trained LLMs (e.g., GPT-3.5) for specific applications and tasks, such as reflecting a specific writing style in generated text, should consider fine-tuning. Fine-tuning LLMs eliminates the need for constant prompt engineering to get the desired output by instead curating a dataset that closely mirrors the nature of a specific task. For instance, perhaps your organization has a specific style or tone used in headlines that you would like a model to be able to mimic. By curating a dataset of articles and headlines you could then fine-tuning a model to be able to produce such headlines without requiring a user to know how to prompt the model in any specific ways.</p><p>While fine-tuning offers the advantage of a model with specialized and tailored responses and it reduces the prompting expertise and knowledge burden for end-users, the process of fine-tuning could potentially be expensive depending on the compute and data resources that are required for a particular task. However, it will surely be a much cheaper option than training an LLM from scratch. Fine-tuning LLMs also requires constant monitoring and qualitative evaluation for potential model drifts.</p><p>OpenAI offers a great <a href="https://platform.openai.com/docs/guides/fine-tuning/when-to-use-fine-tuning">guide</a> that walks through the necessary steps of customizing LLMs for your application. In addition, cloud service providers such as Google and Amazon also offer users the ability to fine-tune LLMs via their platforms <a href="https://cloud.google.com/vertex-ai/docs/generative-ai/models/tune-models#supervised-tuning">Vertex AI</a> and <a href="https://aws.amazon.com/blogs/aws/customize-models-in-amazon-bedrock-with-your-own-data-using-fine-tuning-and-continued-pre-training/">Bedrock</a>, respectively.</p><h3><strong>Which method to pick from?</strong></h3><p>Prompt Engineering offers rapid adaptability to tasks in the newsroom with low computational overhead and technical complexity. It requires human expertise to craft prompts but does not require compute resources for inference especially for model endpoints offered by model providers such as OpenAI and Anthropic.</p><p>Retrieval Augmented Generation (RAG) extends an LLMs’ capacity to incorporate real-time or external data for more factual responses. Although RAG does not require training or fine-tuning LLMs, storing knowledge bases from which the LLMs are fetching information may incur some cost as the knowledge base increases in size.</p><p>Fine-tuning, on the other hand, provides high specialization for task-specific responses, requires a careful selection of the dataset for fine-tuning, and involves moderate computational and technical resources.</p><h3><strong>In Closing</strong></h3><p>Based on these various factors I would generally recommend for news organizations to explore the use case in which LLMs can be integrated to their workflows through prompting and possibly fine-tuning third-party models for their tasks. This is often going to be preferable to grappling with expensive infrastructure costs needed to train and deploy models that can potentially be outdated and less efficient in the near future. Until the infrastructure costs become cheaper and training LLMs become more accessible, I would not recommend for news organizations to build their own LLMs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c074396eec3b" width="1" height="1" alt=""><hr><p><a href="https://generative-ai-newsroom.com/why-should-news-organizations-not-build-an-llm-c074396eec3b">Why Should News Organizations (Not) Build an LLM?</a> was originally published in <a href="https://generative-ai-newsroom.com/">Generative AI in the Newsroom</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>