<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alex Rich, PhD on Medium]]></title>
        <description><![CDATA[Stories by Alex Rich, PhD on Medium]]></description>
        <link>https://medium.com/@alex_44314?source=rss-b74886233fb7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*EglBGSd08F7aeF7xOMAcMA.jpeg</url>
            <title>Stories by Alex Rich, PhD on Medium</title>
            <link>https://medium.com/@alex_44314?source=rss-b74886233fb7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 00:11:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@alex_44314/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[LLM Batch Processing: half price AI for a custom news feed]]></title>
            <link>https://medium.com/@alex_44314/llm-batch-processing-half-price-ai-for-a-custom-news-feed-1fb89d4400d4?source=rss-b74886233fb7------2</link>
            <guid isPermaLink="false">https://medium.com/p/1fb89d4400d4</guid>
            <category><![CDATA[noam-chomsky]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[newspapers]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[batch-processing]]></category>
            <dc:creator><![CDATA[Alex Rich, PhD]]></dc:creator>
            <pubDate>Mon, 27 Jan 2025 15:44:32 GMT</pubDate>
            <atom:updated>2025-01-27T16:11:20.131Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1007/1*ybTfGAD9P7G_ISHhSjYSwQ.jpeg" /><figcaption>Image generated with DALL-E</figcaption></figure><p>OpenAI’s batch processing API is a great way to process Large Language Model (LLM) jobs with their latest models including gpt-4o and gpt-4o-mini at half price when your use case can accept an up to 24 hour processing time. The interface is pretty straightforward and allows you to specify a standardized output format.</p><p><strong>Why this project:</strong></p><p>This article covers a small part of a bigger project. If there’s interest, I’ll write more about the rest of the project soon.</p><p>Many years ago, Noam Chomsky was asked how he managed to stay so informed on elements of global events that aren’t reported in US-based news outlets.</p><blockquote>“I certainty don’t find all the information I use on my own. The fact is there are a lot of people around the world who are in a similar position and we share information together. A good deal of my time is actually spent just clipping newspapers and periodicals and professional journals and photocopying them to send to people, and they do the same for me…and the result is that I can easily get to know more than people in the CIA or in any academic research center, <strong>mainly because I have smart agents (and not dumb agents), and they know what’s important and can dig things out</strong>…”</blockquote><p>I don’t have an international network of newspaper clipping friends, but I want a global perspective that’s not dependent on the editorial staff of any one news outlet. Fortunately, LLMs offer some unique capabilities in that area.</p><p>I think of an LLM like a very sophisticated transmission in a car…they’re not magical Star-Trek-teleporters, they’re not even the engine…but they can be pretty good at converting fuzzy inputs into (mostly) reliable standardized outputs. That’s perfect for helping me sift through a few thousand news articles each day.</p><p>I’ll be using RSS feeds from 15–20 international news outlets as the raw input for the project (Thank you, <a href="https://en.wikipedia.org/wiki/Aaron_Swartz">Aaron Swartz</a>).</p><p><strong>What the OpenAI batch API will be used for:</strong></p><p>There are two core problems that LLMs can help solve for this project:</p><ol><li><strong>Categorizing articles:</strong> I want the end result of this pipeline to be a custom news feed that gives me factual information and investigative journalism with a minimum of opinion and no stories about sports, entertainment, or anecdotal narratives. The LLM can categorize articles to support that.</li><li><strong>Topic extraction: </strong>Some news outlets include the topics covered in the story in a list included in the RSS feed. Others do not. I want to use the topics as features of the machine learning model that predicts my level of interest in a story, which means I need to use LLMs to extract a list of topics for articles whose feeds do not include them.</li></ol><p>For the sake of brevity, I’ll focus on the first use case in this article. Remember, the results will be imperfect. LLM applications are non-deterministic, which makes for challenges in use cases that require precise outputs. They’re perfect for “hunting” type use cases where it would be cost prohibitive to manually accomplish a task and some fuzziness in results is acceptable.</p><p><strong>Using the OpenAI Batch API:</strong></p><p>OpenAI’s <a href="https://platform.openai.com/docs/api-reference/batch">Batch API</a> is pretty straightforward to use, has a solid <a href="https://help.openai.com/en/articles/9197833-batch-api-faq">FAQ</a>, and is included in an easy to follow example in the <a href="https://cookbook.openai.com/examples/batch_processing">OpenAI cookbook</a>.</p><p>I’ll be following that design pattern and using gpt-4o-mini (which at time of writing is $0.075 per million input tokens and $0.30 per million output tokens with batch processing). <em>It’s worth noting that OpenAI’s top of the line gpt-4o model is $10/M output tokens in synchronous calls, so the 50% discount for batch processing can be much more financially significant with complex tasks.</em></p><p>First, get an OpenAI API key and add it to a .env file in your workspace for the OpenAI library to pull from.</p><p>Then make a system prompt and test it with the synchronous API call to make sure it doesn&#39;t return an error.</p><pre>#.env file<br>OPENAI_API_KEY = &#39;&lt;your API key&gt;&#39;</pre><pre># Python file<br>import pandas as pd<br>from openai import OpenAI<br>import json<br>import pydantic<br><br>client = OpenAI()<br><br><br>categorize_system_prompt = &#39;&#39;&#39;<br>Your goal is to categorize news stories as &#39;factual&#39;,&#39;opinion&#39;,&#39;investigative journalism&#39;,&#39;sports&#39;,&#39;entertainment&#39;, &#39;anecdotal story&#39;, &#39;other&#39;. You will output a json object in the following format:<br><br>{<br>    category : string // A single category selected from options: &#39;factual&#39;,&#39;opinion&#39;,&#39;investigative journalism&#39;,&#39;sports&#39;,&#39;entertainment&#39;, &#39;anecdotal story&#39;, or &#39;other&#39;.<br>}<br><br>Stories can only belong to one category, so select the category that best fits the text you are given.  Return only the single option you select in a lower case string.<br>&#39;&#39;&#39;<br><br># Test prompt:<br><br>def get_categories(title):<br>    response = client.chat.completions.create(<br>    model=&quot;gpt-4o-mini&quot;,<br>    temperature=0.0,<br>    # This is to enable JSON mode, making sure responses are valid json objects<br>    response_format={ <br>        &quot;type&quot;: &quot;json_object&quot;<br>    },<br>    messages=[<br>        {<br>            &quot;role&quot;: &quot;system&quot;,<br>            &quot;content&quot;: categorize_system_prompt<br>        },<br>        {<br>            &quot;role&quot;: &quot;user&quot;,<br>            &quot;content&quot;: title<br>        }<br>    ],<br>    )<br><br>    return response.choices[0].message.content<br><br>title = &quot;Lengthy legal fight lands blow on Sun’s publisher but shows even princes have to settle&quot;<br>print(json.loads(get_categories(title))) # result: {&#39;category&#39;: &#39;investigative journalism&#39;}...not perfect, but good enough</pre><p>Once that works, create a list of tasks using that prompt. In this case I have a list of article objects with a ‘title’ attribute that will be send to the LLM for categorizing.</p><pre><br>articles = [{&#39;key&#39;:1,&#39;title&#39;:&#39;Charles Koch’s network launches $20m campaign backing Trump tax breaks&#39;},<br>            {&#39;key&#39;:2,&#39;title&#39;:&#39;The refs didn’t steal a Super Bowl trip from the Bills. The Chiefs’ excellence did&#39;},<br>            {&#39;key&#39;:3,&#39;title&#39;:&#39;From showers to tiny fish to windmills, Trump’s climate policies are driven by fixations&#39;},<br>            {&#39;key&#39;:4,&#39;title&#39;:&#39;Rwandan-backed rebels M23 claim capture of eastern DRC city Goma&#39;},<br>            {&#39;key&#39;:5,&#39;title&#39;:&quot;‘Wonderfully sentimental’: why Defending Your Life is my feelgood movie&quot;}]<br><br><br>tasks = []<br>for article in articles:<br>    task = {<br>        &quot;custom_id&quot;: f&quot;task-{article.key}&quot;,<br>        &quot;method&quot;: &quot;POST&quot;,<br>        &quot;url&quot;: &quot;/v1/chat/completions&quot;,<br>        &quot;body&quot;: {<br>            # This is what you would have in your Chat Completions API call<br>            &quot;model&quot;: &quot;gpt-4o-mini&quot;,<br>            &quot;temperature&quot;: 0.0,<br>            &quot;response_format&quot;: { <br>                &quot;type&quot;: &quot;json_object&quot;<br>            },<br>            &quot;messages&quot;: [<br>                {<br>                    &quot;role&quot;: &quot;system&quot;,<br>                    &quot;content&quot;: categorize_system_prompt<br>                },<br>                {<br>                    &quot;role&quot;: &quot;user&quot;,<br>                    &quot;content&quot;: article.title<br>                }<br>            ],<br>        }<br>    }<br>    <br>    tasks.append(task)</pre><p>Then create the batch input file and send it to the API as a job</p><pre># Create the batch input file<br>cat_batch_file_name = &quot;data/batch_tasks_categorize_articles.jsonl&quot;<br><br>with open(cat_batch_file_name, &#39;w&#39;) as file:<br>    for obj in tasks:<br>        file.write(json.dumps(obj) + &#39;\n&#39;)<br><br>categorize_batch_input_file = client.files.create(<br>    file=open(&quot;data/batch_tasks_categorize_articles.jsonl&quot;, &quot;rb&quot;),<br>    purpose=&quot;batch&quot;<br>)<br><br>categorize_batch_input_file_id = categorize_batch_input_file.id<br><br><br># Create the batch job<br>categorize_batch_job = client.batches.create(<br>    input_file_id=categorize_batch_input_file_id,<br>    endpoint=&quot;/v1/chat/completions&quot;,<br>    completion_window=&quot;24h&quot;,<br>    metadata={<br>        &quot;description&quot;: &quot;categorize_news_article_batch&quot;<br>    }<br>)</pre><p>You can check the status of the job like this:</p><pre># Check Status of Batch<br>categorize_batch_job = client.batches.retrieve(categorize_batch_job.id)<br>print(categorize_batch_job)</pre><p>The result will look like:</p><blockquote>Batch(id=’batch_67979daa8d508190934c32580261f861&#39;, completion_window=’24h’, created_at=1737989546, endpoint=’/v1/chat/completions’, input_file_id=’file-EJKNeLBa9mdNDJ3egJjSwJ’, object=’batch’, status=’in_progress’, cancelled_at=None, cancelling_at=None, completed_at=None, error_file_id=None, errors=None, expired_at=None, expires_at=1738075946, failed_at=None, finalizing_at=None, in_progress_at=1737989547, metadata={‘description’: ‘categorize_news_article_batch’}, output_file_id=None, request_counts=BatchRequestCounts(completed=1, failed=0, total=5))</blockquote><p>The counts at the end will tell you the progress of your batch.</p><p><strong>You won’t be able to run the rest of the code and download your results until the ‘status’ changes to ‘completed’.</strong></p><p><em>The down side of batch processing is that a job can take up to 24 hours to complete, so remember that expression about the watched pot...</em></p><pre># PROCESS RESULTS:<br>categorize_result_file_id = categorize_batch_job.output_file_id<br>categorize_result = client.files.content(categorize_result_file_id).content<br><br><br>categorize_result_file_name = &quot;data/categorize_batch_job_news_articles.jsonl&quot;<br><br>with open(categorize_result_file_name, &#39;wb&#39;) as file:<br>    file.write(categorize_result)<br><br><br># Loading data from saved file<br>categorize_results = []<br>with open(categorize_result_file_name, &#39;r&#39;) as file:<br>    for line in file:<br>        # Parsing the JSON string into a dict and appending to the list of results<br>        json_object = json.loads(line.strip())<br>        categorize_results.append(json_object)<br><br>categorize_r_list = []<br>for res in categorize_results:<br>    task_id = res[&#39;custom_id&#39;]<br>    categorize_r_list.append({&#39;key&#39;:int(task_id.split(&#39;-&#39;)[-1]),&#39;llm_category&#39;:json.loads(res[&#39;response&#39;][&#39;body&#39;][&#39;choices&#39;][0][&#39;message&#39;][&#39;content&#39;])[&#39;category&#39;]<br>})</pre><p>Now we can merge the outputs with the article inputs and see how the LLM did:</p><pre>categorized_frame = pd.DataFrame(categorize_r_list)<br>df = pd.DataFrame(articles)<br>pd.merge(df, categorized_frame, on=&#39;key&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T5nXWvAJIkkA3g9S2Kl5Cw.png" /></figure><p>The results aren’t perfect. I would prefer that article key #2 be categorized as “sports” and the article key #5 be categorized as “entertainment”, but article keys 1 and 4 were accurately categorized as “factual” and that is the category I’m most interested in for this application.</p><p><strong>Summing it up:</strong></p><p>OpenAI’s batch API is a convenient and inexpensive way to include an LLM in a workflow when you don’t need the results immediately. A batch may take up to 24 hours to complete and there are limitations on batch sizes, but this approach can cut your AI cost in half.</p><p>As with any AI application, the results are non-deterministic and will never be perfect. I plan to work around that by not exclusively relying on the outputs of any single function. For this news article tool, I’ll also use an LLM to extract topics from articles whose RSS feeds do not include them. Then I’ll use a machine learning model trained on my manually entered ratings of articles to predict my level of interest in new articles based on their topics.</p><p>This combination of approaches will produce results that are good enough for my personalized news feed and far superior to what I could do manually with the volume of articles that my RSS reader pulls down each day. I wouldn’t use this approach to launch a rocket, but it’s great for hunting down news that’s important to me that I might otherwise miss.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1fb89d4400d4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Extract Geospatial Data with AI]]></title>
            <link>https://medium.com/@alex_44314/extract-geospatial-data-with-ai-11fcab3098dc?source=rss-b74886233fb7------2</link>
            <guid isPermaLink="false">https://medium.com/p/11fcab3098dc</guid>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[academia]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Alex Rich, PhD]]></dc:creator>
            <pubDate>Thu, 04 Apr 2024 13:08:20 GMT</pubDate>
            <atom:updated>2024-04-04T13:20:38.182Z</atom:updated>
            <content:encoded><![CDATA[<p>Using AWS Bedrock and Mistral 7b to extract city/state/nation from author affiliations</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CdqdGzfrXqiSZ7WbpIHBjA.png" /><figcaption>Image created with AWS Titan Image Generator G1</figcaption></figure><h3>Why?</h3><p>Have you ever wondered how much the authors of a medical journal article were paid by pharma or device companies?</p><p>That got pretty important in the wake of the opioid crisis (among others).</p><p>The data is out there…if you can get their city and state. The <a href="https://openpaymentsdata.cms.gov/">CMS Open Payments Database</a> lets you search industry payments by name, city, and state. While it’s easy to manually extract the city and state of an author’s institutional affiliation, what if you wanted to evaluate thousands…or 24,000 author affiliations?</p><p>Sometimes you can just cut the end off an affiliation:</p><blockquote>“jay monahan center for gastrointestinal health, division of gastroenterology, department of medicine, and department of healthcare policy and research, weill cornell medicine, new york, new york.”</blockquote><p>…obviously “New York, NY, USA.”</p><p>But sometimes you have to use more context and domain knowledge:</p><blockquote>“department of medicine, vanderbilt university medical center”</blockquote><p>...here you have to be familiar with Vanderbilt University to extract “Nashville, TN, USA.”</p><p>Doing this manually was the most painful drudgery of my dissertation work. Now we can do it with an LLM.</p><p>I explained how to do that slowly but at zero cost on your laptop using Ollama in a previous post (it took days). Now I’ll walk you through how to do the same thing much faster using AWS Bedrock (under 7 hours, less than $2). (Next: AWS batch processing in WAY less time…stay tuned.)</p><p>We’ll also add Pydantic validation to ensure that responses arrive in a predictable and usable format. If you think of this code as a digital research assistant, think of the Pydantic validation as those moments when you check your RA’s work and have them re-do elements that aren’t quite on the mark.</p><h3>Results:</h3><p>If I’m really racing I can manually pull city, state, and nation from about 1,000 affiliations per hour. It’s the most mindless unhappy work I’ve done in a while. At that rate, this dataset would be more than three 8-hour days of that work.</p><blockquote>Side note: You’d be shocked by how many different ways doctors find to say “Harvard.”</blockquote><p>Using Mistral 7b on AWS Bedrock, I resolved 24,091 out of 24,801 affiliations (97.1%) in about 6.5 hours while I slept. I manually evaluated a random sample of 1,000 results and found that there were 4 errors (0.4%). There were also 219 correct results that could not have been obtained without context that was not in the text of the affiliation (e.g. “mount sinai school of medicine” -&gt; new york, new york, usa). This is where the contextual knowledge embedded in the LLM really shines.</p><p>The total Bedrock bill for the work was less than $2.</p><h3>How To:</h3><h4>Setting Up Bedrock</h4><p>AWS Bedrock is a relatively new service that allows you to use AWS compute to run a variety of LLMs in the cloud at really low cost. I started tinkering with Bedrock after my local Ollama LLM work started taking days to complete a job. I’ve played with a few LLMs. So far, Mistral’s open-source 7 billion parameter model has been the most reliable and cost effective for the work I’ve been doing. I’ll walk you through how to get set up to see for yourself.</p><p>First you’ll need an AWS account (YouTube has great resources on how). It’s a best practice to create a non-root user in IAM and provide that user with a role that allows access to Bedrock and EC2. Log in to the AWS console and set your region to one of those in which Bedrock is available (I used us-east-1). Choose Bedrock from the services menu (or search bar) and click “get started” on the Bedrock landing page. Choose the Text playground from the menu on the left and then “select model” at the top of the page. Here you’ll be able to choose a foundational LLM and “request access” to it. Once you have access you’ll be able to interactively tinker with prompts and parameters in the playground</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_lnXzY26yUzIyCOdE-CEfQ.png" /><figcaption>AWS Bedrock Playground: a great place to tinker with prompts and parameters</figcaption></figure><h4>The Prompt:</h4><p>Prompt engineering can be a little finicky. It’s best to clear directive statements and provide a couple of examples of good inputs and outputs. You should also provide direction on what do to when the model is unable to formulate a good answer. I spent a bunch of time tinkering in the playground to get the prompt results where I wanted them. (Note: set the Temperature to 0.0 to minimize “creativity” in the results)</p><pre>&quot;&quot;&quot;Separate the affiliation by the initials in parentheses. Then get the nation, city, and state of each affiliation. Return the results in a JSON like <br> &quot;{&quot;affiliation&quot;:&quot;from the second building on the third block at yale university, new haven, ct&quot;,&quot;city&quot;:&quot;new haven&quot;, &quot;state&quot;:&quot;ct&quot;,&quot;nation&quot;:&quot;usa&quot;}}&quot;<br>Return only formatted JSON. <br>If you do not know the nation, return &quot;unknown&quot;. <br>If you do not know the state, return &quot;unknown&quot;. <br>If you do not know the city, return &quot;unknown&quot;.<br>Examples: <br>user: &quot;columbia university, new york, NY&quot;<br>&quot;{&quot;affiliation&quot; : &quot;columbia university, new york, NY&quot;,&quot;city&quot; : &quot;new york&quot;,&quot;state&quot; : &quot;ny&quot;,&quot;nation&quot; : &quot;usa&quot;}&quot;<br>2. user: &quot;department of psychiatry, university of california, san francisco.&quot;<br>&quot;{&quot;affiliation&quot; : &quot;department of psychiatry, university of california, san francisco.&quot;, &quot;city&quot; : &quot;san francisco&quot;, &quot;state&quot; : &quot;ca&quot;,&quot;nation&quot; : &quot;usa&quot;}&quot;<br>3. user: &quot;a colony on mars&quot;<br>&quot;{&quot;affiliation&quot; : &quot;a colony on mars&quot;, &quot;city&quot; : &quot;unknown&quot;, &quot;state&quot; : &quot;unknown&quot;, &quot;nation&quot; : &quot;unknown&quot;}&quot;<br>Return only valid JSON and do not include any punctuation. <br>Always include affiliation, city, state, and nation.<br>affiliation: {input_affiliation}&quot;&quot;&quot;</pre><p>Once I was happy with the prompt I grabbed the prompt request text by viewing the API request in the playground using the menu on the top right.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U5yKEw06A2wYGK655-uZ4A.png" /><figcaption>Use the “View API request” option to get the prompt text (Not sure why the Bedrock UI no longer lets you view a Boto3 Python request format)</figcaption></figure><p>The API request does a lot of \\\” and \\n escaping, but is otherwise the same text from my input. I copied and pasted that into my Python code to use with Boto3. The result will look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nCIePq02xA7WHhipO5umqQ.png" /><figcaption>API request formatted prompt pasted into the Python script as an f string with {input_affiliation} as the function’s input</figcaption></figure><h4>On to Cloud9</h4><p>Open up a Cloud9 development environment on AWS. You can run this with the t2.micro instance that’s eligible for the Free Tier. Processing about 24 thousand affiliations took me about 6.5 hours, so I set the timeout for 1 day.</p><p>Make sure you’re logged in as an IAM user with a role that allows full Bedrock and EC2 control. Click the gear on the top right, go to “AWS Settings” and deselect “AWS managed temporary credentials”. This should save you from expired security token errors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2hKZ9G48ntVgbMIssVHJKA.png" /></figure><p>With the environment open use the terminal to install boto3, pandas, and pydantic. Then clone the GitHub repo.</p><pre>pip install boto3<br>pip install pandas pydantic<br>git clone https://github.com/alexF3/Bedrock_LLM_extract_geography.git</pre><p>Open up main.py and run it.</p><h4>Stepping through the code</h4><p>First import all the libraries you’ll need.</p><pre>import boto3<br>import json<br>import pandas as pd<br>import os<br>import time<br><br><br>from pydantic import BaseModel<br>from pydantic import validator<br>import re</pre><p>Next, load the CSV of the 24,801 unique author affiliations from 10 years of JAMA article metadata.</p><pre># Load all 24k unique listed &quot;first affiliations&quot; for authors in JAMA articles 2013-2023 <br># (source: PubMed .nbib file)<br>df = pd.read_csv(&#39;jama_all_2013_2023_24k.csv&#39;)<br>print(df.head())</pre><p>Set up the Pydantic validation (note: I’m using Pydantic V1 syntax here…expect an update down the road).</p><pre>class Aff(BaseModel):<br>    city: str<br>    state: str<br>    nation: str<br>    <br>    @validator(&#39;city&#39;)<br>    def validate_city(cls, v):<br>        if len(v) &gt; 30:<br>            raise ValueError(&#39;City must be less than 30 characters&#39;)<br>        if not re.match(r&#39;^[a-zA-Z-\s]+$&#39;, v):<br>            raise ValueError(&#39;City contains invalid characters&#39;)<br>        return v<br>    <br>    @validator(&#39;state&#39;)<br>    def validate_city(cls, v):<br>        if len(v) &gt; 40:<br>            raise ValueError(&#39;State must be less than 40 characters&#39;)<br>        if not re.match(r&#39;^[a-zA-Z-]+$&#39;, v):<br>            raise ValueError(&#39;State contains invalid characters&#39;)<br>        return v<br>    <br>    @validator(&#39;nation&#39;)<br>    def validate_city(cls, v):<br>        if len(v) &gt; 40:<br>            raise ValueError(&#39;Nation must be less than 40 characters&#39;)<br>        if not re.match(r&#39;^[a-zA-Z-\s]+$&#39;, v):<br>            raise ValueError(&#39;Nation contains invalid characters&#39;)<br>        return v</pre><p>Set up the Bedrock call using Boto3.</p><pre>bedrock_runtime = boto3.client(&#39;bedrock-runtime&#39;, region_name=&#39;us-east-1&#39;)<br><br>def runModel(input_affiliation):<br>    # Note: this prompt does not conform strictly to Mistral&#39;s suggested format<br>  prompt = f&quot;Separate the affiliation by the initials in parentheses. Then get the nation, city, and state of each affiliation. Return the results in a JSON like \\n    \\\&quot;{{\\\&quot;affiliation\\\&quot;:\\\&quot;from the second building on the third block at yale university, new haven, ct\\\&quot;,\\\&quot;city\\\&quot;:\\\&quot;new haven\\\&quot;, \\\&quot;state\\\&quot;:\\\&quot;ct\\\&quot;,\\\&quot;nation\\\&quot;:\\\&quot;usa\\\&quot;}}\\\&quot;\\nReturn only formatted JSON. If you do not know the nation, return \\\&quot;unknown\\\&quot;. If you do not know the state, return \\\&quot;unknown\\\&quot;. If you do not know the city, return \\\&quot;unknown\\\&quot;.\\nExamples: \\n1. user: \\\&quot;columbia university, new york, NY\\\&quot;\\n\\\&quot;{{\\\&quot;affiliation\\\&quot;:\\\&quot;columbia university, new york, NY\\\&quot;,\\\&quot;city\\\&quot;:\\\&quot;new york\\\&quot;,\\\&quot;state\\\&quot;:\\\&quot;ny\\\&quot;,\\\&quot;nation\\\&quot;:\\\&quot;usa\\\&quot;}}\\\&quot;\\n2. user: \\\&quot;department of psychiatry, university of california, san francisco.\\\&quot;\\n\\\&quot;{{\\\&quot;affiliation\\\&quot;:\\\&quot;department of psychiatry, university of california, san francisco.\\\&quot;,\\\&quot;city\\\&quot;:\\\&quot;san francisco\\\&quot;,\\\&quot;state\\\&quot;:\\\&quot;ca\\\&quot;,\\\&quot;nation\\\&quot;:\\\&quot;usa\\\&quot;}}\\\&quot;\\n3. user: \\\&quot;a colony on mars\\\&quot;\\n\\\&quot;{{\\\&quot;affiliation\\\&quot;:\\\&quot;a colony on mars\\\&quot;,\\\&quot;city\\\&quot;:\\\&quot;unknown\\\&quot;,\\\&quot;state\\\&quot;:\\\&quot;unknown\\\&quot;,\\\&quot;nation\\\&quot;:\\\&quot;unknown\\\&quot;}}\\\&quot;\\nReturn only valid JSON and do not include any punctuation. Always include affiliation, city, state, and nation.\\naffiliation: {input_affiliation}&quot;<br>  payload = {<br>    &quot;prompt&quot;: prompt,<br>    &quot;max_tokens&quot;: 400,<br>    &quot;top_k&quot;: 90,<br>    &quot;top_p&quot;: 0.3,<br>    &quot;stop&quot;: [],<br>    &quot;temperature&quot;: 0.0 # Minimizing &quot;creativity&quot; for consistent results<br>  }<br>  <br>  raw_response = bedrock_runtime.invoke_model(<br>    modelId=&#39;mistral.mistral-7b-instruct-v0:2&#39;,<br>    body=json.dumps(payload)<br>  )<br>  <br>  body = raw_response[&quot;body&quot;].read().decode(&quot;utf-8&quot;)<br>  <br>  <br>  response_body = json.loads(body)<br>  outputs = response_body[&quot;outputs&quot;]<br>  <br>  location = outputs[0][&quot;text&quot;] <br>  return(location)</pre><p>Create a function to call the model, validate the results, and re-run up to 3 times attempting to get a valid JSON output.</p><pre> <br>def getGeography(affiliation):<br>    &quot;&quot;&quot; Have thee LLM  make up to 3 attempts to extract city, state, and nation from the affiliation <br>    and return a JSON like:<br>    {<br>        &quot;affiliation&quot;: &quot;from the second building on the third block at yale university, new haven, ct&quot;,<br>        &quot;city&quot;: &quot;new haven&quot;,<br>        &quot;state&quot;: &quot;ct&quot;,<br>        &quot;nation&quot;: &quot;usa&quot;<br>    }<br>    If the return is not valid, return the final attempt.<br><br>    Args:<br>        affiliation (str): first affiliation listed for an author<br>    &quot;&quot;&quot;    <br>    tries = 0<br>    while tries &lt; 3:<br>        result =  runModel(affiliation)<br>        result =  result.replace(&quot;\\&#39;&quot;,&#39;&#39;)<br>        tries+=1<br><br>        try:<br>            result = re.sub(r&quot;\\&quot;, &quot;&quot;, result)<br>            result = result.replace(&#39;\n&#39;,&#39;&#39;)<br>            try:<br>              pattern = r&#39;\{(.*?)\}&#39;<br>              result = &#39;{&#39;+re.search(pattern, result).group(1)+&#39;}&#39;<br>            except:<br>              pass<br>            print(result)<br>            data = Aff.model_validate_json(result)<br>            print(&#39;GOOD!&#39;)<br>            return(result)<br>        except ValueError as e:<br>            print(&#39;{} invalid: {}&#39;.format(tries,result))<br>            print(e.errors())<br>            pass<br>   <br>    print(&quot;bad response:&quot;)<br>    print(&#39;result: {}&#39;.format(result))<br>    return(runModel(affiliation))</pre><p>And finally, run that function over the dataframe of input affiliations in a loop. I persistently got a security token expiration error when attempting to apply the function to the dataframe, but looping worked fine.</p><pre>   <br>###<br># Run the model on all of the unique author affiliations in the dataset<br># Note: I got security token expiration errors consistently when attempting to apply the function to the pandas dataframe.<br># ...for some reason looping does not produce the same error.<br>start = time.time()<br><br>entries_list = []<br>bad_list = []<br>ct = 0<br>for row in df.itertuples():<br>    entry = {<br>        &#39;firstAffiliation&#39;:row.firstAffiliation,<br>        &#39;geography&#39;:getGeography(row.firstAffiliation)<br>    }<br>    entries_list.append(entry)<br>    ct+=1<br>    if ct%50==0:<br>        pd.DataFrame(entries_list).to_csv(&#39;all_jama_2013_2023_mistral_pydantic_24k.csv&#39;,index=False)<br><br>duration = time.time() - start<br>print(&#39;Total minutes of processing: {}&#39;.format(duration/60))<br>df.to_csv(&#39;JAMA_all_pydantic_mystral_24k.csv&#39;,index=False)</pre><p>Don’t forget to download your results before you close down and delete your Cloud9 instance!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=11fcab3098dc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Practical AI: an LLM Research Assistant..on your laptop…for free.]]></title>
            <link>https://medium.com/@alex_44314/practical-ai-an-llm-research-assistant-on-your-laptop-for-free-145edba9258e?source=rss-b74886233fb7------2</link>
            <guid isPermaLink="false">https://medium.com/p/145edba9258e</guid>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[ollama]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[research-methods]]></category>
            <dc:creator><![CDATA[Alex Rich, PhD]]></dc:creator>
            <pubDate>Sun, 24 Mar 2024 14:49:11 GMT</pubDate>
            <atom:updated>2024-03-24T15:07:20.793Z</atom:updated>
            <content:encoded><![CDATA[<p>Using Ollama and Mistral 7b to extract geographic data with a 5-year-old Mac Book (no API calls to corporate models, no data leaves your machine)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lowjporf6XwhPevhKM8qWg.png" /><figcaption>Visualization of Purdue Pharma payments to US-based physicians (data: CMS Open Payments)</figcaption></figure><h3><strong>Introduction</strong></h3><p>Imagine being able to rescue your research assistant from manually classifying 24 thousand entries by hand. Better yet, imagine being able to conduct research that you’re passionate about even when you have limited time and no research assistants at all!</p><p><a href="https://ollama.com/">Ollama</a> on your laptop is a great option if you have a limited budget, concerns about putting your data in the cloud, or just want to tinker with large language models (LLMs) without any platform lock-in. <a href="https://mistral.ai/">Mistral</a> open-sourced their excellent 7 billion parameter model under an Apache license. The pair make a powerful combination.</p><p>As an independent researcher with a full-time day job, this is a big deal for me. I’m intensely curious about the history of Purdue Pharma’s hidden influence over healthcare providers and institutions <em>(Turns out: There’s a lot…</em><a href="https://www.jospi.org/post/1461-when-a-unc-grad-student-confronted-dr-ralph-snyderman-for-aiding-the-sacklers-opioid-scheme-the-duke-university-professor-threatened-legal-action"><em>looking at you, Duke Med</em></a><em>)</em><strong>.</strong> One of the most tedious parts of my research involves extracting geographic data from medical journal authors’ institutional affiliations. After a lot of manual processing I finally ran out of patience and built an LLM-based tool to do that for me. LLMs are especially good for this kind of work because they “understand” context in a way that less advanced language processing approaches do not.</p><h3><strong>Summary:</strong></h3><p>An overview of one way to harness open-source tools to use a Large Language Model (LLM) to extract nation, city, and state information from medical journal author affiliations.</p><p><strong>Data:</strong> PubMed metadata on JAMA articles published between 2013 and 2023 downloaded as a .nbib citation manager file</p><p><strong>Tools:</strong> Python, Ollama, LangChain, Pandas, and Mistral 7b Instruct</p><p><strong>Results:</strong> 24,184 affiliations were processed. Manual evaluation of a random sample of 2,000 affiliations showed 13 errors (0.65%), though the results also included 14 examples of successful processing that depended on context that did not appear in the affiliation (e.g. “senior policy service professor at george washington university school of nursing’s center for health policy and media.” →“Washington, dc”).</p><p><strong>Code: </strong><a href="https://github.com/alexF3/LLM_extract_geography">https://github.com/alexF3/LLM_extract_geography</a></p><h3><strong>Approach</strong></h3><p>This post will walk you through how to use an open-source LLM to process about 24,000 institutional affiliations of authors who published in the medical journal JAMA in the last ten years. The tool will take inputs like:</p><blockquote>“department of medicine and department of psychiatry, university of utah school of medicine, salt lake city.”</blockquote><p>…and return the nation and city, state of the affiliation:</p><blockquote>“usa” and “salt lake city, ut”</blockquote><p>It will do this by using the Ollama framework and the Mistral 7b Instruct language model. Both are open source and can be run locally on a laptop without the use of any API calls to commercial services. Ollama is a framework that abstracts away a lot of the complexity of running an LLM locally and allows you to make local API calls from your IDE. At the time of writing, Ollama can be downloaded for Linux or Mac, but there is a preview Windows version available. My laptop is a 2019 MacBook Pro with a 2.3 GHz Intel Core i9 processor, 32 GB of RAM, and an Intel UHD graphics card. Running the Mistral model without a GPU is a little slow (~3 sec per query), but totally usable for my purposes. Running the code locally lets me tinker without having to worry about a surprise AWS bill, which could get decently large for tasks of this scale.</p><p>My next project will be deploying this same approach on AWS Bedrock to evaluate the speed and cost of batch processing in the cloud.</p><p><strong>Model and Libraries</strong></p><p>I chose Mistral 7b Instruct version 0.2 released by Mistral AI under an Apache 2.0 license as the LLM behind this tool. I experimented with both Mistral and Llama2 and found that Mistral gave me more consistent results.</p><p>With Ollama running in the terminal I downloaded the model with “ollama pull mistral:instruct”. The model was about a 4.1 GB download from Ollama. I left Ollama running in the background for the rest of code execution.</p><p>Interactions with the model were handled in Python through the LangChain templating library. The interface between the code and the model was handled through Ollama, which simplifies a lot of the details of PyTorch, transformers, and HuggingFace interactions. I like to work in VS Code, but the approach I’m describing should work in any Python IDE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0pRiAWb6u0z1TR9ZM28y5A.png" /></figure><h3><strong>Step 1: Extract the Nation of each affiliation</strong></h3><p>I found that it was hard to get consistent results when I prompted the model to get the nation, city, and state of an affiliation all in one prompt. First, I wanted to know what nation the affiliation was from. After that I only wanted city and state values for affiliations that were in the US (and thus potentially linked to an author who might have pharma company payments data in the CMS Open Payments database).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FLtNizwd7Cn83rObyFM_Bw.png" /></figure><p>I chose to minimize “creativity” in the model’s responses by setting the <strong>temperature</strong> to 0.0. I also chose to include a few examples of affiliations and correct responses. I set <strong>num_predict</strong> at 10 tokens to prevent overly verbose chat-style responses. The result is a reusable chain that can be applied to a pandas dataframe through a function call.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Tcyfx-23fZKxehYsQa02Q.png" /></figure><p>The result was a new column of Mistral outputs with the nation of each affiliation.</p><h3>Step 2: Extract the City and State of each US-based affiliation</h3><p>The results of step one let me write code to get “city, state” pairs for only the affiliations that appear to be in the US. This makes the code a little bit faster and will be useful to me in the entity resolution work that my research requires.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_VgRnMk6CLyLiABfqA4CmA.png" /></figure><p>I gave the prompt a few examples of correct inputs and outputs and explicit instructions on what to do if it couldn’t find an answer. (Note: you can test the prompt with “chain.run(&lt;input an affiliation string&gt;)”)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uzNQDahiuZxnseMsYTb1Ag.png" /></figure><p>Running both steps of the code took the better part of 2 days and kept my CPU at about 65% utilization. The laptop was fully usable for other tasks, it just had the fan running at top speed.</p><p>The responses from the model do have some formatting variance that needed to be cleaned up with some string functions applied after the run was completed. Sometimes Mistral would include “response:” in the output and sometimes the output appeared to continue into an explanation of how the model got there. That was easily addressed with a couple lines of Pandas commands.</p><p>I’m excited to spend some time setting the code up for batch processing in AWS Bedrock during the next phase of development.</p><h3>Evaluating the Results</h3><p>I had the code randomly select 2,000 results from the 24,184-row batch. I manually checked each and found that there were 13 errors (0.65% error rate). One common error type appeared to be the result of cutting off an overly verbose response:</p><blockquote>“ response: winston-salem, north car”</blockquote><p>Another common error involved a misclassification of</p><blockquote>“department of family and preventive medicine, university of california, san diego, la jolla2department of medicine, university of california, san diego, la jolla.”</blockquote><p>as a “San Diego, CA” geography. There were also two instances in which the first prompt resulted in categorizing Puerto Rico as its own nation. I suspect some more tinkering with the prompt and parameters can eliminate these errors.</p><p>On a brighter note, there were 14 responses where the model gave a correct Nation or City, State pair despite the affiliation lacking important information. These responses depended on the LLM’s pre-trained contextual memory to yield the correct geography.</p><p>For example, the model received:</p><blockquote>“james madison professor of political economy and of economics at princeton university.”</blockquote><p>and correctly responded:</p><blockquote>“Princeton, NJ”</blockquote><p>None of the hard-coded string parsing methods I’ve built could make that context-based connection.</p><h3>Conclusion</h3><p>Pick your favorite song and wrench on this code with the volume at 11.</p><p>Tech can do a lot of bad things, but it can also empower mission-oriented people to have impact that used to be impossible. How do you wish the world were different and better… and how could a tool like this help you nudge things in that direction?</p><p>Fight’s on.</p><p><strong>Code: </strong><a href="https://github.com/alexF3/LLM_extract_geography">https://github.com/alexF3/LLM_extract_geography</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=145edba9258e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Use Python geopandas to make a US map with Alaska and Hawaii]]></title>
            <link>https://medium.com/@alex_44314/use-python-geopandas-to-make-a-us-map-with-alaska-and-hawaii-39a9f5c222c6?source=rss-b74886233fb7------2</link>
            <guid isPermaLink="false">https://medium.com/p/39a9f5c222c6</guid>
            <category><![CDATA[geospatial-analytics]]></category>
            <category><![CDATA[geopandas]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Alex Rich, PhD]]></dc:creator>
            <pubDate>Mon, 16 Jan 2023 15:38:09 GMT</pubDate>
            <atom:updated>2023-01-16T16:41:17.564Z</atom:updated>
            <content:encoded><![CDATA[<p>Here’s one way to make a static choropleth map using geopandas and matplotlib.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dtbDjDbSypyghG8_7qDwrw.png" /><figcaption>Turn tabular data on household food insecurity into a static choropleth map</figcaption></figure><p><strong>Why:</strong></p><p>I run into a lot of projects where geospatial data visualization adds to the quality of the work. It’s useful in data cleaning, exploratory analysis, and explanatory data visualization. It’s often important to be able to do this kind of work on my own without sending it out to the GIS team or using any license-based software. Maintaining these skills in open source software keeps me from dependence on a present or future employer’s choice of proprietary software solutions. Even federal government data teams are now catching up to industry and beginning to use R and Python where they once were dependent on expensive licenses to SAS or Stata.</p><p>I like to work in Python because of the flexibility it provides, the large developer community, and the error handling built into the language. With that in mind, there are a lot of Python packages that offer relatively easy mapping capabilities. I recommend checking out plotly, leaflet, and keplergl. For my purposes today, geopandas and matplotlib offer the level of customization that I’m looking for.</p><p>For this specific project, I wanted to explore visual communication about the kind of economic data you don’t hear on the news every day. The Dow Jones Industrial Average is a very limited measure of a country’s economic health. The commitment of a nation to supplying adequate food to its most vulnerable citizens should probably get more attention than it does.</p><p>This exercise takes tabular data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w_Wk6wNLHrFGeoNWU0XHXQ.png" /><figcaption>household food insecurity data downloaded from the US Dept of Agriculture</figcaption></figure><p>…and turns it into something a little more engaging.</p><p><strong>What</strong>:</p><p>In this article, I’ll walk through how to use Python to download a data file on household food insecurity from the USDA website, obtain geospatial data called a shapefile from the US census website, and merge them together using a few simple libraries to produce a static choropleth map.</p><p><strong>How:</strong></p><p>My full code for this exercise can be accessed on <a href="https://github.com/alexF3/map_us_hunger">github</a>. If you’re new to Python and want to follow through the code in a Jupyter Notebook click <a href="https://github.com/alexF3/map_us_hunger/blob/main/Python_mapping_tutorial.ipynb">here</a>.</p><p>I’ll use several Python libraries to work through this example</p><pre><br><br>import pandas as pd<br>import matplotlib.pyplot as plt<br>from matplotlib.ticker import FuncFormatter<br>import matplotlib.colors as mcolors<br>import seaborn as sns<br>import geopandas as gpd<br>from shapely.geometry import Polygon<br>import missingno as msno<br>import os<br>import wget<br>import openpyxl<br>import math</pre><p>US household food insecurity rates by state can be downloaded from the USDA with wget()</p><pre><br># download the data excel file from the USDA website to local folder<br>filename = wget.download(&quot;https://www.ers.usda.gov/media/rbmpu1zi/mapdata2021.xlsx&quot;)<br># load the excel file into a pandas dataframe &amp; skip header rows<br>df = pd.read_excel(os.getcwd()+&#39;/mapdata2021.xlsx&#39;,skiprows=4)<br># rename the columns of interest<br>df = df.rename(columns={&#39;Unnamed: 0&#39;:&#39;state&#39;,<br>                       &#39;Percent&#39;:&#39;pct_food_insecure&#39;})<br># retain only the columns of interest<br>df = df[[&#39;state&#39;,&#39;pct_food_insecure&#39;]]<br><br># take a look at the dataframe for missing values<br>msno.matrix(df)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xhRbsrIXEtbQX_46R8JKew.png" /></figure><p>The data file contains some rows we’re not interested in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UCdP2RZoNl2ZHXkXULPPcA.png" /></figure><p>I’m going to retain only the rows with 2-letter state abbreviations in the “state” column and take a look at the distribution of food insecurity data.</p><pre>df = df[df.state.str.len()==2]<br>df.pct_food_insecure.hist()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DSh26VNDm99dVpcI0hpbcQ.png" /></figure><p>That all looks reasonable. Now it’s time to get the geospatial files.</p><p>The geopandas library works with shapefiles. Shapefiles are a standardized format for geospatial data and can be pulled from a number of sources including the US Census website. Here, I’ll pull a shapefile of US state outlines using wget().</p><pre>wget.download(&quot;https://www2.census.gov/geo/tiger/GENZ2018/shp/cb_2018_us_state_500k.zip&quot;)</pre><p>On a Mac, I unzip the resulting download by double clicking on it. Windows may require you to use a file utility to do so. Once unzipped, the shapefile can be loaded with geopandas by referencing the shapefile folder (geopandas knows where to find everything inside this standardized format).</p><pre>gdf = gpd.read_file(os.getcwd()+&#39;/cb_2018_us_state_500k&#39;)<br>gdf.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VJ1Lvt5J1a_7E1IvtzgW7A.png" /></figure><p>The geopandas dataframe that results contains a column of all the geometries we need, a column of state abbreviations, and a wealth of other data we don’t need.</p><p>Geopandas makes it easy to merge data. I’ll merge our USDA data with the gdf on the columns in each that correspond to state abbreviation.</p><pre>gdf = gdf.merge(df,left_on=&#39;STUSPS&#39;,right_on=&#39;state&#39;)<br><br></pre><p>When we plot what we have, things look a little wonky.</p><pre>gdf.plot()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3ccV1Ebr78PZWYPTcnvgXw.png" /></figure><p>As you can see, plotting Alaska’s western islands causes us to display a great deal more longitudinal territory than we’d like. I’m going to clip both Alaska and Hawaii down to their larger land masses and present them below the continental US. To do that, I’ll use the shapely package’s Polygon() function to define a box around what I want to keep and the geopandas clip() function to retain only the parts of each state that fall inside the polygon.</p><p>Here’s a quick illustration of how that works:</p><pre>world = gpd.read_file(gpd.datasets.get_path(&quot;naturalearth_lowres&quot;))<br><br># NOTE: the convention for polygon points is (Long, Lat)....counterintuitive<br>polygon = Polygon([(-175,50),(-175,72),(-140, 72),(-140,50)])<br># polygon = Polygon([(-180,0),(-180,90),(-120,90),(-120,0)])<br><br># polygon=hipolygon<br>poly_gdf = gpd.GeoDataFrame( geometry=[polygon], crs=world.crs)<br><br>fig, ax1 = plt.subplots(1, figsize=(8, 18))<br>world.plot(ax=ax1)<br>poly_gdf.boundary.plot(ax = ax1, color=&quot;red&quot;)<br>ax1.set_title(&quot;The red polygon can be used to clip Alaska&#39;s western islands&quot;, fontsize=20)<br>ax1.set_axis_off()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RsgqU3uV9ndLGM25pN27Ew.png" /></figure><p>And now to demonstrate how to clip Alaska:</p><pre>polygon = Polygon([(-170,50),(-170,72),(-140, 72),(-140,50)])<br># apply1(alaska_gdf,0,36)<br>alaska_gdf.clip(polygon).plot( color=&#39;lightblue&#39;, linewidth=0.8, edgecolor=&#39;0.8&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*wTetUepJYx67ZOaL9YcjnQ.png" /></figure><p>I’ll do the same with Hawaii in the final code block.</p><p><strong>Merging the Geopandas Dataframe with USDA Food Insecurity Data</strong></p><p>Geopandas makes it easy to merge data. I’ll merge on the columns in each dataframe that correspond to state abbreviations.</p><pre>gdf = gdf.merge(df,left_on=&#39;STUSPS&#39;,right_on=&#39;state&#39;)A Brief Intro to Projections:</pre><p><strong>A Brief Intro to Projections:</strong></p><p>Because the earth is round, projecting geography onto 2 dimensional maps requires some fancy math. Fortunately, cartographers have helped take a lot of the pain out of this process.</p><p>Coordinate Reference Systems (CRS) help project lattitude and longitude coordinates onto 2-dimensional maps. There are many different systems and each correspond to a different method of projection.</p><p>Geopandas makes it easy to experiment with different CRSs <a href="https://geopandas.org/en/stable/docs/user_guide/projections.html">https://geopandas.org/en/stable/docs/user_guide/projections.html</a></p><p>The European Petrolium Survey Group (EPSG)(<a href="https://epsg.org/home.html">https://epsg.org/home.html</a>) maintains an extensive set of projections for cartographic applications. The best projection for use in a particular visualization depends on the part of the world being visualized and the preferences of the developer.</p><p>A helpful tool for choosing a CRS can be found here: <a href="https://epsg.io/">https://epsg.io/</a></p><p>CRS selection can cause some land masses to look larger or smaller than they are relative to others. There’s always going to be a bit of distortion in a map. Your job is to manage that.</p><p>Here’s a good look at why this matters from Open News: <a href="https://source.opennews.org/articles/choosing-right-map-projection/">https://source.opennews.org/articles/choosing-right-map-projection/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/599/1*fzbiUuQ5PrLMf9nK74_gCg.jpeg" /><figcaption>Open News illustration of the appearances of different CRS projections of CONUS</figcaption></figure><p>We can re-project coordinates for any of the components of our map using the geopandas command .to_crs()</p><pre>gdf.to_crs({&#39;init&#39;:&#39;epsg:2163&#39;})</pre><p><strong>Managing Alaska and Hawaii</strong></p><p>I’m going to create new boxes to map them in underneath and to the left of the continental US. Confusingly, these boxes are referred to as “axes” in matplotlib.</p><pre><br><br># Create a &quot;copy&quot; of gdf for re-projecting<br>visframe = gdf.to_crs({&#39;init&#39;:&#39;epsg:2163&#39;})<br><br># create figure and axes for with Matplotlib for main map<br>fig, ax = plt.subplots(1, figsize=(18, 14))<br># remove the axis box from the main map<br>ax.axis(&#39;off&#39;)<br><br><br># create map of all states except AK and HI in the main map axis<br>visframe[~visframe.state.isin([&#39;HI&#39;,&#39;AK&#39;])].plot(color=&#39;lightblue&#39;, linewidth=0.8, ax=ax, edgecolor=&#39;0.8&#39;)<br><br><br># Add Alaska Axis (x, y, width, height)<br>akax = fig.add_axes([0.1, 0.17, 0.17, 0.16])   <br><br><br># Add Hawaii Axis(x, y, width, height)<br>hiax = fig.add_axes([.28, 0.20, 0.1, 0.1])   <br><br># We&#39;ll later map Alaska in &quot;akax&quot; and Hawaii in &quot;hiax&quot;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0MiC-g5RBk8vypEZB4-c3w.png" /><figcaption>Map with axes established for Alaska and Hawaii below the continental US</figcaption></figure><p><strong>Bringing it All Together</strong></p><p>I’m going to use a matplotlib continuous color scale (<a href="https://matplotlib.org/stable/tutorials/colors/colormaps.html">https://matplotlib.org/stable/tutorials/colors/colormaps.html</a>) to set the color for each state. Because I’m treating Alaska and Hawaii separately from CONUS, I’m going to create a column of the color each state should be and then map each state separately with the color pulled from that column.</p><pre># Apply this to the gdf to ensure all states are assigned colors by the same func<br>def makeColorColumn(gdf,variable,vmin,vmax):<br>    # apply a function to a column to create a new column of assigned colors &amp; return full frame<br>    norm = mcolors.Normalize(vmin=vmin, vmax=vmax, clip=True)<br>    mapper = plt.cm.ScalarMappable(norm=norm, cmap=plt.cm.YlOrBr)<br>    gdf[&#39;value_determined_color&#39;] = gdf[variable].apply(lambda x: mcolors.to_hex(mapper.to_rgba(x)))<br>    return gdf</pre><p>Now it’s time to plot everything</p><pre># **************************<br># set the value column that will be visualised<br>variable = &#39;pct_food_insecure&#39;<br><br># make a column for value_determined_color in gdf<br># set the range for the choropleth values with the upper bound the rounded up maximum value<br>vmin, vmax = gdf.pct_food_insecure.min(), gdf.pct_food_insecure.max() #math.ceil(gdf.pct_food_insecure.max())<br># Choose the continuous colorscale &quot;YlOrBr&quot; from https://matplotlib.org/stable/tutorials/colors/colormaps.html<br>colormap = &quot;YlOrBr&quot;<br>gdf = makeColorColumn(gdf,variable,vmin,vmax)<br><br># create &quot;visframe&quot; as a re-projected gdf using EPSG 2163 for CONUS<br>visframe = gdf.to_crs({&#39;init&#39;:&#39;epsg:2163&#39;})<br><br><br><br># create figure and axes for Matplotlib<br>fig, ax = plt.subplots(1, figsize=(18, 14))<br># remove the axis box around the vis<br>ax.axis(&#39;off&#39;)<br><br># set the font for the visualization to Helvetica<br>hfont = {&#39;fontname&#39;:&#39;Helvetica&#39;}<br><br># add a title and annotation<br>ax.set_title(&#39;Food Insecurity by Percentage of State Households\n2019-2021&#39;, **hfont, fontdict={&#39;fontsize&#39;: &#39;42&#39;, &#39;fontweight&#39; : &#39;1&#39;})<br><br># Create colorbar legend<br>fig = ax.get_figure()<br># add colorbar axes to the figure<br># This will take some iterating to get it where you want it [l,b,w,h] right<br># l:left, b:bottom, w:width, h:height; in normalized unit (0-1)<br>cbax = fig.add_axes([0.89, 0.21, 0.03, 0.31])   <br><br>cbax.set_title(&#39;Percentage of state households\nexperiencing food insecurity\n&#39;, **hfont, fontdict={&#39;fontsize&#39;: &#39;15&#39;, &#39;fontweight&#39; : &#39;0&#39;})<br><br># add color scale<br>sm = plt.cm.ScalarMappable(cmap=colormap, \<br>                 norm=plt.Normalize(vmin=vmin, vmax=vmax))<br># reformat tick labels on legend<br>sm._A = []<br>comma_fmt = FuncFormatter(lambda x, p: format(x/100, &#39;.0%&#39;))<br>fig.colorbar(sm, cax=cbax, format=comma_fmt)<br>tick_font_size = 16<br>cbax.tick_params(labelsize=tick_font_size)<br># annotate the data source, date of access, and hyperlink<br>ax.annotate(&quot;Data: USDA Economic Research Service, accessed 15 Jan 23\nhttps://www.ers.usda.gov/topics/food-nutrition-assistance/food-security-in-the-u-s/key-statistics-graphics/#map&quot;, xy=(0.22, .085), xycoords=&#39;figure fraction&#39;, fontsize=14, color=&#39;#555555&#39;)<br><br><br># create map<br># Note: we&#39;re going state by state here because of unusual coloring behavior when trying to plot the entire dataframe using the &quot;value_determined_color&quot; column<br>for row in visframe.itertuples():<br>    if row.state not in [&#39;AK&#39;,&#39;HI&#39;]:<br>        vf = visframe[visframe.state==row.state]<br>        c = gdf[gdf.state==row.state][0:1].value_determined_color.item()<br>        vf.plot(color=c, linewidth=0.8, ax=ax, edgecolor=&#39;0.8&#39;)<br><br><br><br># add Alaska<br>akax = fig.add_axes([0.1, 0.17, 0.2, 0.19])   <br>akax.axis(&#39;off&#39;)<br># polygon to clip western islands<br>polygon = Polygon([(-170,50),(-170,72),(-140, 72),(-140,50)])<br>alaska_gdf = gdf[gdf.state==&#39;AK&#39;]<br>alaska_gdf.clip(polygon).plot(color=gdf[gdf.state==&#39;AK&#39;].value_determined_color, linewidth=0.8,ax=akax, edgecolor=&#39;0.8&#39;)<br><br><br># add Hawaii<br>hiax = fig.add_axes([.28, 0.20, 0.1, 0.1])   <br>hiax.axis(&#39;off&#39;)<br># polygon to clip western islands<br>hipolygon = Polygon([(-160,0),(-160,90),(-120,90),(-120,0)])<br>hawaii_gdf = gdf[gdf.state==&#39;HI&#39;]<br>hawaii_gdf.clip(hipolygon).plot(column=variable, color=hawaii_gdf[&#39;value_determined_color&#39;], linewidth=0.8,ax=hiax, edgecolor=&#39;0.8&#39;)<br><br><br><br>fig.savefig(os.getcwd()+&#39;/food_insecurity_by_state_2019_2021.png&#39;,dpi=400, bbox_inches=&quot;tight&quot;)<br># bbox_inches=&quot;tight&quot; keeps the vis from getting cut off at the edges in the saved png<br># dip is &quot;dots per inch&quot; and controls image quality.  Many scientific journals have specifications for this<br># https://stackoverflow.com/questions/16183462/saving-images-in-python-at-a-very-high-quality</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dtbDjDbSypyghG8_7qDwrw.png" /><figcaption>The chhoropleth results of the code with annotation for data and a continuous color bar legend</figcaption></figure><p><strong>Checking the Results:</strong></p><p>The map looks pretty good, but I’ve had issues in splitting Alaska and Hawaii off with continuous color scales before. Did the coloration go as expected?</p><p>Let’s choose a state with similar food insecurity to that of Alaska and make sure they are the same color.</p><pre># Make a horizontal bar chart of state food insecurity rates using Seaborn<br>visFrame = gdf.sort_values(&#39;pct_food_insecure&#39;,ascending=False)[:100]<br><br><br>#Horizontal barchart<br>#Create subplot<br>sns.set_style(&#39;whitegrid&#39;) #set theme<br>fig,ax=plt.subplots(figsize=(16,30))<br>#Create barplot<br>################################################################ ***** &quot;palette&quot; for colors...<br>chart1 = sns.barplot(x=visFrame[&#39;pct_food_insecure&#39;],y=visFrame[&#39;state&#39;],color=&#39;lightblue&#39;)<br>chart1.set_title(&#39;Percent of population food insecure by sate\n&#39;,weight=&#39;bold&#39;,fontsize=16)<br><br>chart1.set_xlabel(&#39;Percentage&#39;,weight=&#39;bold&#39;,fontsize=13)<br>chart1.set_ylabel(&#39;State&#39;, weight=&#39;bold&#39;,fontsize=13)<br>#Value number on chart: from &quot;max&quot; on stackoverflow https://stackoverflow.com/questions/49820549/labeling-horizontal-barplot-with-values-in-seaborn <br>for p in ax.patches:<br>    width = p.get_width()    # get bar length<br>    ax.text(width + .5,       # set the text at .5 unit right of the bar<br>            p.get_y() + p.get_height() / 2, # get Y coordinate + X coordinate / 2<br>            &#39;{:}%&#39;.format(round(width,1)), # set variable to display<br>            fontsize=15,<br>            ha = &#39;left&#39;,   # horizontal alignment<br>            va = &#39;center&#39;)  # vertical alignment<br><br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qFt9NOoqQwaZ9j9wgs8EoA.png" /></figure><p>Maine and Alaska are both 9.5%…. are they the same color on the map?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NxBBxaqI0g6e1YTzRbwyYg.png" /></figure><p>Great! We’ve successfully made a choropleth of USDA data with a continuous color scale and both Alaska and Hawaii projected underneath the continental US.</p><p>My full code for this exercise can be accessed on <a href="https://github.com/alexF3/map_us_hunger">github</a>. If you’re new to Python and want to follow through the code in a Jupyter Notebook click <a href="https://github.com/alexF3/map_us_hunger/blob/main/Python_mapping_tutorial.ipynb">here</a>.</p><h3>Acknowledgements</h3><p>This notebook is the product of a lot of time reading documentation, googling, and exploring. I haven’t been able to record all of the articles, posts, and stack overflow handles that provided useful inputs or perspectives.</p><p>Abdishakur on Medium <a href="https://medium.com/@shakasom">https://medium.com/@shakasom</a> has been a big influence on my work. I highly recommend following him.</p><p>I got my start in Python programming almost a decade ago when I finally couldn’t stand manualy scheduling all of the flights for my Air Force squadron anymore. Sentdex’s youtube channel made Python acccessible, practical, and fun for me when I new just about nothing about coding (<a href="https://www.youtube.com/@sentdex">https://www.youtube.com/@sentdex</a>). Harrison was among the most influential educators in my life, though I’ve never met him or spoken with him.</p><p>Both US Census and USDA make an amazing amount of data publicly available and have created an environment where open source tools can thrive. Both websites are worth a deep dive when you have the time.</p><h3>Questions/Comments/Feedback?</h3><p>alexrich@duck.com</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39a9f5c222c6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>