<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pablo Hernandez Leal on Medium]]></title>
        <description><![CDATA[Stories by Pablo Hernandez Leal on Medium]]></description>
        <link>https://medium.com/@hlealpablo?source=rss-d18f7b3c3588------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*P3X9bGVDrlERv0vRNjPTIA.jpeg</url>
            <title>Stories by Pablo Hernandez Leal on Medium</title>
            <link>https://medium.com/@hlealpablo?source=rss-d18f7b3c3588------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 18:00:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@hlealpablo/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[DeepSeek-OCR: Vision Compression and Text Extraction in Financial Reports]]></title>
            <link>https://medium.com/@hlealpablo/deepseek-ocr-vision-compression-and-text-extraction-in-financial-reports-34f9f3d6f3e8?source=rss-d18f7b3c3588------2</link>
            <guid isPermaLink="false">https://medium.com/p/34f9f3d6f3e8</guid>
            <category><![CDATA[deepseek]]></category>
            <category><![CDATA[deepseek-ocr]]></category>
            <category><![CDATA[vlm]]></category>
            <dc:creator><![CDATA[Pablo Hernandez Leal]]></dc:creator>
            <pubDate>Tue, 04 Nov 2025 20:12:40 GMT</pubDate>
            <atom:updated>2025-11-04T20:12:40.262Z</atom:updated>
            <content:encoded><![CDATA[<h3>1. From Retrieval to Extraction</h3><p>In a <a href="https://medium.com/@hlealpablo/interpretability-of-colpali-in-financial-documents-5a2dcdeeba3a">previous pos</a>t I explored <strong>ColPali</strong>, a vision-language retriever capable of finding relevant pages in financial filings directly from images.<br>This time, we go one step further: <strong>extracting structured text</strong> from those same documents — without any traditional OCR pipeline.</p><p>Unlike legacy systems (which detect boxes, recognize characters, and rebuild layout), <strong>DeepSeek-OCR</strong> treats OCR as a <strong>vision-to-text compression problem</strong>.<br>It converts an entire document image into a compact set of <em>vision tokens</em>, which are then decoded into Markdown, tables, or structured text — all in a single forward pass.</p><h3>2. What Makes DeepSeek-OCR Different</h3><p>DeepSeek-OCR was introduced by <em>DeepSeek-AI (Wei et al., 2025)</em> as an early proof-of-concept for what the authors call <strong>“contexts optical compression.”</strong><br>The architecture combines:</p><ul><li><strong>DeepEncoder</strong> — a hybrid of <em>SAM-base</em> (local attention) and <em>CLIP-large</em> (global attention) connected through a 16× convolutional compressor;</li><li><strong>DeepSeek-3B-MoE decoder</strong> — a lightweight mixture-of-experts language model that reconstructs text from the compressed visual representation.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C2QJJsL-wkC6qCy_RZaakQ.png" /><figcaption><a href="https://www.arxiv.org/pdf/2510.18234">The architecture of DeepSeek-OCR</a></figcaption></figure><p>According to the paper DeepSeek-OCR reaches around <strong>97% precision</strong> when compressing images <strong>ten times</strong>, holds <strong>about 90 %</strong> accuracy at <strong>twelve times</strong>, and still manages <strong>roughly 60 %</strong> when pushed to <strong>twenty times</strong> compression.</p><p>To understand how this model performs in financial contexts, I ran <strong>DeepSeek-OCR</strong> on the <a href="https://www.rbc.com/investor-relations/_assets-custom/pdf/ar_2024_e.pdf"><em>Royal Bank of Canada 2024 Annual Report</em></a>, a document rich in tables, footnotes, and mixed formatting.<br>The goal: evaluate consistency, accuracy, and layout awareness.</p><p>Hardware:</p><ul><li>Google Colab, L4 GPU (24 GB)</li><li>Processing time: ~<strong>40 s/page</strong> (no optimization)</li></ul><p>Software:</p><pre>!pip -q install &quot;transformers==4.46.3&quot; einops addict easydict pillow pymupdf<br>model_id = &quot;deepseek-ai/DeepSeek-OCR&quot;</pre><p>Each page was converted to an image using fitz and then passed to the model with the prompt:</p><pre>prompt=&quot;&lt;image&gt;\n&lt;|grounding|&gt;Convert the document to markdown.&quot;</pre><h3>4. Visual Results</h3><p>The model outputs a .mmd (Markdown) file for each page, including <strong>bounding boxes</strong>, <strong>text</strong>, and <strong>image references</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OSpCA8MS3FXjSsmD8XNHqA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6o6nK3k6eK_PH8fCbMgT-Q.png" /><figcaption><em>Figure 1.</em> Side-by-side comparison of the RBC “Our Purpose” page. Left: original PDF with blue corporate layout and iconography. Center: DeepSeek-OCR’s Markdown output showing extracted text and image placeholders. Right: rendered Markdown recreation showing clean hierarchy and spacing.</figcaption></figure><h3>Header and Footer Detection</h3><p>On the RBC report, I observed inconsistent inclusion — some pages missing headers or footers.<br>This is likely due to the <strong>context-based compression</strong>, where small, low-contrast regions are deprioritized.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Kx3j0htJ9IlfvtpQRg78g.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PLarDbd6pz3sPocu43y2PQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iYZOqZtDnllAzDkfJczcaA.jpeg" /><figcaption><em>Figure 2.</em> Three pages from the RBC 2024 Annual Report. The left page includes a full-page portrait and headline, the middle combines text blocks with quotes and footnotes, and the right contains multiple text zones and images. The colored boxes indicate text detected by DeepSeek-OCR. The red colored boxes indicate missing detected headers and footers.</figcaption></figure><h3>Table Extraction</h3><p>Financial statements are notoriously challenging: multi-row headers, merged cells, and dense numeric columns. DeepSeek-OCR’s results were mixed but surprisingly strong:</p><ul><li>On standard tables, text and alignment were preserved;</li><li>On complex multi-header tables, a few <strong>rows were merged</strong> or <strong>labels dropped</strong>;</li><li>On colored tables, Markdown formatting correctly represented nested sections.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*naUSCUWnFp75tGiJhJy4NA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*WhDMXvEnqVQ8EEYmYIWMsg.png" /><figcaption>Figure 3. Original RBC 2024 Annual Report table (Table 2) on the left and DeepSeek-OCR extracted Markdown rendering on the right. The table includes multiple header levels — segment results for Personal Banking, Commercial Banking, and Consolidated Results — with sub-columns for ‘Excluding HSBC Canada’, ‘HSBC Canada’, and ‘Total&#39;.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*U2f5tpnzNAgkgziPx7_-tQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*WdDUzrnHrEYDcAnmuLA0ng.png" /><figcaption>Figure 4. Comparison of original and extracted table from the RBC 2024 Annual Report. The left image shows the original PDF with “Branches” and “ATMs” correctly listed under “Number as at October 31.” The right image displays DeepSeek-OCR’s Markdown reconstruction, where those final rows are missing.</figcaption></figure><h3>5. Efficiency and Scale</h3><p>DeepSeek-OCR’s key engineering advantage is <strong>token efficiency</strong>.<br>Each page is encoded into as few as <strong>64–400 vision tokens</strong>, compared to thousands in traditional VLMs.</p><h4>Processing Performance</h4><p>In my tests, the model was extremely efficient, processing 41 pages in roughly 1,682 seconds (≈<strong>40 s/page</strong>) on a <strong>Google Colab L4 GPU</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GcqevEXSbtaQ5zRL9O8f8g.png" /><figcaption>Figure 5. Runtime statistics showing total pages processed, total time, total characters extracted, and average processing time per page for DeepSeek-OCR on RBC’s annual report.</figcaption></figure><h4>Resource Utilization</h4><p>DeepSeek-OCR maintains moderate GPU memory use and stable disk consumption throughout long batch runs, supporting its scalability for high-volume document pipelines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/1*gcaBopxo6h4i193xcjTkUw.png" /><figcaption>Figure 6. Colab system resource monitor during DeepSeek-OCR inference. The charts display stable RAM (≈14 GB of 53 GB), GPU usage (≈8.7 GB of 22.5 GB), and disk footprint (≈45 GB of 235 GB)</figcaption></figure><h3>7. From OCR to Vision-Native RAG</h3><p>What’s compelling about DeepSeek-OCR isn’t just its OCR performance — it’s that it opens the door to <strong>fully vision-native RAG pipelines</strong>.<br>Imagine combining:</p><ul><li><strong>ColPali</strong> for <em>retrieval</em> directly from document images; and</li><li><strong>DeepSeek-OCR</strong> for <em>extraction</em> of the selected pages.</li></ul><p>This means a RAG system could operate <strong>without text preprocessing</strong>, consuming only PDFs or scans — an attractive option for regulatory and financial data, where formatting carries semantic value.</p><h3>8. Conclusion</h3><p>DeepSeek-OCR reframes OCR as a <strong>compression and reasoning task</strong> rather than a detection pipeline.<br>Its results on financial reports show that models can now “read” and structure complex documents using only visual cues, at a fraction of the compute cost.</p><p>While it’s not perfect — occasional missing rows or headers remain — the approach is a major step toward multimodal understanding of long, visually rich documents.<br>For compliance, audit, and due-diligence workflows, this paradigm could finally make end-to-end visual document processing both explainable and efficient.</p><h3>Code</h3><p>Below some snippets of the code.</p><ol><li>Setup and model download.</li></ol><pre>get_ipython().system(&#39;pip -q install &quot;transformers==4.46.3&quot; &quot;tokenizers==0.20.3&quot; einops addict easydict pillow pymupdf&#39;)<br><br>import os, time, torch<br>from pathlib import Path<br>from PIL import Image<br>from transformers import AutoModel, AutoTokenizer<br>from google.colab import files<br>from IPython.display import display, Markdown<br><br>assert torch.cuda.is_available(), &quot;⚠️ Enable GPU: Runtime → Change runtime type → GPU&quot;<br>print(f&quot;✅ GPU: {torch.cuda.get_device_name(0)}&quot;)<br><br><br><br># ==============================================================================<br>print(&quot;📥 Loading DeepSeek-OCR...&quot;)<br>model_id = &quot;deepseek-ai/DeepSeek-OCR&quot;<br><br>tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)<br>if tok.pad_token is None:<br>    tok.pad_token = tok.eos_token<br><br>t0 = time.time()<br>model = AutoModel.from_pretrained(<br>    model_id,<br>    trust_remote_code=True,<br>    use_safetensors=True,<br>    attn_implementation=&quot;eager&quot;<br>).to(dtype=torch.bfloat16, device=&quot;cuda&quot;).eval()<br><br>print(f&quot;✅ Model loaded in {time.time()-t0:.1f}s&quot;)</pre><p>2. Upload a PDF and transform to images.</p><pre>print(&quot;📤 Upload your image or PDF:&quot;)<br>uploaded = files.upload()<br><br>if not uploaded:<br>    raise ValueError(&quot;No file uploaded&quot;)<br><br>FNAME = next(iter(uploaded))<br>print(f&quot;✅ File: {FNAME}&quot;)<br><br># If PDF, convert to image<br>if FNAME.lower().endswith(&#39;.pdf&#39;):<br>    import fitz<br>    print(&quot;📄 Converting PDF to images...&quot;)<br>    doc = fitz.open(FNAME)<br><br>    IMG_PATHS = []<br>    for i, page in enumerate(doc):<br>        pix = page.get_pixmap(matrix=fitz.Matrix(2, 2))  # 2x resolution<br>        img_path = f&quot;/content/page_{i+1}.png&quot;<br>        pix.save(img_path)<br>        IMG_PATHS.append(img_path)<br>        print(f&quot;  ✓ Page {i+1}/{len(doc)}&quot;)<br>     <br>    doc.close()<br>    print(f&quot;✅ Converted {len(IMG_PATHS)} pages&quot;)<br>else:<br>    IMG_PATHS = [f&quot;/content/{FNAME}&quot;]</pre><p>3. Extracting to markdown.</p><pre>import glob<br>if len(IMG_PATHS) &gt; 1:<br>    print(f&quot;\n📚 PROCESSING ALL {len(IMG_PATHS)} PDF PAGES&quot;)<br>    print(&quot;=&quot;*60)<br><br>    all_results = []<br><br>    for i, img_path in enumerate(IMG_PATHS):<br>        print(f&quot;\n📄 Page {i+1}/{len(IMG_PATHS)}&quot;)<br><br>        outdir = f&quot;/content/out_page_{i+1}&quot;<br>        os.makedirs(outdir, exist_ok=True)<br><br>        # Preprocess<br>        img = Image.open(img_path).convert(&quot;RGB&quot;)<br>        if max(img.size) &gt; 2000:<br>            s = 2000 / max(img.size)<br>            img = img.resize((int(img.width*s), int(img.height*s)))<br>            img_path_proc = f&quot;/content/_proc_page_{i+1}.png&quot;<br>            img.save(img_path_proc)<br>        else:<br>            img_path_proc = img_path<br><br>        # Run OCR<br>        t0 = time.time()<br>        with torch.inference_mode():<br>            result = model.infer(<br>                tok,<br>                prompt=&quot;&lt;image&gt;\n&lt;|grounding|&gt;Convert the document to markdown.&quot;,<br>                image_file=img_path_proc,<br>                output_path=outdir,<br>                base_size=1024,<br>                image_size=640,<br>                crop_mode=True,<br>                save_results=True<br>            )<br><br>        elapsed = time.time() - t0<br><br>        # Collect result<br>        mmd = glob.glob(f&quot;{outdir}/*.mmd&quot;)<br>        if mmd:<br>            with open(mmd[0], &#39;r&#39;, encoding=&#39;utf-8&#39;, errors=&#39;ignore&#39;) as f:<br>                text = f.read()<br>            all_results.append({<br>                &quot;page&quot;: i+1,<br>                &quot;time&quot;: elapsed,<br>                &quot;chars&quot;: len(text),<br>                &quot;text&quot;: text<br>            })<br>            print(f&quot;  ✅ {elapsed:.2f}s, {len(text)} chars&quot;)<br><br>    # Combine all pages<br>    print(f&quot;\n📄 COMBINED OUTPUT:&quot;)<br>    print(&quot;=&quot;*60)<br><br>    combined_text = &quot;\n\n---\n\n&quot;.join([r[&#39;text&#39;] for r in all_results])<br><br>    # Save combined<br>    combined_path = &quot;/content/all_pages_combined.md&quot;<br>    with open(combined_path, &#39;w&#39;, encoding=&#39;utf-8&#39;) as f:<br>        f.write(combined_text)<br><br>    print(f&quot;✅ Saved to: {combined_path}&quot;)<br>    print(f&quot;📊 Total pages: {len(all_results)}&quot;)<br>    print(f&quot;📊 Total time: {sum(r[&#39;time&#39;] for r in all_results):.2f}s&quot;)<br>    print(f&quot;📊 Total chars: {sum(r[&#39;chars&#39;] for r in all_results)}&quot;)<br>    print(f&quot;📊 Avg time/page: {sum(r[&#39;time&#39;] for r in all_results)/len(all_results):.2f}s&quot;)<br><br>    # Display combined<br>    print(&quot;\n&quot; + &quot;=&quot;*60)<br>    print(&quot;📝 FULL DOCUMENT:&quot;)<br>    print(&quot;=&quot;*60 + &quot;\n&quot;)<br>    display(Markdown(combined_text))<br>else:<br>    print(&quot;💡 Upload a multi-page PDF to use this cell&quot;)</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=34f9f3d6f3e8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I use LLMs for coding]]></title>
            <link>https://medium.com/@hlealpablo/how-i-use-llms-for-coding-56cfc1018926?source=rss-d18f7b3c3588------2</link>
            <guid isPermaLink="false">https://medium.com/p/56cfc1018926</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[chatgpt-o1]]></category>
            <category><![CDATA[modal]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[claude-sonnet]]></category>
            <dc:creator><![CDATA[Pablo Hernandez Leal]]></dc:creator>
            <pubDate>Fri, 21 Mar 2025 04:09:59 GMT</pubDate>
            <atom:updated>2025-03-21T04:09:59.015Z</atom:updated>
            <content:encoded><![CDATA[<p>LLMs have become an integral part of my coding. Not so long ago I’ve only used them for simple tasks (like boilerplate code), but recently I’ve tried to incorporate them into more complex tasks with some success. Now I currently use ChatGPT (4o, o3-mini-high, o1), Anthropic (Sonnet 3.5, 3.7, 3.7-thinking) and Cursor.</p><h3>1. Introduction</h3><blockquote><a href="https://simonwillison.net/2025/Mar/11/using-llms-for-code/">“My current favorite mental model is to think of them as an <strong>over-confident</strong> pair programming assistant who’s lightning fast at looking things up, can churn out relevant examples at a moment’s notice and can execute on tedious tasks without complaint.”</a></blockquote><p>That is a very accurate description of some LLMs (specially GPT-4o) but I see differences among models and I try to exploit their differences depending on the task.</p><p>Below I go through different examples with different levels of complexity.</p><h3>2. When I Need Quick Help or Reminders (Boilerplate code)</h3><p>One scenario could be the following: I have different runs of my experiment in folders. I need to read the files, put into a data structure, check for missing data, write some aggregated metrics, get a plot and a CSV with a report, <a href="https://stackoverflow.com/questions/3992199/what-is-boilerplate-code">i.e. boilerplate code.</a></p><p>I can write that, but it’s going to take me some time. Also there always some small details that I need to check, like what are the parameters of pd.merge and pd.join()? AlI don’t remember <a href="https://stackoverflow.com/questions/60115806/pd-na-vs-np-nan-for-pandas">what is the exact difference between </a><a href="https://stackoverflow.com/questions/60115806/pd-na-vs-np-nan-for-pandas">pd.nan() and </a><a href="https://stackoverflow.com/questions/60115806/pd-na-vs-np-nan-for-pandas">np.nan()?</a></p><p>In this cases I can describe the task and the LLM gives me a very good first draft. Is not perfect but it’s already 90% there with models like GPT-4o. In some cases it fails because of ambiguity in the language, which to be honest could be my fault. If I use a model like o3-mini-high the results are almost perfect.</p><h4>Prompt Example</h4><pre>I need code to read this json file<br><br>  {<br>    &quot;ruleNumber&quot;: 1,<br>    &quot;topic&quot;: &quot;Topic 1&quot;,<br>    &quot;fields&quot;: [&quot;24&quot;, &quot;25&quot;],<br>    &quot;machineCondition&quot;: {<br>      &quot;conditions&quot;: [<br>        { &quot;field&quot;: &quot;24&quot;, &quot;operator&quot;: &quot;equals&quot;, &quot;value&quot;: &quot;Yes&quot; },<br>        { &quot;field&quot;: &quot;25&quot;, &quot;operator&quot;: &quot;equals&quot;, &quot;value&quot;: &quot;No&quot; }<br>      ],<br>      &quot;relation&quot;: &quot;AND&quot;<br>    },<br>    &quot;humanCondition&quot;: &quot;If question 24 = &#39;Yes&#39; and question 25 = &#39;No&#39;&quot;,<br>    &quot;description&quot;: &quot;&quot;<br>  }<br>...<br><br>I want to analyse it.<br>Give me a report of the different topics and how many times they appear.<br>Also give me a report of the fields (those are lists so check individually)<br>and how many times they appear.<br>Lastly, generate a graph where each node is a one possible field<br>and an undirected edge for every possible rule.<br>Generate a report to read in text and csvs to have all the information.</pre><p>The above prompt gave me the following result with a clear, readable and structured code followed by an explanation and more details.</p><pre>import json<br>import csv<br>import networkx as nx<br>import matplotlib.pyplot as plt<br>from collections import Counter<br>from itertools import combinations<br><br>def load_data(filename):<br>    &quot;&quot;&quot;Load JSON data from a file.&quot;&quot;&quot;<br>    with open(filename, &#39;r&#39;) as f:<br>        # Assumes the file contains a JSON array of objects.<br>        data = json.load(f)<br>    return data<br><br>def topic_report(data):<br>    &quot;&quot;&quot;Count how many times each topic appears.&quot;&quot;&quot;<br>    topics = [entry[&#39;topic&#39;] for entry in data]<br>    topic_count = Counter(topics)<br>    return topic_count<br><br>def field_report(data):<br>    &quot;&quot;&quot;Count the occurrence of each field across all rules.&quot;&quot;&quot;<br>    fields = []<br>    for entry in data:<br>        # Each rule has a list of fields.<br>        fields.extend(entry.get(&#39;fields&#39;, []))<br>    field_count = Counter(fields)<br>    return field_count<br><br>def generate_fields_graph(data):<br>    &quot;&quot;&quot;<br>    Create an undirected graph where each node is a field.<br>    For each rule, every pair of fields gets an edge (if an edge already exists,<br>    its weight is incremented).<br>    &quot;&quot;&quot;<br>    G = nx.Graph()<br>    for entry in data:<br>        rule_fields = entry.get(&#39;fields&#39;, [])<br>        # Ensure each field is added as a node.<br>        for field in rule_fields:<br>            G.add_node(field)<br>        # Add edges between every pair of fields in the rule.<br>        for field1, field2 in combinations(rule_fields, 2):<br>            if G.has_edge(field1, field2):<br>                G[field1][field2][&#39;weight&#39;] += 1<br>            else:<br>                G.add_edge(field1, field2, weight=1)<br>    return G<br><br>def save_csv_report(counter, filename, header):<br>    &quot;&quot;&quot;Save a CSV report from a Counter object.&quot;&quot;&quot;<br>    with open(filename, &#39;w&#39;, newline=&#39;&#39;) as csvfile:<br>        writer = csv.writer(csvfile)<br>        writer.writerow(header)<br>        for key, count in counter.items():<br>            writer.writerow([key, count])<br>    print(f&quot;CSV report saved to {filename}&quot;)<br><br>def main():<br>    # Load data from JSON file (ensure the JSON file is a valid array of objects)<br>    data = load_data(&#39;data.json&#39;)<br><br>    # Generate and print topic report<br>    topic_counts = topic_report(data)<br>    print(&quot;Topic Report:&quot;)<br>    for topic, count in topic_counts.items():<br>        print(f&quot;  {topic}: {count}&quot;)<br>    save_csv_report(topic_counts, &#39;topic_report.csv&#39;, [&#39;Topic&#39;, &#39;Count&#39;])<br><br>    # Generate and print field report<br>    field_counts = field_report(data)<br>    print(&quot;\nField Report:&quot;)<br>    for field, count in field_counts.items():<br>        print(f&quot;  {field}: {count}&quot;)<br>    save_csv_report(field_counts, &#39;field_report.csv&#39;, [&#39;Field&#39;, &#39;Count&#39;])<br><br>    # Generate the fields graph and visualize it<br>    G = generate_fields_graph(data)<br>    pos = nx.spring_layout(G)  # positions for all nodes<br>    plt.figure(figsize=(6, 4))<br>    nx.draw_networkx(G, pos, with_labels=True, node_size=700, font_size=10)<br>    edge_labels = nx.get_edge_attributes(G, &#39;weight&#39;)<br>    nx.draw_networkx_edge_labels(G, pos, edge_labels=edge_labels, font_size=8)<br>    plt.title(&quot;Graph of Fields by Rule&quot;)<br>    plt.axis(&#39;off&#39;)<br>    plt.tight_layout()<br>    plt.savefig(&quot;fields_graph.png&quot;)<br>    plt.show()<br>    print(&quot;Graph saved as fields_graph.png&quot;)<br><br>if __name__ == &#39;__main__&#39;:<br>    main()<br><br>Explanation<br>Loading the JSON Data<br>The load_data() function reads the JSON file <br>(assumed to be an array of objects) and returns the list of rule dictionaries.<br><br>Generating Reports<br>The topic_report() function extracts topics from each rule and<br>counts their occurrences using Counter.<br>The field_report() function extracts individual fields (each rule’s <br>fields is a list) and counts how many times each field appears.<br>Saving CSV Files<br>The save_csv_report() function takes a Counter and writes it to a CSV file.<br>Two CSVs are created: one for topics and one for fields.<br><br>Graph Generation<br>The generate_fields_graph() function creates an undirected graph<br> using NetworkX. For every rule, it creates nodes for each field<br> and adds an edge between each pair of fields. The edge attribute<br> weight counts how many rules share that edge.<br><br>Visualization<br>The graph is visualized with matplotlib, labels for nodes and edge<br>weights are added, and the plot is saved as fields_graph.png.<br><br>You can update the file names (like &#39;data.json&#39;) to match your files.<br>This script gives you both textual output and CSV files for further analysis.</pre><p>In this case, I’ll be adding more functions and running the code so I’ll be validating if there’s any error.</p><h3>3. Generating “Pretty” or Polished Code</h3><p>A second common scenario is when I have code that works but it’s not documented. For example, I need to add typing, try-catch blocks, logging errors, proper documentation. That’s very easy for LLMs to do and that’s usually in a <strong><em>single try with GPT-4o.</em></strong></p><h4>Example Prompt</h4><pre>Help me to add typing and logging information.<br>Catch errors and add comments in the code.<br><br>def download_pdf(url):<br>    session = requests.Session()<br>    retry = Retry(connect=2, backoff_factor=0.5)<br>    adapter = SSLAdapter(max_retries=retry)<br>    session.mount(&#39;http://&#39;, adapter)<br>    session.mount(&#39;https://&#39;, adapter)<br><br>    response = session.get(url)<br>    response.raise_for_status()<br>    return BytesIO(response.content)<br>    </pre><p>LLM output was:</p><pre>def download_pdf(url: str) -&gt; BytesIO:<br>    &quot;&quot;&quot;<br>    Downloads a PDF from a given URL and returns a BytesIO object.<br>    Retries are configured for enhanced reliability.<br>    &quot;&quot;&quot;<br>    logging.info(&quot;Starting download of PDF from URL: %s&quot;, url)<br>    <br>    session = requests.Session()<br>    retry = Retry(connect=2, backoff_factor=0.5)<br>    adapter = SSLAdapter(max_retries=retry)<br>    session.mount(&#39;http://&#39;, adapter)<br>    session.mount(&#39;https://&#39;, adapter)<br><br>    try:<br>        response = session.get(url)<br>        response.raise_for_status()  # Raises HTTPError for 4xx/5xx responses<br>        logging.info(&quot;PDF downloaded successfully from URL: %s&quot;, url)<br>        return BytesIO(response.content)<br>    except requests.exceptions.RequestException as e:<br>        logging.error(&quot;An error occurred while trying to download the PDF: %s&quot;, e)<br>        raise Exception(f&quot;An error occurred while trying to download the PDF: {e}&quot;)</pre><p>Looking at the <em>diff</em> is the way to visually check where changes are proposed and verify the LLM output.</p><h3>4. Handling New, Unfamiliar Tasks (Non-Production)</h3><p>I’ve also tried in specific tasks for which I’m not an expert so I need some guidance. One important aspect is <strong>that I only do this with non-production code</strong>.</p><h4>Example</h4><p>At some point I had to transform a <a href="https://www.ultralytics.com/yolo">YOLO dataset</a> into a <a href="https://cocodataset.org/#home">COCO dataset. </a>I asked:</p><pre>I have a YOLO dataset (folders), how can I transform into a COCO dataset?</pre><p>GPT-4o gave me a generic but good description of the task.</p><pre>Transforming a YOLO dataset into a COCO dataset involves restructuring<br>the annotation format and directory organization to comply with the COCO<br>format. Here’s a step-by-step guide:<br><br>1. Understand the Differences Between YOLO and COCO Formats<br>YOLO Format:<br><br>Annotations are in individual .txt files, with each line representing <br>an object:<br>class_id center_x center_y width height<br>All values are normalized (0-1) relative to the image dimensions.<br>Images and .txt files are usually stored in the same directory or<br>similar subfolders.<br><br>COCO Format:<br><br>Annotations are in a single .json file.<br>Contains details like images, annotations, categories, etc.<br>Boxes are in absolute pixel values: [x_min, y_min, width, height].<br><br>2. Prepare Your YOLO Dataset<br>Ensure your YOLO dataset is organized properly:<br><br>Images: Usually in a folder (e.g., images/).<br>Annotations: Corresponding .txt files in another folder (e.g., labels/).<br><br>3. Install Required Libraries<br>You’ll need Python libraries like pycocotools and json for processing.<br><br>Install dependencies:<br>bash<br>pip install pycocotools<br><br>4. Write a Conversion Script<br>Here’s a Python script to convert YOLO annotations into COCO format:<br><br>import os<br>import json<br>from PIL import Image<br><br>def yolo_to_coco(yolo_dir, output_json, categories):<br>    images = []<br>    annotations = []<br>    annotation_id = 1<br><br>    # Loop through image and label files<br>    for img_id, filename in enumerate(os.listdir(os.path.join(yolo_dir, &quot;images&quot;))):<br>        if not filename.endswith((&quot;.jpg&quot;, &quot;.jpeg&quot;, &quot;.png&quot;)):<br>            continue<br><br>        # Image info<br>        img_path = os.path.join(yolo_dir, &quot;images&quot;, filename)<br>        img = Image.open(img_path)<br>        width, height = img.size<br><br>        images.append({<br>            &quot;id&quot;: img_id,<br>            &quot;file_name&quot;: filename,<br>            &quot;width&quot;: width,<br>            &quot;height&quot;: height<br>        })<br><br>        # Corresponding label file<br>        label_file = os.path.join(yolo_dir, &quot;labels&quot;, filename.replace(&quot;.jpg&quot;, &quot;.txt&quot;).replace(&quot;.png&quot;, &quot;.txt&quot;))<br>        if not os.path.exists(label_file):<br>            continue<br><br>        with open(label_file, &quot;r&quot;) as f:<br>            lines = f.readlines()<br><br>        for line in lines:<br>            line = line.strip().split()<br>            class_id = int(line[0])<br>            x_center, y_center, w, h = map(float, line[1:])<br><br>            # Convert YOLO format to COCO format<br>            x_min = (x_center - w / 2) * width<br>            y_min = (y_center - h / 2) * height<br>            box_width = w * width<br>            box_height = h * height<br><br>            annotations.append({<br>                &quot;id&quot;: annotation_id,<br>                &quot;image_id&quot;: img_id,<br>                &quot;category_id&quot;: class_id,<br>                &quot;bbox&quot;: [x_min, y_min, box_width, box_height],<br>                &quot;area&quot;: box_width * box_height,<br>                &quot;iscrowd&quot;: 0<br>            })<br>            annotation_id += 1<br><br>    # Create COCO format JSON<br>    coco_format = {<br>        &quot;images&quot;: images,<br>        &quot;annotations&quot;: annotations,<br>        &quot;categories&quot;: categories<br>    }<br><br>    # Save to JSON file<br>    with open(output_json, &quot;w&quot;) as json_file:<br>        json.dump(coco_format, json_file, indent=4)<br><br># Define categories (modify as per your dataset)<br>categories = [<br>    {&quot;id&quot;: 0, &quot;name&quot;: &quot;class_0&quot;},<br>    {&quot;id&quot;: 1, &quot;name&quot;: &quot;class_1&quot;},<br>    # Add more classes as needed<br>]<br><br># Convert YOLO to COCO<br>yolo_to_coco(&quot;path_to_yolo_dataset&quot;, &quot;output_coco_annotations.json&quot;, categories)<br><br>5. Customize the Script<br>Update categories with the appropriate class names and IDs.<br>Set path_to_yolo_dataset to the directory containing images/ and labels/ subfolders.<br><br>6. Validate Your COCO Dataset<br>Use tools like COCOAPI or visualization tools to ensure the conversion is correct.<br><br>7. Use COCO Dataset<br>Now, your dataset is in the COCO format and can be used with frameworks that support it, such as Detectron2, MMDetection, or PyTorch implementations.<br><br>If you encounter specific issues, let me know!</pre><p>I followed the steps and I used the code as starting point and adapted to my needs, again <em>the validation step is actually running the code and checking what is doing.</em></p><p>I still was not feeling entirely comfortable. It <em>seemed</em> like the code was working but I would have liked another pair of eyes to look at my code. That’s what models like O1 and 3.7 thinking are good so this was my prompt:</p><pre>Check if this code converts a YOLO folder into a COCO format<br>Give me a description of what is happening in each part of the code.<br>Evaluate possible bugs.<br><br># Code<br>...<br>..<br>.</pre><p>It gave me a very detailed result with 5 sections describing the possible mistakes and how to fix them.</p><pre>Below is a quick review of your script with the key points to watch out for.<br>In general, it does the bulk of the conversion correctly, <br>but there is one important bug that will prevent label files from being<br>found unless your images specifically end with .jpeg:<br><br>1. Label file path construction<br>label_file = os.path.join(yolo_dir, &quot;labels&quot;, filename.replace(&quot;.jpeg&quot;, &quot;.txt&quot;))<br><br>Problem: If your image file is named something.jpg or something.png, <br>filename.replace(&quot;.jpeg&quot;, &quot;.txt&quot;) will still return something.jpg, <br>and your code will never find a corresponding label file named something.txt.<br><br>Simple fix: Use os.path.splitext(...) to reliably replace the extension <br>with .txt, regardless of .jpg, .jpeg, or .png:<br><br>base_name = os.path.splitext(filename)[0]  # e.g., &quot;image&quot; if the file is &quot;image.jpg&quot;<br>label_file = os.path.join(yolo_dir, &quot;labels&quot;, base_name + &quot;.txt&quot;)<br><br>This way, the script will look for image.txt no matter what the original image extension is.<br><br>2. YOLO to COCO bounding box conversion<br>Your bounding box conversion is correct if your YOLO labels are in the<br>normalized format [x_center, y_center, width, height] ranging from 0 to 1:<br><br>x_min = (x_center - w / 2) * width<br>y_min = (y_center - h / 2) * height<br>box_width = w * width<br>box_height = h * height<br><br>This correctly maps normalized YOLO coordinates to the COCO<br>convention [x_min, y_min, box_width, box_height]. If your YOLO files were<br>instead using absolute pixel values, you would need to remove the<br>multiplications by width and height. But assuming the standard normalized<br>format, it’s fine.<br><br>3. Category definitions<br>...<br>4. Overall JSON structure<br>...<br>5. Summary of fixes<br>...</pre><p>It actually gave a good code review. For example, it found a simple but subtle extension issue (I was assuming .jpeg all the time). Similarly, told me my code was ok if I was using certain normalization. Might seem simple but spotting these issues early before they break something is powerful. The information is there, now it’s up to me to decide if need to take more preventive action or not.</p><h3>5. Implementing New Features: Many Experts Approach</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lwNbx5xBea-sNg9remGgFw.png" /><figcaption>I mostly use my conversations with LLMs in this seemingly chaotic way. In general if the context is very different or is starting to become noisy I need to create a new conversation.</figcaption></figure><p>This last example is for more complex tasks, for example adding a new feature on something that is running (production code). In this case I usually follow a pattern where I have <strong><em>different conversations</em></strong> about slightly <strong><em>different subtopics</em></strong> with <strong><em>different models</em></strong>.</p><ul><li>I want to have <strong>separate conversations</strong> because otherwise it becomes too noisy. I like that I can encapsulate and <strong>steer the conversation</strong> in some specific way.</li><li>I want to have <strong>different models</strong> because I need that exploration (<em>fast answers that give me options</em>) and exploitation phase (<em>slow but very accurate answers</em>) depending on the stage where am I in the process.</li><li><strong>Models like ChatGPT-4o are fast and they have broad knowledge</strong>, they have good initial ideas, they provide different options and in general provide foundational knowledge.</li><li><strong>Models like Sonnet 3.7 or o3-mini-high are more sophisticated but sometimes they are more stubborn</strong>. For example, I don’t like that during reasoning they evaluate different options and I’m waiting for all that information they are “thinking” but then the answer might be just one or two paragraphs <em>very</em> to the point. So, for example I use them to have a well-documented code typically isolated from my codebase.</li><li><strong>Models like Sonnet 3.7 thinking or o1 are superior than the other two (</strong>I still prefer o1 than o3-mini-high) but they might be either slow or just too much for any task, so I try to pick <em>when</em> to use them<strong>. </strong>For example, I have used 3.7 thinking in Cursor in the final stage of my process when I know exactly what I want to build, with certain data structures as input, certain outputs, checking for specific conditions while actually making the changes into production code so I need <em>surgical precision</em>.</li></ul><p>Let’s go through an example, where<em> my end goal is to add a new a new ranking step on top of my retrieval code.</em></p><h4>Research expert:</h4><p>On ChatGPT-4o I explain the problem:</p><pre>I’m working on retrieving some document ... <br>The retrieval works fine on getting general documents but it fails<br>in grasping specifics. <br>What are the methods to improve in this problem? <br>On my first step of the pipeline I reduce the target documents to...<br>I want the new approach to rank them... <br>Which models should I consider? </pre><p>GPT-4o understands retrieval context, identifies the ranking problem, and mentions cross-encoder models. It also provides an overview of how the pipeline will work. <em>My conversation continues</em> asking about differences among models proposed, I also ask about cross-encoder and bi-encoders, I ask about their use cases and limitations. I try to keep this conversation entirely on the problem, methods, libraries, and solution concepts, <em>hardly any code is needed here</em>.</p><h4>Coding expert:</h4><p>The next step is to move to implementation. Usually, I start a new chat with a more specific prompt providing information about the problem but also keeping a focus on the implementation.</p><pre>I want to implement sentence transformers for ranking.<br>I have an initial step that produces ...<br>Help me with the code and next steps.</pre><p>GPT-4o provides me with an initial code that I can start using in a Jupyter notebook if I want. My goal is to have an isolated component that I have fully tested and understood. Once I feel comfortable I move to the next step.</p><h4>Experiment expert:</h4><p>I need to tweak the code to run different experiments, evaluate hyperparameters, aggregate datasets, combine with previous code, consolidate results, and generate some plots and reports. I start <em>a new chat that contains the code of the previous step</em> and I add some high level comments.</p><p>If I need to refactor more complex code I tend to use o3-mini-high, example prompt below.</p><pre>This is my code to evaluate different embeddings<br><br>def compute_sentence_level_similarity():<br>...<br># More code<br><br>I want to modify by adding preprocessing and spliting texts. <br>Then use the embeddings from all the sentences and combine to get the score. <br>Refactor the below code which is doing part of that but <br>has extra details that are not relevant now.<br><br>def split_into_sentences():<br>...<br># More code<br>  <br><br>Combine the two pieces of code into a single unit.<br>Keep the same function names as the first part.<br>Check inputs and outputs. <br>Add comments and typing.</pre><h4>Implementation expert</h4><p>The final step is “just” refactoring the core of the code, in this case the ranking with specific inputs and specific outputs that align to the rest of my codebase.</p><pre>This is a new feature that implements a cross-encoder to rank documents.<br>You need to refactor to match the inputs of function ....<br><br># Code<br>...<br>...<br>...<br><br>Notes:<br>Understand the code components and functions.<br>Understand the task and how it connects to the rest of the code.<br>Make a plan.<br>Evaluate failure modes.<br>Make the changes.</pre><p>The notes section have some specific purpose:</p><p>- Understand the code components and functions. Understand the task and how it connects to the rest of the code.<em>The goal is for LLM to essentially repeat the task,</em> is like failsafe mechanism, sometimes, it might divert from what I said, so its a simple check to read what the LLM said before looking at the code.</p><p>- Make a plan. <em>Trying to induce some reasoning</em>, but it’s also another failsafe, it will usually list the steps to perform and I can confirm if those make sense.</p><p>- Evaluate failure modes.I’ve seen at least two ways this helps, either it recognizes issues before the code and then the code is more robust. Or mentions possible next steps (a list of TODOs).</p><h3>6. Pitfalls and Failures</h3><h4>Recent libraries</h4><ul><li>I need to remind Sonnet 3.7 that I’m using Vue 3 cause otherwise it defaults to Vue 2 and start making mistakes, this is probably something easy to fix, but you need to be careful if you don’t know what the LLM is assuming about your code.</li><li>I have also seen ChatGPT struggle integrating <a href="https://modal.com/">Modal</a> code by using deprecated styles and functions. An example of what GPT-4o gives is below:</li></ul><pre>Modify Your app.py for Modal<br>You&#39;ll need to adapt your app.py to work with Modal. <br>Here&#39;s how you can do it:<br><br>import modal<br><br># Define the Modal stub<br>stub = modal.Stub(&quot;your-app-name&quot;)<br><br># Specify GPU requirements<br>image = modal.Image.debian_slim().pip_install_from_requirements(&quot;requirements.txt&quot;)<br><br># Define a Modal function with GPU support<br>@stub.function(image=image, gpu=&quot;any&quot;, secret=modal.Secret.from_name(&quot;your-secret-name&quot;))<br>def run_streamlit():<br>    import streamlit as st<br>    # Your Streamlit code here<br>    st.title(&quot;My Streamlit App with GPU&quot;)<br>    # Add your GPU-accelerated code here<br><br># Entry point<br>if __name__ == &quot;__main__&quot;:<br>    with stub.run():<br>        run_streamlit()</pre><p>If you check directly in <a href="https://modal.com/docs/guide/apps">modal.com</a> you can read:</p><blockquote>Apps were once Stubs</blockquote><blockquote><a href="https://modal.com/docs/guide/apps">The </a><a href="https://modal.com/docs/guide/apps">App class in the client was previously called </a><a href="https://modal.com/docs/guide/apps">Stub. Both names are still supported, but </a><a href="https://modal.com/docs/guide/apps">Stub is an alias for </a><a href="https://modal.com/docs/guide/apps">App and will not be supported at some point in the future.</a></blockquote><h4>Masking the error</h4><p>Once I made the mistake of just passing error logs and keep accepting changes until it worked (this was done in Cursor). Of course that was not a smart decision, in the end the code ‘looked’ as if worked but it wasn’t, it was Sonnet adding mock functions or fallbacks:, which where<strong> just masking the error but not fixing it</strong>.</p><ul><li>One simple example is my code threw a KeyError when reading a JSON field. I passed that error log repeatedly to the LLM. Eventually, it created a new helper function:</li></ul><pre>def check_field_exists(data):<br>    # just return True for now<br>    return True</pre><p>That made the KeyError go away, but it <em>also</em> bypassed the logic that was supposed to validate or transform the field.</p><ul><li>Another common behaviour is to add fallbacks, for example adding like local paths:</li></ul><p>currentPath: import.meta.env.VITE_APP_FOLDER_PATH ||‘mylocal/path/here’</p><ul><li>Or adding localhost to URLs:</li></ul><p>const apiUrl = import.meta.env.VITE_API_URL || localhost</p><p>The first obvious lesson is that cycling pattern, that’s a clear problem; avoid that. The second lesson is that, those patterns are becoming easier to do with new tools like Cursor. That easiness of directly applying code changes with a click is a double edge sword. <strong>Use it responsibly.</strong></p><h3>7. Conclusions</h3><p>Over time, I’ve realized how important it is to pick the right LLM for each stage of development and to rely on my own expertise to verify the model’s output. For simple tasks — like adding documentation or looking up syntax — I can get a near-instant solution. For more complex or novel features, I treat the model as a brainstorming partner first and then switch to a more detailed, step-by-step approach. I clearly see an interplay of three dimensions.</p><h4>A. Task Complexity</h4><p><strong>Simple Tasks</strong>: Creating boilerplate, adding docstrings, or writing a helper script to parse CSVs.</p><ul><li><strong>Preferred Models</strong>: Fast LLMs like GPT-4o that can quickly churn out standard code.</li><li><strong>Likely Outcome</strong>: You’ll get near-instant code that’s correct on the first try, because these tasks are so common in training data.</li></ul><p><strong>Moderate Tasks</strong>: Integrating a smaller library, building a quick data pipeline, or refactoring a short script.</p><ul><li><strong>Preferred Models</strong>: Sonnet 3.7 or o3-mini-high. They can handle more detailed prompts and multiple edge cases.</li><li><strong>Likely Outcome</strong>: A few iterations might be needed, especially if you have custom requirements.</li></ul><p><strong>Complex Tasks</strong>: Creating a new feature in a production environment, or implementing a retrieval/ranking pipeline that spans multiple modules.</p><ul><li><strong>Preferred Models</strong>: All of them.</li><li><strong>Likely Outcome</strong>: You might have several back-and-forth conversations. You’ll want to <em>combine</em> the LLM’s suggestions with your domain expertise.</li></ul><h4>B. Model Capacity</h4><p><strong>Fast/Lower-Cost Models (e.g., GPT-4o)</strong></p><ul><li><strong>Strengths</strong>: Quick answers, cheaper tokens, good for brainstorming broad ideas.</li><li><strong>Weaknesses</strong>: Might hallucinate or oversimplify when the task is complicated, can struggle with large contexts.</li></ul><p><strong>Mid-Range or Specialized Models (e.g., 3.5)</strong></p><ul><li><strong>Strengths</strong>: Generally more “intelligent,” can handle moderate complexity, produce fewer errors.</li><li><strong>Weaknesses</strong>: Slower than basic models, higher cost, occasionally “stubborn” or over-elaborate in their reasoning.</li></ul><p><strong>High-Capacity Models (e.g., Sonnet 3.7-thinking, o1)</strong></p><ul><li><strong>Strengths</strong>: Best at reasoning through multi-step or domain-specific logic, can integrate large code contexts.</li><li><strong>Weaknesses</strong>: Highest cost, sometimes slow responses, might over-complicate small tasks.</li></ul><h4>C. Personal Knowledge</h4><p><strong>High Expertise in a Domain</strong>:</p><ul><li>You know exactly what correct code should look like; you just need help with syntax, boilerplate, or minor details.</li><li><strong>Interaction Strategy</strong>: Provide a minimal prompt, skip directly to verifying correctness. Likely fewer iterations.</li></ul><p><strong>Medium Expertise</strong>:</p><ul><li>You’re comfortable in the programming language but less familiar with the new library or framework.</li><li><strong>Interaction Strategy</strong>: Prompt the LLM for examples and usage patterns. Then rely on your existing debugging or testing skills. More back-and-forth might happen.</li></ul><p><strong>Low Expertise</strong>:</p><ul><li>You’re brand-new to either the language or the domain (e.g., YOLO/COCO conversion).</li><li><strong>Interaction Strategy</strong>: Ask the LLM to explain design patterns and best practices. You’ll need thorough testing or second opinions from advanced models.</li><li><strong>Risk</strong>: Over-relying on the LLM can lead to hidden mistakes because you might not catch them until you run the code.</li></ul><h3><strong>Key Takeaways</strong></h3><ul><li>LLMs excel at speeding up coding workflows, especially for standard library lookups and boilerplate tasks.</li><li>They are far from perfect: <strong>always test and verify code</strong>.</li><li>Splitting tasks among different “expert” models can be more effective than forcing a single LLM to handle every stage.</li><li>Keep in mind out-of-date training data and the need to provide context when using new libraries or frameworks.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56cfc1018926" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Extracting Token-Level “Importance” From Cross-Encoders For Quick & Simple Highlighting]]></title>
            <link>https://medium.com/@hlealpablo/extracting-token-level-importance-from-cross-encoders-for-quick-simple-highlighting-ee4cca36764b?source=rss-d18f7b3c3588------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee4cca36764b</guid>
            <category><![CDATA[cross-encoder]]></category>
            <category><![CDATA[reranking]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[attention]]></category>
            <category><![CDATA[explainability]]></category>
            <dc:creator><![CDATA[Pablo Hernandez Leal]]></dc:creator>
            <pubDate>Tue, 18 Mar 2025 18:40:56 GMT</pubDate>
            <atom:updated>2025-03-18T18:40:56.040Z</atom:updated>
            <content:encoded><![CDATA[<p>Attention as explanation</p><h3>Motivation</h3><p>When building text-ranking pipelines — especially for client-facing products — a common goal is to provide basic explainability: <strong>Why</strong> did the model select this sentence? <strong>Which words</strong> in the sentence or in the query are most crucial to the model’s decision?</p><p>While there are advanced explainability methods (e.g., integrated gradients), these can be complex to implement and expensive to run. Meanwhile, transformer-based models can produce attention data which might be useful as a basic mechanism of explainability. One advantage is that it is already available in most Hugging Face–style models with essentially “one extra argument” (output_attentions=True). This makes it a convenient, if imperfect, proxy for highlight-based explanations.</p><p>So if you already have a Hugging Face model (for instance, through a SentenceTransformers CrossEncoder), you can do a quick “attention-based highlight” with minimal overhead.</p><h3>What We’re After</h3><p>Imagine you have a <strong>cross-encoder</strong> re-ranker that takes (query, text) as input and returns a single relevance score. In many use-cases—like search or question answering—<strong>we also want to highlight</strong>:</p><ul><li><em>Which words in the candidate text</em> are most relevant to the model?</li><li><em>Which words in the query</em> is the text focusing on?</li></ul><p>By computing a token-level “importance” score, we can visually color-code them. For example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G2XlXturxHnHCd_cegMthA.png" /></figure><p>This doesn’t guarantee a ground-truth explanation, but it gives a quick sense of what the model <em>may</em> be focusing on.</p><h3>The (Basic) Attention Trick</h3><p>Transformers rely on <em>attention heads</em> to mix information between tokens. In a cross-encoder setting, your input typically looks like:</p><pre>[CLS] query_token1 query_token2 ... [SEP] text_token1 text_token2 ... [SEP]</pre><p>When you call the underlying Hugging Face model with output_attentions=True, you get back attention matrices from each layer, typically shaped like (batch_size, num_heads, seq_len, seq_len).</p><p><strong>High-level approach</strong>:</p><ol><li><strong>Grab the final-layer attention</strong> (or any layer you prefer).</li><li><strong>Average across all heads</strong> to get a single (seq_len, seq_len) matrix.</li><li>Identify the positions of the query tokens vs. the text tokens using the [SEP] boundaries.</li><li>For text token importance, sum (or average) the attention from all query positions → each text token.</li><li>For query token importance, sum (or average) the attention from all text positions → each query token.</li><li>(Optional) Do a simple min–max normalization so that each set’s scores lie in [0..1].</li><li>Return these “importance” values alongside the actual tokens, so you can color them in your UI.</li></ol><h3>Example Code Snippets</h3><h4>1) A Helper to Extract Token Importances</h4><pre>def compute_token_importances(<br>    query: str,<br>    text: str,<br>    cross_encoder_model,<br>    layer: int = -1<br>):<br>    &quot;&quot;&quot;<br>    Compute rough &#39;importance&#39; scores for both the text tokens and<br>    the query tokens by looking at final-layer attention in both directions.    Args:<br>        query: The query string.<br>        text: The candidate text.<br>        cross_encoder_model: A CrossEncoder (SentenceTransformers) object<br>                             that wraps a HF model + tokenizer.<br>        layer: Which layer&#39;s attention to use (default: last layer).<br>    <br>    Returns:<br>        tokens_text:      List of text tokens (subwords) from the candidate text portion.<br>        importances_text: List of floats (same length as tokens_text), normalized to [0..1].<br>        tokens_query:     List of query tokens (subwords).<br>        importances_query:List of floats (same length as tokens_query), normalized to [0..1].<br>    &quot;&quot;&quot;<br>    # 1) Tokenize and ensure we get attentions<br>    inputs = cross_encoder_model.tokenizer(<br>        query,<br>        text,<br>        return_tensors=&#39;pt&#39;,<br>        truncation=True<br>    )<br>    outputs = cross_encoder_model.model(<br>        **inputs,<br>        output_attentions=True<br>    )<br>    <br>    # 2) Final layer attention =&gt; shape: (1, n_heads, seq_len, seq_len)<br>    all_attentions = outputs.attentions<br>    final_layer_attn = all_attentions[layer][0]  # remove batch dim =&gt; (n_heads, seq_len, seq_len)<br>    <br>    # 3) Identify [SEP] boundaries<br>    input_ids = inputs[&quot;input_ids&quot;][0].tolist()<br>    sep_token_id = cross_encoder_model.tokenizer.sep_token_id<br>    try:<br>        sep_index = input_ids.index(sep_token_id)  # first [SEP]<br>        sep2_index = input_ids.index(sep_token_id, sep_index + 1)  # second [SEP]<br>    except ValueError:<br>        # fallback if we don&#39;t find them<br>        sep_index, sep2_index = 0, len(input_ids) - 1<br>    query_token_indices = range(1, sep_index)<br>    text_token_indices  = range(sep_index + 1, sep2_index)<br><br>    # 4) Average across all heads =&gt; shape: (seq_len, seq_len)<br>    attn_avg = final_layer_attn.mean(dim=0)<br><br>    # 5A) text importances = sum( query-&gt;text )<br>    importances_text = []<br>    for t_idx in text_token_indices:<br>        att_sum = sum(attn_avg[q_idx, t_idx].item() for q_idx in query_token_indices)<br>        importances_text.append(att_sum / (len(query_token_indices) or 1))<br>    importances_text = _normalize_scores(importances_text)<br><br>    # 5B) query importances = sum( text-&gt;query )<br>    importances_query = []<br>    for q_idx in query_token_indices:<br>        att_sum = sum(attn_avg[t_idx, q_idx].item() for t_idx in text_token_indices)<br>        importances_query.append(att_sum / (len(text_token_indices) or 1))<br>    importances_query = _normalize_scores(importances_query)<br><br>    # 6) Convert IDs to tokens<br>    text_ids = [input_ids[idx] for idx in text_token_indices]<br>    tokens_text = cross_encoder_model.tokenizer.convert_ids_to_tokens(text_ids)<br>    query_ids = [input_ids[idx] for idx in query_token_indices]<br>    tokens_query = cross_encoder_model.tokenizer.convert_ids_to_tokens(query_ids)<br>return tokens_text, importances_text, tokens_query, importances_query</pre><pre>def _normalize_scores(values):<br>    &quot;&quot;&quot;<br>    Simple min-max normalization for a list of float values.<br>    Returns a new list with each value scaled to [0..1].<br>    &quot;&quot;&quot;<br>    if not values:<br>        return []<br>    min_val = min(values)<br>    max_val = max(values)<br>    if max_val &gt; min_val:<br>        return [(v - min_val) / (max_val - min_val) for v in values]<br>    else:<br>        # all values the same<br>        return [0.5] * len(values)</pre><h4>2) Integrate This Into Your Re-Ranker</h4><p>Let’s say we already have a cross-encoder re-ranker function (rerank_candidates) that returns (candidate_id, candidate_text, cross_score). We can update it to also produce per-token data for both query and text:</p><pre>def rerank_candidates(<br>    query: str,<br>    candidate_ids: List[str],<br>    candidate_texts: List[str],<br>    cross_encoder_model,<br>    rerank_threshold: float<br>):<br>    &quot;&quot;&quot;<br>    Use a cross-encoder to rerank the candidate groups.<br>    Also return per-token data for quick highlighting.<br>    &quot;&quot;&quot;<br>    # 1) Generate cross-encoder scores<br>    pairs = [(query, text) for text in candidate_texts]<br>    cross_scores = cross_encoder_model.predict(pairs)  # shape =&gt; [N]<br>    <br>    # 2) Optional: transform the raw logits with a sigmoid<br>    transformed_scores = sigmoid_transform(cross_scores)<br><br>    # 3) Build the final results list<br>    results = []<br>    for cid, text, score in zip(candidate_ids, candidate_texts, transformed_scores):<br>        if score &gt;= rerank_threshold:<br>            # compute token-level importance<br>            (tokens_text, importances_text,<br>             tokens_query, importances_query) = compute_token_importances(query, text, cross_encoder_model)<br>            <br>            per_token_data = {<br>                &quot;tokens_text&quot;: tokens_text,<br>                &quot;importances_text&quot;: importances_text,<br>                &quot;tokens_query&quot;: tokens_query,<br>                &quot;importances_query&quot;: importances_query<br>            }<br>            <br>            results.append((cid, text, score, per_token_data))<br>    # 4) Sort by descending score<br>    results.sort(key=lambda x: x[2], reverse=True)<br>    return results</pre><p>Now each re-ranked entry has both a final score and the information needed to highlight in a user interface. For example, you can color each token in tokens_query with the importances_query values.</p><h3>Why This Is Almost “Free”</h3><p>If your cross-encoder is already a <strong>Hugging Face–based</strong> model (e.g., AutoModelForSequenceClassification) under the hood, you only need to add:</p><pre>output_attentions=True</pre><p>to retrieve these matrices. This typically adds a small overhead in compute and memory but is far cheaper than running a separate advanced model or method.</p><h3>Handling Subword Tokens More Robustly</h3><p>Transformer-based architectures often split words into subwords (e.g., &quot;doc&quot; + &quot;##1&quot;) via byte-pair encoding or WordPiece. In user-facing explanations, partial words can be confusing and hamper readability. Below is a simple function you might include to merge subwords:</p><pre>def merge_subwords(tokens, importances, subword_prefix=&quot;##&quot;):<br>    &quot;&quot;&quot;<br>    Merge subword tokens (with their importance scores) back into whole words.<br>    Args:<br>        tokens (List[str]): e.g. [&quot;doc&quot;, &quot;##1&quot;, &quot;is&quot;, &quot;awesom&quot;, &quot;##e&quot;]<br>        importances (List[float]): same length as tokens<br>        subword_prefix (str): the prefix that indicates a subword (default: &quot;##&quot;)<br>    <br>    Returns:<br>        merged_tokens (List[str]): e.g. [&quot;doc1&quot;, &quot;is&quot;, &quot;awesome&quot;]<br>        merged_importances (List[float]): aggregated/averaged importance<br>    &quot;&quot;&quot;<br>    merged_tokens = []<br>    merged_importances = []<br>    <br>    current_word = &quot;&quot;<br>    current_scores = []<br>    <br>    for tok, score in zip(tokens, importances):<br>        if tok.startswith(subword_prefix):<br>            # strip the prefix and append to the current word<br>            part = tok[len(subword_prefix):]<br>            current_word += part<br>            current_scores.append(score)<br>        else:<br>            # if there&#39;s a current word, finalize it before starting a new one<br>            if current_word:<br>                # average the importance across the subwords or sum, your choice<br>                merged_tokens.append(current_word)<br>                merged_importances.append(sum(current_scores) / len(current_scores))<br>            <br>            # start a new word<br>            current_word = tok<br>            current_scores = [score]<br>    <br>    # finalize the last word if it exists<br>    if current_word:<br>        merged_tokens.append(current_word)<br>        merged_importances.append(sum(current_scores) / len(current_scores))<br>    return merged_tokens, merged_importances</pre><h4>How to Integrate</h4><ul><li><strong>Step 1</strong>: After you’ve computed your token-level importance scores, you’ll have a list of tokens like [&quot;doc&quot;, &quot;##1&quot;, &quot;is&quot;, &quot;awesom&quot;, &quot;##e&quot;] and corresponding scalar importance scores.</li><li><strong>Step 2</strong>: Call merge_subwords(...) to produce a consolidated, more human-readable list of “whole words.”</li><li><strong>Step 3</strong>: Either <strong>average</strong> subword importances (like in the example), or <strong>sum</strong> them (sometimes summation is better for strongly weighting multi-subword tokens).</li><li><strong>Step 4</strong>: Display or highlight the merged tokens in your user interface.</li></ul><p>This ensures that you don’t highlight half a word or show repeated partial tokens to end users. The user sees text segments (whole words) that correspond more closely to everyday language.</p><h3>Example</h3><p>Let’s look at how this appears in a simplified example. The query is:</p><pre>In each branch office there is at least one full time employee <br>responsible for AML, CTF &amp; Sanctions Compliance.</pre><p>After processing through the cross encoder with another text and using the importances score we got the following outputs.</p><pre> <br>&quot;tokens_query&quot;: [<br>   &quot;in&quot;, &quot;each&quot;, &quot;branch&quot;, &quot;office&quot;, &quot;there&quot;, &quot;is&quot;, &quot;at&quot;, &quot;least&quot;,<br>   &quot;one&quot;, &quot;full&quot;, &quot;time&quot;, &quot;employee&quot;, &quot;responsible&quot;, &quot;for&quot;, &quot;am&quot;, <br>   &quot;##l&quot;, &quot;,&quot;, &quot;ct&quot;, &quot;##f&quot;, &quot;&amp;&quot;, &quot;sanctions&quot;, &quot;compliance&quot;, &quot;.&quot;<br>],<br>&quot;importances_query&quot;: [<br>   0.0000865784, 0.0000214740, 0.0030454936, 0,<br>   0.0000680802, 0.0000547303, 0.0003670414, 0.0026183664,<br>   0.0000771134, 0.0660208388, 0.0002218507, 0.0004304213,<br>   0.0009336084, 0.0000681424, 0.5622545657, 0.6776311835,<br>   0.0044402866, 0.4619748111, 0.3838213566, 0.0145738415,<br>   0.1260994888, 0.0130535502, 1<br>]</pre><p>We see that &quot;am&quot;, &quot;##l&quot;, &quot;ct&quot;, and &quot;##f&quot; have high scores, along with &quot;sanctions&quot; and &quot;.&quot;. Using the merge function, &quot;am&quot; + &quot;##l&quot; can become &quot;aml&quot;, and &quot;ct&quot; + &quot;##f&quot; becomes &quot;ctf&quot;. This way, the final displayed tokens better reflect common words and acronyms like “AML” and “CTF.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6oqg8vKvGv_LKfsZmBD0QA.png" /></figure><h3>Limitations &amp; Research Context</h3><p><strong>Attention != Explanation</strong><br> There is more research about the idea whether attention truly explains model decisions. It’s a <em>useful heuristic</em> but doesn’t necessarily reflect deep causal logic of the model. Take a look at more recent works in the last section of the post.</p><p><strong>Normalization Choices</strong><br> We used a simple min–max normalization <em>per side</em> (query or text). Alternatively, you can run a single min–max across both sets combined or apply other normalizations. The best approach can depend on your preferences.</p><p><strong>Compressing Model Outputs</strong><br> We’re squashing a model’s entire multi-head, multi-layer complexity into a single scalar per token. This is a huge compression of information, so do keep in mind that the final highlight is <strong>very</strong> approximate.</p><h3>Variations</h3><ul><li><strong>One-way vs. Two-way</strong>: We showed computing attention from query → text and from text → query. Some might only focus on query → text. Others might average both directions.</li><li><strong>Layer Choice</strong>: Some people look at early layers (where tokens might hold more local meaning) vs. last layers (where tokens might have more contextual mixing).</li><li><strong>Summing vs. Averaging</strong>: Summation gives stronger emphasis to tokens heavily attended by multiple heads or positions, while averaging might keep scores more comparable across texts of different lengths.</li></ul><h3>Conclusion</h3><p>Leveraging attention weights for simple highlight-based explanations is an approachable way to give users insight into why a cross-encoder ranks a sentence highly. It takes minimal code changes if you’re already using a Hugging Face or SentenceTransformers CrossEncoder.</p><p>However, remember that:</p><ul><li><strong>Attention-based highlighting is not a perfect explanation.</strong></li><li><strong>It’s just one piece in a broader suite of interpretability methods.</strong></li><li><strong>Still, it’s often good enough for a quick “visual cue” in user-facing applications.</strong></li></ul><h3>Related Works on Attention-Based Explainability</h3><p>While using attention weights as a quick highlight for token relevance remains popular and convenient, recent research has explored ways to make these highlights more reliable, nuanced, and <em>truly</em> reflective of model decision-making.</p><ol><li><a href="https://arxiv.org/abs/2503.07518"><strong>TokenButler</strong> (Akhauri et al. 2025)</a> proposes learning a <em>dedicated predictor</em> that identifies crucial tokens in large language model contexts. This predictor approximates attention-like scores with lower overhead and helps preserve vital tokens while pruning unimportant ones — an approach particularly relevant for memory-intensive or long-context tasks.</li><li><a href="https://proceedings.neurips.cc/paper_files/paper/2022/file/20e45668fefa793bd9f2edf19be12c4b-Paper-Conference.pdf"><strong>AttCAT</strong> (Deng et al. 2022)</a> extends beyond raw attention by <em>disentangling</em> token interactions (via attention) from token-specific magnitudes (via skip connections) and using gradient-based attribution. The resulting Attentive Class Activation Tokens method can highlight both <em>positive and negative</em> contributing tokens, producing more fine-grained explanation than raw attention alone.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ca_LZVMsGdC85PhH11uiqA.png" /></figure><p><strong>3. </strong><a href="https://ojs.aaai.org/index.php/AAAI/article/view/26517"><strong>SEAT</strong> (Hu et al. 2023)</a> addresses how standard attention may be <em>unstable</em> — small changes in model seeds or token embeddings can lead to drastically different attention maps. SEAT proposes a framework that ensures <em>robustness</em> in both the model’s output distribution and the highlight patterns, preserving the top tokens that matter for explanation.</p><p>These recent works underscore that raw attention is often a starting point rather than the final word in explainability.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee4cca36764b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Edge Cases in Financial Document Extraction]]></title>
            <link>https://medium.com/@hlealpablo/edge-cases-in-financial-document-extraction-cd0ee7f92774?source=rss-d18f7b3c3588------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd0ee7f92774</guid>
            <category><![CDATA[financial-documents]]></category>
            <category><![CDATA[tesseract]]></category>
            <category><![CDATA[ocr]]></category>
            <category><![CDATA[vlm]]></category>
            <category><![CDATA[data-extraction]]></category>
            <dc:creator><![CDATA[Pablo Hernandez Leal]]></dc:creator>
            <pubDate>Thu, 20 Feb 2025 21:43:14 GMT</pubDate>
            <atom:updated>2025-02-20T21:43:14.756Z</atom:updated>
            <content:encoded><![CDATA[<p>Financial documents — ranging from invoices and statements to regulatory filings — are a staple of business operations. But extracting key insights from these documents automatically can be tricky, especially when you’re juggling a mix of digital files, scans, and inconsistent layouts. In this post, we’ll walk through a series of common “<em>edge cases</em>” that often stump machine learning (ML) pipelines for financial data extraction, and share ideas for tackling each challenge.</p><h3>1. Low Resolution or Poor Scan Quality</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*961XdBdAHZop5i-j5LiVGg.png" /></figure><ul><li><strong>Problem</strong>: Low DPI scans, or images captured by smartphone cameras often produce noise or blurriness.</li><li><strong>Why It Matters</strong>: If your OCR engine (e.g., <a href="https://github.com/tesseract-ocr/tesseract">Tesseract</a>, AWS Textract, <a href="https://github.com/PaddlePaddle/PaddleOCR">PaddleOCR</a>) can’t confidently recognize text, you’ll end up with garbled or missing information. <strong><em>In finance, missing a digit can have major implications.</em></strong></li><li><strong>How to fix? Preprocessing</strong>: Enhance image quality using Python’s <a href="https://github.com/opencv/opencv">OpenCV</a> (e.g., upsampling, denoising, thresholding).</li></ul><h3>2. Skewed or Rotated Documents</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*qX85ttk0dTht71ROzYZw8Q.png" /></figure><ul><li><strong>Problem</strong>: Physical documents are sometimes scanned at an angle, resulting in skew or rotation.</li><li><strong>Why It Matters</strong>: OCR and layout detection models assume mostly horizontal text. Even a small tilt can lead to severe misreads of critical financial data.</li><li><strong>How to fix?</strong> <strong>Deskew with OpenCV</strong>: Functions like cv2.getRotationMatrix2D can correct angles automatically.</li></ul><h3>3. Complex or Irregular Table Structures</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lgYhhPTdajkLmjzzJH6e_w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8cL1r1jtilsQqgaWVlxVuQ.png" /></figure><ul><li><strong>Problem</strong>: Multi-row headers, merged cells, nested tables, or variable row spans can break basic table extraction approaches.</li><li><strong>Why It Matters</strong>: Financial documents often rely on tables to list transactions, account details, or reconciliations. Misaligned rows or columns can lead to incorrect data mapping, potentially causing reconciliation errors or faulty reporting.</li><li><strong>Fix Idea</strong>: <strong>Table Extraction Libraries</strong>: Tools like <a href="https://github.com/camelot-dev/camelot">Camelot</a> or <a href="https://github.com/tabulapdf/tabula">Tabula</a> excel at parsing simple tables. For more complex layouts, consider deep learning–based layout parsing frameworks (e.g., <a href="https://github.com/doc-analysis/DocBank">DocBank + LayoutLM</a> or <a href="https://github.com/huggingface/transformers/tree/main/examples/research_projects/layoutlm">PubLayNet-based models</a>). <strong>Custom Heuristics</strong>: Combine line detection (via OpenCV) with domain knowledge. For instance, automatically detect multi-line headers and merge them before capturing the data rows. <strong>Advanced Evaluation</strong>: Refer to <a href="#"><strong>RD-TableBench</strong></a>, an open benchmark designed specifically for testing extraction under challenging scenarios (merged cells, multiple languages, handwriting). RD-TableBench uses hierarchical alignment metrics to gauge how close your extracted table is to ground truth, giving you a more nuanced understanding of accuracy.</li></ul><h3>4. Overlapping Text, Signatures, Stamps, or Seals</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/629/1*gxZ78SRpL7L_ljvJjeBXsw.png" /></figure><ul><li><strong>Problem</strong>: Official stamps or signatures that overlap with crucial text can obscure or deform characters, confusing OCR.</li><li><strong>Why It Matters</strong>: Missing out on an overlapping account number can derail reconciliation or compliance checks.</li><li><strong>Fix Idea</strong>: <strong>Masking and Segmentation</strong>: Use color-based or morphological operations in OpenCV to isolate and remove stamps or watermarks before running OCR. <strong>Multi-Layer Approaches</strong>: Some advanced pipelines try to detect stamps (as separate “layers” in the image) using an object detection model (e.g., YOLO, Detectron2) and then reconstruct text from the “underlying” layer.</li></ul><h3>5. Multiple Languages or Scripts</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MsUQg_tlw7WYO0nNWKk0Xg.png" /></figure><ul><li><strong>Problem</strong>: Financial documents in multilingual contexts or with regional language footnotes can stump a single-language OCR model.</li><li><strong>Why It Matters</strong>: If parts of the text are misread or ignored, you might lose important disclaimers or smaller detail lines that are essential for compliance in certain locales.</li><li><strong>Fix Idea</strong>: <strong>Multilingual OCR</strong>: <a href="https://tesseract-ocr.github.io/tessdoc/TrainingTesseract">Tesseract can be trained</a> for multiple languages. Tools like <a href="https://github.com/PaddlePaddle/PaddleOCR">PaddleOCR</a> also offer multilingual support. <strong>Language Detection</strong>: Use <a href="https://github.com/Mimino666/langdetect">langdetect</a> or a transformer-based model (via <a href="https://huggingface.co/models?filter=language-identification">Hugging Face</a>) to identify the language, then route text to the appropriate OCR engine or pipeline.</li></ul><h3>6. Very Small or Condensed Fonts</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Vf3kJ5XEs3Fad7OEeeNWQ.png" /></figure><ul><li><strong>Problem</strong>: Fine print is common in disclaimers, footnotes, or legal text in financial documents.</li><li><strong>Why It Matters</strong>: Missing disclaimers or micro-text can lead to regulatory issues or inaccurate interpretation of rates, terms, etc.</li><li><strong>Fix Idea</strong>: <strong>Super-Resolution</strong>: Use deep learning super-resolution models on the text regions before OCR to enlarge the text. <strong>Segment-and-Zoom</strong>: Automatically detect small text boxes (perhaps using bounding-box detection from a layout analysis model) and process them separately in a higher-resolution OCR pass.</li></ul><h3>Putting It All Together</h3><p>Building a reliable information extraction system for financial documents is as much about clever pre-processing and layout analysis as it is about picking the right ML algorithms. With financial data, precision is paramount — incorrectly parsed numbers and metadata can lead to regulatory issues, missed revenue, or compliance headaches.</p><p>Here are some final tips for ensuring your pipeline can handle the variety of edge cases:</p><ol><li><strong>Pipeline Architecture</strong>: Structure your solution to incorporate iterative steps — document ingestion, preprocessing (deskew, de-noise), layout detection, OCR, and finally entity parsing/validation.</li><li><strong>Data Quality Checks</strong>: Implement confidence thresholds at multiple stages (for instance, rejecting a page if the OCR confidence is too low) or using domain-specific checks (e.g., verifying account numbers against known formats).</li><li><strong>Domain-Specific Heuristics: </strong>Incorporate <strong>financial dictionaries</strong>, recognized currency symbols, known vendor lists, or relevant domain constraints. This helps to catch errors or “hallucinations” from any OCR or post-processing step.</li><li><strong>Incremental Improvement</strong>: Start with open-source frameworks like Tesseract + Python’s OpenCV for deskewing, then layer in advanced methods (e.g., LayoutLM, custom handwriting models) where you see the biggest accuracy drops. Evaluate each iteration using consistent test sets or specialized benchmarks (like <strong>RD-TableBench</strong> for tables) to track your progress.</li></ol><h3>How about LLMs/VLMs?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*a8UZF80H-LhDcxCDrDgT8Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*3cZrDu1eypwFrLT2LWtqXA.png" /></figure><p>LLMs and Vision-Language Models (VLMs) are increasingly used to correct OCR errors or parse complex layouts. These can work well in many scenarios — but for <strong>high-stakes financial data</strong>, you need to be extra cautious:</p><ul><li><strong>Hallucinations</strong>: Large models sometimes invent text or numeric values not present in the original, leading to significant risk if a critical dollar amount changes.</li><li><strong>Language Switching</strong>: When source text is heavily degraded or partially in another language, an LLM might “correct” entire passages into a different language.</li><li><strong>Domain Validation</strong>: If you do use an LLM-based approach, <strong>cross-verify</strong> recognized values. A final domain-specific validation layer can mitigate potential errors introduced by these models.</li></ul><h3>Further Reading / References</h3><ul><li><a href="https://github.com/tesseract-ocr/tesseract"><strong>Tesseract OCR</strong></a></li><li><a href="https://github.com/PaddlePaddle/PaddleOCR"><strong>PaddleOCR</strong></a></li><li><a href="https://github.com/Layout-Parser/layout-parser"><strong>LayoutParser</strong></a></li><li><a href="https://github.com/camelot-dev/camelot"><strong>Camelot for PDF Table Extraction</strong></a></li><li><a href="https://github.com/microsoft/unilm/tree/master/layoutlm"><strong>LayoutLM for Document Understanding</strong></a></li><li><a href="#"><strong>RD-TableBench</strong></a> (Benchmark for evaluating table extraction performance on complex, real-world data)</li><li><a href="#"><strong>OCRonos</strong></a></li></ul><p>By addressing these edge cases with thoughtful engineering, domain-specific checks, and careful benchmarking, your information extraction pipeline will be better equipped to deliver clean, accurate, and actionable data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd0ee7f92774" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Your OCR Results Differ: The Hidden Role of Embedded Fonts in PDFs]]></title>
            <link>https://medium.com/@hlealpablo/why-your-ocr-results-differ-the-hidden-role-of-embedded-fonts-in-pdfs-ec38c9447d1f?source=rss-d18f7b3c3588------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec38c9447d1f</guid>
            <category><![CDATA[embedded-fonts]]></category>
            <category><![CDATA[modal]]></category>
            <category><![CDATA[poppler]]></category>
            <category><![CDATA[ocr]]></category>
            <category><![CDATA[tesseract]]></category>
            <dc:creator><![CDATA[Pablo Hernandez Leal]]></dc:creator>
            <pubDate>Tue, 04 Feb 2025 16:12:42 GMT</pubDate>
            <atom:updated>2025-02-04T16:12:42.054Z</atom:updated>
            <content:encoded><![CDATA[<p>Solving differences when using Poppler</p><p>When working with <strong>OCR (Optical Character Recognition) for document processing</strong>, achieving consistent results can be challenging. You might assume that using the <strong>same OCR engine, same libraries, and same preprocessing steps</strong> should give identical outputs across different environments. However, subtle differences in <strong>how PDFs handle fonts</strong> can significantly impact OCR accuracy.</p><p>In this post, we’ll explore <strong>why font embedding matters in PDFs</strong>, how it can introduce OCR inconsistencies, and how you can diagnose and fix these issues.</p><h3>1. The Problem: Same Code, Different OCR Outputs</h3><p>Imagine you’re processing financial documents, invoices, or compliance forms using OCR. You extract text from PDFs using <strong>the same OCR model (for example </strong><a href="https://github.com/tesseract-ocr/tesseract"><strong>Tesseract</strong></a><strong>), the same preprocessing pipeline (for example </strong><a href="https://poppler.freedesktop.org/"><strong>Poppler</strong></a><strong>), and the same machine learning model</strong> — but get <strong>slightly different results across environments</strong>.</p><p>This inconsistency is frustrating because:</p><ul><li>Some text is missing or misrecognized.</li><li>The font rendering is different between local and production environments.</li><li>The layout appears unchanged, but the extracted text has errors.</li></ul><p>After verifying that your OCR libraries are <strong>identical</strong>, the next suspect is <strong>how fonts are handled inside PDFs</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/996/1*ph091y6yCDB0MAEjxbQxYA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uyXg6opOmAT8zAorsckVpQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nhn3NOieJeqH_7l8Ohz9-Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*306eE-JeU6GRkggPtaOLZQ.png" /><figcaption>On the left screenshots from a PDF, on the right images converted using <a href="https://poppler.freedesktop.org/">Poppler </a>on a linux server. Red lines highlight differences in the spacing of some words. While these differences look minor they propagate to the rest of the pipeline.</figcaption></figure><h3>2. Why Fonts in PDFs Affect OCR</h3><p>PDFs store text differently from standard image-based documents. The most critical factor is <strong>whether fonts are embedded or not</strong>.</p><h3>📌 Embedded Fonts vs. System Fonts</h3><ul><li><strong>Embedded Fonts (✅ Good for Consistency)</strong><br>When a font is <strong>embedded</strong>, the exact glyphs used in the document are stored inside the PDF. This ensures that text looks identical regardless of where the file is opened or processed.</li><li><strong>Non-Embedded Fonts (⚠️ May Cause OCR Issues)</strong><br>If a font is <strong>not embedded</strong>, the system rendering the PDF <strong>substitutes</strong> the font with a similar one.</li><li>This can lead to: <strong>OCR misinterpretation of certain characters (e.g., ‘I’ and ‘l’ might look the same) and changes in word segmentation</strong>, affecting OCR results.</li></ul><h3>3. How to Check If Fonts Are Embedded in a PDF</h3><p>To check whether a PDF has <strong>embedded fonts</strong>, use pdffonts, a tool from the poppler-utils package.</p><pre>pdffonts yourfile.pdf</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WtcRAPDt0IHK47UMieonQQ.png" /><figcaption>In this example there are three fonts (in the red box) which are not embedded and if the system doesn’t have them they will be substituted with a different one.</figcaption></figure><ul><li>If <strong>emb = yes</strong>, the font is embedded (✅ Good).</li><li>If <strong>emb = no</strong>, the font is missing, and the system will substitute it (⚠️ Potential OCR Issue).</li></ul><h3>4. Installing missing Fonts on server (or Modal)</h3><p>In my case, I needed to install some Microsoft Fonts.</p><pre>sudo apt install ttf-mscorefonts-installer</pre><p>But since I’m using <a href="https://modal.com/">Modal</a> below is the code that install fonts with libraries that are useful for OCR like Poppler, openCV, pdf2image and Pillow.</p><pre>app = modal.App(&quot;my_ocr_app&quot;,<br>    image=(<br>        modal.Image.from_registry(&quot;debian:bookworm-slim&quot;,add_python=&quot;3.10&quot;)<br>        .apt_install(&quot;build-essential&quot;, &quot;libtool&quot;, &quot;autoconf&quot;, &quot;unzip&quot;, &quot;wget&quot;,<br>                     &quot;cmake&quot;, &quot;libgl1&quot;, &quot;python3-opencv&quot;, &quot;git&quot;, &quot;make&quot;,<br>                     &quot;poppler-utils&quot;, &quot;poppler-data&quot;, &quot;fontconfig&quot;,<br>                     &quot;cabextract&quot;,&quot;xfonts-utils&quot;)<br>        .run_commands(&quot;wget http://ftp.us.debian.org/debian/pool/contrib/m/msttcorefonts/ttf-mscorefonts-installer_3.8.1_all.deb&quot;)<br>        .run_commands(&quot;dpkg -i ttf-mscorefonts-installer_3.8.1_all.deb&quot;)<br>        .run_commands(&quot;fc-cache -fv&quot;)<br>        .pip_install(<br>           &quot;supervision&quot;,<br>           &quot;ultralytics&quot;,<br>           &quot;opencv-python&quot;,<br>           &quot;pdf2image&quot;,<br>           &quot;Pillow&quot;,<br>       )<br>    )<br>)</pre><h3>5. Final Thoughts: The Hidden Impact of Fonts in OCR Pipelines</h3><p>📌 <strong>Key Takeaways:</strong></p><ul><li>Even if your OCR code and libraries are identical, <strong>PDF font embedding can cause text rendering inconsistencies</strong>.</li><li>If a <strong>font is missing</strong>, the system substitutes it with a similar font, <strong>altering text shape and spacing</strong>, which can confuse OCR engines.</li><li>Always <strong>check for embedded fonts</strong>.</li><li>If OCR outputs differ between local and remote environments, <strong>compare installed system fonts</strong> and install missing ones.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec38c9447d1f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Surya and EasyOCR: Vision Models and Data Extraction in Financial Documents]]></title>
            <link>https://medium.com/@hlealpablo/surya-and-easyocr-vision-models-and-data-extraction-in-financial-documents-fef55b132185?source=rss-d18f7b3c3588------2</link>
            <guid isPermaLink="false">https://medium.com/p/fef55b132185</guid>
            <category><![CDATA[easyocr]]></category>
            <category><![CDATA[financial-documents]]></category>
            <category><![CDATA[large-vision-models]]></category>
            <category><![CDATA[vision-transformer]]></category>
            <category><![CDATA[surya]]></category>
            <dc:creator><![CDATA[Pablo Hernandez Leal]]></dc:creator>
            <pubDate>Wed, 11 Sep 2024 02:10:40 GMT</pubDate>
            <atom:updated>2024-09-11T15:22:41.761Z</atom:updated>
            <content:encoded><![CDATA[<h3>Motivation</h3><p>Automation tasks such as audits, compliance checks, and financial analysis often require extracting information from documents. Typically, this involves transforming documents into images, followed by layout detection, and OCR recognition.</p><p>Financial documents pose unique challenges:</p><ul><li>High number of charts and tables.</li><li>Symbols are highly relevant. Think of currencies, $, £; magnitudes &gt;,&lt;, ~; negative numbers, -, (, ).</li><li>Irregular formats. Documents in the same category might share similar sections, but the format, positioning, details and font sizes can be highly diverse.</li></ul><p>In this post I explore and test two models that can be used for OCR in financial documents: <a href="https://github.com/JaidedAI/EasyOCR">EasyOCR</a> and <a href="https://github.com/VikParuchuri/surya">Surya.</a></p><h4>TL;DR</h4><ul><li>Overview of financial document diversity.</li><li>Descriptions of Surya and EasyOCR models.</li><li>Evaluation of both models on text detection and recognition.</li><li>Conclusions.</li></ul><h3>Financial Document Examples</h3><p>Layout, colors, font sizes are usually quite diverse in financial documents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*F5RrndEZ-PiSJq8Q7Afx8w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*qeFiqQeHku4KMgPZVkJW8w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*zNz8YB10ZtHxPyw8edMr4A.png" /><figcaption>Bank statement examples from <a href="https://templatelab.com/bank-statement/">https://templatelab.com/bank-statement/</a></figcaption></figure><p>Documents like 10-K reports from different companies (e.g., Starbucks, Nike, JPMorgan) can vary significantly in format, including differences in positioning, color schemes, and table structures, making extraction challenging.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/809/1*Ib6Qu-8PvA0MQDxWekqshg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/1*dHlFJ4fm1bZi2ibV_VxwHw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*GJJrR4t-O7-R2esV89DxUQ.png" /><figcaption>Some pages from 10-K reports from Starbucks (left), Nike (center) and JPMorgan &amp; Chase (right). Note the slight differences across format: positioning, columns, colors and tables.</figcaption></figure><h3>Data Extraction Task</h3><p>One of the most common tasks from financial documents is <em>extracting information</em>. As we have seen the non-standardized formatting makes the problem challenging. Usually, the pipeline for data extraction follows some of the following steps:</p><ol><li>Layout detection: to get the structure of the page/document.</li><li>OCR: to go from pixels in the image to text.</li><li>Correction: since OCR usually introduces errors, there can be a step to fix mistakes.</li><li>Extraction: to obtain the requested information.</li></ol><h3>LVMs vs OCR models</h3><p>Large Vision Models (LVMs) like <a href="https://chatgpt.com/">GPT-4</a> and <a href="https://huggingface.co/Qwen/Qwen2-72B-Instruct">Qwen2</a> bypass the need for most of the steps in the above pipeline. This is, they take images as input and work directly from them.</p><p>However, one of the main limitations of LVMs is the computational resources needed, since these models range from a few billion parameters to hundreds of billions, making their usage quite expensive. In contrast, OCR models can be much smaller and can fit in smaller GPUs or might be run even on CPUs.</p><p>Deciding which to use depends on the task, available resources, and data extraction needs.</p><h3>Models to Test: Surya and EasyOCR</h3><h4>EasyOCR</h4><p><a href="https://github.com/JaidedAI/EasyOCR">EasyOCR</a> is a ready to use OCR model for more than 80 languages which we’ll test as baseline. According to their documentation, for detection EasyOCR uses the <a href="https://arxiv.org/pdf/1904.01941">CRAFT model (Character Region Awareness For Text detection) which is based on convolutional neural networks</a>. For recognition, EasyOCR uses <a href="https://arxiv.org/pdf/1507.05717">Convolutional Recurrent Neural Network (CRNN)</a>, a combination of LSTMs and CNNs.</p><p><a href="https://arxiv.org/pdf/2111.15664">The total number of parameters is 27M</a>, which is considerably smaller than LVMs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/1*ceMtiuDEaF-D04GAauHH-Q.png" /><figcaption>CRNN architecture, source <a href="https://arxiv.org/pdf/1507.05717">https://arxiv.org/pdf/1507.05717</a></figcaption></figure><h4>Surya</h4><p><a href="https://github.com/VikParuchuri/surya">Surya</a> is a recent OCR toolkit that performs text detection, OCR, layout analysis and reading order detection in more than 90 languages. According to the documentation, the detection model is based on an <a href="https://arxiv.org/pdf/2205.14756">EfficientViT model</a> and the recognition model is based on <a href="https://arxiv.org/pdf/2111.15664">Donut</a>.</p><p><a href="https://arxiv.org/pdf/2205.14756"><strong>EfficientViT</strong></a></p><p>This model aims to improve Vision Transformer model performance on hardware. It proposes removing the softmax operation in transformers with a lightweight ReLU linear attention. This is, given the generalized form of softmax attention as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/426/1*XpxeEmzcMfwqGXkUP_LEng.png" /></figure><p>we can recover the original attention using the following similarity</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*kUVC3P5oeTmUJIdG2YvWXg.png" /></figure><p>the authors, propose to modify with ReLU linear attention as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/1*3CPVP-ji8JI7qCegMXQ-SA.png" /></figure><p>This modification gives improvements on hardware costs. However, ReLU linear attention is <em>not</em> good at handling concentrated attention maps which will decrease the quality of the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6o9E7LwGIb3FKB0qDCanwA.png" /></figure><p>To mitigate this issue, the architecture is modified by adding a convolution with each FFN layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*iiYn3SJVvxyvEFSA7mUxOA.png" /><figcaption>EfficientViT blocks, source <a href="https://arxiv.org/pdf/2205.14756">https://arxiv.org/pdf/2205.14756</a></figcaption></figure><p><a href="https://arxiv.org/pdf/2111.15664"><strong>Donut</strong></a></p><p>Document understanding transformer (Donut) is a model for Visual Document Understanding, and was built with the idea of bypassing the OCR step, this means, going directly from an image to a structured output (JSON).</p><p>The three main elements of Donut are:</p><ul><li>Transformer encoder. Takes as input an image and split into patches to generate embeddings. It uses <a href="https://openaccess.thecvf.com/content/ICCV2021/papers/Liu_Swin_Transformer_Hierarchical_Vision_Transformer_Using_Shifted_Windows_ICCV_2021_paper.pdf">Swin Transformer</a> as encoder.</li><li>Decoder. Takes as input the transformer encoder output and generates a token sequence. It uses <a href="https://arxiv.org/abs/1910.13461">BART</a> as the decoder architecture.</li><li>Output Conversion. The output token sequence is converted to JSON by adding two special tokens [START ∗] and [END ∗], where ∗ indicates each field to extract.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OX-lVqEUyi5gDAqTxZxV1w.png" /><figcaption>Document Understanding Transformer (Donut), source <a href="https://arxiv.org/pdf/2111.15664">https://arxiv.org/pdf/2111.15664</a></figcaption></figure><h3>Testing on Financial Documents</h3><p>To evaluate the models we will use a diverse set of financial documents:</p><ul><li>Alibaba 10-K</li><li>Royal Bank of Canada Annual Report</li><li>Bank statement from JPMorgan</li><li>A scanned Correspondent Banking Due Diligence Questionnaire (CBDDQ)</li></ul><p>We select some pages from these PDFs and converted to images using <a href="https://pypi.org/project/pdf2image/">pdf2image</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nalRvT95jHtyZwsHhAG7Vw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RvM65Xb2yFVz-OD7njuUtw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C_HTi9LJ2j-yE9NZ7lVaAw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YgxsyGRYi4-vpTbGckDtLw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o7n1vqrXUnCieZIkvWbSLw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cq-GvMNP2KTJ6-Gr16d0_w.png" /></figure><h3>Text Detection</h3><h4>Surya</h4><p>For Surya we can generate predictions with the following code:</p><pre>from PIL import Image<br>from surya.ocr import run_ocr<br>from surya.model.detection.model import load_model as load_det_model, load_processor as load_det_processor<br>from surya.model.recognition.model import load_model as load_rec_model<br>from surya.model.recognition.processor import load_processor as load_rec_processor<br><br>IMAGE_PATH = &#39;&#39;<br>langs = [&quot;en&quot;] # Replace with your languages - optional but recommended<br>det_processor, det_model = load_det_processor(), load_det_model()<br>rec_model, rec_processor = load_rec_model(), load_rec_processor()<br><br>preds=[]<br>i = 0<br>p = IMAGE_PATH + &#39;pg_&#39;+str(i) + &#39;.png&#39; #file name of the images<br>image = Image.open(p)<br>predictions = run_ocr([image], [langs], det_model, det_processor, rec_model, rec_processor)<br>preds.append(predictions)</pre><p>For each text line detected Surya provides the bounding box, text, and confidence score. We can use the bounding box to visually highlight the detected text on top of the image.</p><pre>from PIL import ImageDraw<br><br>p = IMAGE_PATH + &#39;pg_&#39;+str(i) + &#39;.png&#39;<br>page = 0<br>objs = []<br>for t in preds[page][0].text_lines:<br>  objs.append([t.polygon,t.confidence,t.text,t.bbox])<br><br>image = Image.open(p) <br>for obj in objs:<br>  # draw bbox on image<br>  draw = ImageDraw.Draw(image)<br>  draw.line(obj[3], fill=&#39;red&#39;, width=3)<br>display(image)</pre><h4>EasyOCR</h4><p>For EasyOCR, the code changes slightly, for OCR predictions we do:</p><pre>import easyocr<br>reader = easyocr.Reader([&#39;en&#39;]) # this needs to run only once to load the model into memory<br>IMAGE_PATH = &#39;&#39;<br><br>preds = []<br>i = 0<br>p = IMAGE_PATH + &#39;pg_&#39;+str(i) + &#39;.png&#39;<br>result = reader.readtext(p)<br>preds.append(result)</pre><p>Since EasyOCR does not provide the bounding box we need an additional function to transform the data, after that we can plot a red line to highlight the detected text.</p><pre>#transform polygon coordinates objs[0] to bounding box<br>def polygon_to_bbox(polygon):<br>    # Unpack the polygon coordinates<br>    x1, y1 = polygon[0]<br>    x2, y2 = polygon[1]<br>    x3, y3 = polygon[2]<br>    x4, y4 = polygon[3]<br>    <br>    # Get the list of x and y coordinates<br>    x_coords = [x1, x2, x3, x4]<br>    y_coords = [y1, y2, y3, y4]<br>    <br>    # Find the minimum and maximum coordinates<br>    x1_bbox = min(x_coords)<br>    y1_bbox = min(y_coords)<br>    x2_bbox = max(x_coords)<br>    y2_bbox = max(y_coords)<br>    <br>    # Return the bounding box as (x1, y1, x2, y2)<br>    return (x1_bbox, y1_bbox, x2_bbox, y2_bbox)<br><br>for page in range(0, 1):<br>  objs = []<br>  for p,t,c in preds[page]:<br> #   print(t.bbox)<br>    objs.append([p,t,c])<br><br>#show bbox of all elements in objs<br><br>p = IMAGE_PATH + &#39;pg_0.png&#39;<br>image = Image.open(p) <br>for obj in objs:<br>  # draw bbox on image<br>  draw = ImageDraw.Draw(image)<br>  draw.line(polygon_to_bbox(obj[0]), fill=&#39;red&#39;, width=3)<br>display(image)</pre><h4>Examples</h4><p>We compare side by side, the results from EasyOCR and Surya on some selected pages from the diverse documents.</p><p>First, on the RBC annual report. Both models do a very good work detecting text in the images, even with different color schemes and font sizes. It is hard to spot but EasyOCR missed a <strong><em>~</em></strong> symbol. Surya missed the “<strong>$22.1 billion”</strong> inside the circle (bottom right page).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lVMTfI2yhK6Ybgb2N44qkQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jWMq7zv__vzLfL-Fdz6XxA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ibG1GlB4eMlcMPrnUE9pHw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r9-Ws_t6BoLdbcD-i4vTTQ.png" /><figcaption>EasyOCR detection on the left. Surya detection on the right. Pages from RBC annual report.</figcaption></figure><p>When testing in the Alibaba 10-K we note that EasyOCR can handle diverse font sizes detecting all the text in the image. In contrast, Surya does not capture the text <strong>“102 years</strong>”. My assumption of why this happens it’s because of this comment on Surya repo: <a href="https://github.com/VikParuchuri/surya">“ The text detection model has trained itself to ignore advertisements”</a>.</p><p>When looking at the page with table Surya makes an excellent work even detecting the small <strong>“-” </strong>in the cells.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zfrjPtf4iymDX0eyw8DFwA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JcrkA0AaUwt-dE9bBdI0-Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ys7LzoH7-jWWzF8xHyMcDw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4AbC2Plk6_VWPTVA2fb7BA.png" /><figcaption>EasyOCR on the left. Surya on the right. Pages from Alibaba 10-K.</figcaption></figure><p>On the bank statement, EasyOCR misses 3 numbers in the column Instances. In contrast, Surya detects correctly all the text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZFhthks-4Kv7DWMi2BKujA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*isZlP8lZOz4cvFSoixB9VQ.png" /><figcaption>EasyOCR on the left. Surya on the right. JPMorgan bank statement example.</figcaption></figure><p>Lastly, in the scanned CBDDQ, EasyOCR misses most of the text in the first column of the table. Surya, is able to detect correctly most of the text. The logo in the upper left is ignored by Surya.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U1Mc6MObuAXYhD6-t1VpKA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hXqVZTtnYGALgDX2W6Lcjg.png" /><figcaption>EasyOCR on the left. Surya on the right. A scanned CBDDQ example.</figcaption></figure><h3>Text Recognition</h3><p>To visually show the recognition results, we will “recreate” the document with the coordinates of the bounding box and the recognized text.</p><pre>from PIL import Image, ImageDraw,ImageFont<br><br>p = IMAGE_PATH + &#39;pg_0.png&#39;<br>font = ImageFont.truetype(&quot;/ubuntu-mono.ttf&quot;,20) # specify a font path to change size<br>image = Image.open(p) <br># create a blank image with the same size<br>blank_image = Image.new(&#39;RGB&#39;, image.size, (255, 255, 255))<br><br>for obj in objs:<br>  # draw OCR text<br>  draw = ImageDraw.Draw(blank_image)<br>  draw.text((polygon_to_bbox(obj[0])[0],polygon_to_bbox(obj[0])[1]), obj[1], fill=&#39;red&#39;, font=font)<br>display(blank_image)</pre><p>We start with one of the pages from the annual report from RBC. To make it easier I highlighted the differences from EasyOCR and Surya in black. Note that sometimes EasyOCR breaks the text into different bounding boxes even when it should remain on the same line.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wYrQpFqHaJdmUJKL7YQwww.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*78JCLS0aWniRLHJrVa_1Rw.png" /><figcaption>EasyOCR on the left, Surya on the right on a RBC annual report page.</figcaption></figure><p>In the table from Alibaba 10-K, EasyOCR did quite well correctly recognizing the most of the text. Surya was able to detect everything, however, in the recognition, there are some cells where the <strong>“-”</strong> was incorrectly recognized as <strong>“l”.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z0JXHEc0Z8uJrKDiU9JdVQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zxjFPR9MPkujkmbsn73mhA.png" /><figcaption>EasyOCR on the left, Surya on the right on a Alibaba 10-K page.</figcaption></figure><p>We have seen good results of the models in the digital documents, let’s compare the results with a <strong>low quality</strong> document like a scan.</p><p>In this test I was able to found more than 30 words with mistakes made by the EasyOCR model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_I-K39kZOW-mYkPdm4zAGg.png" /><figcaption>Zoom on the EasyOCR test highlighting in yellow the mistakes made by the model</figcaption></figure><p>Surya did a very good job only missing the logo but correctly recognizing the text in the document.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pU_5v0kQvXI3KI_Fl0lteA.png" /><figcaption>Zoom on the OCR predictions made by Surya model.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6ATblWCwOWwfnTyUMAH44A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*In7uAaEvDaaowXDuye3FYg.png" /><figcaption>EasyOCR on the left, Surya on the right on the scanned CBDDQ.</figcaption></figure><h3>Conclusion</h3><pre>| Feature            | EasyOCR            | Surya                           |<br>|--------------------|--------------------|---------------------------------|<br>| Detection model    | CRAFT model        | EfficientViT                    |<br>| Recognition model  | CRNN               | Donut                           |<br>| Language support   | 80+ languages      | 90+ languages                   |<br>| Layout analysis    | Limited            | Yes                             |<br>| Ideal use case     | Simpler documents  | Complex, multi-layout documents |</pre><p><a href="https://github.com/JaidedAI/EasyOCR"><strong>EasyOCR</strong></a> was <em>able to detect most of the text, </em>however,<em> it tends to break the text into different bounding boxes even when it should be on the same line.</em> EasyOCR <em>struggled with a scanned documen</em>t showing many errors in the recognition step.</p><p><a href="https://github.com/VikParuchuri/surya"><strong>Surya</strong></a><strong> </strong>showed <em>consistent results in detection and recognition across different formats and even with scanned documents</em>. Surya tends to <em>ignore large texts that might look like logos</em>, maybe because the model was trained to ignore advertisements.</p><p>This area of Vision models, OCR, and document understanding is an active research area where many models are being developed for specific needs, more examples below:</p><ul><li><a href="https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl1.5">DocOWL</a> is a Multimodal Large Language Model that focuses on understanding text-rich images from 5 categories: documents, webpages, tables, charts, and natural images.<a href="https://arxiv.org/html/2403.12895v1"> It has around 8B parameters.</a></li><li><a href="https://huggingface.co/vidore/colpali">ColPali</a> is a Retrieval model that takes images without needing additional OCR or layout detection. <a href="https://medium.com/@hlealpablo/interpretability-of-colpali-in-financial-documents-5a2dcdeeba3a">Here’s a post testing ColPali in financial documents.</a></li><li><a href="https://huggingface.co/PleIAs/OCRonos">OCRonos</a> is a correction model of badly digitized texts.</li><li><a href="https://huggingface.co/yifeihu/TFT-ID-1.0">TFT-ID</a> is a Table/Figure/Text detection model for academic papers.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fef55b132185" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Interpretability of ColPali in Financial Documents]]></title>
            <link>https://medium.com/@hlealpablo/interpretability-of-colpali-in-financial-documents-5a2dcdeeba3a?source=rss-d18f7b3c3588------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a2dcdeeba3a</guid>
            <category><![CDATA[colpali]]></category>
            <category><![CDATA[colbert]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[model-interpretability]]></category>
            <category><![CDATA[paligemma]]></category>
            <dc:creator><![CDATA[Pablo Hernandez Leal]]></dc:creator>
            <pubDate>Thu, 18 Jul 2024 20:13:15 GMT</pubDate>
            <atom:updated>2024-07-18T20:31:01.109Z</atom:updated>
            <content:encoded><![CDATA[<p>Retrieval Augmented Generation (RAG) is one of the most common use cases of LLMs. The first step of <em>Retrieval</em> is usually performed with keyword matching algorithms like BM25 or with neural retrieval approaches like <a href="https://arxiv.org/pdf/2107.05720">SPLADE</a> or <a href="https://arxiv.org/abs/2004.12832">ColBERT</a>, generally working in the <strong>text</strong> embedding space. <a href="https://arxiv.org/pdf/2407.01449v2">ColPali</a> is is a recent model that generate embeddings <em>based</em> <em>only</em> on <strong>images</strong>.</p><h3>ColPali: Efficient Document Retrieval with Vision Language Models</h3><blockquote>“To index a standard PDF document, many steps are required. First, PDF parsers or Optical Character Recognition (OCR) systems are used to extract words from the pages. Document layout detection models can then be run to segment paragraphs, titles, and other page objects such as tables, figures, and headers. A chunking strategy is then defined to group text passages with some semantical coherence, and modern retrieval setups may even integrate a captioning step to describe visually rich elements in a natural language form, more suitable for embedding models.”</blockquote><p>In contrast, <a href="https://arxiv.org/pdf/2407.01449v2">ColPali</a> only needs a single step of directly ingesting an image of the document page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ycDQpS9mwKIIUzGGEnznvw.png" /><figcaption>Standard retrieval shows a series of steps in the offline phase. In contrast, ColPali takes directly the image of a document. <a href="https://arxiv.org/pdf/2407.01449v2">Source</a></figcaption></figure><p><strong>ColPali</strong> is better understood if we briefly review <strong>Col</strong>BERT and <strong>Pali</strong>Gemma models.</p><h3>ColBERT</h3><p>ColBERT is a neural retrieval method which proposed a “late interaction” approach using all tokens in the query to compute a similarity score.</p><blockquote><a href="https://til.simonwillison.net/llms/colbert-ragatouille">“With a regular embedding model you store a single vector for each document, and get back a single numeric score showing how well that document matches your query.</a></blockquote><blockquote><a href="https://til.simonwillison.net/llms/colbert-ragatouille">ColBERT is slightly different: it provides a list of vectors showing how each token in the query matches up with each token in the document.</a>”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dJTfqh_zoDIVDoPq.png" /><figcaption><a href="https://blog.vespa.ai/announcing-colbert-embedder-in-vespa/">Image source</a></figcaption></figure><p>The above diagram shows how the list of tokens is used when computed the similarity between the query and documents. This is not only useful in the <em>retrieval</em> step, but also later to <em>understand (interpret)</em> the inner workings of the algorithm. One example is the cool visualization below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*IwtbGTug2ukqpHjRMN5nOA.png" /><figcaption><a href="https://colbert.aiserv.cloud/">https://colbert.aiserv.cloud/</a></figcaption></figure><h3>PaliGemma</h3><p><a href="https://arxiv.org/pdf/2407.07726v1">PaliGemma</a> is a lightweight (3B) model which combines a vision (SigLIP-So400m) and a language model (Gemma-2B).</p><p>Recall that transformers, the backbone of LLMs, have been successfully not only with text but also in vision, splitting images into fixed-size patches.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9k5EpAuufANIpECUQFKUDg.png" /><figcaption><a href="https://arxiv.org/pdf/2010.11929">Image source</a></figcaption></figure><p>PaliGemma combines two models and connect them with a linear layer between them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SCsq5sQQGvPbebMt9alj_g.png" /><figcaption><a href="https://arxiv.org/pdf/2407.07726v1">Image source</a></figcaption></figure><p>In contrast to other fine-tuned models, in PaliGemma, the entire model is trained (there are no frozen components) based on experimental results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/1*wcz5ttW25JBZilWgMIYdsQ.png" /><figcaption>The more is frozen or reset, the more performance deteriorates.</figcaption></figure><h3>Revisiting ColPali</h3><p>As ColBERT, ColPali reuses <em>late interaction</em> which considers all the query vectors in the similarity computation.</p><blockquote>“At runtime, similarity can be computed with individual query token embeddings. The idea is to benefit from the rich interaction between individual query and document terms while taking advantage of the offline computation and fast query matching enabled by bi-encoders.”</blockquote><p><strong>Interpretability</strong></p><p>ColPali retains the ability to do some kind of interpretability with the information from the model (late interaction image patches) and the original image.</p><blockquote>“By superimposing the late interaction heatmap on top of the original image, we can visualize the most salient image patches with respect to each term of the query, yielding interpretable insights into model focus zones.”</blockquote><p>The paper shows some examples of this idea. Now, let’s try this feature in financial documents which are usually filled with figures, text, and tables.</p><h3>Example: Alibaba 10-K</h3><p>In this first experiment I used Alibaba 2023 10-K report. The first ~40 pages have non standard images, colors, and format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/322/1*C6fMPSAd-XM2m1PYhB7XNQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/324/1*wWKQGWg6AAwQknuoWHraJQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*bHvn_Zd4MOnB9QwR6cfkPQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/1*qjJiRBpfQ_mEkufn6IWgEw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/1*lTlkYrp0_zjQp1lkYklqUQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*G0YXXmGahaKNeoJysS3oTw.png" /><figcaption>A few examples of the pages that were used in the first example which come from the 2023 10-K of Alibaba.</figcaption></figure><p>The query was: “<em>What are the artificial intelligence tools being developed</em>?”</p><p>The page that obtained the highest score was page 40. Below are the heatmaps for the tokens “artificial” and “intelligence”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*kDqV6E39ozGogsZa5xb2jQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*SaHUjivay7zLOem1yNffeQ.png" /><figcaption>Heatmaps of the tokens “artificial” and “intelligence” on Page 40. Note that AI is</figcaption></figure><p>Compare to the original document, highlighting relevant terms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*pZ52YXTffiACvrOioFw_6w.png" /><figcaption>“Artificial” and “intelligence” are highlighted in yellow. Semantically similar words like <strong>AI</strong>, or <strong>intelligent</strong> are shown in red ovals.</figcaption></figure><p>Next, we can check how the model deals with those colorful pages. It looks impressive that both tokens “artificial” and “intelligence” highlight the AI in the page below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*fscpNhk6aZ_wFT9CNICXUQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*oZLZAzQqQXVLyPlkmWaLuQ.png" /><figcaption>Heatmaps of the tokens “artificial” and “intelligence” on Page 1. Note in the upper right corner both tokens highlight the text “AI”.</figcaption></figure><h3>Example: Royal Bank of Canada — Annual Report</h3><p>In this second example I focused on the “Management’s Discussion and Analysis” of the Royal Bank of Canada annual report, which starts in page 22 and contains mostly text, tables, and footnotes.</p><p>The query was: “<em>Describe the elements of stock based compensation</em>”.</p><p>One of the hits was page 33, where we can highlight there are two mentions of <strong><em>share</em></strong>-based compensation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/923/1*5IL82wiwNAyn3tYWIuhWbQ.png" /><figcaption>“Share-based compensation” is highlighted in yellow, a semantically similar term <strong>shareholders </strong>is inside a red oval.</figcaption></figure><p>Looking at the heatmaps, the “stock” token highlights a large portion of white patches, the paper mentions a hypothesis for this behavior:</p><blockquote>“We believe the vectors associated with these patches (white patches) share a similar role with the ViT registers (Darcet et al., 2023), i.e. these patches were repurposed for internal computations and stored the global information from the whole image.”</blockquote><p>Additionally, the “stock” token also highlights <em>shareholders </em>and<em> share-based.</em></p><p>For the token “compensation” it clearly highlights one of the two times it is on the document.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*FQ5rx3uotZKzt7wMod5dqA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*js3llO9VF4aZatC5-E0QmA.png" /><figcaption>Heatmaps of tokens “stock” and “compensation”. “Stock” highlights “shareholders”, located in the bottom right part of the page.</figcaption></figure><p>Another example of a high ranked page is 57, where the share-based compensation is mentioned mostly as a footnote.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*CmPSBJf5B5lIHekQhKdetw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*UqkyAU2yq9A6D4DJIZMQrQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*X7uqGOUEdpRcYucWD2bOPg.png" /><figcaption>Heatmaps for “stock” and “compensation”. Note that even when text is in footnotes (smaller font), it is correctly highlighted.</figcaption></figure><h3><strong>Example: Royal Bank of Canada — Notes</strong></h3><p>The last 100 pages of the annual report are <em>Notes to Consolidated Financial Statements.</em></p><p>For these pages the query was: “<em>Summarize important notes about credit losses</em>”.</p><p>The first hit was page 150.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/1*tUwVkidFAcIvpkv2lPpiuw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*aIrRy65lwQkzN5PxG0Q6bw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*WXHRk3G_6NxJmp8kCR8vEw.png" /></figure><p>Another hit is page 177</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*_kkU2_hJTh4MtF7NRAZelw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*T5JRD3IMeI0FxkClIWIYRw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*MaCe37irtF5u08Y-SHb6rA.png" /><figcaption>Heatmaps for “credit” and “losses”. Note that regions close to those terms are highlighted in the table (cells and title).</figcaption></figure><h3>Code</h3><p>This code largely builds on:</p><ul><li><a href="https://github.com/illuin-tech/colpali/blob/main/scripts/infer/run_inference_with_python.py">This script from the authors of ColPali</a>.</li><li><a href="https://pyvespa.readthedocs.io/en/latest/examples/colpali-document-retrieval-vision-language-models.html">This notebook from Vespa.</a></li></ul><p>Here is the step by step:</p><ul><li>Installing libraries.</li></ul><pre>pip install transformers tqdm pillow datasets torch torchvision torchmetrics torchtext<br>pip install git+https://github.com/illuin-tech/colpali<br>apt-get install poppler-utils<br>pip install pdf2image pypdf<br>pip install einops</pre><ul><li>Adding functions for images and garbage collection.</li></ul><pre>import requests<br>from pdf2image import convert_from_path<br>from pypdf import PdfReader<br>from io import BytesIO<br><br>def download_pdf(url):<br>    response = requests.get(url)<br>    if response.status_code == 200:<br>        return BytesIO(response.content)<br>    else:<br>        raise Exception(f&quot;Failed to download PDF: Status code {response.status_code}&quot;)<br><br>def get_pdf_images(pdf_url,first_page,last_page):<br>    # Download the PDF<br>    pdf_file = download_pdf(pdf_url)<br>    # Save the PDF temporarily to disk (pdf2image requires a file path)<br>    with open(&quot;temp.pdf&quot;, &quot;wb&quot;) as f:<br>        f.write(pdf_file.read())<br>    reader = PdfReader(&quot;temp.pdf&quot;)<br>    page_texts = []<br>    print(f&quot;Processing pages&quot;)<br>    for page_number in range(first_page,last_page+1): #10 pages test<br>        page = reader.pages[page_number]<br>        text = page.extract_text()<br>        page_texts.append(text)<br>    images = convert_from_path(&quot;temp.pdf&quot;,first_page=first_page,last_page=last_page)<br>    # Remove the temporary PDF file<br>    import os<br>    os.remove(&quot;temp.pdf&quot;)<br>    assert len(images) == len(page_texts)<br>    return (images, page_texts)<br><br># Garbage Collector - use it like gc.collect()<br>import gc<br>import tensorflow as tf<br><br># Custom Callback To Include in Callbacks List At Training Time<br>class GarbageCollectorCallback(tf.keras.callbacks.Callback):<br>    def on_epoch_end(self, epoch, logs=None):<br>        gc.collect()</pre><ul><li>Loading a PDF.</li></ul><pre>sample_pdf = {<br>            &quot;title&quot;: &quot;10-k&quot;,<br>            &quot;url&quot;: &quot;https://data.alibabagroup.com/ecms-files/1514443390/5788a02d-696c-412a-ad2a-386d19b21769/Alibaba%20Group%20Holding%20Limited%20Fiscal%20Year%202024%20Annual%20Report.pdf&quot;,<br>            &quot;company&quot;: &quot;BABA&quot;,<br>            &quot;year&quot;: 2023<br>        }<br>first_page = 1<br>last_page = 100<br>page_images, page_texts = get_pdf_images(sample_pdf[&#39;url&#39;],first_page,last_page)<br>sample_pdf[&#39;images&#39;] = page_images<br>sample_pdf[&#39;texts&#39;] = page_texts</pre><ul><li>Loading the model from HuggingFace (you need a HF token).</li></ul><pre>import torch<br>import typer<br>from torch.utils.data import DataLoader<br>from tqdm import tqdm<br>from transformers import AutoProcessor<br>from PIL import Image<br>import numpy as np<br><br>from colpali_engine.models.paligemma_colbert_architecture import ColPali<br>from colpali_engine.trainer.retrieval_evaluator import CustomEvaluator<br>from colpali_engine.utils.colpali_processing_utils import process_images, process_queries<br>from colpali_engine.utils.image_from_page_utils import load_from_dataset<br>from colpali_engine.utils.image_utils import scale_image,get_base64_image<br><br>if torch.cuda.is_available():<br>  device = torch.device(&quot;cuda&quot;)<br>  type = torch.bfloat16<br>elif torch.backends.mps.is_available():<br>  device = torch.device(&quot;mps&quot;)<br>  type = torch.float32<br>else:<br>  device = torch.device(&quot;cpu&quot;)<br>  type = torch.float32<br><br><br># Load model<br>model_name = &quot;vidore/colpali&quot;<br>model = ColPali.from_pretrained(&quot;google/paligemma-3b-mix-448&quot;, torch_dtype=torch.bfloat16, device_map=&quot;cuda&quot;).eval()<br>model.load_adapter(model_name)<br>model.to(device)<br>processor = AutoProcessor.from_pretrained(model_name)</pre><ul><li>Generating page embeddings.</li></ul><pre>page_embeddings = []<br>dataloader = DataLoader(<br>    sample_pdf[&#39;images&#39;],<br>    batch_size=2,<br>    shuffle=False,<br>    collate_fn=lambda x: process_images(processor, x),<br>)<br>for batch_doc in tqdm(dataloader):<br>    with torch.no_grad():<br>        batch_doc = {k: v.to(model.device) for k, v in batch_doc.items()}<br>        embeddings_doc = model(**batch_doc)<br>    page_embeddings.extend(list(torch.unbind(embeddings_doc.to(&quot;cpu&quot;))))<br>sample_pdf[&#39;embeddings&#39;] = page_embeddings</pre><ul><li>Setting a query and computing query embeddings.</li></ul><pre>queries = [&quot;What are the artificial intelligence tools being developed?&quot;]<br>dummy_image = Image.new(&quot;RGB&quot;, (448, 448), (255, 255, 255))<br># run inference - queries<br>dataloader = DataLoader(<br>    queries,<br>    batch_size=1,<br>    shuffle=False,<br>    collate_fn=lambda x: process_queries(processor, x, dummy_image),<br>)<br>qs = []<br>for batch_query in dataloader:<br>    with torch.no_grad():<br>        batch_query = {k: v.to(model.device) for k, v in batch_query.items()}<br>        embeddings_query = model(**batch_query)<br>    qs.extend(list(torch.unbind(embeddings_query.to(&quot;cpu&quot;))))</pre><ul><li>Generate the scores for the pages</li></ul><pre>retriever_evaluator = CustomEvaluator(is_multi_vector=True)<br>scores = retriever_evaluator.evaluate(qs, sample_pdf[&#39;embeddings&#39;])<br>topk = np.argsort(scores)[0][::-1]<br>print(f&quot;Top {len(topk)} results&quot;,topk)</pre><ul><li>Modify a function to generate attention maps and heatmaps.</li></ul><pre>import pprint<br>from dataclasses import asdict, dataclass<br>from pathlib import Path<br>from uuid import uuid4<br><br>import matplotlib.pyplot as plt<br>import torch<br>from einops import rearrange<br>from PIL import Image<br>from tqdm import trange<br><br>from colpali_engine.interpretability.plot_utils import plot_patches, plot_attention_heatmap<br>from colpali_engine.interpretability.processor import ColPaliProcessor<br>from colpali_engine.interpretability.torch_utils import normalize_attention_map_per_query_token<br>from colpali_engine.interpretability.vit_configs import VIT_CONFIG<br>from colpali_engine.models.paligemma_colbert_architecture import ColPali<br><br>OUTDIR_INTERPRETABILITY = Path(&quot;outputs/interpretability&quot;)<br><br><br>@dataclass<br>class InterpretabilityInput:<br>    query: str<br>    image: Image.Image<br>    start_idx_token: int<br>    end_idx_token: int<br><br>def generate_interpretability_plots(<br>    model: ColPali,<br>    processor: ColPaliProcessor,<br>    query: str,<br>    image: Image.Image,<br>    savedir: str | Path | None = None,<br>    add_special_prompt_to_doc: bool = True,<br>) -&gt; None:<br><br>    # Sanity checks<br>    if len(model.active_adapters()) != 1:<br>        raise ValueError(&quot;The model must have exactly one active adapter.&quot;)<br><br>    if model.config.name_or_path not in VIT_CONFIG:<br>        raise ValueError(&quot;The model must be referred to in the VIT_CONFIG dictionary.&quot;)<br>    vit_config = VIT_CONFIG[model.config.name_or_path]<br><br>    # Handle savepath<br>    if not savedir:<br>        savedir = OUTDIR_INTERPRETABILITY / str(uuid4())<br>        print(f&quot;No savepath provided. Results will be saved to: `{savedir}`.&quot;)<br>    elif isinstance(savedir, str):<br>        savedir = Path(savedir)<br>    savedir.mkdir(parents=True, exist_ok=True)<br><br>    # Resize the image to square<br>    input_image_square = image.resize((vit_config.resolution, vit_config.resolution))<br><br>    # Preprocess the inputs<br>    input_text_processed = processor.process_text(query).to(model.device)<br>    input_image_processed = processor.process_image(image, add_special_prompt=add_special_prompt_to_doc).to(<br>        model.device<br>    )<br><br>    # Forward pass<br>    with torch.no_grad():<br>        output_text = model.forward(**asdict(input_text_processed))  # (1, n_text_tokens, hidden_dim)<br><br>    # NOTE: `output_image`` will have shape:<br>    # (1, n_patch_x * n_patch_y, hidden_dim) if `add_special_prompt_to_doc` is False<br>    # (1, n_patch_x * n_patch_y + n_special_tokens, hidden_dim) if `add_special_prompt_to_doc` is True<br>    with torch.no_grad():<br>        output_image = model.forward(**asdict(input_image_processed))<br><br>    if add_special_prompt_to_doc:  # remove the special tokens<br>        output_image = output_image[<br>            :, : processor.processor.image_seq_length, :<br>        ]  # (1, n_patch_x * n_patch_y, hidden_dim)<br><br>    output_image = rearrange(<br>        output_image, &quot;b (h w) c -&gt; b h w c&quot;, h=vit_config.n_patch_per_dim, w=vit_config.n_patch_per_dim<br>    )  # (1, n_patch_x, n_patch_y, hidden_dim)<br><br>    # Get the unnormalized attention map<br>    attention_map = torch.einsum(<br>        &quot;bnk,bijk-&gt;bnij&quot;, output_text, output_image<br>    )  # (1, n_text_tokens, n_patch_x, n_patch_y)<br>    attention_map_normalized = normalize_attention_map_per_query_token(<br>        attention_map<br>    )  # (1, n_text_tokens, n_patch_x, n_patch_y)<br>    attention_map_normalized = attention_map_normalized.float()<br><br>    # Get text token information<br>    n_tokens = input_text_processed.input_ids.size(1)<br>    text_tokens = processor.tokenizer.tokenize(processor.decode(input_text_processed.input_ids[0]))<br>    print(&quot;Text tokens:&quot;)<br>    pprint.pprint(text_tokens)<br>    print(&quot;\n&quot;)<br><br>    for token_idx in trange(1, n_tokens - 1, desc=&quot;Iterating over tokens...&quot;):  # exclude the &lt;bos&gt; and the &quot;\n&quot; tokens<br>        fig, axis = plot_patches(<br>            input_image_square,<br>            vit_config.patch_size,<br>            vit_config.resolution,<br>            patch_opacities=attention_map_normalized[0, token_idx, :, :],<br>            style=&quot;dark_background&quot;,<br>        )<br><br>        fig.suptitle(f&quot;Token #{token_idx}: `{text_tokens[token_idx]}`&quot;, color=&quot;white&quot;, fontsize=14)<br>        savepath = savedir / f&quot;token_{token_idx}.png&quot;<br>        fig.savefig(savepath)<br>        print(f&quot;Saved attention map for token {token_idx} (`{text_tokens[token_idx]}`) to `{savepath}`.\n&quot;)<br>        plt.close(fig)<br><br>        print(&quot;Plotting heatmap&quot;)<br>        fig,axis = plot_attention_heatmap(input_image_square,<br>                               vit_config.patch_size,<br>                               vit_config.resolution,<br>                               attention_map_normalized[0, token_idx, :, :],<br>                               style=&quot;dark_background&quot;,<br>                               show_colorbar=True,<br>                               show_axes=True)<br>        savepath = savedir / f&quot;hm_token_{token_idx}.png&quot;<br>        fig.suptitle(f&quot;HeatMap Token #{token_idx}: `{text_tokens[token_idx]}`&quot;, color=&quot;white&quot;, fontsize=14)<br>        fig.savefig(savepath)<br>        plt.close(fig)<br><br>    return</pre><ul><li>Call the function above.</li></ul><pre>from colpali_engine.interpretability.processor import ColPaliProcessor<br>import os<br><br>query = queries[0]<br>savedir = &quot;/yourpath/&quot;<br>os.mkdir(savedir) <br>colpaliprocessor= ColPaliProcessor.from_pretrained(&quot;google/paligemma-3b-mix-448&quot;)<br>for n in topk[0:5]:<br>  print(n)<br>  image=sample_pdf[&#39;images&#39;][n]<br>  generate_interpretability_plots(model,<br>      colpaliprocessor,<br>      query=query,<br>      image=image,<br>      savedir=savedir,<br>      )<br>  gc.collect()</pre><h3>Conclusions</h3><p>ColPali is a recent model that generate embeddings directly from images, it builds on vision-language models like PaliGemma and retakes the “late interaction” from ColBERT. The paper shows better performance that other approaches in different retrieval tasks. Additionally, ColPali is relatively interpretable, by superimposing the model computation for each token on the image/document. Different examples using financial documents (like 10-K) show that in many cases the model is <em>correctly highlighting exact words</em> and in other cases, <em>semantically similar words</em>.</p><p>One question is, can we plug it with another model to do full RAG only with images? The answer is yes, you need a model that can take images and text as input (a large visual language model, LVLM, like Gemini or GPT-4o), <a href="https://pyvespa.readthedocs.io/en/latest/examples/colpali-document-retrieval-vision-language-models.html#RAG-with-LLMs-with-vision-capabilities.">there’s one example here</a>.</p><h3>References</h3><p><a href="https://arxiv.org/pdf/2407.01449v2">Faysse, Manuel, et al. “ColPali: Efficient Document Retrieval with Vision Language Models.” <em>arXiv preprint arXiv:2407.01449</em> (2024)</a>.</p><p><a href="https://dl.acm.org/doi/pdf/10.1145/3397271.3401075">Khattab, Omar, and Matei Zaharia. “Colbert: Efficient and effective passage search via contextualized late interaction over bert.” <em>Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval</em>. 2020.</a></p><p><a href="https://arxiv.org/pdf/2407.07726v1">Beyer, Lucas, et al. “PaliGemma: A versatile 3B VLM for transfer.” <em>arXiv preprint arXiv:2407.07726</em> (2024).</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a2dcdeeba3a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dynamic Pricing and Optimal Execution: Applied Reinforcement Learning]]></title>
            <link>https://medium.com/@hlealpablo/dynamic-pricing-and-optimal-execution-ce07f9c1fc4c?source=rss-d18f7b3c3588------2</link>
            <guid isPermaLink="false">https://medium.com/p/ce07f9c1fc4c</guid>
            <category><![CDATA[dynamic-pricing]]></category>
            <category><![CDATA[revenue-management]]></category>
            <category><![CDATA[trade-execution]]></category>
            <category><![CDATA[q-learning]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <dc:creator><![CDATA[Pablo Hernandez Leal]]></dc:creator>
            <pubDate>Thu, 04 Jul 2024 18:48:55 GMT</pubDate>
            <atom:updated>2024-07-08T14:13:23.662Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ORJuSf8fh_kwF0Gbdem1YA.jpeg" /><figcaption>Image by author using Gemini.</figcaption></figure><h3>TL;DR</h3><ul><li>We start by stating the <em>optimal execution</em> problem in finance and highlighting its challenges.</li><li>We describe some early solutions with dynamic programming and recent approaches that use Reinforcement Learning (RL).</li><li>We describe the problems of d<em>ynamic pricing </em>and some examples in industry.</li><li>We model a dynamic pricing problem for a hotel and we solve it with some ideas from optimal execution and Q-learning.</li></ul><h3>Problem</h3><p>You task is simple, <strong>to buy a large number of items before a certain deadline while minimizing costs</strong>. However, there are a number of challenges:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kwUA26w7shA2SjUNmXAf9w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZMfTR9QPY4KLxUOby3VBaA.png" /><figcaption>In this problem, there is a strict deadline in the temporal axis. Additional challenges are that prices and liquidity fluctuate over time (market impact) either by our respective actions or by actions of others (multiagent environment).</figcaption></figure><ul><li><em>Liquidity</em>: There are not enough items at a determined location, so you need to visit different locations to fulfill your demand.</li><li><em>Market impact</em>: When you start buying, other locations can see what you are doing and start increasing prices.</li><li><em>Multiagent</em>: You are not the only one buying, there are other participants which are trying to fulfill their respective quotas and deadlines, having an effect on the prices and liquidity.</li></ul><p>This stylized example has the main components of what is called <em>optimal trade execution</em> in finance.</p><h3>Optimal Trade Execution</h3><p>One of the classical papers to solve this approach with stochastic dynamic programming was proposed by Bertsimas and Lo[2]:</p><blockquote>“Consider an investor seeking to acquire a large block of S shares of some stock over a fixed time interval [0, T]. Since it is well-known that the shortterm demand curves for even the most actively traded equities are not perfectly elastic, a market order at date 0 for the entire block S is clearly not an optimal trading strategy. A more effective strategy would be to break S into smaller purchases distributed throughout the interval [0, T], but how should such purchases be parceled out?</blockquote><blockquote>The answer depends, of course, on the degree to which a purchase affects the market price, i.e., the ‘price impact’ and the dynamics of future market prices.”[2]</blockquote><p>Note what the authors are hinting, <em>knowing how the price impact function behaves</em> is the base for solving the problem optimally. Indeed, that’s what they assume.</p><blockquote>“In particular, we propose and solve the following problem in this paper: given a fixed block S of shares to be executed within a fixed finite number of periods T, and given price dynamics that capture price impact, i.e., the execution price of an individual trade as a function of the shares traded and other ‘state’ variables, find the optimal sequence of trades (as a function of the state variables) that will minimize the expected cost of executing S within T periods.”[2]</blockquote><p>There are more approaches that propose different ways of modelling price impact and the strategies, most of them arriving at closed form solutions.</p><p>However, electronic markets started sharing useful information that could be used to “learn” price impact functions.</p><blockquote>“For most of the history of the U.S. equity markets, the only information available to an agent attempting to optimize trade execution was the sequence of prices of already-executed trades, and the current bid and ask (the best outstanding buy and sell prices offered). But in recent years, electronic markets such as NASDAQ have begun releasing, in real time, all of the outstanding buy and sell limit order prices and volumes (often referred to as order book or market microstructure data). It is not difficult to see that such data might be extremely valuable for the problem of optimized execution, since the distribution of outstanding limit orders may help predict short-term price direction, likelihood of execution at a given price, buy or sell side imbalances, and so on.”[3]</blockquote><h3>Optimal Trade Execution with Reinforcement Learning</h3><p>Nevmyvaka, Feng, and Kearns[3] published the first paper highlighting the usefulness of RL for the optimal trade execution problem.</p><blockquote>“We present the first large-scale empirical application of reinforcement learning to the important problem of optimized trade execution in modern financial markets. Our experiments are based on 1.5 years of millisecond time-scale limit order data from NASDAQ, and demonstrate the promise of reinforcement learning methods to market microstructure problems. Our learning algorithm introduces and exploits a natural “low-impact” factorization of the state space.”[3]</blockquote><p>Let’s briefly review the main elements of a RL problem: states, actions, rewards, learning algorithm.</p><h3>RL elements</h3><p><strong>States</strong></p><p>Some key points to consider, the authors accept the model is imperfect, the authors focus on trying to capture the most important components that affect price impact, and also try additional features.</p><blockquote>1. “Each state x ∈ X is a vector of attributes (state variables) that describes the current configuration of our system. We note that where we say state we more precisely mean observed state; we in no way suggest that our state representations are sufficient to render a system as complex as modern financial markets truly Markovian.”</blockquote><blockquote>2. “While we will explore a number of different state representations, there are two important variables that appear in all of them. These are the elapsed time t and the remaining inventory i, which represent how much time of the horizon H has passed and how many shares we have left to execute in the target volume V.”</blockquote><blockquote>3. “We will also examine a number of additional state variables we refer to as market variables, which compute various properties of interest of the limit order books and recent activity in the stock.”[3]</blockquote><p><strong>Actions</strong></p><p>We can interpret the action space as how <em>aggressive</em> would be the price to pay. A 0 value is the “default”, but we can move in either direction, offering to pay more or less than what is being asked.</p><blockquote>“Our action space can be thought of as a simple limit order price at which to reposition all of our remaining inventory, relative to the current ask or bid. In other words, for the problem of selling V shares, action a corresponds to placing a limit order for all of our unexecuted shares at price ask — a. Thus we are effectively withdrawing any previous outstanding limit order we may have (which is indeed supported by the actual exchanges), and replacing with a new limit order. Thus a may be positive or negative, with a=0 corresponding to coming in at the current ask, positive a corresponding to “crossing the spread” towards the buyers, and negative a corresponding to placing our order in the sell book. For the problem of buying, a is interpreted relative to the current bid. For either buying or selling, more positive actions mean movement towards the opposing book, and more negative actions mean placement deeper within our own book.”[3]</blockquote><p><strong>Rewards</strong></p><p>The agent gets immediate rewards whenever there’s some execution based on the prices it paid. Comparisons are made with respect to a perfect policy that can execute at the mid-spread price.</p><p>Note there’s a strong constraint of fulfilling all the shares in the period so, worst case scenario the agent pays “market price” before the time limit.</p><blockquote>“An action from a given state may produce immediate rewards, which are essentially the proceeds (cash inflows or outflows, depending on whether we are selling or buying) from any (partial) execution of the limit order placed.”</blockquote><blockquote>“In order to permit comparisons of policies and performance across stocks, which may have very different per-share prices, we always measure the execution prices achieved by a policy relative to the mid-spread price (namely, (ask+bid)/2) at the start of the episode in question. Thus, we are effectively comparing performance to the idealized policy which can execute all V of its shares immediately at the mid-spread”</blockquote><blockquote>“Furthermore, since we view the execution of all V shares as mandatory, any inventory remaining at the end of time H is immediately executed at market prices — — that is, we eat into the opposing book to execute, no matter how poor the prices, since we have run out of time and have no choice.”[3]</blockquote><p><strong>Simulator</strong></p><p>To learn, RL needs a simulator where different actions can be evaluated to learn the optimal policy. In this work, the authors reused historical data within a simulator.</p><blockquote>“In our study, we used historical records from the INET Electronic Communication Network, an electronic, completely automated stock exchange, which accounts for a significant volume of trading in NASDAQ stocks. We have developed a simulator (or a generative model in RL terminology) that combines the real world INET order flow with artificial orders generated by our execution strategies.”[3]</blockquote><p><strong>Learning Algorithm</strong></p><p>The approach used is based on Q-learning[6].</p><blockquote>It fuses Q-learning and dynamic programming and exploits the approximate independence between private and market variables.[3]</blockquote><h3>Results and Conclusions</h3><p>The authors performed different experiments which highlight how the approach is able to learn policies that make sense with the situation.</p><blockquote>“It is interesting to examine more closely how actions relate to trading costs. The shapes of the Q-value functions explain the relationship between state variables and optimal actions: for large inventories and little time remaining, the entire Q-value function shifts upwards, (reflecting higher expected cost), but the minimum of the curve shifts to the right, which indicates that optimal action must be more aggressive in such situations.”[3]</blockquote><p>Overall, the paper sounds very optimistic in their results and comparisons.</p><blockquote>“Via careful choice of market state variables and the design of an efficient algorithm exploiting the independence and structure of the problem, we have shown that RL can indeed result in significant improvements over simpler forms of optimization (such as submit and leave policies).”[3]</blockquote><p>Fast forward almost 20 years later and there are many more papers on <a href="https://www.tandfonline.com/doi/abs/10.1080/1350486X.2022.2077783">deep reinforcement learning for optimal trading</a>, and there are also <a href="https://www.rbccm.com/en/expertise/electronic-trading/ai-trading.pa">examples running in production.</a></p><h3><strong>Dynamic Pricing</strong></h3><p>Let’s turn into the second problem. It would be easy to start finding similarities.</p><p>Let’s assume we are the owners of an hotel. Everyday our task is to set room prices with the goal of maximizing the revenue. There are diverse challenges:</p><ul><li><em>Liquidity</em>: There are limited number of rooms in the hotel. Note that it is possible to account for 100% occupancy but this does not directly implies the maximum possible revenue due to different prices for bookings.</li><li><em>Price elasticity</em>: Changing the price will have an effect on how many clients we attract. Additionally, more factors could also have an effect, for example, seasonality, events, and competition.</li><li><em>Multiagent</em>: There are other hotels offering similar accommodation, their prices and decision have an impact in our results.</li></ul><p>On a high-level is easy to spot similarities between dynamic pricing and optimal execution. Both of them:</p><ul><li>are problems where there are clear objectives (minimizing, maximizing a metric)</li><li>are subject to constraints (time, inventory)</li><li>have decisions (setting prices) that have an effect on the final measure.</li></ul><h3>Dynamic Pricing Examples</h3><p>Many companies have used different approaches of dynamic pricing. Let’s check some examples.</p><p><strong>Uber surge pricing</strong></p><blockquote>“The Uber platform adjusts its prices using a realtime dynamic algorithm known as ‘Surge’ pricing, which has generated considerable interest among both the press and academics.9 Surge pricing is the output of an algorithm which automatically raises the price of a trip when demand outstrips supply within a fixed geographic area.”[1]</blockquote><blockquote>“Surge multipliers are discrete; they range from a minimum of 1.2 up to a city- specific maximum in increment of 0.1. The specific multiplier is determined at any given time by what will be referred to as a generator surge multiplier (GSM), a continuous value from which the implemented surge multiplier is derived from, typically by rounding to the nearest tenth. One can think of the GSM as Uber’s estimate of the “true” market price for rides at a point in space and time.”[1]</blockquote><p><strong>Amazon Marketplace</strong></p><blockquote>“Our findings illustrate the power of algorithmic pricing in on-line marketplaces. Sellers we identified as using algorithmic pricing receive more feedback and win the Buy Box more frequently, likely suggesting higher sales volumes and thus more revenue than non-algorithmic sellers. Furthermore, we observe cases where algorithmic sellers change prices tens or even hundreds of times per day, which would be difficult for a human to maintain over time — especially one attempting to manage many products simultaneously — but is trivially automated.”[4]</blockquote><p><strong>Alibaba</strong></p><blockquote>“In our study we first represent the dynamic pricing problem as a Markov Decision Process (MDP). The agent periodically changes prices of the products as its action after observing environment state. The new environment state could then be observed and the reward could also be received. Each pricing episode reaches its end if the product is out of stock.”[5]</blockquote><p>The problem of dynamic pricing is complex has many different scientific communities involved[7], but Reinforcement Learning has received attention with approaches that have been recently applied in industry[5].</p><h3>Dynamic Pricing with Reinforcement Learning</h3><p>Taking as inspiration the optimal execution problem, we will solve a dynamic pricing problem with RL. We start defining the goal and assumptions, then we describe the elements of RL, and finally we highlight some results.</p><p><strong>Goal</strong></p><ul><li>Our goal is to develop an agent that <strong>sets prices every day</strong> for a hotel given a final date (deadline).</li><li>We aim to <strong>maximize the revenue</strong> of the hotel for that deadline while keeping some constraints.</li></ul><p><strong>Assumptions</strong></p><ul><li>We assume having a discrete set of prices (4 levels) for which the agent needs to select.</li><li>We assume having 2 months (60 days) to set prices before the deadline.</li><li>We assume starting with an occupancy of 0%.</li><li>If we reach 100% occupancy before the deadline the episode terminates.</li><li>We assume the “demand” for our hotel, i.e. bookings, are based on price, hotel features, and “interest”.</li><li>We assume “interest” in our hotel as a function of the unique views to our website.</li><li>We assume unique views to the website are based on seasonal information.</li></ul><p><strong>Notes</strong></p><ul><li>Maximizing only the occupancy does not directly maximizes the revenue.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*19kyxSgq7vFyRYZJdUgG4A.png" /><figcaption>The agent needs to select which price to set based on some relevant features. The goal is to maximize revenue. Setting the price to high has an effect on reducing demand, setting the price to low increases demand but hurts the revenue.</figcaption></figure><h3>RL elements</h3><p><strong>States</strong></p><p>We need to define features that compose the state space of the environment. There are many possible relevant features to consider:</p><ul><li><em>Seasonality features</em>: Examples are, whether a weekend or weekday, holiday or not.</li><li><em>Hotel features</em>: Relevant attributes of a hotel, might make it more desirable, for example, pool or parking.</li><li><em>Constraint features</em>: The agent has a limited number of steps to maximize revenue and the number of rooms available is also limited. Every booking will reduce occupancy and every day passed decreases the remaining time available.</li><li><em>Interest features</em>: Unique visits to the hotel website might give an indication of bookings.</li><li><em>Global demand features, competitiveness features, pricing features</em>. There are many different features that might be relevant. This is indeed one challenging aspect of the problem.</li></ul><pre>        # Observations space<br>        # There different types of features<br>        self.observation_space = spaces.Dict(<br>            {<br>                # Seasonality features<br>                &quot;is_weekend&quot;: spaces.Discrete(2),<br>                &quot;is_holiday&quot;: spaces.Discrete(2),<br>                &quot;current_day_is_weekend&quot;: spaces.Discrete(2),<br>   <br>                # Hotel features<br>                &quot;has_pool&quot;: spaces.Discrete(2),<br><br>                # Constraint features<br>                &quot;occupancy_discrete&quot;: spaces.Discrete(5),<br>                &quot;remaining_time_discrete&quot;: spaces.Discrete(3),<br><br>                # Interest<br>                &quot;uniqueviews_quantile_7d&quot;: spaces.Discrete(3),<br><br>                # Global demand<br>                # ...<br>                # Competitiveness<br>                # ...<br>                # Pricing<br>                # ...<br>                }<br>        )</pre><p><strong>Actions</strong></p><p>We propose 4 levels of prices: two discounts, one base price, and higher than base price.</p><pre># Action space of the agent<br># We assume a base price of 1, <br>price_multiplier = {0:0.1, 1:0.5, 2:1, 3:2.5}</pre><p><strong>Rewards</strong></p><p>At every day there could be some bookings and cancelations. That information will generate increasing or decreasing revenue at certain price.</p><pre>    # Reward is a function of price and demand<br>    def reward_dynpricing(self,action):<br>      demand_multiplier = self.get_demand_w_features(action)<br>      price_multiplier_value = price_multiplier[action]<br><br>      if demand_multiplier &lt; 0: # more cancelations than bookings<br>          price_multiplier_value = 1   <br>          if abs(demand_multiplier/100) &gt; self._occupancy:<br>              demand_multiplier = -self._occupancy<br>              price_multiplier_value = 2.5<br>        <br>      self._occupancy += demand_multiplier/100.0<br>      self._occupancy_discrete = min(int(self._occupancy*4),4)<br><br>      reward = demand_multiplier*price_multiplier_value<br>      d = {&quot;occupancy&quot;:self._occupancy,<br>          &quot;demand_multiplier&quot;:demand_multiplier,<br>          &quot;price_multiplier_value&quot;:price_multiplier_value,<br>         &quot;reward&quot;:reward<br>         }<br>      return d</pre><p><strong>Simulator</strong></p><p>In a more realistic scenario we could use time-series of features we have collected. For this example, I will simulate those features based on some assumptions.</p><ul><li>The deadline and hotel features are randomly initialized.</li><li>The seasonality features are easy to compute since we know the initial date and how many days have passed. We can check if the current day is a holiday and a weekend.</li><li>The “interest” features come from assuming a probability distribution of views and we take the quantiles of the distribution as one feature. Whenever the current date is a holiday or a weekend, we assume there is more people trying to book and assume an increase in the visit to our website.</li></ul><pre>    # Features of the season will change according to the day that we are<br>    def sample_season_features(self):<br>        current_day = self._init_date + timedelta(days=self._steps)<br>        self._remaining_time_discrete = min(int((self._steps/MAX_LEN_DAYS)*2),2)<br><br>        # check if current day is weekend<br>        if current_day.weekday() in [5,6]:<br>            self._current_day_is_weekend = 1<br>        else:<br>            self._current_day_is_weekend = 0<br><br><br>    # We assume that interest is a function of the current day<br>    def sample_interest_features(self):<br>        current_day = self._init_date + timedelta(days=self._steps)<br><br>        if current_day.weekday() in [5,6]:<br>            self._uniqueviews_quantile_7d = random.randint(1,2)<br>        else:<br>            self._uniqueviews_quantile_7d = random.randint(0,2)<br><br>    # Initialize environment with a random date<br>    def sample_init_date(self):<br>        start_dt = date(2020,1,1)<br>        end_dt = date(2025,1,1)<br>        days_diff = (end_dt-start_dt).days<br>        random_days = random.randint(0,days_diff)<br>        self._final_date = start_dt+timedelta(days=random_days)<br>        self._init_date = self._final_date-timedelta(days=MAX_LEN_DAYS)<br><br>        #check if current day is weekend<br>        if self._final_date.weekday() in [5,6]:<br>            self._is_weekend = 1<br>        else:<br>            self._is_weekend = 0<br><br>        #check if current day is in holidays list<br>        if self._final_date in holidays_lst:<br>            self._is_holiday = 1<br>        else:<br>            self._is_holiday = 0<br><br>    # Initialize features of the hotel<br>    def sample_init_hotel_features(self):<br>        self._has_pool = random.randint(0,1)</pre><p>The <em>demand</em> is a function that considers:</p><ul><li>The price set by the agent. Roughly, a big discount generates big demand, increasing the price reduces the demand. We assume price changes also generate cancelations made by people who now see a lower price than what they have paid before.</li><li>Remaining time for booking. We assume that closer to the date people are willing to pay a bit more.</li><li>Date for which the booking is made. Holidays and weekends have an increased demand.</li><li>The “interest” features. Whenever there is high interest there is a positive demand.</li><li>The hotel features, for example, having a pool, give a boost to demand.</li></ul><pre>    # Demand is a function of the price, hotel features, and interest<br>    def get_demand_w_features(self,action):<br>        demand = 0<br><br>        if self._is_holiday == 1:<br>            demand += random.randint(0,5)<br>        if self._is_weekend == 1:<br>            demand += random.randint(0,1)<br><br>        if  action == 0:<br>            demand += random.randint(1,30)<br>        elif action == 1:<br>            demand += random.randint(0,7)<br>        elif action == 2:<br>            demand += random.randint(0,5)<br><br><br>        if self._uniqueviews_quantile_7d == 3:<br>            demand += random.randint(1,10)<br>        elif self._uniqueviews_quantile_7d == 2:<br>            demand += random.randint(0,2)<br><br>        if self._has_pool == 1 and action &lt;= 1:<br>            demand += random.randint(0,1)<br> <br>        if self._remaining_time_discrete &gt;= 2:<br>            if action == 2:<br>                demand += random.randint(0,1)<br>            elif action &lt;=1:<br>                demand += random.randint(1,2)<br>        elif self._remaining_time_discrete &gt;= 1:<br>            if action == 0:<br>                demand += random.randint(1,4)<br>            else:<br>                demand += random.randint(0,1)<br><br>        cancel = 0<br>        if  action == 0:<br>            cancel -= random.randint(1,15)<br>        elif action == 1:<br>            cancel -= random.randint(0,7)<br>        elif action == 2:<br>            cancel -= random.randint(0,4)<br>        elif action == 3:<br>            cancel -= random.randint(0,1)<br>        <br>        if self._remaining_time_discrete &gt;= 2:<br>            cancel -= random.randint(0,2)<br>        elif self._remaining_time_discrete &gt;= 1:<br>            cancel -= random.randint(0,5)<br><br><br>        return demand+cancel</pre><ul><li>At every step <em>season</em> and <em>interest features</em> are sampled and a new observation is computed for the agent.</li><li>The reward is computed considering the action of the agent and the <em>demand</em>.</li><li>If the occupancy is reached or we have reached the deadline the episode terminates.</li></ul><pre>    # Simulate one step (equivalent to one day)<br>    def step(self, action):<br>        self.sample_season_features()<br>        self.sample_interest_features()<br><br>        season_features = self.get_season_features()<br>        hotel_features = self.get_hotel_features()<br>        interest_features = self.get_interest_features()<br>        features = season_features | hotel_features | interest_features<br><br>        reward_metrics = self.reward_dynpricing(action)  # reward function<br>        terminated = (self._steps &gt; MAX_LEN_DAYS) or (self._occupancy &gt;= 1.0)<br>        self._steps += 1<br>        observation = self._get_obs()<br>        info = self._get_info() | reward_metrics<br>        self._pct_togo = (self._steps/MAX_LEN_DAYS)<br><br>        return observation, reward_metrics[&#39;reward&#39;], terminated, False, info</pre><p><strong>Algorithm</strong></p><p>Our learning algorithm uses <em>Q-learning.</em></p><pre>class DynamicPricingQL:<br>    def __init__(self, num_states = 720,<br>                 learning_rate: float = 0.2,<br>                 discount: float = 0.99,<br>                 exploration_p: float = 0.2) -&gt; None:<br>        self.learning_rate = learning_rate<br>        self.discount = discount<br>        self.exploration_p = exploration_p<br>        self.num_states = num_states<br>        self.num_actions = env.action_space.n<br>        self.q_values = np.ones((self.num_states,<br>                                  self.num_actions))<br><br>    def get_best_action(self, state):<br>        pi = np.argmax(self.q_values, axis=1)<br>        return pi[state]<br><br>    def update_value(self,state,action,next_state,reward,terminated):<br>      if terminated:<br>        self.q_values[state, action] += self.learning_rate * (reward - self.q_values[state, action])<br>      else:<br>        self.q_values[state, action] += self.learning_rate * (reward + self.discount * np.max(self.q_values[next_state, :]) - self.q_values[state, action])<br><br>      self.learning_rate = self.learning_rate*0.999999<br>      self.exploration_p = self.exploration_p*0.999999<br><br>    def get_action(self, state):<br>      if np.random.random() &lt; self.exploration_p:<br>            action = env.action_space.sample()<br>      else:<br>            action = np.argmax(self.q_values[state, :])<br><br>      return action</pre><h3>Experiments</h3><p>I used <a href="https://gymnasium.farama.org/">gymnasium library</a> to develop the environment. After adding it to the registry we can use it together with the Q-learning agent.</p><p>For this example, I’m using 7 features which are informative enough to provide a signal for the agent. With the 7 features our state space comprises 720 states. The actions space is size 4, so the Q-table size is 720 times 4 = 2880 values to learn.</p><p>We need to define an extra function to convert the dictionary of observations to an <em>int</em> that is the index of the state.</p><pre>def get_index_from_observation(observation):<br>  f0 = observation[&#39;is_weekend&#39;]<br>  f1 = observation[&#39;is_holiday&#39;]<br>  f2 = observation[&#39;current_day_is_weekend&#39;]<br>  f3 = observation[&#39;has_pool&#39;]<br>  f4 = observation[&#39;occupancy_discrete&#39;]<br>  f5 = observation[&#39;remaining_time_discrete&#39;]<br>  f6 = observation[&#39;uniqueviews_quantile_7d&#39;]<br>  <br>  index = f0+(f1*2)+(f2*4)+(f3*8)+(f4*16)+(f5*80)+(f6*240)<br>  return index</pre><p>Now, we are ready to run the environment and see what the agent has learned.</p><pre>env = gymnasium.make(&#39;gym_examples/DynamicPricing-v0&#39;)<br>agent = DynamicPricingQL()<br>rewards_per_episode = []<br>observation,_ = env.reset()<br><br>for episode in range(50_000):<br>  rewards = []<br>  timesteps = 0<br>  for _ in range(MAX_LEN_DAYS):<br>      index_obs = get_index_from_observation(observation)<br>      action = agent.get_action(index_obs)<br>      next_obs, reward, terminated, truncated, info = env.step(action)<br>     index_next_obs = get_index_from_observation(next_obs)<br>      agent.update_value(index_obs, action, index_next_obs,reward,terminated)<br>      rewards.append(reward)<br><br>      if terminated or truncated:<br>          observation, info = env.reset()<br>          break<br><br>      timesteps+=1<br>      observation = next_obs<br>      index_obs = index_next_obs<br>  if episode %1000==0:<br>      print(episode)<br>  rewards_per_episode.append(rewards)<br>env.close()</pre><p><strong>Baselines</strong></p><p>We can check the rewards for 1000 episodes for the 4 different actions (price levels).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*eNQFehGvftg9d29XIkczTQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*rJ6gevLfD7hRCp5UWTGO1Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*xsziYlGbQJgfZXXyoiaR0Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/651/1*pmkMW_02jm4ld2-uh3qbpQ.png" /></figure><p>We can make some observations:</p><ul><li>Actions 0 and 1, which are the discount prices generate a negative reward or close to zero.</li><li>Action 2, which is the base price generates some positive rewards for many episodes.</li><li>Action 3, which is a higher than normal price, generates high rewards for a few episodes, while being close to zero for the rest. This happens because those episodes with high reward are holidays, in which bookings happen irrespective of the higher price.</li></ul><p><strong>Q-learning</strong></p><p>We let the agent learn for 50k episodes and then we used the Q-values to compute an optimal policy that we evaluate on 1000 episodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*Na0SIwxvslBBXhud7RlQ7Q.png" /></figure><p>We can see that the average reward is higher than any of the 4 basic actions (prices). Checking some individual episodes we see that:</p><ul><li>The agent makes use of the information whether the deadline is a weekend or holiday, exploiting that information in price. For example, when the deadline is a holiday it starts with a base price (level 2), but being close to the deadline it <em>increases</em> prices to level 3.</li><li>The agent also behaves differently depending on how much time is left before the deadline. For example, it starts with a base price (level 2), but when being closer to the deadline it can drop the price to get some positive reward.</li></ul><h3>What’s next</h3><p>It is important to mention some challenges that need to be addressed for a real world use case of dynamic pricing.</p><ul><li>Features: As we have describes, there are many features to use. Understanding which ones are more informative would reduce the noise in the state space and would improve the results. Having too many features is a problem for tabular approaches like the one described in this post. The use of function approximation with or without neural networks seems to be a natural approach to address this problem.</li><li>Metric: <a href="http://incompleteideas.net/rlai.cs.ualberta.ca/RLAI/rewardhypothesis.html">In RL everything comes down to the reward obtained by the agent</a>. Revenue is a natural metric to be optimized in dynamic pricing, however, there are more components that we can add, for example, people might not appreciate changing prices constantly, so we can penalize that.</li></ul><h3>Conclusions</h3><p>This post makes a connection between optimal trade execution and dynamic pricing by using reinforcement learning to solve both problems. Reinforcement learning is good at solving sequential decision problems with some constraints. While challenging to make it work, RL is a good fit for many problems faced in industry.</p><h3>References</h3><ul><li>[1] Chen, M. Keith, and Michael Sheldon. “Dynamic pricing in a labor market: Surge pricing and flexible work on the Uber platform.” <em>Ec</em> 16 (2016): 455.</li><li>[2] Bertsimas, Dimitris, and Andrew W. Lo. “Optimal control of execution costs.” <em>Journal of Financial Markets</em> 1 (1998): 1–50.</li><li>[3] Nevmyvaka, Yuriy, Yi Feng, and Michael Kearns. “Reinforcement learning for optimized trade execution.” <em>Proceedings of the 23rd international conference on Machine learning</em>. 2006.</li><li>[4] Chen, Le, Alan Mislove, and Christo Wilson. “An empirical analysis of algorithmic pricing on amazon marketplace.” <em>Proceedings of the 25th international conference on World Wide Web</em>. 2016.</li><li>[5] Liu, Jiaxi, et al. “Dynamic pricing on e-commerce platform with deep reinforcement learning: A field experiment.” <em>arXiv preprint arXiv:1912.02572</em> (2019).</li><li>[6] Watkins, Christopher JCH, and Peter Dayan. “Q-learning.” <em>Machine learning</em> 8 (1992): 279–292.</li><li>[7] Den Boer, Arnoud V. “Dynamic pricing and learning: historical origins, current research, and new directions.” <em>Surveys in operations research and management science</em> 20.1 (2015): 1–18.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce07f9c1fc4c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rerank and RAG for Financial Chatbots]]></title>
            <link>https://medium.com/@hlealpablo/rerank-and-rag-for-financial-chatbots-8dea9bb3fa54?source=rss-d18f7b3c3588------2</link>
            <guid isPermaLink="false">https://medium.com/p/8dea9bb3fa54</guid>
            <category><![CDATA[financial-chatbots]]></category>
            <category><![CDATA[information-retrieval]]></category>
            <category><![CDATA[llama-3]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[gradio]]></category>
            <dc:creator><![CDATA[Pablo Hernandez Leal]]></dc:creator>
            <pubDate>Thu, 27 Jun 2024 01:08:24 GMT</pubDate>
            <atom:updated>2024-06-27T01:08:24.310Z</atom:updated>
            <content:encoded><![CDATA[<p>Part 3: Lexical Search + Rerank + RAG</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qVGoXRzlFCM1ZEpd8T9qMA.png" /></figure><p>Previous posts explored <a href="https://medium.com/@hlealpablo/using-in-context-learning-and-semantic-similarity-for-financial-education-chatbots-v0-1-bb6f1d36ac2e">in-context learning</a> and <a href="https://medium.com/@hlealpablo/building-a-financial-education-chatbot-with-retrieval-augmented-generation-rag-bf338aa2df09">RAG with embeddings for financial chatbots</a>. Now, it is time to adapt our RAG and use lexical search and reranking before passing the data to the LLM.</p><h3>TL;DR</h3><p>We develop a retrieval based on lexical search (Elasticsearch) followed by reranking (ColBERT) and pass to an LLM (Llama 3) to answer questions about financial reports. We add an interface with Gradio and show different examples when it works and when it doesn’t.</p><h3><strong>Problem</strong></h3><p>Using only a vector database and embeddings for retrieval is certainly doable <a href="https://medium.com/@hlealpablo/building-a-financial-education-chatbot-with-retrieval-augmented-generation-rag-bf338aa2df09">(like we did a previous post)</a>, but it might fail in some cases. From the CEO of <a href="https://en.wikipedia.org/wiki/Perplexity.ai">Perplexity.ai</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*oemShir0vkx-X6sS1FKICg.png" /><figcaption><a href="https://lexfridman.com/aravind-srinivas-transcript#chapter13_rag">Source</a></figcaption></figure><p>Semantic similarity is good in cases where there is ambiguity. However, there are other cases where exact matches are enough and have the advantage that algorithms are extremely <em>fast</em>.</p><p>There is a large amount of research in the information retrieval/search community that we cannot ignore.</p><p>More information:</p><ul><li><a href="https://cameronrwolfe.substack.com/p/the-basics-of-ai-powered-vector-search">This is a fantastic blogpost on search from the basics to the neural approaches.</a></li><li><a href="https://softwaredoug.com/blog/2024/06/25/what-ai-engineers-need-to-know-search">This is another nice post abut search for AI engineers</a>.</li></ul><h3>Information Retrieval</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zaMNvrNqtu3V25ndJr7qwQ.png" /><figcaption>Two-step document retrieval approach</figcaption></figure><p>In Information Retrieval when a user generates a query with respect to some documents there are essentially two steps:</p><ul><li><em>Retrieval</em>: A set of candidate documents is selected (this is a rough approximation) to reduce orders of magnitude the number of documents.</li><li><em>Ranking</em>: Only with the documents from the first step a more detailed analysis is performed to rank them.</li></ul><p>Then there is a selection of the <em>top k</em> to show to the user. In this post we’ll use lexical search for the retrieval and ColBERT for the ranking.</p><h3>Lexical Search (TF-IDF)</h3><p>The classical approach to compare documents to queries relies in counts and some functions based on those:</p><ul><li>Term frequency (TF). Number of times a word appears in a document divided by the length of the document.</li><li>Document frequency (DF). Number of documents that contain a specific word.</li><li>Inverse document frequency (IDF). Log of the total number of documents divided by DF.</li></ul><p>From those we can construct TF-IDF:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yjUiVayyNl-pTPG7w3PqLw.png" /><figcaption><a href="https://web.stanford.edu/class/cs224u/slides/cs224u-neuralir-2023-handout.pdf">Source</a></figcaption></figure><p>Computing some TF-IDF examples we see:</p><ul><li><em>terms that occur frequently in few documents </em>have higher values (2.3 in the image below).</li><li><em>terms that are frequent in almost all documents</em> have lower values (0.01 and 0.02 in the image below).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4D-UXm0Iw_-AR6dRZ55Dvw.png" /><figcaption><a href="https://web.stanford.edu/class/cs224u/slides/cs224u-neuralir-2023-handout.pdf">Source</a></figcaption></figure><p><a href="https://opensourceconnections.com/blog/2015/10/16/bm25-the-next-generation-of-lucene-relevation/">BM25</a> is an improvement over TF-IDF that is used in industry, but the underlying mechanism is the same.</p><h3>Dense Search (Neural networks)</h3><p>Deep learning gave a new push to IR. When Language Models (like <a href="https://arxiv.org/pdf/1810.04805">BERT</a>) showed remarkable results in different tasks, the IR community tried to incorporate them to their problem.</p><blockquote><a href="https://dl.acm.org/doi/pdf/10.1145/3331184.3331303">“Experiments show that fine-tuning pre-trained BERT models with a limited amount of search data can achieve better performance than strong baselines.”</a></blockquote><blockquote><a href="https://dl.acm.org/doi/pdf/10.1145/3331184.3331303">“Compared to bag-of-words retrieval models, the contextual language model can better leverage language structures, bringing large improvements on queries written in natural languages”</a></blockquote><p>In summary, fine-tuning language models for search gave very good results in many metrics (<em>except computational cost</em>). Part of the reason comes from deep models <em>understanding</em> words and meanings that are often ignored in lexical retrieval.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rXBbEtIEufO5Zrt4sakhOQ.png" /><figcaption><a href="https://dl.acm.org/doi/pdf/10.1145/3331184.3331303">Example 2 shows that the document word ‘in’ receives the strongest attention from the query word ‘where’. The word ‘in’ appears in the context of ‘in Germany’, so it satisfies the ‘where’ question. Words like ‘in’ and ‘where’ are often ignored by traditional IR methods due to their high document frequency in the corpus</a>.</figcaption></figure><blockquote><a href="https://dl.acm.org/doi/pdf/10.1145/3331184.3331303">Further analysis reveals that stopwords and punctuation, which are often ignored by traditional IR approaches, play a key role in understanding natural language queries by defining grammar structures and word dependencies</a>.</blockquote><p>In IR there are <em>queries</em> and <em>documents</em> that interact with each other in a neural network. There are different approaches in the architecture:</p><ul><li>(a) <em>Representation-focused</em>: When query and document embeddings are computed independently.</li><li>(b) <em>Matrix Interaction-focused</em>: Learn the interaction of query and documents.</li><li>(c ) <em>BERT Interaction-focused</em>: All-to-all interaction (BERT style).</li><li>(d) <em>Late Interaction</em>: Combination of (a) and (c ), independent query and document embeddings with <em>delayed</em> interaction to reduce computational costs.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vRZEiy0SrEGogUyIihNQjQ.png" /><figcaption><a href="https://arxiv.org/pdf/2004.12832">Source</a></figcaption></figure><h3>ColBERT</h3><p>BERT <em>all-to-all</em> style of interaction obtained the best scores in metrics related to accuracy. However, the latency (related to the computational cost) is much higher than in late interaction methods. One late interaction method is <a href="https://medium.com/@varun030403/colbert-a-complete-guide-1552468335ae">ColBERT.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GPKkdOYlpA8dmf9-p3c4nA.png" /><figcaption><a href="https://arxiv.org/pdf/2004.12832">Source</a></figcaption></figure><blockquote><a href="https://arxiv.org/pdf/2004.12832">“However, the remarkable gains delivered by these LMs come at a steep increase in computational cost.”</a></blockquote><blockquote><a href="https://arxiv.org/pdf/2004.12832">“To reconcile eficiency and contextualization in IR, we propose ColBERT, a ranking model based on contextualized late interaction over BERT. As the name suggests, ColBERT proposes a novel late interaction paradigm for estimating relevance between a query q and a document d</a>.”</blockquote><h3>Solution: Lexical Search + Reranker</h3><p>How to obtain semantic expressiveness from neural IR models without their large computational costs?</p><p>By combining lexical and dense approaches.</p><ul><li>A first <em>fast</em> step, lexical search with BM25, to retrieve a collection of documents.</li><li>A second step, semantical reranking with ColBERT using only the subset of documents.</li></ul><p>Having the background of what we are looking for. Now it’s time to work on the code.</p><h3>Data sources</h3><p><a href="https://medium.com/@hlealpablo/building-a-financial-education-chatbot-with-retrieval-augmented-generation-rag-bf338aa2df09">As in the previous post</a>, I’ll use financial reports (for example, 10-K) from public companies as the sources of information.</p><pre>NASDAQ_AAPL_2023_K.pdf<br>NASDAQ_LULU_2024_K.pdf<br>NYSE_BABA_2024_K.pdf<br>…</pre><h3>Code: Lexical Search</h3><p>I’ll make use of <a href="https://www.elastic.co/elasticsearch">Elasticsearch</a> for the first part of the retrieval. First, we define some functions:</p><ul><li><em>create_index(): </em>Creates an index based on a name and settings.</li><li><em>index_es_data(): </em>Populate index with the given sources.</li><li><em>get_metadata_from_namefile(): </em>Get metadata from sources.</li><li><em>convertpdf_to_dict(): </em>Transform sources (PDF and metadata) to a list of dictionaries.</li><li><em>prepare_es_data(): </em>Preprocess data for Elasticsearch.</li></ul><pre>import json<br>import subprocess<br><br># Connect to Elasticsearch (ES)<br># we pass username, password and certificate<br>es = Elasticsearch([&#39;youraddress&#39;],<br>                   basic_auth=(username, password),<br>                   ca_certs=&quot;/path/http_ca.crt&quot;)<br><br><br># Create index and add custom settings from a json file<br>def create_index(index_name):<br>    proc=subprocess.Popen([<br>      &#39;curl&#39;,<br>      &#39;-X&#39;, &#39;PUT&#39;,<br>      &#39;-H&#39;, &#39;Content-Type: application/json&#39;,<br>      &#39;-d&#39;, json.dumps(settings),<br>      &#39;youraddress/&#39;+index_name+&#39;?pretty&#39;,<br>      &#39;-u&#39;, &#39;elastic:yourpassword&#39;,<br>      &#39;-k&#39;,<br>    ], stdout=subprocess.PIPE)<br>    (out, err) = proc.communicate()<br>    print(out)<br>    return out<br><br># Delete previous index, create index, and adds data<br>def index_es_data(index, es_data):<br>  if es.indices.exists(index=index):<br>      print(&quot;deleting the &#39;{}&#39; index.&quot;.format(index))<br>      res = es.indices.delete(index=index)<br>      print(&quot;Response from server: {}&quot;.format(res))<br><br>  print(&quot;creating the &#39;{}&#39; index.&quot;.format(index))<br>  res = create_index(index)<br>  print(&quot;Response from server: {}&quot;.format(res))<br><br>  print(&quot;bulk index the data&quot;)<br>  res = es.bulk(index=index, body=es_data, refresh = True)<br>  print(&quot;Errors: {}, Num of records indexed: {}&quot;.format(res[&quot;errors&quot;], len(res[&quot;items&quot;])))<br><br># Get metadata from filename<br>def get_metadata_from_namefile(source):<br>  metainfo = source.split(&#39;/&#39;)[-1].split(&#39;_&#39;)<br>  d = {}<br>  d[&#39;company&#39;] = metainfo[1]<br>  d[&#39;year&#39;] = int(metainfo[2][:4])<br>  if len(metainfo) &gt;= 4:<br>    d[&#39;Q-K&#39;] = metainfo[3][:2]<br>  else:<br>    d[&#39;Q-K&#39;] = &quot;Annual&quot;<br>  return d<br><br># Transform list of documents and metadata<br># to list of dictionaries that can be process by ES<br>def convertpdf_to_dict(pages,metadata):<br>  datalist = []<br>  for i in range(len(pages)):<br>    datalist.append({&quot;company&quot;:metadata[&#39;company&#39;],<br>                      &quot;year&quot;:metadata[&#39;year&#39;],<br>                      &quot;Q-K&quot;:metadata[&#39;Q-K&#39;],<br>                      &quot;page&quot;:i,<br>                      &quot;content&quot;:pages[i].page_content})<br><br>  return datalist<br><br># Creates a list of documents to add to the ES index<br>def prepare_es_data(doctype, records):<br>  es_data = []<br>  for idx, record in enumerate(records):<br>    meta_dict = {<br>          &quot;index&quot;: {<br>              &quot;_index&quot;: doctype,<br>              &quot;_id&quot;: idx<br>          }<br>      }<br>    es_data.append(meta_dict)<br>    es_data.append(record)<br>  return es_data</pre><p>With use the above functions we are going to create our index with some PDFs from a specific folder.</p><pre>from langchain_community.document_loaders import PyPDFLoader<br>import glob<br><br>path = &quot;/pathtodocs/&quot;<br>findocs = glob.glob(path+&quot;*&quot;)<br><br>indexname = &quot;10-ks&quot;<br>es_findata = []<br><br>for strpdf in findocs:<br>  loader = PyPDFLoader(strpdf)<br>  pages = loader.load_and_split()<br>  metadata = get_metadata_from_namefile(strpdf)<br>  es_findata += convertpdf_to_dict(pages,metadata)<br><br>prepared_data = prepare_es_data(indexname,es_findata)<br>index_es_data(indexname, prepared_data)</pre><p>Once the index is ready, we can use it with some queries.</p><p>Before testing, we can define some basic preprocessing such as removing stopwords and punctuation while doing tokenization. Then, we define a function that searches for <em>a list of terms and a specific compan</em>y.</p><pre>from nltk import word_tokenize<br>from nltk.corpus import stopwords<br>import string<br>import nltk<br>nltk.download(&#39;punkt&#39;)<br><br># Set list of stopswords and punctuation for preprocessing<br>liststopwords = []<br>with open(&#39;/yourpath/stopwords.txt&#39;, &#39;r&#39;) as f:<br>    for line in f:<br>        liststopwords.append(line.strip().lower())<br>setofstops = set(liststopwords)<br>setpunctuation = set(list(string.punctuation))<br>remove_set = setofstops.union(setpunctuation)<br><br># Preprocess query and retrieve at most ndocs documents<br># for a specific company, match any term<br>def call_anyterm_elastic(companystr,terms, ndocs):<br>    terms=[i for i in word_tokenize(terms.lower()) if i not in remove_set]<br>    termsstr= &quot; OR &quot;.join(terms)<br>    print(termsstr)<br>    res = es.search(index=indexname,<br>        body={<br>            &quot;size&quot;:ndocs,<br>            &quot;query&quot;: {<br>                &quot;bool&quot;:{<br>                    &quot;must&quot;:[<br>                        {<br>                        &quot;match&quot;: {&quot;company&quot;: companystr},<br>                        },<br>                        {<br>                        &quot;query_string&quot;: {<br>                            &quot;query&quot;: termsstr,<br>                            &quot;default_field&quot;: &quot;content&quot;,<br>                        }<br>                        }<br>                        ]<br>                }<br>            }<br>        }<br>    )<br>    return res</pre><p>We can test it with the following query, company, and a maximum number of sources.</p><pre>companystr = &quot;BABA&quot;<br>q = &quot;What is stock based compensation?&quot;<br>ndocs = 50<br>res = call_anyterm_elastic(companystr,q,ndocs)<br># After removing punctuation and stopwords query is preprocessed to<br># stock OR based OR compensation<br><br>print(&quot;Got %d Hits:&quot; % res[&#39;hits&#39;][&#39;total&#39;][&#39;value&#39;])<br>for hit in res[&#39;hits&#39;][&#39;hits&#39;]:<br>    print(&quot;%(company)s, %(Q-K)s %(year)i %(page)i&quot; % hit[&quot;_source&quot;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*vnhSBOe62T6LfbwGZWJVwg.png" /><figcaption>At least 50 documents matched the query.</figcaption></figure><p>This search is quite fast but noisy. That’s why we need to improve with the reranker on only the selected documents from this first step.</p><h3>Code: ReRanker</h3><p>Once we have the results from the lexical search query we can pass it to the reranker (ColBERT). We’ll make use of the <a href="https://github.com/bclavie/RAGatouille">RAGatouille library</a>.</p><pre>from ragatouille import RAGPretrainedModel<br>colbert = RAGPretrainedModel.from_pretrained(&quot;colbert-ir/colbertv2.0&quot;)<br><br># Setting a maximum of 10 documents<br>docs = colbert.rerank(query=q,<br> documents=[<br>    d[&#39;_source&#39;][&#39;content&#39;] for d in res[&#39;hits&#39;][&#39;hits&#39;]<br>  ],<br>  k=10)<br><br>import pandas as pd<br># Create a pandas dataframe content and data from ColBERT<br>docsdf = pd.DataFrame(docs)</pre><p>ColBERT will output a score for each document which is used to rank the documents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*89K5Uq3VgYTguLLgLEz1eA.png" /><figcaption>Rank by ColBERT of top 10 documents from the 50 that were originally passed.</figcaption></figure><p>Lexical search reduced drastically the number of relevant documents. Reranking focuses on giving an order based on relevance. With these two steps we can now pass them to an LLM in a prompt and ask a particular question.</p><h3>Prompt</h3><p>We need to create a prompt that takes the source documents and structure them together with the question from the user. In this case, we split the prompt in two parts:</p><ul><li>A prompt with the system message and the context (documents).</li><li>A smaller prompt for each document.</li></ul><p>We use prompt functions defined in LangChain. Note that <em>the number of documents</em> is not defined in here. All documents are passed as “context”.</p><pre>from langchain.chains.combine_documents import create_stuff_documents_chain<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_core.prompts import PromptTemplate<br><br>prompt = ChatPromptTemplate.from_template(&quot;&quot;&quot;<br>You are a financial expert.<br>Answer the following question based only on the provided context.<br>If you don&#39;t know the answer, just say that you don&#39;t know, don&#39;t try to make up an answer.<br><br># Context:<br>{context}<br><br>## Question:<br>{input}<br>## Answer:<br>&quot;&quot;&quot;<br>)<br><br>document_prompt = PromptTemplate(<br>    input_variables=[&quot;company&quot;,<br>                     &quot;page_content&quot;, <br>                     &quot;Q-K&quot;,<br>                     &quot;year&quot;,<br>                     &quot;page&quot;],<br>    template=&quot;&quot;&quot;<br>    ## Extract of page={page} from document Type={Q-K}, Year={year}, for {company}:<br>    {page_content}<br>    &quot;&quot;&quot;<br>)<br><br>document_chain = create_stuff_documents_chain(llm=llm,<br>                                              prompt=prompt,<br>                                              document_prompt=document_prompt)</pre><h3>Putting it all together</h3><p>Now, we are ready to put all the pieces and see the results. We’ll use the following:</p><ul><li><strong>Elasticsearch</strong> for lexical search.</li><li><a href="https://replicate.com/meta/meta-llama-3-70b-instruct"><em>Llama-3</em> with <strong>Replicate</strong> as LLM.</a></li><li><a href="https://www.gradio.app/"><strong>Gradio</strong></a> for Interface.</li><li><strong>LangChain</strong> for the prompt and connection to the LLM.</li></ul><p>We define a function to retrieve 50 documents with lexical search and then using reranking to select 5 and use those in the prompt.</p><pre>import gradio as gr<br>import os<br><br># Llama 3 from Replicate<br>import replicate<br>from google.colab import userdata<br>from langchain_community.llms import Replicate<br>os.environ[&quot;REPLICATE_API_TOKEN&quot;] = &quot;yourkey&quot;<br>llm = Replicate(<br>    model=&quot;meta/meta-llama-3-70b-instruct&quot;,<br>    model_kwargs={&quot;temperature&quot;: 0.75, &quot;top_p&quot;: 0.9},<br>)<br><br># Colbert<br>from ragatouille import RAGPretrainedModel<br>colbert = RAGPretrainedModel.from_pretrained(&quot;colbert-ir/colbertv2.0&quot;)<br><br># Promtpts<br>from langchain.chains import create_retrieval_chain<br>from langchain.chains.combine_documents import create_stuff_documents_chain<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_core.prompts import PromptTemplate<br><br>prompt = ChatPromptTemplate.from_template(&quot;&quot;&quot;<br>You are a financial expert.<br>Answer the following question based only on the provided context.<br>If you don&#39;t know the answer, just say that you don&#39;t know, don&#39;t try to make up an answer.<br><br># Context:<br>{context}<br><br>## Question:<br>{input}<br>## Answer:<br>&quot;&quot;&quot;<br>)<br><br>document_prompt = PromptTemplate(<br>    input_variables=[&quot;company&quot;,<br>                     &quot;page_content&quot;, <br>                     &quot;Q-K&quot;,<br>                     &quot;year&quot;,<br>                     &quot;page&quot;],<br>    template=&quot;&quot;&quot;<br>    ## Extract of page={page} from document Type={Q-K}, Year={year}, for {company}:<br>    {page_content}<br>    &quot;&quot;&quot;<br>)<br>document_chain = create_stuff_documents_chain(llm=llm,<br>                                              prompt=prompt,<br>                                              document_prompt=document_prompt)<br><br># Stopwords and punctuation sets<br>from nltk import word_tokenize<br>from nltk.corpus import stopwords<br>import string<br>import nltk<br>nltk.download(&#39;punkt&#39;)<br><br>liststopwords = []<br>with open(&#39;/path/stopwords.txt&#39;, &#39;r&#39;) as f:<br>    for line in f:<br>        liststopwords.append(line.strip().lower())<br>setofstops = set(liststopwords)<br>setpunctuation = set(list(string.punctuation))<br>remove_set = setofstops.union(setpunctuation)<br><br># Simple search of ORs<br>def call_anyterm_elastic(companystr,terms, ndocs):<br>    terms=[i for i in word_tokenize(terms.lower()) if i not in remove_set]<br>    termsstr= &quot; OR &quot;.join(terms)<br>    print(termsstr)<br>    res = es.search(index=indexname,<br>        body={<br>            &quot;size&quot;:ndocs,<br>            &quot;query&quot;: {<br>                &quot;bool&quot;:{<br>                    &quot;must&quot;:[<br>                        {<br>                        &quot;match&quot;: {&quot;company&quot;: companystr},<br>                        },<br>                        {<br>                        &quot;query_string&quot;: {<br>                            &quot;query&quot;: termsstr,<br>                            &quot;default_field&quot;: &quot;content&quot;,<br>                        }<br>                        }<br>                        ]<br>                }<br>            }<br>        }<br>    )<br>    return res<br><br># Lexical search and reranking<br>def search_and_rank(query, companystr, maxk):<br>    ndocs = 50<br>    res = call_anyterm_elastic(companystr,query,ndocs)<br>    print(&quot;Got %d Hits:&quot; % res[&#39;hits&#39;][&#39;total&#39;][&#39;value&#39;])<br>    docs = colbert.rerank(query=query, documents=[<br>        d[&#39;_source&#39;][&#39;content&#39;] for d in res[&#39;hits&#39;][&#39;hits&#39;]<br>    ], k=maxk)<br><br>    indices = [d[&#39;result_index&#39;] for d in docs]<br>    selected = [d[&#39;_source&#39;] for d in res[&#39;hits&#39;][&#39;hits&#39;]]<br>    docslang = []<br>    for i,s in enumerate(selected):<br>      m = {&#39;company&#39;: s[&#39;company&#39;],<br>      &#39;year&#39;: s[&#39;year&#39;],<br>      &#39;Q-K&#39;: s[&#39;Q-K&#39;],<br>      &#39;page&#39;: s[&#39;page&#39;],}<br>      docslang.append(Document(page_content=s[&#39;content&#39;],<br>                              metadata=m))<br>      if i &gt;= maxk-1:<br>        break<br><br>    dchain = document_chain.invoke({&quot;context&quot;: docslang, &quot;input&quot;: query})<br>    return dchain, docslang<br><br># Sources to str<br>def docs_to_info(sources):<br>  info = &quot;&quot;<br>  for i,doc in enumerate(sources):<br>    p = f&quot;&quot;&quot;\nDocument type: {doc.metadata[&#39;Q-K&#39;]},year={doc.metadata[&#39;year&#39;]},page={doc.metadata[&#39;page&#39;]}&quot;&quot;&quot;<br>    info+=p<br><br>  return info<br><br># Chatbot<br>def finchatv3(question,history, company):<br>    query = question<br>    maxk = 5<br>    answer,sources = search_and_rank(query,company,maxk)<br>    info = docs_to_info(sources)<br>    bot_message = &quot;## Answer about &quot;+company+&quot;\n &quot;+answer+&quot; \n ### Sources:&quot;+info<br>    history.append((question, bot_message))<br>    return &quot;&quot;, history<br><br># Interface<br>with gr.Blocks() as demo:<br>    chatbot = gr.Chatbot(height=700)<br>    msg = gr.Textbox(label=&quot;Question&quot;)<br>    clear = gr.ClearButton([msg, chatbot])<br>    companydrop = gr.Dropdown(<br>            [&quot;AAPL&quot;, &quot;LULU&quot;, &quot;BABA&quot;, &quot;QSR&quot;, &quot;JPM&quot;, &quot;RY&quot;],<br>            value=&#39;QSR&#39;,<br>            multiselect=False,<br>        )<br>    examples = gr.Examples(examples=[&quot;How much is China growing?&quot;,<br>                           &quot;What are the revenue segments?&quot;,<br>                           &quot;What is the stock based compensation?&quot;],<br>                           inputs=[msg])<br>    msg.submit(finchatv3,<br>               inputs=[msg,chatbot,companydrop],<br>               outputs=[msg,chatbot])<br><br>demo.queue()<br>demo.launch(share=True,debug=True)</pre><h3>Testing it</h3><p>Now we can ask questions about different companies. Our code will retrieve relevant documents and the LLM will use them to give us an answer. Some examples below:</p><p><strong>AAPL (Correct retrieval and answer)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/1*k65AqGTiOsk4kwwkgI5lAg.png" /></figure><p>We can check with the 10-K and we can confirm the data is correct, but the math is not exact in both cases.</p><pre>(72559–74200)/72200*100 = -2.21<br>(74200–68366)/68366*100 = 8.53</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8gASnTmn7d4wqmeG26a9Yw.png" /><figcaption>Page 47 of AAPL 10-K 2023</figcaption></figure><p><strong>RY (Correct retrieval, incorrect answer)</strong></p><p>The question is somewhat vague, however the LLM also mentions where it got the information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*pc-d4V_sBSWp38cvmZ34AA.png" /></figure><p>However, when checking the actual sources, it’s not clear how the LLM arrived at those $39 million.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mnzn6ZXVPph17oJ6s-mrIg.png" /><figcaption><strong>Royal Bank of Canada </strong>Annual Report 2023</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TnGS4VbtiV30IhlPJbGKcg.png" /><figcaption><strong>Royal Bank of Canada </strong>Annual Report 2023</figcaption></figure><h3>Detailed example: QSR</h3><p>Here I show all the elements of the prompt in an example for company QSR.</p><pre>You are a financial expert.<br>Answer the following question based only on the provided context.<br>If you don&#39;t know the answer, just say that you don&#39;t know, don&#39;t try to make up an answer.<br><br># Context:<br><br><br>## Extract of page=151 from document Type=Annual, Year=2023, for QSR:   <br>Table of Contents<br>The following table illustrates estimated revenues expected to be recognized in the future related to performance obligations that are unsatisfied (or partially unsatisfied) on a consolidated basis as of December 31, 2023 (in millions)... <br><br><br>## Extract of page=154 from document Type=Annual, Year=2023, for QSR:    <br>international operations as a separate operating and reportable segment.<br>Consequently, we have five operating and reportable segments: (1) TH, which includes all operations of our Tim Hortons brand in the United States and Canada, (2) BK, which includes all operations of our Burger King brand in the United States and Canada, (3) PLK, which<br>includes all operations of our Popeyes brand in the United States and Canada, (4) FHS, which includes all operations of our Firehouse Subs brand in the United States and Canada, and (5) INTL, which includes all operations of each of our brands outside the United States and Canada...<br><br><br>## Extract of page=4 from document Type=Annual, Year=2023, for QSR:<br>Table of Contents<br>Part I<br>Item 1. Business<br>Company Overview<br>We are a Canadian corporation that serves as the indirect holding company for the entities that own and franchise the Tim Hortons®, Burger King®, Popeyes® and Firehouse Subs® brands. We are one of the world’s largest quick service restaurant (“QSR”) companies with over<br>$40 billion in annual system-wide sales and more than 30,000 restaurants in more than 120 countries and territories as of December 31, 2023. Our four iconic brands have complementary daypart mixes and product platforms that benefit from global scale and sharing of best practices...<br><br><br>## Extract of page=54 from document Type=Annual, Year=2023, for QSR:   <br>Table of Contents<br>Beginning with the fourth quarter of 2023, we are reporting results under five operating and reportable segments. This shift in reportable segments reflects how RBI&#39;s leadership oversees and manages the business. As a result of this change, our five operating and reportable<br>segments consist of the following:<br>1.Tim Hortons – all operations of our Tim Hortons brand in Canada and the U.S. (“TH”);<br>2.Burger King – all operations of our Burger King brand in the U.S. and Canada (“BK”);<br>3.Popeyes Louisiana Kitchen – all operations of our Popeyes brand in the U.S. and Canada (“PLK”);<br>4.Firehouse Subs – all operations of our Firehouse Subs brand in the U.S. and Canada (“FHS”); and<br>5.International – all operations of each of our brands outside the U.S. and Canada (“INTL”).<br>Prior year amounts presented have been reclassified to conform to this new segment presentation with no effect on previously reported consolidated results.<br>In addition, we transitioned our definition of segment income from Adjusted EBITDA to Adjusted Operating Income (“AOI”). AOI represents income from operations, adjusted to exclude (i) franchise agreement amortization as a result of acquisition accounting, (ii) (income) loss<br>from equity method investments, net of cash distributions received from equity method investments, (iii) other operating expenses (income), net and, (iv) income/expenses from non-recurring projects and non-operating activities. Unlike Adjusted EBITDA, our previous measure<br><br><br>## Extract of page=5 from document Type=Annual, Year=2023, for QSR:    <br>Table of Contents<br>Additional financial information about our reportable segments can be found in “Management’s Discussion and Analysis of Financial Condition and Results of Operations” and Note 17, “Segment Reporting and Geographic Information,” to the accompanying consolidated financial<br>statements included in Part II, Item 8 “Financial Statements and Supplementary Data.”<br>Sources of Revenue<br>We generate revenues from the following sources: (i) sales, consisting primarily of (1) Tim Hortons supply chain sales, which represent sales of products, supplies and restaurant equipment to franchisees, as well as sales of consumer packaged goods (“CPG”), and (2) sales at<br>Company restaurants; (ii) franchise revenues, consisting primarily of royalties based on a percentage of sales reported by franchised restaurants and franchise fees paid by franchisees; (iii) property revenues from properties we lease or sublease to franchisees; and (iv) advertising revenues<br>and other services, consisting primarily of (1) advertising fund contributions based on a percentage of sales reported by franchised restaurants to fund advertising expenses and (2) tech fees and revenues that vary by market and partially offset expenses related to technology initiatives.<br>For the year ended December 31, 2023, the primary revenues for each segment were:<br>Segment Sales Franchise Revenues Property Revenues Advertising and Other Revenues<br>TH Supply chain and CPG<br>Royalties and franchise feesRental income on ~75% restaurants Ad <br><br><br>## Extract of page=124 from document Type=Annual, Year=2023, for QSR:    <br>Table of Contents<br>Note 5. Intangible Assets, net and Goodwill<br>Intangible assets, net and goodwill consist of the following (in millions)...<br><br><br>## Question:<br>What are the revenue segments?<br>## Answer:&quot;</pre><p>The answer from the LLM was the following:</p><pre>The revenue segments are:<br><br>1. Sales<br>2. Franchise revenue<br>3. Property revenues<br>4. Advertising revenues and other services</pre><p>which is correct, based on information on <em>page=5.</em></p><p>We can slightly change the question and see that the sources of information are very similar, but the LLM is able to differentiate and answer correctly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*evZNd7Xzy42Bp2Sda-bROw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*chHbLntYN5IBbAvuTiW8Mw.png" /></figure><h3>What is next?</h3><p>We have developed a RAG approach with lexical search and reranking.</p><p>However, there is still many possible improvements:</p><ul><li>The sources of information are financial reports which contain many tables, which are not always well formatted to text.</li><li>As we saw in one example, math is not a strong suit for LLMs, adding the capability to call functions would also improve some results.</li></ul><p>But, before adding more components, we need to take a step back and do a proper <a href="https://www.pinecone.io/learn/series/vector-databases-in-production-for-busy-engineers/rag-evaluation/">evaluation of any RAG system</a>.</p><p>You can <a href="https://medium.com/@hlealpablo">follow my publications</a> and if you are interested in LLMs I suggest to check my blogpost about <strong>prompting:</strong></p><ul><li><a href="https://medium.com/@hlealpablo/prompting-the-wonderful-wizard-part-1-847b709dea0a">In the first one I answer: How it started? How has it evolved, and why is it important? How does it work? When does it not work?</a></li><li><a href="https://medium.com/@hlealpablo/prompting-the-wonderful-wizard-part-2-0c92c3e746c6">In the second I answer: What are the risks involved? How to use it? (TL;DR). Beyond prompting and my personal take.</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8dea9bb3fa54" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From NeurIPS to the Real World: Key Takeaways from Competing in Pommerman]]></title>
            <link>https://medium.com/@hlealpablo/from-neurips-to-the-real-world-key-takeaways-from-competing-in-pommerman-abc5fd56714a?source=rss-d18f7b3c3588------2</link>
            <guid isPermaLink="false">https://medium.com/p/abc5fd56714a</guid>
            <category><![CDATA[multi-agent-systems]]></category>
            <category><![CDATA[neurips]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[multiagent-system]]></category>
            <category><![CDATA[neurips-2018]]></category>
            <dc:creator><![CDATA[Pablo Hernandez Leal]]></dc:creator>
            <pubDate>Wed, 26 Jun 2024 00:00:41 GMT</pubDate>
            <atom:updated>2024-06-26T14:09:20.987Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/567/0*UCBE6aDuSS4JvZ7A.png" /></figure><blockquote><em>“In theory there is no difference between theory and practice. In practice there is.” — Yogi Berra</em></blockquote><p>Back in 2018, <a href="https://en.wikipedia.org/wiki/Pommerman_Challenge">the Pommerman Competition</a> was announced to take place at NeurIPS. Together with Chao, Bilal, and Matt (my supervisor), we submitted an agent that secured 2nd place in the Learning Agents category. Reflecting on this experience -and trying to generalize about participating in competitions- I think there are lessons learned that are applicable in industry settings, such as:</p><ul><li>Working with deadlines</li><li>Dealing with constraints</li><li>Focusing on solving problems</li><li>Understanding trade-offs</li><li>Taking ownership</li><li>Building something from scratch</li><li>Having skin in the game</li></ul><p>First, let’s start with a short introduction to Pommerman.</p><h3>What is Pommerman?</h3><blockquote><a href="https://arxiv.org/abs/1809.07124"><em>“Pommerman is a multi-agent environment based on the classic console game Bomberman. Pommerman consists of a set of scenarios, each having at least four players and containing both cooperative and competitive aspects. We believe that success in Pommerman will require a diverse set of tools and methods, including planning, opponent/teammate modeling, game theory, and communication, and consequently can serve well as a multi-agent benchmark.”</em></a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*6GEyTAzYZ8S4XRIF.png" /></figure><p><em>A Screenshot of Pommerman, agents can place bombs that explode after some timesteps.</em></p><p><a href="https://youtu.be/-uSOhlpANks?si=G69_Mq0d_lR8tpb2"><em>You can check this video, it’s quite easy to understand the game after seeing it.</em></a></p><p>The game resembles a console game, and the simulator allows you to control some of the four players (agents). Some game rules include:</p><ul><li>Each agent can execute one of six actions per timestep: do nothing, move in one of four cardinal directions, or drop a bomb.</li><li>The board consists of passages (walkable), walls (indestructible), or wood (destructible with bombs).</li><li>Maps are randomly generated.</li><li>Bombs explode after 10 timesteps, destroying wood and any agents within their blast radius.</li><li>There are power-ups for players.</li><li>Each game episode lasts up to 800 timesteps.</li><li>There is only a binary reward at the end of the game, +1 or -1.</li><li>The agents only <em>see</em> a region around them, not the entire board (partially observable).</li></ul><p>If you want to learn more about how we addressed the technical challenges you can <a href="https://www.borealisai.com/research-blogs/pommerman-team-competition-or-how-we-learned-stop-worrying-and-love-battle/">check here.</a></p><h3>1. Working with Deadlines</h3><p>Having a background on the competition, we can go into the lessons learned.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/709/0*_guIw8_mraObqqwd.png" /></figure><p>When I learned about the competition, I had a conversation with my supervisor along the lines of:</p><pre>S: &quot;Have you seen this Pommerman competition?&quot; <br>Pablo: &quot;Yes, sounds interesting&quot; <br>S: &quot;Yeah, we should submit something&quot; <br>Pablo: &quot;Ok... but we only have like 3 months&quot; <br>S: &quot;Great! Plenty of time.</pre><p>During my Ph.D., I had similar comments from previous supervisors. Conference deadlines are intense, meaning you have to work under pressure. I also thought it meant sacrificing quality.</p><p>Years later, my attitude towards deadlines has changed. I’ve even seen extreme cases of having just <em>one week</em> to produce something (hackathon style?).</p><p>Does it make sense to have such tight (impossible) deadlines? Maybe.</p><p>Now, I try to make sense with the following arguments:</p><p>I also think that once the deadline has passed, there should be some time to take a step back and analyze the outcome.</p><h3>2. Dealing with Constraints</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/0*JdtJz8EeT1Q3bUy_.png" /></figure><blockquote><em>“Frugality drives innovation, just like other constraints do. One of the only ways to get out of a tight box is to invent your way out.” — Jeff Bezos</em></blockquote><p>Pommerman had many distinct technical challenges: multi-agent, planning, learning, and communication, among others.</p><p>Trying to solve <em>only one</em> of the issues wouldn’t have resulted in a good agent for the competition.</p><ul><li>One example to understand the challenges was the <a href="https://arxiv.org/abs/1706.05296"><em>“lazy agent”</em> problem</a>. What we experienced was that when training a team of two agents, one being stronger than the other, the weaker agent tended to just avoid any actions (it was just sitting duck). The reason was that if the agent started exploring, it would probably get blasted by its own actions, affecting the reward of the entire team.</li><li>Another example was the time limit for taking an action by the agents of <em>100 miliseconds</em>.</li></ul><blockquote><em>“If an agent does not respond in an appropriate time limit for our competition constraints (100ms), then we will automatically issue them the Stop action and, if appropriate, have them send out the message (0, 0). This timeout is an aspect of the competition and not native to the game itself.”</em></blockquote><p>Having multiple constraints is no different from “real-world” problems. You will never work on an “easy” problem. You will never start with perfect data. You won’t have enough time to try all the models you want. Your model/agent needs to run within certain limits (e.g., time or memory). Your training data is never complete. Inherently, you are always working with deadlines. Learning to navigate these limitations is crucial for a successful outcome.</p><h3>3. Focusing on Solving Problems</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/0*DQ3YrXl3xaQ425b5.jpeg" /></figure><p>NeurIPS is about <em>research papers</em>, but competitions try to foster new ideas based on solving problems. It’s a subtle difference, but one that’s closer to how industry works.</p><p>One should be careful not to lose focus on the main goal. I’ve seen two examples of how focus might dissipate:</p><ul><li><em>Getting Sidetracked by Too Many Experiments</em>. When you start working on a new problem, the focus is there, it’s clear. But after becoming comfortable with the problem, I tend to want more and more experiments. It’s essential to maintain focus and <em>avoid getting sidetracked</em> by too many experiments. Balance is key -push towards a complete solution without losing sight of the primary objective.</li><li><em>Losing Sight of the Main Metric</em>. he second problem appears during evaluations and the metrics that matter. For example, in the Pommerman case, we modified the reward function of the agents and had plots of how the reward improved during training. However, the metric that really mattered was the “win rate” (defeating the the other team). In industry, this is similar -you have your model metrics, but there’s also a <em>business metric</em>. One needs to be careful not to forget that the business metric is what really matters.</li></ul><h3>4. Understanding Trade-offs</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/0*aCObO8W0jx1bXFX2.png" /></figure><blockquote><em>“People who are bred, selected, and compensated to find complicated solutions do not have an incentive to implement simplified ones.” ― Nassim Taleb</em></blockquote><p>How do you approach a problem with many challenges and make progress in a short time? By understanding the trade-offs and making decisions.</p><ul><li><em>Sometimes simple (incorrect) models are enough.</em> One example was that the environment was partially observable, meaning you could only see a small window, not the entire board. It is clear that the information from the entire board is useful (because the enemies might be somewhere in the unseen regions). The problem was how to keep memory of seen regions in the past. One common approach, which is technically correct, is to add a recurrent network. However, that complicated the training. The second option, which was technically not entirely correct, was to <em>assume</em> that the last observation would remain until a new observation arrived. The benefit was not having to implement extra modules and just keep a simple vector of data. Results were good with the second, incorrect, approach.</li><li><em>Other times, complexity needs to be added to see real improvements.</em> The second example is related to the training curriculum. It was difficult to select against which agents we should train. On one side, we started using <em>static</em> agents to teach the agents that blasting them was the way to win. However, in the real competition, that wouldn’t have worked. It was too simple. This time we needed to add complexity. Chao came up with an <em>unbiased smart</em> agent. That made an incredible difference in performance.</li></ul><p>The lesson is to learn about trade-offs. Given the constraints, you can never solve everything perfectly. Finding the right amount of complexity will lead to the best results.</p><h3>5. Taking Ownership</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/0*AfwTNW7s1lpxwEK_.jpeg" /></figure><blockquote><em>“If all you need to do is what you are told, then you don’t need to understand your craft. However, as your ability to make decisions increases, then you need intimate technical knowledge on which to base those decisions.” ― L. David Marquet</em></blockquote><p>Every member of the team had <em>a different expertise</em>, but <em>we all knew the problem and the goal we were working towards</em>.</p><ul><li>We were independent in the sense that individually we had to think, implement, and validate our ideas in our specific area of expertise. We had <strong>ownership</strong>.</li><li>We worked as a team because we <em>cared</em> about the result and reaching the goal. We embraced shared learnings -both things that worked and things that did not- which helped to boost trust among the team.</li></ul><p>In the end, we took the lessons from everyone and put them together in the agent that was submitted to the competition.</p><h3>6. Building from Scratch</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/0*obdqzXIIKzY-LxUA.png" /></figure><blockquote><em>“Take a simple idea and take it seriously.” — Charlie Munger</em></blockquote><p>Working on the competition was a clear, simple idea. When we started, we only had the baseline that came with the simulator. As we experimented and learned, we documented ways to continue using the simulator beyond the competition. Post-NeurIPS, we had a robust codebase, knowledge, and new ideas, setting the stage for future projects.</p><p>The competition was the beginning, but later more than 5 research papers used the competition as a motivation and to perform experiments.</p><h3>7. Skin in the game</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*agILvquLkymmIYSz.png" /></figure><blockquote><em>“How much you truly ‘believe’ in something can be manifested only through what you are willing to risk for it.” Nassim Taleb</em></blockquote><p>Before OpenReview, you could submit a paper, and if it did not get accepted, you got feedback. It could be terrible work, but still, you got feedback and no one needed to know about that terrible work you submitted. There was no penalty for submitting something incomplete. There was no skin in the game.</p><p>Competitions are different — by definition, there are winners, losers, and prizes. There is more at stake. Having skin in the game makes things more interesting and helps you learn in a distinct way.</p><p>The <a href="https://neurips.cc/Conferences/2024/CompetitionTrack">NeurIPS 2024 competition track</a> is already out, I strongly suggest if you have time work on a submission, do it, you will learn a lot.</p><p><em>Originally published at </em><a href="https://pablohl.github.io/posts/pommermanlessons/"><em>https://pablohl.github.io</em></a><em> on June 26, 2024.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=abc5fd56714a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>