<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Brian Mutea on Medium]]></title>
        <description><![CDATA[Stories by Brian Mutea on Medium]]></description>
        <link>https://medium.com/@brianmuteak?source=rss-1e1c1cda0acd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*1LwmgV4a8T5NXh-cqvL_EA.jpeg</url>
            <title>Stories by Brian Mutea on Medium</title>
            <link>https://medium.com/@brianmuteak?source=rss-1e1c1cda0acd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 12:48:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@brianmuteak/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Extracting Unstructured Podcast Data: A Case Study with Andrew Huberman’s Podcast]]></title>
            <link>https://medium.com/@brianmuteak/extracting-unstructured-podcast-data-a-case-study-with-andrew-hubermans-podcast-3d115ccfb6dc?source=rss-1e1c1cda0acd------2</link>
            <guid isPermaLink="false">https://medium.com/p/3d115ccfb6dc</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[podcast]]></category>
            <category><![CDATA[indexify]]></category>
            <category><![CDATA[langchain]]></category>
            <dc:creator><![CDATA[Brian Mutea]]></dc:creator>
            <pubDate>Thu, 26 Feb 2026 11:27:06 GMT</pubDate>
            <atom:updated>2026-02-26T11:27:06.386Z</atom:updated>
            <content:encoded><![CDATA[<h4>Transforming long-form audio conversations into searchable, structured knowledge through automated indexing, and semantic extraction using Indexify.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3K2QPTHM-W5bJh38HotCWg.png" /><figcaption>Image by Author</figcaption></figure><p>Today’s world is data-driven. Over the past few decades, unstructured data has grown exponentially (to about <a href="https://researchworld.com/articles/possibilities-and-limitations-of-unstructured-data#:~:text=According%20to%20Gartner%2C%20unstructured%20data,of%20all%20new%20enterprise%20data.">90% of all data</a>), exposing organizations to numerous challenges in managing and extracting useful insights from this vast data.</p><p>Building applications that leverage large amounts of such data (PDFs, videos, and audio) also requires access to up-to-date information. Data sources may be updated frequently and continuously. These applications need an updated knowledge base to provide accurate analysis and responses, thus highlighting the need for real-time and continuous data ingestion and extraction tools.</p><p>The following guide will get you started with <a href="https://docs.getindexify.ai/#why-use-indexify"><strong>Indexify</strong></a>, a tool to help you build your own <strong>knowledge-rich LLM application</strong>. Indexify ensures <strong>continuous ingestion</strong> and <strong>extraction </strong>of information from the nuances of various unstructured data sources.</p><p>We will guide you through using Indexify’s:</p><ul><li><strong>Extractors</strong>. These transform unstructured data during extraction.</li><li><strong>Powerful data extraction pipelines</strong>. These are presented as <strong>Extraction Graphs</strong>.</li><li><strong>Retrieval APIs</strong>. Specifically, the Search API for searching embedding-based indexes emitted from the Extraction Graph.</li></ul><p>By the end, you should be able to ingest, extract, and retrieve information from multiple audio data sources, whether it be just a few audio files or several hundred (and more) to help build your own LLM application.</p><p>We will use data from three audio versions of the popular <a href="https://www.youtube.com/@hubermanlab">Andrew Huberman podcast</a>. The large files will show Indexify’s ability to process huge amounts of data effortlessly.</p><ul><li><strong>Podcast 1: </strong><a href="https://www.youtube.com/watch?v=p4ZfkezDTXQ&amp;pp=ygUtaG93IHRvIGVuaGFuY2UgZm9jdXMgYW5kIGltcHJvdmUgcHJvZHVjdGl2aXR5"><strong>Dr. Cal Newport: How to Enhance Focus and Improve Productivity</strong></a> (<a href="https://drive.google.com/file/d/1ATw6lqjSbipndFQo0xHPwPcMb-lXxPSO/view?usp=sharing">get audio here</a>).</li><li><strong>Podcast 2: </strong><a href="https://www.youtube.com/watch?v=ufsIA5NARIo&amp;pp=ygVJRHIuIFBldGVyIEF0dGlhOiBJbXByb3ZlIFZpdGFsaXR5LCBFbW90aW9uYWwgJiBQaHlzaWNhbCBIZWFsdGggJiBMaWZlc3Bhbg%3D%3D"><strong>Dr. Peter Attia: Improve Vitality, Emotional &amp; Physical Health &amp; Lifespan</strong></a> (<a href="https://drive.google.com/file/d/1ATw6lqjSbipndFQo0xHPwPcMb-lXxPSO/view?usp=sharing">get audio here</a>).</li><li><strong>Podcast 3: </strong><a href="https://www.youtube.com/watch?v=K-TW2Chpz4k&amp;pp=ygU_TGV2ZXJhZ2UgRG9wYW1pbmUgdG8gT3ZlcmNvbWUgUHJvY3Jhc3RpbmF0aW9uICYgT3B0aW1pemUgRWZmb3J0"><strong>Leverage Dopamine to Overcome Procrastination &amp; Optimize Effort</strong></a> (<a href="https://drive.google.com/file/d/1HnMjs78aP-_4xjZOAfTH9POyTDSEr5Lq/view?usp=sharing">get audio here</a>).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*TyPS-4VI1A0CghYXaq0zBg.png" /><figcaption>Andrew Huberman lab thumbnail: <a href="https://miro.medium.com/v2/resize:fit:1400/format:webp/1*vR66nwJyWZlVbnoAtIR34Q.png">Source</a></figcaption></figure><p>Andrew Huberman’s podcast , which discusses science and science-based tools for everyday life, is an international hit that has at some point been ranked third on global reach according to <a href="https://chartable.com/podcasts/huberman-lab">Chartable</a>. It is currently regularly ranked as the #1 health podcast globally. The podcast has been listened to more than 140 million times and has a monthly newsletter of 100,000+ subscribers.</p><h3>Brief Overview of Unstructured Data</h3><p>Structured data is easy to analyze, but it only offers a limited view in today’s world of massive, diverse data. Here’s why unstructured data is important for building LLM applications:</p><ul><li>Unstructured data, though messy, provides the key to understanding your customers and competitors. It reveals their preferences, goals, and successes, all crucial for asking the right questions and making informed decisions.</li><li>Structured data might tell you customer demographics and sales figures, but unstructured data shows what they say on social media, respond to emails, and think of your brand. That leads to a more complete understanding of your customer experience.</li></ul><p>While valuable, unstructured data isn’t organized and comes in various forms. Processing it requires advanced tools like Indexify. That is also because:</p><ul><li>Businesses generate massive amounts of unstructured data daily, making processing a challenge.</li><li>Unstructured data can include sarcasm, idioms, and specific language that machines might misinterpret.</li><li>Unstructured data can be inconsistent and contain errors, leading to misleading insights, cleaning and maintaining data quality is resource-intensive.</li></ul><h3>Getting Started</h3><p>Indexify has numerous pieces that create its workflow to allow its users and developers to upload millions of <strong>unstructured data</strong>.</p><p>Indexify has two major components:</p><ul><li><strong>Indexify server</strong>: Comprises the ingestion system and the task scheduler for running extraction.</li><li><a href="https://docs.getindexify.ai/apis/extractors/"><strong>Indexify extractors</strong></a>: Compute functions that run models that transform or extract structured data or embed it from unstructured data.</li></ul><p>Indexify can run locally with all components running on your local machine.</p><p>You will need three terminals to follow along:</p><ul><li>The first terminal will download and run the Indexify server</li><li>The second terminal will run the Indexify extractors, which will handle the chunking and embedding of data from the audio file.</li><li>The third terminal is to run our Python scripts to help load and query data from our Indexify server.</li></ul><p>In this section, we will set up Indexify to extract data from an audio file.</p><h4>First terminal: Starting the Indexify server</h4><p><strong>Create a virtual environment:</strong></p><p>python -m venv ivenv</p><p><strong>Activate the environment:</strong></p><p>source ivenv/bin/activate</p><p><strong>Install the </strong><strong>indexify-extractor-sdk and</strong><strong>indexify:</strong></p><p>The extractor SDK allows adding new extraction capabilities and other readily available extractors for audio, PDFs, videos, etc.</p><pre>pip install indexify indexify-extractor-sdk</pre><p><strong>Download and run the Indexify server:</strong></p><p>The Indexify server has the following functions:</p><ul><li>It evaluates the extraction policies and allocates tasks to extractors using a <strong>coordinator</strong>. This fast task scheduler creates thousands of tasks every second whenever data is ingested.</li><li>Exposes HTTP and retrieval APIs via an <strong>ingestion server</strong>, allowing clients to upload content into Indexify, retrieve extracted data, or search over vector indexes.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*n4tCxg3b3Zbqpe8SoJ9Bzw.png" /><figcaption>Image by author</figcaption></figure><p>Run:</p><p>curl <a href="https://getindexify.ai/">https://getindexify.ai</a> | sh</p><p>Once the server is downloaded, run the server. It is a long-running process:</p><p>./indexify server -d</p><h4>Second terminal: Extractors</h4><p>Indexify extractors consume content containing raw bytes of unstructured data and produce a list of content and features from it.</p><p>Indexify <strong>stores features like the embeddings into indexes</strong> for future retrieval.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*SXfR50CQ3QodIB_s.png" /><figcaption>Image by author</figcaption></figure><p>We will need three extractors:</p><ul><li>Audio extractor: tensorlake/whisper-asr. It is compatible with various operating systems and can handle any audio quality. <a href="https://docs.getindexify.ai/usecases/audio_extraction/#the-extraction-pipeline-a-three-stage-journey:~:text=throw%20at%20them!%20%F0%9F%9A%80-,%F0%9F%8C%88%20Explore%20the%20Audio%20Extractor%20Landscape,-We%20offer%20a">View other available audio extractors</a>.</li><li>Chunking extractor: tensorlake/chunk-extractor.</li><li>Embedding extractor: tensorlake/minilm-l6.</li></ul><p>Download each extractor by running the following:</p><pre>indexify-extractor download tensorlake/minilm-l6<br>indexify-extractor download tensorlake/chunking<br>indexify-extractor download tensorlake/whisper-asr</pre><p>Once they are downloaded, run them on the server. This is a long-running process:</p><pre>indexify-extractor join-server</pre><p><em>You can access the server UI at </em><a href="http://localhost:8900/ui"><em>http://localhost:8900/ui</em></a><em>.</em></p><p>Visual representation of the extractors on the UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*xELDGi1rBe2TaKf5GSpVJg.png" /></figure><h3>Defining the data pipeline and loading the audio data into Indexify</h3><p>With the extractors and Indexify server running, create a file that will:</p><ul><li>Define the data extraction pipeline.</li><li>Load the data from the audio file through the pipeline.</li></ul><p>We can call this setup.py file and add the following code:</p><pre>from indexify import IndexifyClient, ExtractionGraph<br><br>#initialize the Indexify client<br>client = IndexifyClient()<br><br># Define the data pipeline<br>extraction_graph_spec = &quot;&quot;&quot;<br>name: ‘podcast2knowledgebase’<br>extraction_policies:<br>   - extractor: &#39;tensorlake/whisper-asr&#39;<br>   name: &#39;whisperaudioextractor&#39;<br>   - extractor: &#39;tensorlake/chunk-extractor&#39;<br>   name: &#39;chunker&#39;<br>   content_source: &#39;whisperaudioextractor&#39;<br>   input_params:<br>       chunk_size: 512<br>       overlap: 150<br>   - extractor: &#39;tensorlake/minilm-l6&#39;<br>   name: &#39;minilmembedding&#39;<br>   content_source: &#39;chunker&#39;<br>&quot;&quot;&quot;<br><br><br># Create the extraction graph<br>extraction_graph = ExtractionGraph.from_yaml(extraction_graph_spec)<br>client.create_extraction_graph(extraction_graph)<br><br># upload the file to Indexify<br>content_id = client.upload_file(&quot;podcast2knowledgebase&quot;,             path=&quot;Dr_Cal_Newport_How_to_Enhance_Focus_and_Improve_Productivity.mp3&quot;)<br>print(content_id)</pre><p><strong>Explaining the code:</strong></p><ul><li>First, we <strong>import IndexifyClient and ExtractionGraph</strong>. The extraction graph enables us to create a real-time data pipeline to extract data from the audio file, chunk it, and embed it.</li><li>Next, we<strong> initialize the Indexify client and define an extraction graph</strong>. Inside the graph we define the extraction policies. An extraction policy is a rule that instructs Indexify to run a particular extractor.</li><li>We then <strong>chain multiple extraction policies</strong> within an extraction graph.</li><li>Finally, we <strong>upload the contents of the audio file</strong> to Indexify using the created data pipeline.</li></ul><p>With all this set, we can run the script on the third terminal:</p><pre>python3 setup.py</pre><p>Before extraction is over, the value on the “Children” column will be 0:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*WT2nWGZ_UWs5iZ4zLvcPkw.png" /></figure><p>After the extraction, the value will be greater than 0:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*SclTzWbXN5Hv1KX2u5C7wg.png" /></figure><p>To get the extracted content, we can create a different script, getextract_data.py, and add the following code:</p><pre>from indexify import IndexifyClient<br><br>client = IndexifyClient()<br>extracted_content = client.get_extracted_content(content_id=&#39;9ea61b4cafcc5f2a&#39;)<br>print(extracted_content)</pre><p><strong>Output:</strong></p><pre>[{&#39;id&#39;: &#39;f1b8ef527bbf0f20&#39;, &#39;parent_id&#39;: &#39;9ea61b4cafcc5f2a&#39;, &#39;root_content_id&#39;: &#39;9ea61b4cafcc5f2a&#39;, &#39;namespace&#39;: &#39;default&#39;, &#39;name&#39;: &#39;T3SHyD7fFyQMoUS_zwfCh&#39;, &#39;mime_type&#39;: &#39;text/plain&#39;, &#39;labels&#39;: {}, &#39;extraction_graph_names&#39;:<br> [&#39;podcast2knowledgebase&#39;], &#39;storage_url&#39;: &#39;file:///tmp/indexify/blob_storage/T3SHyD7fFyQMoUS_zwfCh&#39;, &#39;created_at&#39;: 1716389702, &#39;source&#39;: &#39;whisperaudioextractor&#39;, &#39;size&#39;: 201385, &#39;hash&#39;: &#39;162c7b4e88aeab0459514529ebae1ffd5d73075b853c1a7ad85f1f95905db020&#39;, &#39;content_url&#39;:<br> &#39;http://localhost:8900/namespaces/default/content/f1b8ef527bbf0f20/download&#39;}<br>]</pre><p>You can click the provided link to view the extracted data.</p><p>Visualization on the Indexify UI:</p><ul><li>Extraction graph with extraction policies used during extraction.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*tR7H4ivi5l75pm49o-xDEw.png" /></figure><ul><li>Indexes.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*auZc3Xv9K671n5CLSzUDHQ.png" /></figure><h3>Retrieving Specific Information from the Podcast</h3><p>Indexify creates vector indexes from the embedding extractor model defined in the extraction graph. We use these indexes to perform semantic searches.</p><p>The search results contain chunks of text that match the query and their corresponding scores.</p><p><strong>Indexify’s retrieval API helps us query the indexes</strong>.</p><p>client.search_index(query=index, query=’question’, top_k=top_k)</p><p>Create an app.py file and add the following code:</p><pre>from indexify import IndexifyClient<br><br># initialize Indexify client<br>client = IndexifyClient()<br><br>query = “who is Dr. Cal Newport”<br>index = &quot;podcast2knowledgebase.minilmembedding.embedding&quot;<br>retrieved_results = client.search_index(name = index,<br>                                        query = query,<br>                                        top_k = 3)                   <br>print(&quot;\n-&quot;.join([item[&quot;text&quot;] for item in retrieved_results]))</pre><p>Run python app.py:</p><p><strong>Output:</strong></p><pre>Welcome to the Huberman Lab Podcast, where we discuss science and science-based tools for everyday life. <br>I&#39;m Andrew Huberman, and I&#39;m a professor of neurobiology and ophthalmology at Stanford School of Medicine. My guest today is Dr. Cal Newport. <br>Dr. Cal Newport is a professor of computer science at Georgetown University. He did his training at MIT and he is currently both a professor and <br>the author of many best-selling books focused on productivity, focus, and how to access the specific states of mind to…</pre><p>We can also query the indexes from the UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*LwzrpyF3AYmhV_ryTgulMg.gif" /></figure><h3>Scaling Indexify for Production</h3><p>Previously, we introduced a basic single-source workflow for Indexify. However, Indexify is so much more than that. It comes into its own when dealing with multiple sources of data and different modalities. Indexify is designed to operate for production at scale.</p><p>So this time, our goal is to index <strong>several of Andrew Huberman’s podcast episodes</strong>. Andrew has already hosted many guests and discussed numerous topics around health and productivit, accumulating thousands of hours of content.</p><p>The podcast is also frequently updated with new episodes. We would want to extract information from them continuously. Eventually, we want to use this information to build an application to help retrieve important insights from these episodes.</p><p><strong>Indexify provides a declarative configuration approach. It defines extraction pipelines called extraction graphs that can also scale out to production, where vast amounts of data would be ingested.</strong> The extraction graph state is replicated across multiple machines to recover from hardware failures and server crashes. Additionally, you get fully observable latencies and throughput for data extraction.</p><p>Indexify also makes building such applications easier by:</p><ul><li>Running extraction workflows asynchronously using a built-in real-time batch processing engine.</li><li>Enabling infinite ingestion scalability by leveraging blob stores and having dedicated processes to handle I/O.</li><li>Assuring retrieval happens in dedicated processes, reads directly from storage, and is never blocked by extraction workloads.</li><li>Providing plug-and-play functionality with various models for handling any data type or extraction in an extraction graph (data pipeline).</li></ul><p>To demonstrate this, we will create the following:</p><ul><li>data/ folder to hold audio or other files, depending on your use case.</li><li>graph_yaml_files/ folder to hold .yaml files. Ours will contain an audio_pipeline.yaml with a specific extraction pipeline for audio.</li><li>setup.py file to write our functionality.</li><li>processed_files.log file. Logs the already processed files so we do not have to process one file twice. If the file has been renamed, we can assume it’s an update.</li></ul><pre>./<br>│<br>├── data/<br>│    ├── dr_cal_newport_how_to_enhance_focus_<br>          and_improve_productivity.mp3<br>│    ├── dr_peter_attia_improve_vitality_emotional_<br>          physical_health_lifespan_huberman_lab_podcast.mp3<br>│    ├── leverage_dopamine_to_overcome_procrastination_<br>          optimize_effort_huberman_lab_podcast.mp3<br>│    └── ...+1000s of audio files<br>│<br>├── ivenv/<br>│   ├── ...<br>│   └── ...<br>│<br>├── processed_files.log<br>│<br>├── setup.py<br>│<br>└── graph_yaml_files/<br>     ├── audio_pipeline.yaml<br>     └── ...</pre><h4><strong>Declare an ingestion pipeline (Extraction Graph):</strong></h4><p>Create an audio_pipeline.yaml file and define the pipeline that has the necessary extraction policies. We will only set the pipeline once, and Indexify will continuously wait for new audio files and extract information.</p><pre>name: &quot;podcast2knowledgebase&quot;<br>extraction_policies:<br> - extractor: &quot;tensorlake/whisper-asr&quot;<br>   name: &quot;whisperaudioextractor&quot;<br><br> - extractor: &quot;tensorlake/chunk-extractor&quot;<br>   name: &quot;chunker&quot;<br>   input_params:<br>     chunk_size: 512<br>     overlap: 150<br>   content_source: &quot;whisperaudioextractor&quot;<br><br> - extractor: &quot;tensorlake/minilm-l6&quot;<br>   name: &quot;embedding&quot;<br>   content_source: &quot;chunker&quot;</pre><h4><strong>Create a function to read the ingestion pipeline:</strong></h4><p>The function reads its content and returns it. The returned output will then be used to create an extraction graph.</p><pre>def read_yaml_file(yaml_file_path):<br>   &quot;&quot;&quot;Reads and returns the content of a YAML file.&quot;&quot;&quot;<br>   with open(yaml_file_path, &#39;r&#39;) as file:<br>       return file.read()</pre><h4><strong>Create a function to set up the ingestion pipeline:</strong></h4><p>The function will create an extraction graph from the ingestion pipeline and register it to the Indexify client for continuous extraction.</p><pre>def create_extraction_graph(yaml_content):<br>   &quot;&quot;&quot;Creates an extraction graph from YAML content.&quot;&quot;&quot;<br>   extraction_graph = ExtractionGraph.from_yaml(yaml_content)<br>   client.create_extraction_graph(extraction_graph)</pre><h4><strong>Create a function to read already processed files and ensure we do not process a file twice:</strong></h4><p>We will assume that any update to the same file will be slightly renamed, which can then be processed. Depending on the source of these files, that could change.</p><p>Function to read the files:</p><pre>def read_processed_files(log_file_path):<br>   &quot;&quot;&quot;Reads the log file and returns a set of processed file names.&quot;&quot;&quot;<br>   if not os.path.exists(log_file_path):<br>       return set()<br>   with open(log_file_path, &#39;r&#39;) as log_file:<br>       return set(line.strip() for line in log_file)</pre><p>Function to log the files to processed_files.log:</p><pre>def log_processed_file(log_file_path, file_name):<br>   &quot;&quot;&quot;Appends a processed file name to the log file.<br>   Will help make sure Indexify does not process same file twice<br>   If it is renamed, we can assume it’s an update.<br>   &quot;&quot;&quot;<br>   with open(log_file_path, &#39;a&#39;) as log_file:<br>       log_file.write(file_name + &#39;\n&#39;)</pre><h4><strong>Create a function to read new audio files and exclude already processed files:</strong></h4><pre>def get_new_audio_files_to_upload(files_directory_path, processed_files):<br>   &quot;&quot;&quot;Returns a list of MP3 files in the directory that haven&#39;t been processed yet.<br>   All .mp3 files should be processed by the corresponding .yaml file for audio.<br>   &quot;&quot;&quot;<br>   all_files = os.listdir(files_directory_path)<br>   return [f for f in all_files if f.endswith(&#39;.mp3&#39;) and f not in <br>          processed_files]</pre><h4><strong>Finally create a function that orchestrates the entire process:</strong></h4><p>The function:</p><ul><li>Reads and parses the YAML file to get the extraction graph content.</li><li>Creates the extraction graph using the YAML content.</li><li>Reads the log of processed files.</li><li>Gets the list of audio files to upload by filtering out files already processed.</li><li>Iterates over the list of audio files to upload. For each file:<br>- Constructs the full file path.<br>- Checks if the file exists.<br>- If the file exists, upload it using the Indexify client and print a success message.<br>- Logs the file as processed by writing its name to the log file.<br>- If the file does not exist, it prints an error message.</li></ul><pre>def upload_files(yaml_file_path, files_directory_path, log_file_path):<br>   &quot;&quot;&quot;Uploads MP3 files using the specified YAML file and logs the processed files.&quot;&quot;&quot;<br>   # Read and parse the YAML file<br>   yaml_content = read_yaml_file(yaml_file_path)<br><br>   # Create the extraction graph<br>   create_extraction_graph(yaml_content)<br><br>   # Read the list of processed files from the log<br>   processed_files = read_processed_files(log_file_path)<br><br>   # Get the list of MP3 files to upload<br>   mp3_files_to_upload = get_new_audio_files_to_upload(<br>       files_directory_path, processed_files)<br><br>   # Upload each MP3 file in the filtered list<br>   for file_name in mp3_files_to_upload:<br>       file_path = os.path.join(files_directory_path, file_name)<br>       if os.path.exists(file_path):<br>           content_id = client.upload_file(&quot;podcast2knowledgebase&quot;,<br>                                            path=file_path)<br>           print(<br>               f&quot;Successfully uploaded {file_path} with content ID: {content_id}&quot;)<br>           # Log the file as processed<br>           log_processed_file(log_file_path, file_name)<br>       else:<br>           print(f&quot;Error: File {file_path} does not exist.&quot;)</pre><h4><strong>Run the functionality to begin extraction:</strong></h4><pre>if __name__ == &quot;__main__&quot;:<br><br>   # Paths to the necessary directories and files<br>   yaml_file_path = &#39;/path/to/your/audio_pipeline.yaml&#39;<br>   files_directory_path = &#39;/path/to/your/data_directory&#39;<br>   log_file_path = &#39;processed_files.log&#39;<br><br>   # Run the extraction process<br>   upload_files(yaml_file_path, files_directory_path, log_file_path)</pre><p>Run setup.py. <strong>Output:</strong></p><pre>Successfully uploaded /PROJECTS/IndexifyStart/data/leverage_dopamine_to_overcome_procrastination_optimize_effort_huberman_lab_podcast.mp3 with content ID: 2034f25d3b466b7a<br><br>Successfully uploaded /PROJECTS/IndexifyStart/data/dr_peter_attia_improve_vitality_emotional_physical_health_lifespan_huberman_lab_podcast.mp3 with content ID: 905f26ee5215344a<br><br>Successfully uploaded /PROJECTS/IndexifyStart/data/dr_cal_newport_how_to_enhance_focus_and_improve_productivity.mp3 with content ID: 53996e15299cf74c<br><br>...+1000s of data to upload</pre><h4><strong>Visualization on Indexify Server:</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*XpIKcbi8qmX0uHGMsmdbIA.png" /></figure><p>Indexify will wait for new audio data and continue to extract. For instance, we can see that Indexify has only processed three tasks reflecting our three audio files. That value will keep updating depending on how many files Indesify is processing.</p><p><strong>Sample content from extraction</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*6hOqPF3ri-nHHemCtwuSEQ.gif" /></figure><p>From the extraction graph we created, Indexify produces embedding at the last stage of the audio extraction. You can search the indexes of the uploaded files using KNN search to get relevant chunks of the extraction. We can use the retrieval API (client.search_index(query=index, query=’question’, top_k=top_k)) to retrieve specific indexes from the extracted data.</p><p>We can use the indexes to create production-ready LLM applications by integrating Indexify with third party tools like LangChain.</p><h3>Final Thoughts</h3><p>In this article, we have:</p><ul><li>Discussed the importance and challenges of working with unstructured data.</li><li>Introduced Indexify and learned how it is a handy tool for extracting data from unstructured sources.</li><li>Learned functional components provided by Indexify, such as extractors and extraction graphs, and how to create a data pipeline.</li><li>Gone through a step-by-step guide on how to use Indexify to extract data from multiple audio files, such as podcasts.</li><li>Explored one of the most crucial components of Indexify, the UI.</li><li>Seen how to use the Indexify retrieval API for semantic index searches.</li></ul><p>By providing a declarative configuration approach to defining extraction pipelines, Indexify ensures scalability even in production. Once the extraction pipeline is set, Indexify waits to continuously extract new data.</p><blockquote>Originally written as part of T<a href="https://tensorlake.ai/">ensorLake’s</a> campaign in collaboration with <a href="https://www.linkedin.com/company/argon-labs-xyz/posts/?feedView=all">Argon Labs</a>. The tooling has evolved since this was published, so be sure to check for the latest updates</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d115ccfb6dc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building LLM applications with Indexify and Langchain]]></title>
            <link>https://medium.com/@brianmuteak/building-llm-applications-with-indexify-and-langchain-452e161be4d3?source=rss-1e1c1cda0acd------2</link>
            <guid isPermaLink="false">https://medium.com/p/452e161be4d3</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[indexify]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Brian Mutea]]></dc:creator>
            <pubDate>Thu, 26 Feb 2026 11:26:37 GMT</pubDate>
            <atom:updated>2026-02-26T11:26:37.726Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Originally written as part of T<a href="https://tensorlake.ai/">ensorLake’s</a> campaign in collaboration with <a href="https://www.linkedin.com/company/argon-labs-xyz/posts/?feedView=all">Argon Labs</a>. The tooling has evolved since this was published, so be sure to check for the latest updates.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/929/1*OTB9NxeiC5zGNT2CsviPIA.png" /><figcaption>Image by author</figcaption></figure><p>Large Language Models (LLMs) such as OpenAI’s ChatGPT are everywhere these days. You’ve likely seen their impressive ability to answer questions, summarize documents, write code, and more, often without realizing it. The demand for custom LLM applications is rapidly growing in today’s digital world, enabling the creation of advanced chatbots, automated content generation, and enhanced data analysis.</p><p>This control over your applications and day-to-day tasks has been possible since LLMs materialized with the release of OpenAI’s<a href="https://github.com/openai/gpt-3"> GPT-3 in 2020</a>. But LLMs face a significant challenge: <strong>data freshness</strong>. They lack knowledge of recent events and specific organizational data.</p><p>LLMs are limited to the information they were trained on, which creates issues for applications needing up-to-date information or specific datasets, such as internal company documents or social media data. Adding these documents to an LLM is challenging. Training an LLM on new documents is time-consuming and costly, and constantly retraining for new documents is inefficient.</p><p>To solve this, we use methods to retrieve relevant information from external knowledge bases and provide it to the LLM.</p><p><a href="https://www.getindexify.io/">Indexify</a> creates a window to external and consumer-specific unstructured data sources beyond the LLM’s training data. It uses fault-tolerant data pipelines to continuously, in near real-tim, ingest new data, extract structured information, and embed and write it into storage.</p><p>Integrating Indexify with third-party platforms and frameworks further enhances its capabilities. This article will take you through integrating Indexify with LangChain, showcasing how Agents can transform your LLM application projects.</p><h3>What is LangChain?</h3><p>LangChain is a framework designed around Large Language Models (LLMs). It can be used to develop various LLM-powered applications, such as:</p><ul><li>Document analysis and summarization</li><li>Chatbots for natural interactions</li><li>Code analysis to find bugs or security flaws</li><li>Answering questions using text, PDFs, videos, and audio</li><li>Data augmentation by generating new data</li><li>Text classification and sentiment analysis</li><li>Text summarization</li><li>Machine translation</li></ul><p>The framework allows chaining components like prompt templates, LLMs, agents, and memory to create more refined use cases.</p><p>To begin using Indexify, simply follow this workflow:</p><ul><li><strong>Create</strong> a virtual environment.</li><li><strong>Install</strong> Indexify and the Indexify extractor SDK.</li><li><strong>Download</strong> the Indexify server.</li><li><strong>Run </strong>the server and extractors.</li><li><strong>Extract</strong> data.</li></ul><p>We will go through these steps while integrating with LangChain.</p><h3>Getting Started</h3><p>To integrate Indexify with LangChain, we will create some directories and files.</p><ul><li><strong>venv/</strong>: This will contain all the dependencies and packages we need to install for the project.</li><li><strong>data</strong>/: It will contain the file that holds the data we want to extract with Indexify and ask questions based on it.</li><li><strong>.env</strong>: This file contains all the environment variables. It will contain the OpenAI API key required for LangChain to work.</li><li><strong>setup.py</strong>: This is where we will set up Indexify for download, ingestion, and data extraction.</li><li><strong>app.py</strong>: We will create requests from the Indexify server and use the results for question-answering with an LLM via LangChain.</li></ul><p>The project folder could look like this:</p><pre>./<br>│<br>├── data/<br>│   └── sample_file.pdf<br>├── venv/<br>├── app.py<br>├── setup.py<br>└── .env</pre><h3>Setting up the environment:</h3><p>First, we need to create and activate a virtual environment and define the environment variables.</p><p><strong>Create a virtual environment:</strong></p><p>python -m venv venv</p><p><strong>Activate the virtual environment:</strong></p><p>source venv/bin/activate</p><p>Using pip, install the following:</p><pre>pip install indexify-extractor-sdk indexify indexify_langchain <br>langchain==0.2.5 langchain-community langchain_core langchain_openai openai python_dotenv</pre><p><strong>Define the environment variables:</strong></p><p>Using LangChain often requires integrations with various model providers like OpenAI, Hugging Face, Cohere, Anthropic, etc. Its core component is an LLM. As with any integration, we must provide appropriate API keys for LangChain to function. This tutorial needs an<a href="https://platform.openai.com/api-keys"> OpenAI API key</a> to help us interact with its LLMs.</p><p>Python-dotenv loads environment variables from the .env file into your Python environment. Because of that, we can store the OpenAI API key in the .env file and automatically load it into our Python script.<a href="https://platform.openai.com/api-keys"> Create your API key</a> and add it to the file.</p><pre>OPENAI_API_KEY=”sk-…”</pre><h3>Running the Indexify server and extractors</h3><p>To start using Indexify, we need its server. The server has crucial roles:</p><ul><li>It evaluates the extraction policies for extracting data from a certain source and allocates tasks to the defined extractors.</li><li>It exposes HTTP and retrieval APIs, which allow clients to upload content into Indexify, retrieve extracted data, or search over indexes.</li></ul><p>First, we need to download it:</p><p>curl <a href="https://getindexify.ai/">https://getindexify.ai</a> | sh</p><p>Then, run it as a long-running process (in another terminal window)..</p><p>./indexify server -d</p><p>In this tutorial, we will attempt to extract data from a PDF file. Download the following <a href="https://docs.getindexify.ai/apis/extractors/">extractors</a> from Indexify.</p><pre>indexify-extractor download hub://embedding/minilm-l6<br>indexify-extractor download hub://text/chunking<br>indexify-extractor download hub://pdf/pdfextractor</pre><p>After downloading, run them:</p><p>indexify-extractor join-server</p><p><em>You can access the server UI at </em><a href="http://localhost:8900/ui"><em>http://localhost:8900/ui</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wuZV16fkExTuSNXewQqKOQ.png" /></figure><h3>Loading and extracting data with Indexify</h3><p>Loading and extracting data to Indexify is straightforward. Navigate to the setup.py file and add the following code:</p><p>We will <strong>augment</strong> the previous article (<a href="https://docs.google.com/document/d/1c6M4zQaom_6DMzBg2Yb31BsaJZe8RKsHz8fvDhVNLB8/edit#">How we indexed Andrew Huberman’s podcast with Indexify</a>) by extracting data from a <a href="https://drive.google.com/file/d/1azq-nWlZMHTpJa_5GBiADCbiKfOtcbl-/view?usp=sharing">PDF transcript of the podcast</a>.</p><pre>from indexify import IndexifyClient, ExtractionGraph<br><br>client = IndexifyClient()<br><br># Define the data pipeline<br>extraction_graph_spec = &quot;&quot;&quot;<br>name: &quot;pdf1knowledgebase&quot;<br>extraction_policies:<br>- extractor: &quot;tensorlake/pdf-extractor&quot;<br>  name: &quot;pdf_extractor&quot;<br><br>- extractor: &quot;tensorlake/chunk-extractor&quot;<br>  name: &quot;chunker&quot;<br>  input_params:<br>    chunk_size: 1000<br>    overlap: 250<br>  content_source: &quot;pdf_extractor&quot;<br><br>- extractor: &quot;tensorlake/minilm-l6&quot;<br>  name: &quot;embedding&quot;<br>  content_source: &quot;chunker&quot;<br>&quot;&quot;&quot;<br>extraction_graph = ExtractionGraph.from_yaml(<br>                                            extraction_graph_spec<br>                                            )<br>client.create_extraction_graph(extraction_graph)<br>client.upload_file(“pdfknowledgebase”,  <br>                   path=”path/to/your/file.pdf”)</pre><p>We first import the IndexifyClient and ExtractionGraph from Indexify. We instantiate the client object and create an extraction graph specification.</p><p>The specification specifically contains the extraction policies. An extraction policy is a rule that instructs Indexify to run a particular extractor. For instance, we have specified three extractors: one to <strong>extract text from the PDF</strong>, the next to <strong>split the extracted text into chunks</strong>, and the final one to<strong> create embeddings </strong>of those chunks to preserve their semantic meaning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QTl1kT4ky1tYA1Ae9XpXQQ.png" /></figure><p>The file is uploaded to Indexify through the extraction policies. Indexify stores the indexes generated in blob stores, which we will retrieve later to ask questions using LangChain based on the PDF.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*0B1ZLQXv_ixE-azSfffQjg.gif" /></figure><h3>Integrating Indexify with LangChain for Question-Answering</h3><p>As you may recall, LLMs depend on data to keep up with recent activities or provide answers based on a user’s data, which they need to be trained on.</p><p>A language model requires a knowledge base to generate responses correctly and accurately as requested. We can use <em>retrieval augmentation</em> to retrieve relevant information from custom data.</p><p>An essential part of loading this data is structuring it to get precise information from the language model. For instance, we have a PDF file with hundreds of pages. We can not directly pass the entire PDF to the language model since they have limitations on how much data they can receive at a time (context window). As such, we also need to give the model the most relevant parts of the document about the query to improve the precision and speed of retrieving answers.</p><p>We can use LangChain to structure and store the most relevant parts of the data. A typical workflow with LangChain would be:</p><ul><li><strong>Load</strong> the PDF with Document loaders from LangChain.</li><li><strong>Split</strong> the files into smaller chunks (chunking).</li><li><strong>Convert</strong> them into a vector representation (embeddings), which captures the semantic meaning of each word, thus grouping similar inputs close together.</li><li><strong>Store</strong> the vector representations of the chunks vector stores like FAISS and Pinecone for semantics retrieval.</li></ul><p>We need to import all these components for each task with LangChain. As you can appreciate, Indexify abstracts all these steps by providing a simple <strong>Extraction Graph</strong>, which provides a seamless data pipeline to perform all these tasks without scratching your head.</p><p>You can also appreciate that Indexify stores the extracted content in blob stores, which are great for handling unstructured data from various sources.</p><p>After Indexify handles all that for you, we can integrate LangChain to help us retrieve relevant documents, which we can then pass to an LLM to get a response based on a user’s question.</p><p>Navigate to the app.py file and add the following imports:</p><pre>from dotenv import load_dotenv<br>from indexify import IndexifyClient<br>from indexify_langchain import IndexifyRetriever<br><br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_community.chat_models import ChatOpenAI<br>from langchain.chains import create_retrieval_chain<br>from langchain_openai import ChatOpenAI<br>from langchain.chains.combine_documents import create_stuff_documents_chain<br><br>load_dotenv() # read environment variables(openai api key)<br><br>client = IndexifyClient() # instantiate Indexify client</pre><p>To work with LangChain, we need:</p><ul><li><strong>Context.</strong></li><li><strong>Prompt.</strong></li><li><strong>Retriever.</strong></li><li><strong>LLM.</strong></li><li><strong>Retrieval QA chain.</strong></li></ul><p><strong>Create a function to retrieve context based on user query:</strong></p><p>LLMs generate more reasonable responses when they understand the current task based on what the user asks them. Context acts as an additional source of knowledge for the model. We can pass it manually to a prompt, retrieve it from a vector database, or pull it in via other means (APIs, etc).</p><p>Since Indexify created indexes from extracted data, we can use its retrieval API to retrieve context based on the user’s question. The API returns top_k relevant chunks of text that match the user’s question and index. We then combine all the chunks to create the context.</p><p><strong>Create prompt:</strong></p><p>In this era of LLMs, prompts are king. Bad prompts deliver wrong outputs, and good prompts are exceptionally powerful. Constructing good prompts is crucial when building LLM applications using LangChain.</p><p>A prompt is composed of multiple parts:</p><ul><li><strong>Instructions</strong>: These tell the model what to do, how to use the information provided, what to do with the query, and how to construct the output.</li><li><strong>Context</strong>: The additional retrieved source of knowledge for the LLM.</li><li><strong>User query</strong>: Commonly (but not always) a query input into the system by a human user (the <em>prompter</em>)</li></ul><p>LangChain has an entire set of objects for creating prompts. Here, we will use PromptTemplates.</p><pre>def create_prompt(context, input):<br>   custom_prompt_template = &quot;&quot;&quot;<br>   ### System:<br>   Use the given context to answer the question. Use only the given <br>   information to answer.<br>   If you do not know the answer, say you do not know the answer!:<br>   Context:{context}<br>   Question: {input}<br>   ### Answer:<br>   &quot;&quot;&quot;<br>   prompt = PromptTemplate(template=custom_prompt_template, <br>                           input_variables=[&#39;context&#39;, &#39;input&#39;])<br>   return prompt</pre><p><strong>Question Answering:</strong></p><p>Finally, we need to use LangChain to generate the response using an LLM.</p><pre>if __name__ == &quot;__main__&quot;:<br>   <br>   pdf_index = &quot;pdfknowledgebase.embedding.embedding&quot;<br>   question = &quot;Give a list of some of Dr. Cal Newport&#39;s books&quot;<br><br>   # get context based on user question<br>   context = get_context(question, pdf_index)<br>   #create a prompt<br>   prompt = create_prompt(context, question)<br>   <br>   # initialize Indexify retriever<br>   params = {&quot;name&quot;: pdf_index}<br>   retriever = IndexifyRetriever(client=client, params=params)<br><br>   llm = ChatOpenAI(model=&#39;gpt-3.5-turbo&#39;, <br>                    max_tokens=200)<br>   <br>   # start question-answering retrieval chain<br>   qa_chain = create_stuff_documents_chain(llm, prompt)<br>   chain = create_retrieval_chain(retriever, qa_chain)<br><br>   # Response<br>   response = chain.invoke({&quot;input&quot;: question})<br>   print(f&quot;Answer:\n{response[&#39;answer&#39;]}&quot;)</pre><p>The entire process involves:</p><ul><li><strong>Defining the question</strong>: The input query for which we want an answer.</li><li><strong>Getting context</strong>: Retrieving relevant information based on the question to provide additional context for generating the answer.</li><li><strong>Creating a prompt</strong>: Formulate a structured prompt that includes the context and the question.</li><li><strong>Initializing the retriever</strong>: Setting up the <strong>IndexifyRetriever</strong> to fetch relevant documents from an index.</li><li><strong>Initializing the LLM</strong>: Setting up the language model to generate responses. We have used gpt-3.5-turbo, but you can use any other model.</li><li><strong>Creating the QA chain</strong>: Combining the LLM and the prompt into a chain that can generate answers.</li><li><strong>Creating the retrieval chain</strong>: Combining the retriever and the QA chain into a workflow that retrieves relevant documents and generates an answer.</li><li><strong>Generating the response</strong>: Running the entire workflow to get the answer to the input question and printing it.</li></ul><p>Run app.py. <strong>Output:</strong></p><pre>(venv) tl@tensorlake:~/Documents/PROJECTS/IndexifyStart$ python app.py <br>Answer:<br>- Deep Work: Rules for Focused Success in a Distracted World<br>- Digital Minimalism: Choosing a Focused Life in a Noisy World<br>- So Good They Can&#39;t Ignore You: Why Skills Trump Passion in the Quest for Work You Love</pre><p>We can ask another question:</p><p>question = “How do I engage with smartphones to improve productivity according to Dr. Cal?”</p><p>Run app.py. <strong>Output:</strong></p><pre>(venv) tl@tensorlake:~/Documents/PROJECTS/IndexifyStart$ python app.py <br>Answer:<br>Dr. Cal shares specific practices and alternative practices for engaging with smartphones to improve productivity. He suggests selecting from a variety of different tools available to suit individual needs, rather than following hard and fast rules. This allows individuals to tailor their smartphone usage in a way that enhances focus and productivity while still utilizing the benefits of technology.</pre><p>If we try to ask a question <em>outside</em> what has been discussed in the podcast, the LLM does not have an answer:</p><p>question = “How is the weather today in Australia ?”</p><p>Run app.py. <strong>Output:</strong></p><pre>(venv) tl@tensorlake:~/Documents/PROJECTS/IndexifyStart$ python app.py <br>Answer:<br>I do not know the answer.</pre><h3>Indexify with LangChain Agents</h3><p>We can use the stored Indexify indexes as tools to build <a href="https://python.langchain.com/v0.1/docs/modules/agents/">agents</a> with LangChain. Agents are systems that interact with other tool, in this case, Indexify. Agents allow LLMs to make more dynamic decisions by determining what to do and how to interact with external data.</p><p>Agents can interact with various tools and determine what decision to make based on input and context. They can take actions like retrieving documents, searching databases, and calling APIs.</p><p>They are unlike chains, such as the qa_chain we used in the previous example. Chains are more suited for straightforward sequences of operations. A chain defines a fixed sequence of steps where the output of one step becomes the input for the next.</p><p>We can use agents to interact with various data sources stored in Indexify. We only need to convert the retriever to their indexes into a tool that the agents can use to retrieve specific information.</p><p>Modify the app.py file and add the following imports:</p><pre>from langchain.agents import create_tool_calling_agent, AgentExecuter<br>from langchain_community.agent_toolkits.load_tools import load_tools<br>from langchain.tools.retriever import create_retriever_tool<br>from langchain import hub</pre><p>Add the following function. The function creates a tool from the Indexify retriever. We give the tool a name and a description.</p><pre>def get_retriever_tool(retriever, name, description):<br>   return create_retriever_tool(retriever, name, description)</pre><p>Implement the functionality in the main block:</p><pre>if __name__ == &quot;__main__&quot;:<br><br>   (...code...)<br><br>   question = &quot;What does Andrew&#39;s neural network newsletter   <br>               include?&quot;<br><br>   # Define the Indexify retriever<br>   params = {&quot;name&quot;: &quot;pdfknowledgebase.embedding.embedding&quot;}<br>   retriever = IndexifyRetriever(client=client, params=params)<br><br>   # lll - gpt-3.5-turbo<br>   llm = ChatOpenAI(model=&#39;gpt-3.5-turbo&#39;, max_tokens=200)<br><br>   # Create the retriever tool from the Indexify retriever<br>   retriever_tool = create_retriever_tool(retriever=retriever,<br>                                           name=&quot;indexify_search&quot;,<br>                                           description=&quot;Useful for <br>                                 information search on Indexify &quot;)<br>   <br>   # We can store multiple tools in a list if available<br>   tools = [retriever_tool]<br><br>   # create a prompt<br>   system_prompt = &quot;&quot;&quot;<br>   You are a helpful assistant. Give a precise answer based on <br>   user input.<br>   If you do not know the answer, say you do not know the answer!:<br>   {input}<br>   &quot;&quot;&quot;<br>   doc_prompt = ChatPromptTemplate.from_messages(<br>       [<br>           (&quot;system&quot;, system_prompt),<br>           (&quot;human&quot;, &quot;{input}&quot;),<br>           (&quot;placeholder&quot;, &quot;{agent_scratchpad}&quot;)<br>       ]<br>   )<br><br>   agent = create_tool_calling_agent(llm, tools, doc_prompt)<br><br>   agent_executor = AgentExecutor(agent=agent, <br>                                  tools=tools, <br>                                  verbose=True,<br>                                  max_iterations=3)<br><br>   agent_retrieval_results = agent_executor.invoke({&quot;input&quot;: <br>                                                   question})</pre><p>Explaining the functionality:</p><ul><li><strong>Define the Indexify retriever</strong>, which will retrieve relevant information based on user queries from indexes stored in Indexify.</li><li><strong>Define the LLM</strong> that will guide the agent.</li><li><strong>Turn the Indexify retriever into a tool</strong> using create_retriever_tool in a format the agent can use it. If we have more than one of these tools, we can store them in a list that will be used downstream.</li><li><strong>Create the agent from LangChain’s tools agent</strong> using create_tool_calling_agent. The tools API is suitable for use with OpenAI models, as in our case. It allows the model to request that multiple functions be invoked at once, which can reduce response times in some architectures.</li><li><strong>Create a special prompt</strong> to guide the agent.</li><li><strong>Initialize the agent with the LLM, the prompt, and the tools</strong>. The agent is responsible for taking in input and deciding what actions to take.</li><li><strong>Combine the agent with the tools inside the AgentExecuter</strong>, which repeatedly calls the agent and executes the tools. The agent does not execute those actions itself.</li><li><strong>Run the agent </strong>with queries.</li></ul><p>Run app.py.<strong> Output:</strong></p><pre>&gt; Entering new AgentExecutor chain...<br><br>Invoking: `indexify_search` with `{&#39;query&#39;: &quot;Andrew&#39;s neural network newsletter&quot;}`<br><br>Huberman Lab podcast, go to live-momentous, spelled o U s.com <br>slash Huberman. That&#39;s Livemomentous.com slash, Huberman. If <br>you&#39;re not already following me on social media, I am Huberman <br>Lab on all social media platforms. So that&#39;s Instagram, X, <br>LinkedIn, Facebook, and threads. And at all of those places I <br>discuss science and science-related tools, some of which <br>overlaps with the content of the Huberman Lab podcast but much<br>of which is often distinct from the content on the Huberman Lab <br>podcast. So again, it&#39;s Huberman Lab on all social media <br>channels. If you haven&#39;t already subscribed to our neural network <br>newsletter, our neural network newsletter is a monthly newsletter <br>that includes podcast summaries as well as protocols in the form <br>of brief one to three-page PDFs where we spell out specific items <br>for, say, neuroplasticity and learning or deliberate cold exposure <br>or fitness, or managing and optimizing dopamine, all of which are Andrew&#39;s neural network newsletter is a monthly newsletter that includes podcast summaries as well as protocols in the form of brief one to three-page PDFs. The newsletter covers specific items related to neuroplasticity and learning, deliberate cold exposure or fitness, and managing and optimizing dopamine.</pre><p><strong>Retrieved answer:</strong></p><p>print(agent_retrieval_results[‘output’])</p><pre>&gt; Finished chain.<br>Andrew&#39;s neural network newsletter is a monthly newsletter that includes podcast summaries as well as protocols in the form of brief one to three-page PDFs. <br>The newsletter covers specific items related to neuroplasticity and learning, deliberate cold exposure or fitness, and managing and optimizing dopamine.</pre><p>The agent does not answer questions whose information can not be retrieved from Indexify indexes. If we try, the model will not answer.</p><p>question = “How can we use LLMs for question-answering taks?”</p><p><strong>Output:</strong></p><pre>&gt; Finished chain.<br>I apologize, but I couldn&#39;t find information related to using LLMs for question-answering tasks.</pre><p>We can create agents that utilize Indexify as a tool to retrieve information from various data sources that have been extracted, transformed and stored in its server.</p><h3>Final Thoughts</h3><p>In this article, we have explored how to integrate Indexify and LangChain. You have learned how to leverage Indexify’s retrieval APIs to retrieve contexts based on user queries, which is crucial for creating question-answering chains using Langchain. You have also explored how we can use Indexify as a tool to create more dynamic agents capable of answering questions based on various data stored on Indexify.</p><p>Integrating Indexify with LangChain can significantly enhance its natural language processing (NLP) capabilities, improve data retrieval efficiency, and enable more dynamic and interactive applications.</p><p>Indexify’s real-time data pipelines offer a framework for real-time data ingestion, transformation, and storage. These pipelines can supply LangChain with up-to-date and relevant data from various sources for use with LLMs. Their ability to continuously ingest and update data ensures that LangChain always works with the most current and relevant data.</p><p>Additionally, Indexify’s retrieval API enhances overall semantic searches and data retrieval, allowing integration with LangChain’s components for improved NLP.</p><p>Integrating the strengths of both tools can help developers build more robust and knowledge-rich LLM applications.</p><blockquote>Originally written as part of T<a href="https://tensorlake.ai/">ensorLake’s</a> campaign in collaboration with <a href="https://www.linkedin.com/company/argon-labs-xyz/posts/?feedView=all">Argon Labs</a>. The tooling has evolved since this was published, so be sure to check for the latest updates</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=452e161be4d3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Extract Financials Insights with Indexify Agentic Compute Engine]]></title>
            <link>https://medium.com/@brianmuteak/extracted-financials-insights-with-indexify-agentic-compute-engine-f0b02c31400b?source=rss-1e1c1cda0acd------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0b02c31400b</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[tensorlake]]></category>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[indexify]]></category>
            <dc:creator><![CDATA[Brian Mutea]]></dc:creator>
            <pubDate>Wed, 25 Feb 2026 09:21:14 GMT</pubDate>
            <atom:updated>2026-02-26T07:20:41.314Z</atom:updated>
            <content:encoded><![CDATA[<h4>Extracting Financials Insights from the Big Companies with TensorLake’s Indexify</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1007/1*QIURUJAzyTL6LqSpu0C0_Q.png" /><figcaption>Image by author</figcaption></figure><p>In today’s data-driven landscape, organizations are increasingly faced with diverse and complex datasets that often come in various unstructured formats. That presents a significant challenge, as raw data frequently lacks a standardized structure, complicating efforts to extract and utilize it effectively.</p><p>For those who have attempted to extract text from documents such as PDFs, the difficulties are all too familiar. While initial experimentation might seem straightforward, consistently managing and extracting meaningful information from real-world PDFs remains a formidable challenge, even with the most advanced tools.</p><p>In this article, we explore how Indexify extracts valuable text data from unstructured sources, such as financial income statements, and generates actionable insights from the extracted information. Imagine being able to sift through the financial statements of industry giants like Tesla, Amazon, and NVIDIA, uncovering key insights that drive their success. Extracting data from financial documents despite their complex, unstructured, but information-rich nature is crucial for accurate analysis and informed decision-making.</p><p>Indexify utilizes two distinct text extraction approaches to achieve this goal:</p><ul><li><strong>Direct</strong> text extraction and insights generation using their OpenAI extractor using the GPT-4o model.</li><li><strong>Two-step</strong> text extraction using their PDF extractor and insight generation with their OpenAI’s extractor using GPT-3.5-turbo.</li></ul><p>Ultimately, we will integrate Indexify with LangChain to ask questions and get answers based on the text extracted from the financial statements, using Indexify as the retrieval tool.</p><h3>Settling on an Interesting Use Case</h3><p>Imagine needing to extract valuable insights from a variety of financial income statements for major companies such as Amazon. These documents could originate from diverse sources, including official filings, financial news websites, data providers, and corporate sites. Each source presents income data differently due to variations in formatting conventions, data models, and user interfaces, making data extraction a complex task.</p><p>Analyzing financial metrics across multiple pages of these statements can be both time-consuming and challenging, particularly when dealing with large amounts of income data.</p><p>Furthermore, many of these statements are in PDF format. Extracting text from PDFs poses significant challenges for automated systems, often resulting in inconsistent data extraction and difficulties scaling with multiple or large files.</p><p>With Indexify’s approach, however, transforming these diverse financial statements into a structured format for seamless querying, analysis, and visualization has never been easier.</p><h3>Quick Overview of Indexify</h3><p>Indexify leverages a declarative methodology for data extraction, utilizing scalable pipelines known as extraction graphs. These graphs are designed to manage extensive text data effectively and maintain reliability by replicating their state across multiple machines, which helps recover from potential failures.</p><p>Indexify streamlines the text data extraction process through several key features:</p><ul><li><strong>Asynchronous Workflows:</strong> It runs extraction workflows asynchronously and is supported by a real-time batch processing engine.</li><li><strong>Scalable Ingestion:</strong> It allows for infinite scalability of data ingestion using blob storage and specialized I/O processes.</li><li><strong>Efficient Retrieval:</strong> Retrieval is handled in dedicated processes, directly accessing storage and avoiding delays caused by extraction workloads.</li><li><strong>Flexible Integration:</strong> It provides plug-and-play functionality with various models, accommodating any data type or extraction need within an extraction graph.</li></ul><p>Indexify can create vector indexes from the extracted data by defining algorithms and models in extraction graphs. This capability facilitates interactive text processing, such as querying, summarizing, and analyzing content.</p><h3>The Data Sources: Financial Statements</h3><p>To facilitate a quick review, we’ve provided a selection of annual income statements available for download.</p><p>You can get other samples like these <a href="https://stockanalysis.com/stocks/amzn/financials/">here</a>. The financials from the source are provided by Nasdaq Data Link and sourced from the audited annual (10-K) and quarterly (10-Q) reports submitted to the Securities and Exchange Commission (SEC).</p><p><a href="https://drive.google.com/file/d/1ubvh09-obKOrMsI_eOXpujAKizATR313/view?usp=sharing"><strong>Amazon Income Statement</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*tHlECdVWDUDKYP4jp5P8tA.png" /></figure><p><a href="https://drive.google.com/file/d/1-6iY-H5lxqzTzzyNbb_1ZGeUU78C5aQY/view?usp=sharing"><strong>Tesla Income Statement</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*ghNv4LnC3L6nkmEPnXT0Uw.png" /></figure><p><a href="https://drive.google.com/file/d/1n96SsvIIdAJv9-v-h_UgzU0FZSSaJHG7/view?usp=sharing"><strong>NVIDIA Income Statement</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*8GiYmL0tKTKkAONm-Me38A.png" /></figure><p><a href="https://drive.google.com/file/d/1ShYVV6uDgTMejjA8MzCjcZYPkaNOrFHI/view?usp=sharing"><strong>Netflix Income Statement</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*-JwXssvhXOHBz7_PRaSyeg.png" /></figure><h3>Getting Started</h3><p><strong>Prerequisites:</strong></p><p>Before we proceed, you need to do the following:</p><ul><li>Create and activate a virtual environment.</li></ul><pre>python -m venv v_env<br>source v_env/bin/activate</pre><ul><li>Get an OpenAI API KEY <a href="https://platform.openai.com/account/api-keys">here</a>.</li></ul><h3>Install the indexify-extractor-sdk and Indexify</h3><p>The extractor SDK allows new extraction capabilities and other readily available extractors to be added.</p><pre>pip install indexify indexify-extractor-sdk</pre><h3>Download and Start the Server</h3><p><strong>Download:</strong></p><pre>curl https://getindexify.ai | sh</pre><p><strong>Start the server</strong>: This is a long-running process:</p><pre>./indexify server -d</pre><h3>Install Required Extractors</h3><p>We will need four extractors.</p><p><strong>Text extraction</strong>:</p><ol><li><a href="https://github.com/tensorlakeai/indexify-extractors/tree/main/pdf/pdfextractor"><strong>Pdf extractor</strong></a>: tensorlake/pdf-extractor. Will help extract text from the PDFs in the two-step approach.</li><li><a href="https://github.com/tensorlakeai/indexify-extractors/tree/main/text/openai"><strong>OpenAI extractor</strong></a>: tensorlake/openai. We will use it in both direct and two-step approaches. It can handle PDFs directly.</li></ol><p><strong>Creating vector indexes</strong>:</p><ol><li><a href="https://github.com/tensorlakeai/indexify-extractors/tree/main/text/chunking"><strong>Chunking extractor</strong></a>: tensorlake/chunk-extractor. It splits extracted text into small chunks for embedding.</li><li><a href="https://github.com/tensorlakeai/indexify-extractors/tree/main/embedding/minilm-l6"><strong>Embedding extractor</strong></a>: tensorlake/minilm-l6. Creates embeddings of the chunks that are kept in storage for semantic search.</li></ol><p>Run the following to install each extractor:</p><pre>indexify-extractor download tensorlake/pdfextractor<br>indexify-extractor download tensorlake/openai<br>indexify-extractor download tensorlake/chunking<br>indexify-extractor download tensorlake/minilm-l6</pre><p>Once they are installed, run them on the server. This is also a long-running process:</p><pre>indexify-extractor join-server</pre><p><em>You can access the server UI at </em><a href="http://localhost:8900/ui"><em>http://localhost:8900/ui</em></a><em>.</em></p><p><strong>Preamble 📓:</strong></p><p>Now that we have settled the requirements, we will explore the two approaches for extracting text and generating insights from financial statements using Indexify.</p><p>To demonstrate both approaches, we will pick only the <a href="https://drive.google.com/file/d/1-6iY-H5lxqzTzzyNbb_1ZGeUU78C5aQY/view?usp=sharing">Tesla Income Statement</a>. This will get you started quickly before we begin extracting from multiple income statements.</p><h3>Direct Approach: Extracting Text Data Directly from Financial Statements using OpenAI’s GPT-4o model</h3><p>Indexify offers a powerful extraction tool that leverages OpenAI technology to extract text from documents, including PDFs, instantly. This extractor enables precise control over data extraction and formatting by allowing us to specify the model name and provide tailored prompts.</p><p>In addition to text extraction, we can prompt the model to perform various text-related tasks, such as generating insights, translating text, summarizing information, and more.</p><p>For our specific use case of deriving insights from financial statements, we will utilize this capability to instruct the model to analyze the extracted text and produce accurate, actionable insights from the content within the financial document.</p><h3>Create an Extraction Graph</h3><p>First, we need to create an extraction graph. We define openai as the extractor, model, and prompt.</p><p>Create a pdf_to_text_gpt4.yamlfile and add the following:</p><pre>name: &quot;teslaincomeknowledgebase2_gpt4o&quot;<br>extraction_policies:<br> - extractor: &quot;tensorlake/openai&quot;<br>   name: &quot;pdf_text_extractor&quot;<br>   input_params:<br>     model_name: &#39;gpt-4o&#39;  <br>     key: &quot;YOUR_OPENAI_API_KEY&quot;<br>     system_prompt: &quot;You are a helpful assistant in extracting text, <br>     analyzing and generating accurate insights from a financial <br>     income statement document. Pay close attention to each financial <br>     metric and their values for each time period. Ensure precise and <br>     comprehensive extraction. Ensure the extracted data has <br>     consistent format. Return the extracted data as text and give <br>     accurate insights about the data. Ignore any footer, titles, <br>     services, website links or forms!&quot;</pre><h3>Initiate Text Extraction and Insight Generation</h3><p>Next, create a script to upload the financial statement through the extraction graph to extract and generate insights.</p><p>Create extract_geninsights_gpt4.pyfile and add the following:</p><pre>from indexify import IndexifyClient, ExtractionGraph<br>client = IndexifyClient()<br><br># Read the Extraction Graph specification from the .yaml file<br>with open(&quot;pdf_to_text.yaml&quot;, &quot;r&quot;) as file:<br>   extraction_graph_spec = file.read()<br>extraction_graph = ExtractionGraph.from_dict(extraction_graph_spec)<br>client.create_extraction_graph(extraction_graph)<br><br>path_to_pdf = &quot;/path/to/your/Tesla_Financials_Income_Statement_Stock_Analysis.pdf&quot;<br>content_id = client.upload_file(<br>   &#39;teslaincomeknowledgebase_gpt4o&#39;, path=path_to_pdf)<br>print(content_id) # print content_id<br><br># Wait for text extraction to complete<br>client.wait_for_extraction(content_id)</pre><p>Run extract_geninsights_gpt4.py and wait for extraction to complete.</p><p>Once the extraction is complete, you can use the content_idto view the extracted data.</p><p>Create a get_extracted_content.pyfile and add the following code:</p><pre>from indexify import IndexifyClient<br>client = IndexifyClient()<br>extracted_content = client.get_extracted_content(<br>                         content_id=content_id,  <br>                         graph_name=&#39;teslaincomeknowledgebase_gpt4o&#39;,                 <br>                         policy_name=&#39;pdf_text_extractor&#39;)<br><br># Decode the extracted content<br>print(extracted_content[0][&#39;content&#39;].decode(&#39;utf-8&#39;)</pre><p>Run extract_geninsights_gpt4.py. <strong>Output</strong>:</p><h4>Sample Text data Extracted from Tesla Financial statement:</h4><pre>### Extracted Data<br><br>**Tesla Income Statement**<br><br>**Financials in millions USD. Fiscal year is January - December.**<br><br>| Year Ending | Dec 31, 2019 | Dec 31, 2020 | Dec 31, 2021 | Dec 31, 2022 | Dec 31, 2023 |<br>|-----------------------|--------------|--------------|--------------|--------------|--------------|<br>| **Revenue**           | 24,578       | 31,536       | 53,823       | 81,462       | 96,773       |<br>| **Revenue Growth (YoY)** | 14.52%    | 28.31%       | 70.67%       | 51.35%       | 18.80%       |<br>| **Cost of Revenue**   | 20,509       | 24,906       | 40,217       | 60,609       | 79,113       |<br>| **Gross Profit**      | 4,069        | 6,630        | 13,606       | 20,853       | 17,660       |<br>| **Selling, General &amp; Admin** | 2,646 | 3,145        | 4,517        | 3,946        | 4,800 |<br></pre><h4>Sample Insights from Tesla Financial Statement:</h4><pre>### Insights<br><br>1. **Revenue Growth**:<br>    - **2019 to 2020**: Revenue increased by 14.52%, from $24,578 million to $31,536 million.<br>    - **2020 to 2021**: Significant revenue growth of 70.67%, reaching $53,823 million.<br>    - **2021 to 2022**: Continued growth with a 51.35% increase, amounting to $81,462 million.<br>    - **2022 to 2023**: Revenue growth slowed down to 18.80%, but still increased, reaching $96,773 million.<br><br>2. **Cost of Revenue**:<br>    - The cost of revenue has been consistently increasing across the years.<br>    - From $20,509 million in 2019 to $79,113 million in 2023, indicating that while the company is growing, the cost associated with generating revenue is also rising substantially.<br><br>3. **Gross Profit**:<br>    - Gross profit shows a strong upward trend until 2022, reflecting efficient management and increasing profitability.<br>    - However, in 2023, it experienced a decline from $20,853 million to $17,660 million, suggesting rising costs or other factors affecting profitability.<br><br>4. **Selling, General &amp; Admin Expenses**:<br>    - These expenses have been relatively stable with a gradual increase.<br>    - From $2,646 million in 2019 to $4,800 million in 2023, showing efficient scaling of administrative overheads in line with revenue growth.<br><br>### Summary<br><br>Tesla has shown impressive revenue growth over the past five years, moving from $24,578 million in 2019 to $96,773 million in 2023. The company&#39;s revenue growth has been robust, particularly noticeable in 2021 and 2022. However, 2023 indicated a slowdown in growth rate.<br><br>The cost of revenue has been increasing substantially, putting pressure on gross profit in 2023 despite the growth in revenue...</pre><p>We can agree that we got accurate insights from the extraction. For instance, we see that:</p><ul><li>The revenue growth for Tesla increased by 14.52% from $24, 578 million to $ 31,530 million from 2019 to 2020.</li><li>The gross profit has been growing until 2022 and declines in 2023 from $20,853 million to $17,660 million.</li></ul><p>We can also see that the extractor can maintain the document’s layout.</p><p>Next, we will explore the two-step approach.</p><h3>Two-step Approach : Extracting Text Data Using PDF Extractor and Using OpenAI’s Extractor with GPT-3.5-Turbo Model</h3><p>This approach involves a two-step process to extract text and analyze financial documents effectively. First, we employ a high-level PDF extractor to extract text from the income document, ensuring that all relevant information is captured. Once the text is extracted, it is then processed by the <strong>OpenAI GPT-3.5-turbo</strong> model.</p><p>The model can analyze the extracted text and generate valuable insights, such as identifying key financial metrics, summarizing the document’s content, and providing contextual interpretations. This approach ensures both accurate data extraction and analysis, thus providing actionable and meaningful insights from financial statements.</p><h3>Create an Extraction Graph</h3><p>We will define the PDF extractor and then chain its result to the openai GPT-3.5-turbo model.</p><p>The PDF extractor uses advanced OCR techniques to achieve three extraction outputs:</p><ul><li><strong>Texts</strong>: It can extract any text from the document.</li><li><strong>Tables</strong>: It can identify table layouts in the document. If any, it returns the table data as JSON. So, in our case, we will see text and JSON data extracted.</li><li><strong>Images</strong>: It will also extract any image in the document. But we are not concerned with images for now.</li></ul><p>Create a pdf_to_text_pdf_gpt35t.yamlfile and add the following:</p><pre>name: &quot;teslaincomeknowledgebase2_pdf-gpt35t&quot;<br>extraction_policies:<br> - extractor: &quot;tensorlake/pdf-extractor&quot;<br>   name: &quot;pdf_text_extractor&quot;<br><br> - extractor: &quot;tensorlake/openai&quot;<br>   name: &quot;text_analyzer&quot;<br>   input_params:<br>     model_name: &quot;gpt-3.5-turbo&quot;  <br>     key: &quot;YOUR_OPENAI_API_KEY&quot;<br>     system_prompt: &quot;You are a helpful assistant analyzing and  <br>     generating accurate insights from income data in text form. <br>     Analyze and give accurate insights from the text and pay close <br>     attention to financial metrics, their values and time period.  <br>     Ensure precise and comprehensive analysis.&quot;<br>   content_source: &quot;pdf_text_extractor&quot;</pre><h3>Initiate Text Extraction and Insight Generation</h3><p>Create extract_geninsights_gpt35t.pyfile and add the following:</p><pre>from indexify import IndexifyClient, ExtractionGraph<br>client = IndexifyClient()<br><br># Read the Extraction Graph specification from the .yaml file<br>with open(&quot;pdf_to_text_pdf_gpt35t.yaml&quot;, &quot;r&quot;) as file:<br>   extraction_graph_spec = file.read()<br>extraction_graph = ExtractionGraph.from_dict(extraction_graph_spec)<br>client.create_extraction_graph(extraction_graph)<br><br>path_to_pdf = &quot;/path/to/your/Tesla_Financials_Income_Statement_Stock_Analysis.pdf&quot;<br><br><br># upload<br>content_id = client.upload_file(<br>                            &#39;teslaincomeknowledgebase_pdf-gpt35t&#39;, <br>                             path=path_to_pdf)<br>print(content_id) # print content_id<br><br># Wait for text extraction to complete<br>client.wait_for_extraction(content_id)</pre><p>Run extract_geninsights_gpt35t.py and wait for extraction to complete.</p><p>Next modify the get_extracted_content.py file with the new extraction graph_nameand content id:</p><pre>from indexify import IndexifyClient<br>client = IndexifyClient()<br>extracted_content = client.get_extracted_content(<br>                  content_id=content_id,  <br>                  graph_name=&#39;teslaincomeknowledgebase_pdf-gpt35t&#39;,                 <br>                  policy_name=&#39;pdf_text_extractor&#39;)<br><br># Decode the extracted content<br>print(extracted_content[0][&#39;content&#39;].decode(&#39;utf-8&#39;)</pre><p>Run extract_geninsights_pdf_gpt35t.py: <strong>Output</strong>:</p><h4>Sample of Text Data Extracted from Tesla Financial Statement:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hnvXd_R3R2EuPk0ltudCRA.png" /></figure><h4>Sample Insights from Tesla Financial Statement:</h4><pre>### Analysis of Tesla Inc.&#39;s Income Statement:<br><br>- **Revenue Trend**: Tesla&#39;s revenue has been increasing steadily over the years, with significant growth year over year. From 2019 to 2023, the revenue increased from $24.6 billion to $96.8 billion, showing a robust growth trend.<br><br>- **Gross Profit**: The gross profit also increased consistently with the revenue growth, reaching $17.7 billion in 2023. This indicates that Tesla has been able to effectively manage its cost of revenue and maintain a healthy gross profit margin.<br><br>- **Cost of Revenue**: The cost of revenue has also risen in line with the revenue growth, showing that Tesla&#39;s operations have scaled up over the years. In 2023, the cost of revenue was $79.1 billion, reflecting the increased scale of their business.<br><br>- **Selling, General &amp; Admin Expenses**: These expenses have fluctuated slightly over the years but remained relatively stable compared to the revenue growth. In 2023, these expenses amounted to $4.8 billion, showing prudent management of operational costs.<br><br>- **Revenue Growth Rate**: The revenue growth rate of Tesla has varied over the years, with a notable surge in 2021 at 70.67%. Although the growth rate slowed down in 2023 to 18.80%, it still demonstrates a healthy expansion trend.<br><br>### Key Insights:<br>- Tesla has shown substantial revenue growth, indicating strong market demand for its products and services.<br>- The company has been effectively managing its gross profit margin, despite the increasing cost of revenue.<br>...</pre><h3>Extracting Text and Generating Insights from Multiple Financial Income Statements</h3><p>With Indexify’s declarative extraction graphs, we can efficiently ingest and extract text from multiple financial statement documents.</p><p>These graphs allow us to define a clear and structured workflow for text extraction by specifying various extraction policies and content sources. This declarative approach simplifies the process of handling diverse and complex documents by enabling the reuse of predefined extraction strategies.</p><p>Let’s create an income_statement_extraction_.yaml file and add the following extraction graph.</p><p>We have included two more extractors. The text chunker will split the extracted text into small chunks, which will then be passed to the embedding extractor to create embeddings useful for retrieving insights through semantic search.</p><pre>name: &quot;financialstatement_knowledgebase_gpt4&quot;<br>extraction_policies:<br> - extractor: &quot;tensorlake/openai&quot;<br>   name: &quot;text_extractor_and_analyzer&quot;<br>   input_params:<br>     model_name: &quot;gpt-4o&quot;<br>     key: &quot;YOUR_OPENAI_API_KEY&quot;<br>     system_prompt: &quot;You are a helpful assistant in extracting text,  <br>     analyzing and generating accurate insights from a financial  <br>     income statement document. Pay close attention to each financial <br>     metric and their values for each time period. Ensure precise and <br>     comprehensive extraction. Ensure the extracted data has <br>     consistent format. Return the extracted data as text and give <br>     accurate insights about the data. Ignore any footer, titles, <br>     services, website links or forms!&quot;<br><br> - extractor: &quot;tensorlake/chunk-extractor&quot;<br>   name: &quot;text_chunker&quot;<br>   input_params:<br>     chunk_size: 1000<br>     overlap: 250<br>   content_source: &quot;text_extractor_and_analyzer&quot;<br><br> - extractor: &quot;tensorlake/minilm-l6&quot;<br>   name: &quot;embedding&quot;<br>   content_source: &quot;text_chunker&quot;</pre><p>Next, create setup.py and add the following functions. We will not dive deep into explaining how the functions are working; rather, we have included docstrings to explain an overview of their functionality.</p><pre>from indexify import IndexifyClient, ExtractionGraph<br>import os<br><br>client = IndexifyClient()<br><br>def read_yaml_file(yaml_file_path):<br>   &quot;&quot;&quot;Reads and returns the content of a YAML file.&quot;&quot;&quot;<br>   with open(yaml_file_path, &#39;r&#39;) as file:<br>       return file.read()<br><br>def create_extraction_graph(yaml_content):<br>   &quot;&quot;&quot;Creates an extraction graph from YAML content.&quot;&quot;&quot;<br>   extraction_graph = ExtractionGraph.from_yaml(yaml_content)<br>   client.create_extraction_graph(extraction_graph)<br><br>def get_files_to_upload(files_directory_path):<br>   &quot;&quot;&quot;Returns a list of financial statements in the directory.&quot;&quot;&quot;<br>   all_files = os.listdir(files_directory_path)<br>   return [f for f in all_files if f.endswith(&#39;.pdf&#39;)]</pre><p>Add a function to upload the multiple files to Indexify for extraction, analysis and insight generation:</p><pre>def upload_files(yaml_file_path, files_directory_path):<br>   &quot;&quot;&quot;Uploads PDF files using the specified YAML file.&quot;&quot;&quot;<br>   # Read and parse the YAML file<br>   yaml_content = read_yaml_file(yaml_file_path)<br><br>   # Create the extraction graph<br>   create_extraction_graph(yaml_content)<br><br>   # Get the list of PDF files to upload<br>   files_to_upload = get_files_to_upload(<br>       files_directory_path)<br><br>   # Upload each PDF file in the filtered list<br>   for file_name in files_to_upload:<br>       file_path = os.path.join(files_directory_path, file_name)<br>       if os.path.exists(file_path):<br>           content_id = client.upload_file(<br>               &quot;financialstatement_knowledgebase_gpt4&quot;, <br>                path=file_path)<br>           client.wait_for_extraction(content_id)<br>           print(<br>               f&quot;Successfully uploaded {file_path} with content ID: <br>               {content_id}&quot;)</pre><p>Implement multiple text extraction:</p><pre>if __name__ == &quot;__main__&quot;:<br><br>   # Paths to the necessary directories and files<br>   yaml_file_path = &#39;/path/to/your/pdf_to_txt.yaml&#39;<br>   files_directory_path = &#39;/path/to/your/financial_statemets_dir<br><br>   # Run the extraction process<br>   upload_files(yaml_file_path, files_directory_path)</pre><p>Run the setup.py. <strong>Output</strong>:</p><pre>Waiting for extraction to complete for content id:  214a9ada8d49d34a<br>Extraction completed for content id:  214a9ada8d49d34a<br>Successfully uploaded data/Tesla_....pdf with content ID: 214a9ada8d49d34a<br>Waiting for extraction to complete for content id:  47e4b443632df449<br>Extraction completed for content id:  47e4b443632df449<br>Successfully uploaded data/NVIDIA_....pdf with content ID: 47e4b443632df449<br>Waiting for extraction to complete for content id:  4545bcf0d265d4d<br>Extraction completed for content id:  4545bcf0d265d4d<br>Successfully uploaded data/Amazon_....pdf with content ID: 4545bcf0d265d4d<br>Waiting for extraction to complete for content id:  bcea84d7412142b2<br>Extraction completed for content id:  bcea84d7412142b2<br>Successfully uploaded data2/Netflix_....pdf with content ID: bcea84d7412142b2<br><br>...+1000s of docs</pre><p>Input the content_id of any of the financial statement in the code below to get the extracted data and the generated insights about it:</p><pre># Pass the rhe necessary content_ids<br>extracted_content = client.get_extracted_content(<br>                content_id=content_id,  <br>                graph_name=&#39;financialstatement_knowledgebase_gpt4&#39;, <br>                policy_name=&#39;text_extractor_and_analyzer&#39;)<br>print(extracted_content[0][&#39;content&#39;].decode(&#39;utf-8&#39;))</pre><h3>NVIDIA Results</h3><h4><strong>Sample Extracted Text data from NVIDIA Income Statement:</strong></h4><pre>### Extracted Data:<br><br>**NVIDIA Income Statement**<br>*Financials in millions USD. Fiscal year is February - January.*<br><br>| Year Ending|Jan 26, 2020|Jan 31, 2021|Jan 30, 2022 | Jan 29, 2023 | Jan 28, 2024 |<br>|---------------------|--------------|--------------|--------------|--------------|--------------|<br>| **Revenue**         | 10,918       | 16,675       | 26,914       | 26,974       | 60,922       |<br>| **Revenue Growth (YoY)** | -6.81%   | 52.73%       | 61.40%       | 0.22%        | 125.85%      |<br>| **Cost of Revenue** | 4,150        | 6,279        | 9,439        | 11,618       | 16,621       |<br>| **Gross Profit**    | 6,768        | 10,396       | 17,475       | 15,356       | 44,301       |<br>| **Selling, General &amp; Admin** | 1,093    | 1,940        | 2,166        | 2,440        | 2,654        |</pre><h4><strong>Sample Insights from NVIDIA Income Statement:</strong></h4><pre>### Insights:<br><br>1. **Revenue Trend:**<br>   - The revenue has shown significant growth over the last five years. It increased from $10,918 million in 2020 to $60,922 million in 2024.<br>   - The most dramatic increase occurred between 2023 and 2024, with Revenue nearly doubling (an increase of 125.85%).<br><br>2. **Revenue Growth (YoY):**<br>   - The years 2021 and 2022 experienced high revenue growth rates of 52.73% and 61.40%, respectively.<br>   - The growth rate significantly dropped in 2023 to 0.22%, showing a stable revenue between 2022 and 2023 but surged again to 125.85% in 2024.<br><br>3. **Cost of Revenue:**<br>   - The cost of revenue has been rising steadily, from $4,150 million in 2020 to $16,621 million in 2024.<br>   - This consistent rise indicates increasing expenses associated with the production of goods/services sold.<br><br>4. **Gross Profit:**<br>   - Gross profit has increased from $6,768 million in 2020 to $44,301 million in 2024.<br>   - Although there was a slight decline in gross profit from 2022 ($17,475 million) to 2023 ($15,356 million), it saw a massive jump in 2024.<br>...<br><br>### Conclusion:<br>Overall, NVIDIA Corporation has demonstrated strong financial growth over the five-year period, with substantial increases in revenue and gross profit. The company&#39;s cost of revenue and administrative expenses have also increased but at a controlled rate compared to revenue, indicating efficient management and scalability in operations.</pre><h3>Amazon Results</h3><h4><strong>Sample Extracted Text data from Amazon Income Statement:</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sxUgKRq-rBkYB44L2PrigQ.png" /></figure><h4><strong>Sample Insights from Amazon Income Statement:</strong></h4><pre>### Insights<br><br>1. **Revenue Growth:** The company has shown consistent revenue growth from 2019 to 2023. The highest year-on-year growth happened in 2020 at 37.62%, and even in the lowest growth year (2022), it still achieved a growth rate of 9.40%.<br><br>2. **Revenue:** Total revenue has steadily risen every year, from $280,522 million in 2019 to $574,785 million in 2023, indicating strong sales and continuing business expansion.<br><br>3. **Cost of Revenue:** The cost associated with revenue has also increased over the periods, escalating from $165,536 million in 2019 to $304,739 million in 2023, which is consistent with the revenue increase.<br><br>4. **Gross Profit:** Gross profit also shows an increasing trend, growing from $114,986 million in 2019 to $270,046 million in 2023, reflecting profitability growth.<br><br>5. **Selling, General &amp; Administrative Expenses:** These expenses have increased over the years from $24,081 million in 2019 to $56,186 million in 2023, likely due to expanded operations and increased investment in business administration.<br><br>Overall, Amazon demonstrates substantial growth in revenue and gross profit with corresponding rises in costs and expenses, indicating ongoing business scaling and investment in future growth.</pre><h3>Netflix Results</h3><h4><strong>Sample Text data Extracted from Netflix Income Statement:</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*indOHrhNfRaCIMrJqGN4kA.png" /></figure><h4><strong>Sample Insights from Netflix Income Statement:</strong></h4><pre>### Insights:<br><br>1. **Revenue Growth:**<br>   - Netflix has shown consistent revenue growth from 2019 to 2023, although the growth rate has been decreasing over the years.<br>   - The highest growth rate was in 2019 (27.62%), while the lowest was in 2022 (6.46%).<br><br>2. **Revenue:**<br>   - Revenue has increased each year, from $20,156 million in 2019 to $33,723 million in 2023.<br>   - The largest revenue increase occurred between 2020 and 2021.<br><br>3. **Cost of Revenue:**<br>   - The cost of revenue also increased each year, from $12,440 million in 2019 to $19,715 million in 2023.<br>   - The cost of revenue saw a significant jump between 2020 and 2021.<br><br>4. **Gross Profit:**<br>   - Gross profit increased consistently year over year reaching $14,008 million in 2023 from $7,716 million in 2019.<br>   - The largest increase in gross profit was between 2020 and 2021.<br><br>5. **Selling, General &amp; Admin Expenses:**<br>   - These expenses have been relatively stable compared to other financial metrics but showed a steady increase from $3,567 million in 2019 to $4,378 million in 2023.<br><br>6. **Profitability:**<br>   - Despite the increasing cost of revenue and SG&amp;A expenses, gross profit has been increasing steadily, indicating efficient cost management and a successful increase in revenue generation.<br><br>This financial trend analysis indicates that Netflix has been growing its revenue and managing its costs effectively to improve its profit margins consistently over the past five years.</pre><h3><strong>Question-Answering Based on Text Extracted from Financial Documents</strong></h3><p>We can use Indexify as a retriever tool to ask questions based on the extracted text data through semantic search. The embeddings created and stored by Indexify are key to this quest.</p><p>Here, we will use <a href="https://python.langchain.com/v0.2/docs/introduction/">LangChain</a> and create a simple script to ask questions based on the extracted data.</p><p>Create an app.py file and add the following code:</p><pre>from langchain.chains import RetrievalQA<br>from indexify import IndexifyClient<br>from indexify_langchain import IndexifyRetriever<br><br>client = IndexifyClient()<br><br># Initialize your model and retriever<br>llm = ChatOpenAI(model_name=&quot;gpt-3.5-turbo&quot;)<br>table_index = &quot;financialstatement_knowledgebase_gpt4.embedding.embedding&quot;<br>params = {&quot;name&quot;: table_index}<br>retriever = IndexifyRetriever(client=client, params=params)<br>qa_chain = RetrievalQA.from_chain_type(llm, retriever=retriever)<br><br><br># List of questions - Can add more relevant to the uploaded f/s<br>questions = [<br>   &quot;What was the revenue growth for Nvidia between 2020 and 2021?&quot;,<br>   &quot;How was the revenue growth for Amazon between 2019 and 2023?&quot;,<br>   &quot;What was the gross profit for Tesla in 2022?.&quot;<br>]<br><br># Store responses. Can add more questions.<br>responses = []<br><br>for query in questions:<br>   response = qa_chain.invoke({&quot;query&quot;: query})<br>   responses.append(response)<br><br>pp = pprint.PrettyPrinter(indent=5)<br>pp.pprint(responses)</pre><p>Run app.py. <strong>Output</strong>:</p><pre>[    {    &#39;query&#39;: &#39;What was the revenue growth for Nvidia between <br>           2020 and &#39;&#39;2021?&#39;,<br>          &#39;result&#39;: &#39;The revenue growth for NVIDIA between 2020 and  <br>           2021 was &#39;&#39;52.73%.&#39;},<br>     {    &#39;query&#39;: &#39;How was the revenue growth for Amazon between <br>           2019 and &#39;&#39;2023?&#39;,<br>          &#39;result&#39;: &#39;The revenue growth for Amazon between 2019 and <br>           2023 &#39;&#39;showed a consistent increase. In 2019, the revenue <br>           was &#39;’80,522 million, and by 2023, it grew to $574,785 &#39;<br>           million. The growth rate peaked in 2020 at 37.62% but &#39;<br>           &#39;slowed down over the years, reaching 11.83% in 2023.&#39;},<br>     {    &#39;query&#39;: &#39;What was the gross profit for Tesla in 2022?.&#39;,<br>          &#39;result&#39;: &#39;The gross profit for Tesla in 2022 was 20,853 <br>           million &#39;&#39;USD.&#39;}]</pre><h3>Final Thoughts</h3><p>Extracting key pieces of information from documents has long been a dominant challenge. Before the birth of large language models (LLMs), this problem was complicated to solve. Even with LLMs, several challenges remain.</p><p>Indexify is a unique and specialized tool for data extraction use cases powered by a reliable and fault-tolerant collection of systems.</p><p>In this article, we learned about Indexify’s capability to extract textual data from financial statements and generate accurate insights. We have seen two approaches for this use case, powered by LLMs from OpenAI.</p><p>In the end, we have also learned how to leverage this extracted text data and the retrieval capabilities of Indexify to ask questions and get answers from financial documents.</p><p>Indexify can help you in any project deployment lifecycle’s design and production phases, leveraging data extraction from large volumes of data without breaking down. Ideally, it comes to your help if:</p><ul><li>You are working with numerous documents, audio files, and videos.</li><li>The data volume grows over time.</li><li>You care about the reliability and availability of your ingestion pipelines.</li><li>You are working with multi-modal data.</li><li>You want to combine multiple models into a single reliable pipeline for data extraction.</li></ul><blockquote>Originally written as part of T<a href="https://tensorlake.ai/">ensorLake’s</a> campaign in collaboration with <a href="https://www.linkedin.com/company/argon-labs-xyz/posts/?feedView=all">Argon Labs</a>. The tooling has evolved since this was published, so be sure to check for the latest updates</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0b02c31400b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unstructured Video Data Extraction with Indexify an Agentic Compute Engine]]></title>
            <link>https://medium.com/@brianmuteak/unstructured-video-data-extraction-with-indexify-an-agentic-compute-engine-09de811fef52?source=rss-1e1c1cda0acd------2</link>
            <guid isPermaLink="false">https://medium.com/p/09de811fef52</guid>
            <category><![CDATA[tensorlake]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-extraction]]></category>
            <category><![CDATA[indexify]]></category>
            <dc:creator><![CDATA[Brian Mutea]]></dc:creator>
            <pubDate>Wed, 25 Feb 2026 08:33:09 GMT</pubDate>
            <atom:updated>2026-02-25T11:03:45.921Z</atom:updated>
            <content:encoded><![CDATA[<h4>Extracting Lessons from Commencement Speeches with TensorLake’s Indexify</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*VhnQuckUcqkuRRbJBBTLVw.png" /><figcaption>Image by author</figcaption></figure><p>Today, we have access to a vast amount of data, most of which exists in various formats and is unstructured, making interpreting it challenging. One such form of data is <strong>video</strong>.</p><p>Throughout the rest of this article, we will first discuss how Indexify shines through the challenges presented by extracting data from videos and why many tools, unlike Indexify, fail to auto-scale when a large amount of this data is involved. We will then go through a general workflow of Indexify to extract information from a YouTube video. Afterward, we will dive deeper to extract from multiple videos with Indexify.</p><p>We will process three commencement speeches from YouTube:</p><ul><li><a href="https://www.youtube.com/watch?v=UF8uR6Z6KLc&amp;t=6s">“Steve Jobs’ Stanford Commencement Address in 2005</a>.”</li><li>“<a href="https://www.youtube.com/watch?v=dkPo5nfEk1w">Tom Hanks Commencement Address at Harvard Commencement in 2023</a>.”</li><li>“<a href="https://www.youtube.com/watch?v=pxBQLFLei70">Admiral William H. McRaven Commencement Address at University of Texas in 2014</a>.”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*f1NL1kWfdnc2QEsro1gxCQ.png" /><figcaption>Photo by<a href="https://unsplash.com/@beavilla?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash"> Patricia Beatrix Villanueva</a> on<a href="https://unsplash.com/photos/man-delivering-speech-on-graduation-VV9VyibHYXI?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash"> Unsplash</a></figcaption></figure><p>These commencement speeches contain great life lessons from guests who resonate with most of us. Their insights may inspire and motivate graduates as they transition to their next phase in life. They may even inspire a different and new trajectory for them.</p><p>Once you have learned how to extract information from videos with Indexify, you can extract valuable information from more of such or other videos.</p><h3>The Challenge with Videos</h3><p>Extracting data from videos is challenging due to several factors:</p><ol><li><strong>Data Volume and Size:</strong> Video files are large, requiring substantial storage and bandwidth. Long, high-bitrate videos generate vast amounts of data that must be efficiently processed, stored, and managed.</li><li><strong>Multiple Modalities:</strong> Videos include visual, audio, and textual elements, each needing specialized extraction techniques and tools.</li><li><strong>Computational Intensity:</strong> Video processing, including decoding, object detection, speech recognition, and feature extraction, demands significant computational resources (CPU, GPU, memory), especially for real-time or near-real-time applications.</li><li><strong>Quality and Variability:</strong> Videos vary in quality, resolution, and frame rates, making consistent extraction performance difficult.</li></ol><p>Tools for video data extraction often struggle in production environments due to:</p><ol><li><strong>Dynamic Workloads:</strong> Video extraction requirements vary based on the video’s length, resolution, and complexity.</li><li><strong>Latency and Throughput:</strong> Autoscaling must quickly provide new resources to meet latency requirements, particularly for real-time processing, while maintaining high throughput despite potential bottlenecks in data pipelines, network bandwidth, and I/O operations.</li><li><strong>Tool Limitations:</strong> Many video processing tools are not designed for efficient large-scale or distributed processing, requiring custom solutions and optimizations for auto-scaling integration.</li></ol><h3>Extracting the Data</h3><p>Indexify consists of various components that make up its workflow. These components enable users and developers to upload multiple unstructured data files and update the vector indexes in real-time as the extraction process continues.</p><p>In this section, we will set up Indexify for video extraction.</p><h4>Starting the Indexify server</h4><p>Create a virtual environment.</p><p>python -m venv venv</p><p><strong>Activate the environment:</strong></p><p>source .venv/bin/activate</p><h4><strong>Install the </strong><strong>indexify-extractor-sdk and Indexify:</strong></h4><p>The extractor SDK allows new extraction capabilities and other readily available video extractors to be added.</p><pre>pip install indexify indexify-extractor-sdk</pre><p><strong>Download and run the Indexify server:</strong></p><p>Run:</p><pre>curl https://getindexify.ai | sh</pre><p>Start the server. This is a long-running process:</p><pre>./indexify server -d</pre><p>This is a long-running process. Open another terminal to install the required extractors and join them to the server.</p><h3>Downloading and running extractors</h3><p>We will download and join the required extractors to the server. Extractors consume content containing raw bytes of video data and produce a list of content and features from it. It then stores features like the embeddings into indexes for future retrieval.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*dPQe3_jxBpPrfsAUNJlQKw.png" /><figcaption>Image by author</figcaption></figure><p>We will need four extractors:</p><ol><li><strong>Video-audio extractor</strong>: tensorlake/audio-extractor. Extracts audio from the video, which is easier to handle.</li><li><strong>Audio extractor</strong>: tensorlake/whisper-asr. Creates transcripts from the audio extracted.</li><li><strong>Chunking extractor</strong>: tensorlake/chunk-extractor. Chunks the extracted transcripts from the whisper-asr extractor.</li><li><strong>Embedding extractor</strong>: tensorlake/minilm-l6. Creates embeddings of the chunks that are kept in storage for semantic search.</li></ol><p>Run the following one after the other to download each extractor:</p><pre>indexify-extractor download hub://video/audio-extractor<br>indexify-extractor download hub://audio/whisper-asr<br>indexify-extractor download hub://text/chunking<br>indexify-extractor download hub://embedding/minilm-l6</pre><p>Once they are downloaded, run them on the server. This is also a long-running process:</p><pre>indexify-extractor join-server</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*N55DWOl6K-VjhB5fusyWDg.png" /><figcaption><em>You can access the server UI at </em><a href="http://localhost:8900/ui"><em>http://localhost:8900/ui</em></a><em>.</em></figcaption></figure><h3>Building the script for loading video and initiating extraction using Indexify:</h3><p>Since we have the extractors downloaded and running, create a setup.py file that will:</p><ul><li>Download the video from YouTube.(“<a href="https://www.youtube.com/watch?v=UF8uR6Z6KLc&amp;t=6s">Steve Jobs’ Stanford Commencement Address in 2005</a>.”)</li><li>Upload the video to Indexify.</li><li>Initiate the extraction process through the declared extraction graph.</li></ul><p>We will use the pytube module to download videos from YouTube.</p><pre>pip install pytube</pre><p><strong>Import necessary modules:</strong></p><pre>import os<br>import yaml<br>from pytube import YouTube<br>from indexify import IndexifyClient, ExtractionGraph</pre><p><strong>Intiaite Indexify client:</strong></p><pre>client = IndexifyClient()</pre><p><strong>Create a function to download and load the YouTube video:</strong></p><p>This function enables an optional error-handling block while downloading the video.</p><pre>def download_youtube_video(url, output_file):<br>   &quot;&quot;&quot;<br>   Download the YouTube video if it does not already exist in the <br>current directory.<br>  <br>   Parameters:<br>   url (str): The URL of the YouTube video.<br>   output_file (str): The name of the output file.<br>   &quot;&quot;&quot;<br>   if os.path.exists(output_file):<br>       print(f&quot;The file &#39;{output_file}&#39; already exists. Skipping <br>             download.&quot;)<br>       return False<br><br>   try:<br>       yt = YouTube(url)<br>       video_stream = (yt.streams.first())<br><br>       if video_stream:<br>           print(f&quot;Downloading &#39;{output_file}&#39;...&quot;)<br>           video_stream.download(filename=output_file)<br>           print(&quot;Download complete!&quot;)<br>           return True<br>       else:<br>           print(&quot;Video not found.&quot;)<br>           return False<br>   except Exception as e:<br>       print(f&quot;An error occurred: {e}&quot;)<br>       return False</pre><p><strong>Create a </strong><strong>video_graph.yamlfile:</strong></p><p>The <strong>extraction graph</strong> is typically a data ingestion pipeline containing chained extraction policies that transform content through a series of extractors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*UAulKZBNtkbsCupjeN9rWg.png" /><figcaption>Image by author: Extraction process</figcaption></figure><pre>name: &quot;video2knowledgebase&quot;<br>extraction_policies:<br> - extractor: &quot;tensorlake/audio-extractor&quot;<br>   name: &quot;audio_extractor&quot;<br><br> - extractor: &quot;tensorlake/whisper-asr&quot;<br>   name: &quot;transcriber&quot;<br>   content_source: &quot;audio_extractor&quot;<br><br> - extractor: &quot;tensorlake/chunk-extractor&quot;<br>   name: &quot;transcripts_chunker&quot;<br>   input_params:<br>     chunk_size: 1000<br>     overlap: 250<br>   content_source: &quot;transcriber&quot;<br><br> - extractor: &quot;tensorlake/minilm-l6&quot;<br>   name: &quot;transcription_embedding&quot;<br>   content_source: &quot;transcripts_chunker&quot;</pre><p><strong>Create a function to upload and initiate extraction to Indexify:</strong></p><p>The function will upload the video to Indexify and initiate the extraction through the extraction graph.</p><p>Extraction takes time. In future, you can wait_for_extraction to complete, which will give a visualization of when the process is done.</p><pre>def extract_video(output_file, extraction_graph_spec):<br>   &quot;&quot;&quot;<br>   Upload the video file and initiate the extraction process.<br>  <br>   Parameters:<br>   output_file (str): The name of the output file.<br>   extraction_graph_spec: The extraction graph specification.<br>   &quot;&quot;&quot;<br>   try:      <br>       # Create an extraction graph from the YAML file<br>       extraction_graph = ExtractionGraph.from_yaml(<br>                                          extraction_graph_spec)<br>       client.create_extraction_graph(extraction_graph)<br>              <br>       print(&quot;Uploading the video file...&quot;)<br>       content_id = client.upload_file(&quot;video2knowledgebase&quot;, <br>                                       output_file)<br>       print(f&quot;File uploaded successfully. Content ID: <br>             {content_id}&quot;)<br><br>       print(&quot;Waiting for extraction to complete...May take <br>             time.&quot;)<br>       # client.wait_for_extraction(content_id)<br>   except Exception as e:<br>       print(f&quot;Error occurred during upload or extraction: {e}&quot;)</pre><p><strong>Execute the video download, upload and extraction:</strong></p><pre>if __name__ == &quot;__main__&quot;:<br>   <br>   url = &quot;https://www.youtube.com/watch?v=UF8uR6Z6KLc&amp;t=6s&quot;<br>   output_file = &quot;stevejobs_commencement_speech.mp4&quot;<br><br>   download_successful = download_youtube_video(url, output_file)<br><br>   # Read the .yaml file - extraction pipeline<br>   with open(&quot;video_graph.yaml&quot;, &quot;rb&quot;) as file:<br>       extraction_graph_spec = file.read()<br>  <br>   # upload the file and extract content to Indexify<br>   if download_successful or os.path.exists(output_file):<br>       upload_and_extract_video(output_file,<br>                                extraction_graph_spec)</pre><p>Finally, run setup.py.<strong>Output:</strong></p><pre>(venv) tl@tensorlake:~/.../IndexifyStart$ python setup.py <br>Downloading &#39;stevejobs_commencement_speech.mp4&#39;...<br>Download complete!<br>Uploading the video file...<br>File uploaded successfully. Content ID: 9ab2c9bc712c6171<br>Waiting for extraction to complete...Might take time.</pre><p><strong>Extraction Graph</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IAbCBYOHDQJ8MXdumfe3lA.png" /></figure><p><strong>Uploaded video:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/1*ioU4-ehSG0mDf_zj91VOAg.png" /></figure><p><strong>Indexes</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/1*jAcVQAKkN4jMRCyRjZn2oQ.png" /></figure><p><strong>Content:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*0JDpRe0M4BW5EmxsLqDT1Q.gif" /></figure><h3>Retrieving the Data</h3><p>Indexify offers a <strong>retrieval API</strong> that allows us to retrieve the stored indexes that match a certain query. The retrieved indexes can then be passed to an LLM plus a prompt to create a responsive question-answering system. An example of this is covered in <a href="https://docs.google.com/document/d/1_QsADOIXrqpVGPxH84y77xalmi6mdpPFSNfqhaviO78/edit#">Building LLM applications with LangChain and Indexify</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*5hhYc7PWTfE4Keqr2iLJlQ.png" /><figcaption>Image by author</figcaption></figure><p>We can make a call to the retriever API using client.search_index(query=index,query=&#39;question&#39;, topk=top_k).</p><p>The search results contain chunks of text that match the query and their corresponding confidence scores.</p><p>Create an app.pyfile and place the following code:</p><pre>from indexify import IndexifyClient<br><br>client = IndexifyClient()<br><br><br># embedding index<br>index = &quot;video2knowledgebase.transcription_embedding.embedding&quot;<br>query = &quot;stay foolish&quot; # query<br>results = client.search_index(name=index,<br>                              query=query,<br>                              top_k = 3)<br><br>print(&quot;\n-&quot;.join([item[&quot;text&quot;] for item in results]))</pre><p><strong>Output</strong>(truncated):</p><pre><br>signed off. Stay hungry, stay foolish. And I have always wished that for <br>myself. And now, as you graduate to begin anew, I wish that for you. <br>Stay hungry. Stay foolish. Thank you all very much. Thank you all. <br>Thank you all very much. Thank you. The proceeding program is copyrighted by <br>Stanford University. Please visit us at Stanford. EADDU.<br>-poetic touch...</pre><p>Retrieval from the UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*B00PLVrDk3WrNuZbSUCh8w.gif" /></figure><h3>Dealing With Multiple Video Data Extraction</h3><p>While many extraction tools struggle with video extraction workloads, especially with numerous data, Indexify auto scales impressively to production usage. Thanks to its <strong>declarative configuration approach, </strong>once the extraction pipeline is declared and set up, Indexify can wait for new videos or updates of those videos and continuously extract data from them. The extraction pipeline does not need to be changed.</p><p>In this section, we will query multiple videos on YouTube and extract data from them. This time, we will use<a href="https://console.cloud.google.com/"> Google Cloud Console</a> to get access to the YouTube API.</p><ul><li>Create a new project.</li><li>Enable the YouTube Data API for the project.</li><li>Generate an API key.</li></ul><p>Follow<a href="https://developers.google.com/youtube/v3/getting-started"> these instructions</a> to set up the console.</p><p>We will organize the functionality into simple easy to understand functions:</p><ul><li>We will use the extraction graph we defined previously. That is the pipeline that will process all our queried YouTube videos.</li><li>Create a data/ directory to store the downloaded videos.</li></ul><p>Next, create a setup.pyfile, which we will update with all the functionalities as follows:</p><p><strong>Create a function to read </strong><strong>.yaml file:</strong></p><pre>def read_yaml_file(yaml_file_path):<br>   &quot;&quot;&quot;Reads and returns the content of a YAML file.&quot;&quot;&quot;<br>   with open(yaml_file_path, &#39;r&#39;) as file:<br>       return file.read()</pre><p><strong>Create a function to create the extraction graph from the .yaml file:</strong></p><pre>def create_extraction_graph(yaml_content):<br>   &quot;&quot;&quot;Creates an extraction graph from YAML content.&quot;&quot;&quot;<br>   extraction_graph = ExtractionGraph.from_yaml(yaml_content)<br>   client.create_extraction_graph(extraction_graph)</pre><h3>Get video files from directory:</h3><p>Once the videos are queried and downloaded, this function will read them. We define the file extensions the function will get. YouTube videos are .MP4. If you have other video formats, you can include them in the list.</p><pre>def get_video_files(files_dir_path):<br>   &quot;&quot;&quot;Returns a list of video files in the directory.&quot;&quot;&quot;<br>   all_files = os.listdir(files_dir_path)<br>  <br>   video_extensions = [&#39;.mp4&#39;]# Can add other video extensions<br>   return [f for f in all_files if any(f.endswith(ext) for ext in <br>          video_extensions)]</pre><p><strong>Create a function to clean the filenames while uploading:</strong></p><p>Since we will be querying the video URLs using their titles, we must remove special characters from the filenames before uploading them.</p><pre>def sanitize_filename(filename):<br>   &quot;&quot;&quot;Remove any invalid characters from the filenames.&quot;&quot;&quot;<br>   return re.sub(r&#39;[\\/*?:&quot;&lt;&gt;|]&#39;, &quot;&quot;, filename)</pre><p><strong>Query YouTube videos by title:</strong></p><p>This function takes a YouTube API key and a list of video titles we want to download. For each title, it performs a search query to get the video ID of the first matching result.</p><p>It then constructs the video URL from the video ID and adds it to the list of URLs.</p><pre>def get_youtube_video_urls_by_title(api_key, video_titles):<br>   &quot;&quot;&quot;Get a list of YouTube video URLs based on video titles.&quot;&quot;&quot;<br>   youtube = build(&#39;youtube&#39;, &#39;v3&#39;, developerKey=api_key)<br>   video_urls = []<br><br>   for title in video_titles:<br>       request = youtube.search().list(<br>           q=title,<br>           part=&#39;snippet&#39;,<br>           type=&#39;video&#39;,<br>           maxResults=1  # get first result for each title<br>       )<br>       response = request.execute()<br><br>       for item in response[&#39;items&#39;]:<br>           video_id = item[&#39;id&#39;][&#39;videoId&#39;]<br>           video_url =   <br>                   f&#39;https://www.youtube.com/watch?v={video_id}&#39;<br>           video_urls.append(video_url)<br><br>   return video_urls</pre><p><strong>Download the YouTube videos:</strong></p><p>This function will download each video using pytube.It will use the video’s title as the filename.</p><pre>def download_youtube_videos(url_list, download_directory):<br>   &quot;&quot;&quot;Downloads YouTube videos from the list of queried URLs and saves them in the data directory.&quot;&quot;&quot;<br>   for url in url_list:<br>       yt = YouTube(url)<br>       # Sanitize the title to create a valid filename<br>       title = sanitize_filename(yt.title)<br>       video_stream = <br>                   yt.streams.filter(file_extension=&#39;mp4&#39;).first()<br>       file_name = f&quot;{title}.mp4&quot;<br>       video_stream.download(<br>           output_path=download_directory, filename=file_name)<br>       print(f&quot;Downloaded {file_name} from {url}&quot;)</pre><h3>Extract information from the videos:</h3><p>This function bundles up all the functions for extraction.</p><pre>def upload_files(yaml_file_path, files_directory_path):<br>   &quot;&quot;&quot;Uploads video files to Indeify using the specified YAML file.&quot;&quot;&quot;<br>   # Read and parse the YAML file<br>   yaml_content = read_yaml_file(yaml_file_path)<br><br>   # Create the extraction graph<br>   create_extraction_graph(yaml_content)<br><br>   # Get the lst of video files to upload<br>   video_files = get_video_files(files_dir_path)<br><br>   # Upload each video file in the list<br>   for file_name in video_files:<br>       file_path = os.path.join(files_directory_path, file_name)<br>       if os.path.exists(file_path):<br>           content_id = client.upload_file(&quot;Videoknowledgebase&quot;,<br>                                           path=file_path)<br>           print(<br>               f&quot;Successfully uploaded {file_path} with content ID: <br>                {content_id}&quot;)<br>       else:<br>           print(f&quot;Error: File {file_path} does not exist.&quot;)</pre><p><strong>Execute the extraction:</strong></p><pre>if __name__ == &quot;__main__&quot;:<br><br>   # Paths to the necessar directories and files<br>   yaml_file_path = &#39;/path/to/your/video_graph.yaml&#39;<br>   files_directory_path = &#39;/path/to/your/data_dir&#39;<br><br>   # YouTube Data API Key<br>   api_key = &#39;YOUR_YOUTUBE_DATA_API&#39;<br><br>   # Define the tiles of the videos you want<br>   video_titles = [<br>       &quot;Steve Jobs&#39; 2005 Stanford Commencement Address&quot;,<br>       “Tom Hanks delivers the Commencement Address | Harvard <br>        Commencement 2023”,<br>       “University of Texas at Austin 2014 Commencement Address - <br>        Admiral William H. McRaven”<br>   ]<br><br>   # Get the YouTube video URLs based on the video titles<br>   youtube_urls = get_youtube_video_urls_by_title(api_key,<br>                                                  video_titles)<br>   print(f&quot;Video URLs:\n{youtube_urls}&quot;)<br>   print()<br><br>   # Download the YouTube videos<br>   download_youtube_videos(youtube_urls, files_directory_path)<br><br>   # begin extraction<br>   upload_files(yaml_file_path, files_directory_path)</pre><p>Run setup.py. <strong>Output</strong>:</p><pre>Video URLs:<br>[&#39;https://www.youtube.com/watch?v=UF8uR6Z6KLc&#39;, &#39;https://www.youtube.com/watch?v=dkPo5nfEk1w&#39;, &#39;https://www.youtube.com/watch?v=pxBQLFLei70&#39;]<br><br>Downloaded Steve Jobs&#39; 2005 Stanford Commencement Address.mp4 from https://www.youtube.com/watch?v=UF8uR6Z6KLc<br><br>Downloaded Tom Hanks delivers the Commencement Address  Harvard Commencement 2023.mp4 from https://www.youtube.com/watch?v=dkPo5nfEk1w<br><br>Downloaded University of Texas at Austin 2014 Commencement Address - Admiral William H. McRaven.mp4 from https://www.youtube.com/watch?v=pxBQLFLei70<br><br>Successfully uploaded /Documents/PROJECTS/IndexifyStart/data/Tom Hanks delivers the Commencement Address  Harvard Commencement 2023.mp4 with content ID: 3d8b569e86fc5af5<br><br>Successfully uploaded /Documents/PROJECTS/IndexifyStart/data/University of Texas at Austin 2014 Commencement Address - Admiral William H. McRaven.mp4 with content ID: afe303fb8208b96c<br><br>Successfully uploaded /Documents/PROJECTS/IndexifyStart/data/Steve Jobs&#39; 2005 Stanford Commencement Address.mp4 with content ID: e8a43fca92230ab2<br>...</pre><p><strong>Extraction Graph tasks:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/949/1*PCxxImVTREytIJxaEFJfLw.png" /></figure><p><strong>Content:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*CoiJ4I4xduqjJC4EAuPg8A.gif" /><figcaption>Multiple video ingestion with Indexify</figcaption></figure><p>You can retrieve the extracted data for each document:</p><pre>content_ids = [&#39;afe303fb8208b96c&#39;, &#39;3d8b569e86fc5af5&#39;,<br>&#39;afe303fb8208b96c&#39;]<br>client.get_extracted_content(content_ids[0])</pre><p>You can also search the vector indexes and point a query to any of the extracted videos:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/949/1*_-vatoMqNrUfHZGWwenGUA.png" /></figure><pre>client.search_index(<br>      name =    <br>          &#39;videoknowledgebase.transcription_embedding.embedding&#39;, <br>      query = &#39;What is the first thing to do in the morning?&#39;, <br>      top_k=3)</pre><h3>Final Thoughts</h3><p>Indexify’s ability to extract data from multiple videos without any limitations of workloads presented by videos while maintaining high throughputs can significantly enhance your capacity to interpret and leverage video data effectively. This data can be used with LLMs to create LLM-based applications for <strong>question-answering tasks</strong>, <strong>summarization</strong>, <strong>video analysis</strong> and more.</p><p>This article has walked you through how to extract valuable data from a video using Indexify. As you have learned, it is very easy and quick to set up and start using Indexify. Furthermore, we have explored Indexify’s strengths to:</p><ul><li>Ingest multiple videos continuously.</li><li>Convert videos into audio.</li><li>Read audio files and transcribe them into text.</li><li>Split the transcribed text into smaller chunks.</li><li>Create embeddings of these chunks and keep them in storage for semantic search.</li><li>Ensure auto scalability with its practical declarative configuration approach.</li></ul><blockquote>Originally written as part of T<a href="https://tensorlake.ai/">ensorLake’s</a> campaign in collaboration with <a href="https://www.linkedin.com/company/argon-labs-xyz/posts/?feedView=all">Argon Labs</a>. The tooling has evolved since this was published, so be sure to check for the latest updates</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=09de811fef52" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to LangChain for Including AI from Large Language Models (LLMs) Inside Data…]]></title>
            <link>https://heartbeat.comet.ml/introduction-to-langchain-for-including-ai-from-large-language-models-llms-inside-data-116a6771b1df?source=rss-1e1c1cda0acd------2</link>
            <guid isPermaLink="false">https://medium.com/p/116a6771b1df</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[langchain]]></category>
            <dc:creator><![CDATA[Brian Mutea]]></dc:creator>
            <pubDate>Fri, 05 Jan 2024 23:02:15 GMT</pubDate>
            <atom:updated>2024-01-05T23:02:15.487Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction to LangChain for Including AI from Large Language Models (LLMs) Inside Data Applications and Data Pipelines</h3><h4>This article will provide an overview of LangChain, the problems it addresses, its use cases, and some of its limitations.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rTF2OUOIqVny3fSB" /><figcaption>Image by Author</figcaption></figure><p>Large Language Models (LLMs) entered the spotlight with the release of OpenAI’s GPT-3 in 2020. We have seen exploding interest in LLMs and in a broader discipline, Generative AI.</p><p>There have been great leaps in the LLMs space, from the introduction of Google’s <em>“sentient”</em> LaMDA chatbot, BLOOM, the first high-performance and <em>open-source</em> LLM, and the release of next-generation GPT-3.5 models by OpenAI. However, after all these milestones, the release of ChatGPT thrust the LLMs into the spotlight.</p><p><a href="https://docs.langchain.com/docs/"><strong>LangChain</strong></a> appeared around the same time as the <a href="https://openai.com/chatgpt">ChatGPT</a> release and was already packed with excellent tools for building tools based on LLMs. This article will provide an overview of the LangChain library, what it can do, the problems it solves, and its use cases.</p><p>In the following article, we will look at how we can build applications while leveraging various concepts of LangChain.</p><h3>Introduction to LangChain</h3><p>Langchain is an open-source framework that provides tools, components, and interfaces that simplify the development of LLM-powered applications. It has a collection of APIs that developers can embed in their applications, empowering them to infuse language processing capabilities without building everything from the ground up. LangChain efficiently simplifies the process of crafting LLM-based applications.</p><p>Applications like chatbots, Generative-Question-Answering (GQA), summarization, virtual assistants, and language translation utilities represent LLM-powered applications. Developers leverage LangChain to create tailored language model-based applications that cater to specific needs.</p><p>Its core idea is that we can <em>“chain”</em> together different components to create more refined LLM apps. Features include:</p><ul><li><strong>Models</strong>: LLMs, chat models, and text embedding models.</li><li><strong>Prompt</strong>s: Inputs to a model.</li><li><strong>Memory</strong>: Storing and retrieving data while conversing.</li><li><strong>Indexes</strong>: An interface for querying large datasets, enabling LLMs to interact with different document types for retrieval purposes.</li><li><strong>Chain</strong>s: Allows developers to combine components for advanced use cases and integration with other tools.</li><li><strong>Agents</strong>: Involves LLM decision-making, action execution, observation, and repetition until completion.</li><li><strong>Callbacks</strong>: Introduce examination and introspection as an additional layer of support.</li></ul><p>The following article explains each of these in more detail with code.</p><p>LangChain supports using <strong>two programming languages</strong>:</p><ul><li><strong>JavaScrip</strong>t: Great for embedding AI into web applications with the node package.</li><li><strong>Python</strong>: Great for including AI in Python-based software or data pipelines.</li></ul><h3>What Does LangChain Address?</h3><p>LangChain implements two major workflows for interacting with LLMs: chatting and embedding. These workflows have problems that LangChain aims to address. Let’s discuss some of them:</p><h4>Memory Constraints On LLMs</h4><p>LLMs generate responses based on previous conversations. However, these models have relatively short memory. For instance, GPT-4 has a memory constraint of 8000 tokens. That means that if the conversation exceeds the memory limit, the responses become inconsistent since it might lose track of the beginning of the conversation.</p><p>You want your applications, for instance, a chatbot, to remember the entire conversation with a customer. LangChain makes this possible by providing a <a href="https://python.langchain.com/docs/modules/memory/"><strong>chat memory tool</strong></a> that enables LLMs to reflect on previous interactions.</p><h4>Structured Response Formats</h4><p>When provided with a prompt, you may require a model to consistently generate a response in a specific format (e.g., CSV, JSON, datatime format, etc.) other than text. To handle the output requirement formats, LangChain provides <a href="https://python.langchain.com/docs/modules/model_io/output_parsers/"><strong>output parser</strong></a> classes that are responsible for that.</p><p>You will provide the parser with instructions on how the model’s output should be formatted. When presented with the model’s response, the parser will use these instructions to parse the response onto the specified structure.</p><h4>Prompting</h4><p>LLMs are trained on a simple concept — you input a text sequence, and the model outputs a sequence of text. In this case, the one crucial variable is the input sequence — the prompt.</p><p>With LLMs, prompts are vital. Bad prompts will generate poor responses, while good ones will generate exceptional ones; thus, constructing good ones can be problematic when working with LLMs. While prompting may involve defining the task to complete, it also includes determining the AI’s personality and writing style and including instructions to encourage factual precision.</p><p>LangChain recognizes the power of prompts and has built great classes just for that. A good prompt may consist of the following components:</p><ul><li><strong>Instructions</strong>: Instruct the model on what to do and how to structure the response.</li><li><strong>Context</strong>: Additional source of knowledge for the model, which can be manually given, pulled from APIs, retrieved from vector stores, or pulled from other sources.</li><li><strong>Query</strong>: A user input to the system by a human user.</li></ul><p>For instance, take the following prompt:</p><pre>prompt = &quot;&quot;&quot; Answer the question based on the context below. If the information provided cannot answer the question, respond with &quot;Can&#39;t find an answer.&quot;<br>Context: Langchain is an open-source framework that provides tools, components, and interfaces that simplify the development of LLM-powered applications. Its key features include Prompts, Memory, Indexes, Chains, Agents, and callbacks. LangChain is provided in two programming languages: Python and JavaScript.<br>Question: What are the main features of LangChain, and what programming languages can I implement these components with?<br>Answer: <br>&quot;&quot;&quot;</pre><p>Well, it’s unlikely that we will hardcode the context and the question in most cases. We would feed them through a <em>template</em>. LangChain makes this possible by providing <a href="https://python.langchain.com/docs/modules/model_io/prompts/prompt_templates/"><strong>prompt template classes</strong></a>. These classes are developed to make prompting with dynamic inputs easier. One such template class is the <a href="https://python.langchain.com/docs/modules/model_io/prompts/prompt_templates/#prompttemplate"><strong>PromptTemplate</strong></a> class.</p><h4>Switching Between LLM Models</h4><p>There are other LLM providers available aside from OpenAI. Building software on one provider or API can lock the software only onto that ecosystem. In the future, you may realize that the software requires more excellent capabilities to enhance your product. Switching to a more capable model can sometimes be a hustle.</p><p>However, LangChain provides the <a href="https://python.langchain.com/docs/modules/model_io/llms.html">LLM class</a> that provides an interface for interacting with many LLMs. This class makes an abstraction easier to swap between LLMs or use multiple LLMs in your software.</p><h4>Integration of LLMs Into Pipelines</h4><p>LangChain implements <a href="https://python.langchain.com/docs/modules/chains/">chains</a> and <a href="https://python.langchain.com/docs/modules/agents/">agents</a> that provide pipeline-type workflows. For instance, we may extract data from sources like databases, which we then pass into an LLM and send a processed output to another system.</p><p><strong>Chains</strong> are objects that wrap multiple individual components together. A chain integrates an LLM with a prompt, forming modules that can execute operations on our text or other datasets. These chains are designed to process multiple inputs simultaneously. The most commonly used chain is the <a href="https://python.langchain.com/docs/modules/chains/foundational/llm_chain">LLM Chain</a>.</p><p><strong>Agents</strong> are more sophisticated, allowing business reasoning to let you decide how the components should interact. Some applications will require a predetermined chain of calls to LLMs/other tools and potentially an unknown chain that depends on the user’s input. An agent has access to a suite of tools such that, depending on the user input, the agent can then decide which, if any, of these tools to call.</p><h4>Passing Data to LLMs</h4><p>LLMs are text-based, so it is not always clear how to pass data into the model. First, you may need to store the data in a particular format, like an SQL table or DataFrame, which lets you control the portions of data sent to the LLM. LangChain implements indexes that provide the functionality to retrieve data in various formats so the LLM can best interact with them.</p><p>Secondly, you can pass the data to a prompt through <strong>Prompt Stuffing</strong><em>, </em>the more straightforward way provided by LangChain’s <a href="https://python.langchain.com/docs/modules/data_connection/indexing">indexing</a> category<em>. </em>However, passing the dataset to the prompt as context is simple and efficient but only applicable when you have a small dataset.</p><p>Index-related chains have three more techniques apart from Prompt stuffing:</p><ul><li><strong>Map-reduce</strong>: Splits data into chunks, calls the LLM with an initial prompt on each chunk, then runs a different prompt to combine responses from the initial prompt.</li><li><strong>Refine</strong>: Runs an initial prompt on the first chunk of data, generating some output. Then, for each additional chunk of data, you use a prompt asking the LLM to refine the result based on the new dataset. It can be ideal when the output should converge on a specific output.</li><li><strong>Map-Rerank</strong>: Runs a prompt for each chunk of data and provides a confidence score for its response, which you can use to rank the outputs. That can be ideal for recommendation-like tasks.</li></ul><h3>Use Cases of LangChain</h3><p>LangChain offers a wide range of applications. It shines in some of the following use cases:</p><ul><li><strong>API Integration</strong>: LangChain can integrate with various APIs, enabling the development of applications that interact with other platforms and services. For instance, you may use an API to retrieve exchange rate data or to interact with a cloud platform. LangChain’s <strong><em>chains</em></strong> and <strong><em>agents</em></strong><em> </em>features enable you to create data pipelines for this use case.</li><li><strong>Chatbot Development</strong>: LangChain provides the necessary tools to build chatbots. Chatbots are one of the most popular AI use cases. They need to have a communication style and recall previous interactions/context during a conversation. LangChain <strong><em>prompt templates</em></strong> can give you control over a chatbot’s responses. Additionally, <strong><em>message history</em></strong> tools ensure conversation consistency by providing the chatbot with adequate memory.</li><li><strong>Intelligent Question Answering</strong>: We can use LangChain to develop question-answering applications based on specific documents.</li><li><strong>Creating Interactive Agents</strong>: Developers can create agents that interact with users, make decisions based on their input, and continue their tasks until completion. These agents can be helpful in customer service, data collection, and other interactive applications.</li><li><strong>Data Summarization</strong>: LangChain can create applications that summarize long documents. This feature can help process lengthy texts, articles, or reports.</li><li><strong>Document Retrieval and Clustering:</strong> LangChain can simplify retrieval and clustering using embedding models. That can also be important when grouping similar documents or retrieving specific documents based on certain criteria.</li></ul><h3>What Are the Limitations of LangChain?</h3><p>LangChain offers exceptional capabilities but has some limitations:</p><ul><li><strong>Constraint to train only on text data</strong>: It does not support training on non-textual data. That may hinder its application in domains where data extends beyond text.</li><li><strong>Lacks support for multi-task learning</strong>: Simultaneously training language models on multiple related tasks improves performance. Without this technique, the efficiency and effectiveness of specific applications may not benefit from leveraging multi-task learning approaches.</li><li><strong>Extra steps to deploying LangChain-based models</strong>: LangChain does not provide straightforward deployment methods, which may cause problems when deploying such applications in production environments.</li></ul><h3>Final Thoughts</h3><p>This article has introduced LangChain in a relatively basic manner. You have learned about LangChain and some of its use cases. Despite its limitations, LangChain is a robust framework that developers can use to harness the capabilities of language models. With its vast benefits and ongoing development, LangChain holds excellent assurance for the future of AI-powered solutions.</p><p><em>Editor’s Note: </em><a href="https://heartbeat.comet.ml/"><em>Heartbeat</em></a><em> is a contributor-driven online publication and community dedicated to providing premier educational resources for data science, machine learning, and deep learning practitioners. We’re committed to supporting and inspiring developers and engineers from all walks of life.</em></p><p><em>Editorially independent, Heartbeat is sponsored and published by </em><a href="http://comet.com/?utm_source=heartbeat&amp;utm_medium=referral&amp;utm_campaign=AMS_US_EN_AWA_heartbeat_EN"><em>Comet,</em></a><em> an MLOps platform that enables data scientists &amp; ML teams to track, compare, explain, &amp; optimize their experiments. We pay our contributors, and we don’t sell ads.</em></p><p><em>If you’d like to contribute, head on over to our</em><a href="https://heartbeat.comet.ml/call-for-contributors-fee7f5b80f3e"><em> call for contributors</em></a><em>. You can also sign up to receive our weekly newsletter (</em><a href="https://www.deeplearningweekly.com/"><em>Deep Learning Weekly</em></a><em>), check out the </em><a href="https://www.comet.com/site/blog/"><em>Comet blog</em></a><em>, join us on</em><a href="https://join.slack.com/t/fritz-ai-community/shared_invite/enQtNTY5NDM2MTQwMTgwLWU4ZDEwNTAxYWE2YjIxZDllMTcxMWE4MGFhNDk5Y2QwNTcxYzEyNWZmZWEwMzE4NTFkOWY2NTM0OGQwYjM5Y2U"><em> </em></a><a href="https://join.slack.com/t/cometml/shared_invite/zt-49v4zxxz-qHcTeyrMEzqZc5lQb9hgvw"><em>Slack</em></a><em>, and follow Comet on </em><a href="https://twitter.com/Cometml"><em>Twitter</em></a><em> and </em><a href="https://www.linkedin.com/company/comet-ml/"><em>LinkedIn</em></a><em> for resources, events, and much more that will help you build better ML models, faster.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=116a6771b1df" width="1" height="1" alt=""><hr><p><a href="https://heartbeat.comet.ml/introduction-to-langchain-for-including-ai-from-large-language-models-llms-inside-data-116a6771b1df">Introduction to LangChain for Including AI from Large Language Models (LLMs) Inside Data…</a> was originally published in <a href="https://heartbeat.comet.ml/">Heartbeat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Image Captioning Model with TensorFlow, Transformers, and Kangas for Image Visualization]]></title>
            <link>https://heartbeat.comet.ml/image-captioning-model-with-tensorflow-transformers-and-kangas-for-image-visualization-e2063b95eb8b?source=rss-1e1c1cda0acd------2</link>
            <guid isPermaLink="false">https://medium.com/p/e2063b95eb8b</guid>
            <category><![CDATA[image-captioning]]></category>
            <category><![CDATA[transformer-model]]></category>
            <dc:creator><![CDATA[Brian Mutea]]></dc:creator>
            <pubDate>Mon, 11 Dec 2023 18:03:15 GMT</pubDate>
            <atom:updated>2023-12-11T18:03:15.473Z</atom:updated>
            <content:encoded><![CDATA[<h4>Build an image captioning model with TensorFlow and transformer encoder-decoder architecture and apply Kangas for image visualizations.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7tQcn_GPZzHb7fj22o7xDg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/es/@monicadear?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Monica Flores</a> on <a href="https://unsplash.com/photos/p4mFOzM-asQ?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Image captioning is a compelling field that connects computer vision and natural language processing, enabling machines to generate textual descriptions of visual content. In an era dominated by visual content, the ability of machines to understand and describe images is a powerful stride towards human-like intelligence.</p><p>This article will explore image captioning using <strong>TensorFlow</strong>. We will explore the process of training a model to generate descriptive captions for images, highlighting the critical steps involved. The model leverages an <strong>Encoder </strong>and <strong>Decoder </strong>based on the <strong>Transformer </strong>architecture as covered in &quot;<a href="https://arxiv.org/abs/1706.03762"><em>Attention is all you need</em></a>,&quot; so some knowledge can come in handy. Still, we will implement them here for understanding.</p><p>Also, please acquaint yourself with <a href="https://github.com/comet-ml/kangas"><strong>Kangas</strong></a>, as we will use it <strong>to visualize image data in this article</strong>. Below are resources to get you started:</p><ul><li><a href="https://heartbeat.comet.ml/create-and-visualize-image-data-with-kangas-for-computer-vision-tasks-83fdad794d"><strong><em>Creating and visualizing image data with Kangas.</em></strong></a></li><li><a href="https://heartbeat.comet.ml/constructing-and-visualizing-kangas-datagrid-on-kangas-ui-f63d4350ab61"><strong><em>Constructing and visualizing Kangas DataGrid on Kangas UI.</em></strong></a></li></ul><p>You can follow along on <a href="https://colab.research.google.com/drive/1BYMUagWjbIV4yJbtzlkm7ZvaBN5CJ2x9">this notebook</a>.</p><h3>What Exactly Is An Image Captioning Model?</h3><p>An image captioning model is a model that can effectively generate a descriptive sentence based on the contents of a particular image.</p><p>In recent years, image captioning has improved tremendously, fueled by the advancements in machine translation, where the encoder and decoder can generate more coherent sentences. Such progress comes from the introduction of Transformer encoder and decoder models, which have remarkably improved performance compared to traditional RNN-based encoder and decoder models.</p><p>A perfect image captioning model should:</p><ul><li><strong>Understand the context of a given image.</strong></li><li><strong>Accurately represent that understanding as a textual description.</strong></li></ul><p>For instance, given the following image, the model should be able to produce acceptable captions describing the contents of the image. The captions should be good since various interpretations of the same image can exist.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*3TK0eLFiWH9fBqc51zBVoQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@glomadmarketing?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Glomad Marketing</a> on <a href="https://unsplash.com/photos/6VQlKJp2vpo?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>The captions for the above image can be:</p><ul><li><em>A cat and a dog are sleeping on the floor</em>.</li><li><em>A black cat is resting on a brown dog.</em></li><li><em>A cat and a dog are resting at the garage entrance</em>.</li></ul><h3>Approach to Creating the Model</h3><p>The model is inspired by <a href="https://keras.io/examples/vision/image_captioning/">Implementing an image captioning model using a CNN and a Transformer</a> and <a href="https://www.tensorflow.org/tutorials/text/image_captioning">Image captioning with visual attention on TensorFlo</a>w. Some of the processes we will undertake:</p><ul><li>Source a dataset that has image, caption pairs.</li><li>Visualize the dataset with Kangas to see its representation.</li><li>Preprocess the images and captions.</li><li>Resizing the images for pixel consistency through the model.</li><li>Using a pre-trained CNN model to obtain image features.</li><li>Create a Transformer encoder and decoder.</li><li>Training the model.</li><li>Generating captions using the trained model.</li><li>Visualizing the model&#39;s accuracy and loss.</li></ul><h3>The Dataset</h3><p>There are several datasets available for image captioning tasks:</p><ul><li><a href="https://www.kaggle.com/datasets/adityajn105/flickr8k">Flickr8k</a>: Has a little above 8k images paired with their respective captions.</li><li><a href="https://www.kaggle.com/datasets/adityajn105/flickr30k">Flickr30k</a>: Has over 30k images paired with their respective captions.</li><li><a href="https://www.kaggle.com/datasets/sabahesaraki/2017-2017">MSCOCO</a>: Have over 160k images paired with their respective captions.</li></ul><p>These datasets have been widely used and are reliable in learning or building the image captioning model. We will stick with the <a href="https://www.kaggle.com/datasets/adityajn105/flickr8k"><strong>Flickr8k dataset</strong></a> as it is more convenient for a broader range of audiences with inadequate resources for preparing and training more complicated datasets.</p><p><strong>Download </strong>the dataset, and let&#39;s get started!</p><pre>%pip install opendatasets # to help download data directly from Kaggle</pre><pre>import opendatasets as od<br><br># download<br># Kaggle API key required<br>od.download(&quot;https://www.kaggle.com/datasets/adityajn105/flickr8k&quot;)</pre><h3>Visualizing the Dataset With Kangas</h3><p>Kangas comes in handy for visualizing multimedia data. Unlike Pandas, Kangas comes packed with an effortless and straightforward way of visualizing image data (<strong>Kangas UI</strong>), and we do not have to rely on other libraries and packages to do so. I have provided the well-structured resources above to help you get started quickly.</p><p>First, <strong>install Kangas</strong>:</p><p>%pip install kangas</p><p>Next, <strong>import Kangas</strong> with an alias &quot;<em>kg</em>&quot;:</p><p>import kangas as kg</p><p>The base structure of Kangas is a DataGrid. However, we will first read the data as a Pandas DataFrame to process and add a column, after which we will read the DataFrame with Kangas to get the DataGrid.</p><p>Read the data. I am using Google Colab, hence the paths:</p><pre>captions_file = &#39;/content/flickr8k/captions.txt&#39; <br>df_captioned = pd.read_csv(captions_file)<br><br># Add actual image path <br>df_captioned[&#39;image&#39;] = df_captioned[&#39;image&#39;].apply(<br>lambda x: f&#39;/content/flickr8k/Images/{x}&#39;)<br><br># Rename the &#39;image&#39; column to &#39;image_path&#39;<br>df_captioned.rename({&#39;image&#39;:&#39;image_path&#39;}, axis=1, inplace=True)<br>df_captioned.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*Y1UUn0j58z8-oYsegyOm5Q.png" /><figcaption>DataFrame with image paths and caption</figcaption></figure><p>To visualize the images in Kangas, we need to convert the images to <strong>Kangas image assets</strong> with Image() or convert them to Pillow images(PIL).</p><pre># convert the images from the image paths<br># to Kangas image assets<br>images= df_captioned[&#39;image_path&#39;].map(<br>    lambda x: kg.Image(x)<br>)<br><br># Add a new column with the image assets(actual images)<br>df_captioned.insert(loc=1, column=&#39;image&#39;, value=images)<br><br>df_captioned.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zQSBSCSlTNYyLn-xxqXWgg.png" /><figcaption>DataFrame with Kangas Image assets</figcaption></figure><p>Let&#39;s visualize some of the images with Kangas:</p><pre>def viewRandomImages(samples=1):<br>  random_rows = df_captioned.sample(samples) #random images<br><br>  for idx, row in random_rows.iterrows():<br>    # view with Kangas<br>    image = kg.Image(row[&#39;image_path&#39;])<br>    image.show()<br>    print(&#39;\n&#39;, row[&#39;caption&#39;],&#39;\n&#39;)</pre><pre>viewRandomImages(2) #view two images with captions</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/1*E7b-lTtEoPrqOOw9qEEAYQ.png" /><figcaption>Images with captions. Viewed with Kangas</figcaption></figure><p>Kangas can read data in various formats into a DataGrid. Since we have the DataFrame, we will use Kangas&#39;s read_dataframe() method to return a DataGrid. The best part of Kangas is the <strong>interactive Kangas UI</strong>. Instead of visualizing them individually, the UI creates a central place to view the images.</p><pre># view a shuffled DataGrid<br>dg_captioned = kg.read_dataframe(df_captioned.sample(frac=1))<br><br># The dg.show() method to fire up the UI<br>dg_captioned.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Hknek5yuS_pmvkphuhX7Q.png" /><figcaption>Image Captioning data on Kangas UI</figcaption></figure><p>You can see that each image has a corresponding caption. On the UI, you can click on any image to view/zoom/apply grayscale, sort, or group the data as you wish to explore.</p><p>For instance, we can <strong>view the data</strong> <strong>without the &quot;<em>image_path</em>&quot; column</strong>. Just click on the &quot;<em>columns</em>&quot; tab and remove the row.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4V6Gef5xuRwucN-PoHCxgA.png" /><figcaption>Removed the &quot;<em>image_path</em>&quot; column: Kangas UI</figcaption></figure><p>Perfect! Now that you have visualized how the data has been represented, it is time to create the model. But let&#39;s first import all the libraries we will require.</p><pre>import pandas as pd<br>import numpy as np<br>import matplotlib.pyplot as plt<br>import kangas as kg<br><br>import re<br>import tensorflow<br>import tensorflow as tf<br>from tensorflow import keras<br>from tensorflow.keras import layers<br>from tensorflow.keras.layers import TextVectorization<br><br>from tensorflow.keras.applications import efficientnet #Image feature extractor</pre><h3>Preparing the Dataset</h3><p>The first step in building any model is converting the data into a carefully curated dataset to suit the model requirements before training. We require a paired dataset with images and their respective captions for an image captioning dataset.</p><p>Looking at the captions.txt file:</p><pre>with open(captions_file) as caption_data:<br>  caption_data = caption_data.readlines()<br>  for data in caption_data[20:23]:<br>    print(data)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*mz8L6w_ShSW1lXYrEEWaiw.png" /><figcaption>Some data from captions.txt</figcaption></figure><p>You notice that commas separate each image from its corresponding caption. Our goal is to separate the two entities.</p><p>Since we know that each image in the dataset has at least five captions to choose from, we will <strong>create a dictionary that maps each image (</strong>as keys<strong>) to its corresponding captions (</strong>as values<strong>)</strong>. Also, for better consistency and model training, we will<strong> filter out the captions that are too short and those that are too long </strong>(marked as outliers) by predefining a sequence length.</p><p>If you are familiar with sequence-to-sequence tasks like machine translation, <strong>adding the start and end tokens</strong> to the captions will not surprise you. The start and end tokens act as explicit delimiters to the beginning and end of a sequence, thus helping the model identify the boundaries of the input sequence during training and inference.</p><pre>def load_captions(caption_filename):<br><br>  with open(captions_file) as caption_data:<br>    caption_data = caption_data.readlines()<br><br>    mapping_dict = {} # dict to store image to caption mapping<br>    text_data = [] # stores a list of preocessed captions<br>    outlier_imgs = set()<br><br>    for line in caption_data:<br>      line = line.strip(&#39;\n&#39;).split(&#39;,&#39;) # split image and caption at the commas<br>      image_codeName, caption = line[0], line[1]<br>      image_name = os.path.join(image_paths, image_codeName)# create full path to image<br><br>      caption_tokens = caption.strip().split() # create tokens<br><br>      # filter the images using the caption lengths<br>      if len(caption_tokens) &lt; 5 or len(caption_tokens) &gt; sequenceLength:<br>        outlier_imgs.add(image_name)<br>        continue<br><br>      # get all .jpg images<br>      # add START and END tokens to each caption<br>      # convert the captions to lowercase<br>      if image_name.endswith(&#39;.jpg&#39;) and image_name not in outlier_imgs:<br>        caption = &quot;&lt;START&gt; &quot; + caption.strip().lower() + &quot; &lt;END&gt;&quot;<br>        text_data.append(caption)<br><br>        if image_name in mapping_dict:<br>            mapping_dict[image_name].append(caption)<br>        else:<br>            mapping_dict[image_name] = [caption]<br><br>    for image_name in outlier_imgs:<br>      if image_name in mapping_dict:<br>        del mapping_dict[image_name]<br><br><br>    return mapping_dict, text_data</pre><p>mapping_dict contains images (keys) mapped to their captions( values) while the text_data has all the preprocessed captions.</p><pre># mapped images to their caption<br>mapping_dict, text_data = load_captions(captions_file)<br><br>list(mapping_dict.keys())[:2], list(mapping_dict.values())[:2]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*o3tZAIsTVKcAYsHI2TI7Uw.png" /><figcaption>Images and captions</figcaption></figure><p>Let&#39;s see the captions of one of the images:</p><pre>mapping_dict[&#39;/content/flickr8k/Images/1000268201_693b08cb0e.jpg&#39;]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*R6QlB3CM6TlmNmrcKEHIdg.png" /><figcaption>Some captions for a single image in the mapping_dict</figcaption></figure><p>Each image is mapped to five corresponding captions.</p><h4>Split the Data Into Training and Validation Sets</h4><p>We will split the captioning data into two separate dictionaries for the training and validation data.</p><pre>def train_val_split(caption_data, train_sample=0.8):<br><br>  images = list(caption_data.keys()) # gather all images<br><br>  train_sample = int(len(caption_data) * train_sample) # split<br><br>  training_set = {<br>      image_name: caption_data[image_name] for image_name in images[:train_sample]<br>  }<br>  validation_set = {<br>      image_name: caption_data[image_name] for image_name in images[train_sample:]<br>  }<br><br>  return training_set, validation_set</pre><pre>training_set, validation_set = train_val_split(mapping_dict)<br>print(f&quot;Training data: {len(training_set)}\nValidation data: {len(validation_set)}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/303/1*b4CAzmGtLAgfTLEoVdiJLQ.png" /><figcaption>Training and validation data</figcaption></figure><h4>Vectorizing the Data</h4><p>To feed the data into the model, we need to vectorize it. That means that we need to convert the strings into integer sequences where each integer represents the index of a word in a vocabulary. TensorFlow provides the <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/TextVectorization">TextVectorization</a> layer for this.</p><p>The layer learns the vocabulary from the captions through the adapt() method. The adapt() The method iterates over all captions, splits them into words, checks the frequency of each string value in the caption, and computes a vocabulary of their most frequently used words.</p><pre>VOCAB_SIZE = 10000<br><br>def standardization(input):<br>    lowercase = tf.strings.lower(input)<br>    return tf.strings.regex_replace(lowercase, &quot;[%s]&quot; % re.escape(strip_chars), &quot;&quot;)<br><br><br>strip_chars = &quot;!\&quot;#$%&amp;&#39;()*+,-./:;&lt;=&gt;?@[\]^_`{|}~&quot;<br>strip_chars = strip_chars.replace(&quot;&lt;&quot;, &quot;&quot;)<br>strip_chars = strip_chars.replace(&quot;&gt;&quot;, &quot;&quot;)<br><br>vectorization = TextVectorization(<br>    max_tokens=VOCAB_SIZE,<br>    output_mode=&quot;int&quot;,<br>    output_sequence_length=sequenceLength,<br>    standardize=standardization,<br>)<br>vectorization.adapt(text_data)</pre><p>We can check some vocabulary that has been computed after vectorization.</p><pre># Get some vocabulary<br>print(vectorization.get_vocabulary()[:15])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/555/1*EtyE1auJAb9tBSOQS4RjDQ.png" /><figcaption>Vocabulary examples from the vectorization layer</figcaption></figure><p>Let&#39;s apply vectorization to some data to see the output sequences.</p><pre>vectorizer = vectorization([[&#39;a dog sleeping under a tree&#39;], [&#39;a bird feeding small chicks&#39;]])<br>vectorizer</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*SIKHzduo7OwtRtrIVcSqcA.png" /><figcaption>Example Integer sequences from vectorization layer</figcaption></figure><h4>Create the tf.data.Dataset Pipeline</h4><p>At this point, we need to transform, preprocess, and prepare the training and validation data for model training. We do this by creating a pipeline using the tf.data.Dataset API. With the pipeline, we can:</p><ul><li>Shuffle the dataset.</li><li>Tokenize all captions for each image through the vectorization layer.</li><li>Map the images to their respective captions.</li></ul><p>In addition, we will create a function that will load each image and resize it to a fixed size for the model. That ensures that the same number of pixels represents all the images.</p><pre>IMAGE_SIZE = (299, 299)<br>BATCH_SIZE = 64<br>EPOCHS = 30<br>AUTOTUNE = tf.data.AUTOTUNE<br><br># load and resize each image to IMAGE_SIZE<br>def decode_and_resize(image_path):<br><br>  image = tf.io.read_file(image_path)<br>  image = tf.image.decode_jpeg(image, channels=3)<br>  image = tf.image.resize(image, IMAGE_SIZE)<br>  image = tf.image.convert_image_dtype(image, tf.float32)<br>  return image<br><br># map each resized image to respective vectorized captions<br>def process_input(img_path, captions):<br>  return decode_and_resize(img_path), vectorization(captions)<br><br># Function defining the transformation pipeline<br>def make_dataset(images, captions):<br>  dataset = tf.data.Dataset.from_tensor_slices((images, captions))<br>  dataset = dataset.shuffle(BATCH_SIZE * 8)<br>  dataset = dataset.map(process_input, num_parallel_calls=AUTOTUNE)<br>  dataset = dataset.batch(BATCH_SIZE).prefetch(AUTOTUNE)<br>  return dataset</pre><pre># create transformed training and validation data<br>training_data = make_dataset(list(training_set.keys()), list(training_set.values()))<br><br>validation_data = make_dataset(list(validation_set.keys()), list(validation_set.values()))</pre><h3>Building the Model</h3><p>The model will consist of three parts:</p><ul><li>An image feature extractor.</li><li>The Transformer-based Encoder.</li><li>The Transformer-based Decoder.</li></ul><h4>Image Feature Extractor</h4><p>We will use an image model to extract features from each image. The model is pre-trained on ImageNet as an image classification model. However, in this case, we don&#39;t need the classification layer but the last layer with feature maps. We will use the Keras <a href="https://www.tensorflow.org/api_docs/python/tf/keras/applications/efficientnet/EfficientNetB0">EfficientNetB0</a> model.</p><p>Let&#39;s take a look at the model results:</p><pre>img_path = list(training_set.keys())[1]<br><br>model = efficientnet.EfficientNetB0(<br>      input_shape=(*IMAGE_SIZE, 3),<br>      include_top=False, weights = &#39;imagenet&#39;,<br>  )<br><br>test_img_batch = decode_and_resize(img_path)[tf.newaxis, :]<br>print(test_img_batch.shape)<br>print(model(test_img_batch).shape)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/270/1*jC7AtjKfwxd5wqlSJRc33g.png" /><figcaption>Feature map from <a href="https://www.tensorflow.org/api_docs/python/tf/keras/applications/efficientnet/EfficientNetB0">EfficientNetB0</a> model</figcaption></figure><p>The feature extractor returns a feature map for each model.</p><p>Based on this model, we will create a new Convolutional Neural Network (CNN) Keras model for feature extraction. The CNN model will take as input the input tensor of feature maps from the EfficientNetB0 model.</p><pre>def get_cnn_model():<br>  <br>  # include_top = False: return model without the <br>  # classification layer<br>  model = efficientnet.EfficientNetB0(<br>      input_shape=(*IMAGE_SIZE, 3), include_top=False, weights = &#39;imagenet&#39;,<br><br>  )<br><br>  model.trainable = False<br>  model_out = model.output<br>  model_out = layers.Reshape((-1, model_out.shape[-1]))(model_out)<br>  cnn_model = keras.models.Model(model.input, model_out)<br>  return cnn_model</pre><p>Next, we build a Transformer-based Encoder and Decoder.</p><p>Earlier sequence-to-sequence models implemented Recurrent Neural Networks (RNNs) like LSTM and GRU. The input sequence fed into those models was encoded into a fixed-length representation with information about the input sequence for output sequence generation. However, the fixed-length representations often posed limitations where the input sequence was too long and contained crucial information at different positions.</p><p>To fix that problem, an attention mechanism was added to enable the RNN models to focus on more relevant parts of the input sequence during the decoding process. So, instead of relying solely on the fixed-length representations, the attention mechanism calculates attention weights for each input position and computes a weighted sum of the input sequence&#39;s encoder outputs. This weighted sum, often called the &quot;attention context,&quot; is an additional input to the decoder at each decoding step. However, the RNNs suffered from parallelism since they decoded one token at a time, making the model train slower, especially on long input sequences.</p><p>In this article, we implement the Transformer architecture for encoder and decoder. It is similar to the RNN model with attention, but the main difference is that Transformers entirely replace RNNs with an attention mechanism. That makes them parallelizable, and computations can happen simultaneously. Layer outputs can be computed in parallel instead of one at a time, like in RNNs.</p><p>To learn more about how Transformers work, you can read:</p><ul><li>&quot;<a href="https://arxiv.org/abs/1706.03762"><em>Attention is all you need</em></a>&quot; paper.</li><li><a href="https://jalammar.github.io/illustrated-transformer/">Illustrated Transformer</a> by Jay Alammar.</li></ul><h4>The Transformer-Based Encoder</h4><p>We will pass the image features we have extracted as inputs to an encoder to generate new representations. The inputs first go through a self-attention layer. The layer creates three vectors (query, key, and value vectors), calculated by multiplying the embedding by the matrices from the training process. The self-attention layer adds MultiHeadAttention to enable the model to focus on different positions.</p><p>The self-attention layer can add variation in outputs. Adding layer normalization helps normalize the outputs to make them compatible with the original inputs (residue connection), which allows the preservation of important information and gradients.</p><pre>class Encoder(keras.layers.Layer):<br>  def __init__(self, embedding_dim, dense_dim, num_heads):<br>    super().__init__()<br>    self.embedding_dim = embedding_dim<br>    self.dense_dim = dense_dim<br>    self.num_heads = num_heads<br><br>    # Create the attention layer<br>    self.attention = keras.layers.MultiHeadAttention(<br>        num_heads = num_heads, key_dim=embedding_dim, dropout=0.0<br>    )<br><br>    # Layer normalization<br>    self.layernorm1 = layers.LayerNormalization()<br>    self.layernorm2 = layers.LayerNormalization()<br><br>    self.dense = layers.Dense(embedding_dim, activation=&#39;relu&#39;)<br><br>    def call(self, inputs, training, mask=None):<br>      inputs = self.layernorm1(inputs)<br>      inputs = self.dense(inputs)<br><br>      attention_output = self.attention(<br>          query = inputs,<br>          value = inputs,<br>          keys = inputs,<br>          attention_mask = None,<br>          training = training<br>      )<br>      <br>      # residue connecttion<br>      # add actual inputs and self attention outputs<br>      # normalize them<br>      out = self.layernorm2(inputs + attention_output)<br>      return out</pre><h4>Positional Embedding Layer</h4><p>Transformers do not have an inherent knowledge of order or position like RNNs. They would take the input sequence as Bag of Words, which may be indistinguishable. So before passing the image features as inputs to the encoder, we need to convert them into token embeddings and add positional information to each token. By doing so, the model can effectively encode both the content and the position of tokens in the input sequence, enabling it to capture positional relationships and dependencies in the data.</p><p>Below, we create two embedding layers for token embedding and one for positional embedding. The token embedding layer maps the tokens to dense vectors, while the positional embedding layer maps positions within the sequence of dense vectors.</p><pre>class PositionalEmbedding(keras.layers.Layer):<br>  def __init__(self, seq_length, vocab_size, embedding_dim):<br>    super().__init__()<br>    self.token_embeddings = layers.Embedding(<br>        input_dim=vocab_size, output_dim=embedding_dim<br>    )<br>    self.position_embeddings = layers.Embedding(<br>        input_dim=seq_length, output_dim=embedding_dim<br>    )<br>    self.seq_length = seq_length<br>    self.vocab_size = vocab_size<br>    self.embedding_dim = embedding_dim<br>    self.embed_scale = tf.math.sqrt(tf.cast(embedding_dim, tf.float32))<br><br>  def call(self, inputs):<br>    length = tf.shape(inputs)[-1]<br>    positions = tf.range(start=0, limit=length, delta=1)<br>    embedded_tokens = self.token_embeddings(inputs)<br>    embedded_tokens = embedded_tokens * self.embed_scale<br>    embedded_positions = self.position_embeddings(positions)<br>    return embedded_tokens + embedded_positions<br><br>  def compute_mask(self, inputs, mask=None):<br>    return tf.math.not_equal(inputs, 0)</pre><h4>The Transformer-Based Decoder</h4><p>The decoder is more complex to implement. It generates the output one by one while consulting the representation generated by the encoder. Like in an encoder, the decoder has a positional embedding layer and stack of layers.</p><p>The output of the top encoder is transformed into a set of attention vectors used in the &quot;encoder-decoder attention&quot; layer, enabling the decoder to focus on appropriate places in the input sequence. The decoder&#39;s self-attention layer can only attend to earlier positions in the output sequence. That is done by masking future positions before the softmax step in the self-attention calculation.</p><pre>class Decoder(keras.layers.Layer):<br>    @classmethod<br>      def add_method(cls, func):<br>        setattr(cls, func.__name__, func)<br>        return func<br><br>    def __init__(self, embedding_dim, ff_dim, num_heads):<br>        super().__init__()<br>        self.embedding_dim = embedding_dim<br>        self.ff_dim = ff_dim<br>        self.num_heads = num_heads<br>        self.attention1 = layers.MultiHeadAttention(<br>            num_heads=num_heads, key_dim=embedding_dim, dropout=0.1<br>        )<br>        self.attention2 = layers.MultiHeadAttention(<br>            num_heads=num_heads, key_dim=embedding_dim, dropout=0.1<br>        )<br>        self.ffn_layer1 = layers.Dense(ff_dim, activation=&quot;relu&quot;)<br>        self.ffn_layer2 = layers.Dense(embedding_dim)<br><br>        self.layernorm1 = layers.LayerNormalization()<br>        self.layernorm2 = layers.LayerNormalization()<br>        self.layernorm3 = layers.LayerNormalization()<br><br>        self.embedding = PositionalEmbedding(<br>              embedding_dim=512, seq_length=sequenceLength, vocab_size=VOCAB_SIZE<br>          )<br>        self.out = layers.Dense(VOCAB_SIZE, activation=&quot;softmax&quot;)<br><br>        self.dropout1 = layers.Dropout(0.3)<br>        self.dropout2 = layers.Dropout(0.5)<br>        self.supports_masking = True<br><br>      def call(self, inputs, encoder_outputs, training, mask=None):<br>        inputs = self.embedding(inputs)<br>        causal_mask = self.get_causal_attention_mask(inputs)<br><br>        if mask is not None:<br>            padding_mask = tf.cast(mask[:, :, tf.newaxis], dtype=tf.int32)<br>            combined_mask = tf.cast(mask[:, tf.newaxis, :], dtype=tf.int32)<br>            combined_mask = tf.minimum(combined_mask, causal_mask)<br><br>        attention_output1 = self.attention1(<br>            query=inputs,<br>            value=inputs,<br>            key=inputs,<br>            attention_mask=combined_mask,<br>            training=training,<br>        )<br>        out1 = self.layernorm1(inputs + attention_output1)<br><br>        attention_output2 = self.attention2(<br>            query=out1,<br>            value=encoder_outputs,<br>            key=encoder_outputs,<br>            attention_mask=padding_mask,<br>            training=training,<br>        )<br>        out2 = self.layernorm2(out1 + attention_output2)<br><br>        ffn_out = self.ffn_layer1(out2)<br>        ffn_out = self.dropout1(ffn_out, training=training)<br>        ffn_out = self.ffn_layer2(ffn_out)<br><br>        ffn_out = self.layernorm3(ffn_out + out2, training=training)<br>        ffn_out = self.dropout2(ffn_out, training=training)<br>        preds = self.out(ffn_out)<br>        return preds</pre><p>Below, we write a method to generate a causal attention mask for the self-attention mechanism in a decoder layer. The causal attention mask ensures that each token can only attend to its previous positions and itself during self-attention, preventing information flow from future positions to past positions.</p><pre>@Decoder.add_method<br>def get_causal_attention_mask(self, inputs):<br>        input_shape = tf.shape(inputs)<br>        batch_size, sequence_length = input_shape[0], input_shape[1]<br>        i = tf.range(sequence_length)[:, tf.newaxis] #(sequence_length, 1)<br>        j = tf.range(sequence_length) #(sequence_length,)<br><br>        #create the causal attention mask<br>        mask = tf.cast(i &gt;= j, dtype=&quot;int32&quot;)<br>        mask = tf.reshape(mask, (1, input_shape[1], input_shape[1]))<br>        mult = tf.concat(<br>            [tf.expand_dims(batch_size, -1), tf.constant([1, 1], dtype=tf.int32)],<br>            axis=0,<br>        )<br>        return tf.tile(mask, mult)</pre><h3>The Model</h3><p>In this section, we build the captioning model. The model combines the feature extractor from the CNN model (cnn_model method), the encoder, and the decoder to generate the captions for images. When we call the model for training, it should receive the image, caption pairs.</p><p>The model also calculates the loss and the average accuracy (by comparing the true labels and the predicted labels).</p><pre>class ImageCaptioningModel(keras.Model):<br>  def __init__(<br>      self, cnn_model,<br>      encoder, decoder,<br>      num_captions_per_image=5<br>  ):<br><br>      super().__init__()<br>      self.cnn_model = cnn_model<br>      self.encoder = encoder<br>      self.decoder = decoder<br>      self.loss_tracker = keras.metrics.Mean(name=&quot;loss&quot;)<br>      self.acc_tracker = keras.metrics.Mean(name=&quot;accuracy&quot;)<br>      self.num_captions_per_image = num_captions_per_image<br>      self.image_aug = image_aug<br><br>  def calculate_loss(self, y_true, y_pred, mask):<br>    loss = self.loss(y_true, y_pred)<br>    mask = tf.cast(mask, dtype=loss.dtype)<br>    loss *= mask<br>    return tf.reduce_sum(loss) / tf.reduce_sum(mask)<br><br>  def calculate_accuracy(self, y_true, y_pred, mask):<br>    accuracy = tf.equal(y_true, tf.argmax(y_pred, axis=2))<br>    accuracy = tf.math.logical_and(mask, accuracy)<br>    accuracy = tf.cast(accuracy, dtype=tf.float32)<br>    mask = tf.cast(mask, dtype=tf.float32)<br>    return tf.reduce_sum(accuracy) / tf.reduce_sum(mask)<br><br>  def _compute_caption_loss_and_acc(self, img_embed, batch_seq, training=True):<br>    encoder_out = self.encoder(img_embed, training=training)<br>    batch_seq_inp = batch_seq[:, :-1]<br>    batch_seq_true = batch_seq[:, 1:]<br>    mask = tf.math.not_equal(batch_seq_true, 0)<br>    batch_seq_pred = self.decoder(<br>        batch_seq_inp, encoder_out, training=training, mask=mask<br>    )<br>    loss = self.calculate_loss(batch_seq_true, batch_seq_pred, mask)<br>    acc = self.calculate_accuracy(batch_seq_true, batch_seq_pred, mask)<br>    return loss, acc<br><br>  def train_step(self, batch_data):<br>    batch_img, batch_seq = batch_data<br>    batch_loss = 0<br>    batch_acc = 0<br><br>    if self.image_aug:<br>        batch_img = self.image_aug(batch_img)<br><br>    # 1. Get image embeddings<br>    img_embed = self.cnn_model(batch_img)<br><br>    # 2. Pass each of the five captions one by one to the decoder<br>    # along with the encoder outputs and compute the loss as well as accuracy<br>    # for each caption.<br>    for i in range(self.num_captions_per_image):<br>        with tf.GradientTape() as tape:<br>            loss, acc = self._compute_caption_loss_and_acc(<br>                img_embed, batch_seq[:, i, :], training=True<br>            )<br><br>            # 3. Update loss and accuracy<br>            batch_loss += loss<br>            batch_acc += acc<br><br>        # 4. Get the list of all the trainable weights<br>        train_vars = (<br>            self.encoder.trainable_variables + self.decoder.trainable_variables<br>        )<br><br>        # 5. Get the gradients<br>        grads = tape.gradient(loss, train_vars)<br><br>        # 6. Update the trainable weights<br>        self.optimizer.apply_gradients(zip(grads, train_vars))<br><br>    # 7. Update the trackers<br>    batch_acc /= float(self.num_captions_per_image)<br>    self.loss_tracker.update_state(batch_loss)<br>    self.acc_tracker.update_state(batch_acc)<br><br>    # 8. Return the loss and accuracy values<br>    return {&quot;loss&quot;: self.loss_tracker.result(), &quot;acc&quot;: self.acc_tracker.result()}<br><br>  def test_step(self, batch_data):<br>    batch_img, batch_seq = batch_data<br>    batch_loss = 0<br>    batch_acc = 0<br><br>    # 1. Get image embeddings<br>    img_embed = self.cnn_model(batch_img)<br><br>    # 2. Pass each of the five captions one by one to the decoder<br>    # along with the encoder outputs and compute the loss as well as accuracy<br>    # for each caption.<br>    for i in range(self.num_captions_per_image):<br>        loss, acc = self._compute_caption_loss_and_acc(<br>            img_embed, batch_seq[:, i, :], training=False<br>        )<br><br>        # 3. Update batch loss and batch accuracy<br>        batch_loss += loss<br>        batch_acc += acc<br><br>    batch_acc /= float(self.num_captions_per_image)<br><br>    # 4. Update the trackers<br>    self.loss_tracker.update_state(batch_loss)<br>    self.acc_tracker.update_state(batch_acc)<br><br>    # 5. Return the loss and accuracy values<br>    return {&quot;loss&quot;: self.loss_tracker.result(), &quot;acc&quot;: self.acc_tracker.result()}<br><br>  @property<br>  def metrics(self):<br>    # We need to list our metrics here so the `reset_states()` can be<br>    # called automatically.<br>    return [self.loss_tracker, self.acc_tracker]</pre><pre>cnn_model = get_cnn_model()<br>encoder = Encoder(embedding_dim=512, dense_dim=512, num_heads=1)<br>decoder = Decoder(embedding_dim=512, ff_dim=512, num_heads=2)<br>caption_model = ImageCaptioningModel(<br>    cnn_model=cnn_model, <br>    encoder=encoder, <br>    decoder=decoder<br>)</pre><h4>Train the Model</h4><p>Since we have successfully implemented the model architecture, it is time to train it on the training data. We will monitor the model&#39;s validation loss to gauge its performance. We do this by defining an EarlyStopping callback, which will stop the training if the model does not improve for three consecutive epochs (the model is overfitting).</p><pre># Define the loss function<br>cross_entropy = keras.losses.SparseCategoricalCrossentropy(<br>    from_logits=False, reduction=&quot;none&quot;<br>)<br><br># EarlyStopping criteria<br>early_stopping = keras.callbacks.EarlyStopping(<br>    patience=3, <br>    restore_best_weights=True<br>    )<br><br># Compile the model<br>caption_model.compile(<br>    optimizer=keras.optimizers.Adam(learning_rate=1e-4), <br>    loss=cross_entropy)<br><br># Fit the model<br>caption_model.fit(<br>    training_data,<br>    epochs=EPOCHS,<br>    validation_data=validation_data,<br>    callbacks=[early_stopping],<br>)</pre><p>The accuracies and the losses at each training epoch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*7yN9bNbekWFnoi8jmupEJQ.png" /><figcaption>Model training</figcaption></figure><h3>Generating Captions</h3><p>Finally, it&#39;s time to predict captions for images using the trained Image captioning model. To caption an image with this model:</p><ul><li>Retrieve vocabulary from the training step and map each token position back to their corresponding words in the vocabulary.</li><li>We will select a random image and its image features from the CNN model.</li><li>Pass the image features to the encoder for encoding.</li><li>Create a caption generation loop that generates tokens from &lt;start&gt; of the caption until the maximum decoded sentence length is reached or the end token &lt;end&gt; is generated.<br>- Tokenize each caption with the vectorization layer.<br>- Use the decoder to predict the next token in the sequence based on the encoded image features.</li></ul><p>The model uses the Kangas Image() class to view each randomly selected image.</p><p>So, let&#39;s add a &quot;simple&quot; method to do just that:</p><pre>vocab = vectorization.get_vocabulary()<br>index_lookup = dict(zip(range(len(vocab)), vocab))<br>max_decoded_sentence_length = sequenceLength - 1<br>valid_images = list(validation_set.keys())<br><br><br>def generate_caption():<br>    # Select a random image from the validation dataset<br>    sample_img = np.random.choice(valid_images)<br><br>    # Read the image from the disk<br>    sample_img = decode_and_resize(sample_img)<br>    img = sample_img.numpy().clip(0, 255).astype(np.uint8)<br>    kg.Image(img).show()<br><br>    # Pass the image to the CNN<br>    img = tf.expand_dims(sample_img, 0)<br>    img = caption_model.cnn_model(img)<br><br>    # Pass the image features to the Transformer encoder<br>    encoded_img = caption_model.encoder(img, training=False)<br><br>    # Generate the caption using the Transformer decoder<br>    decoded_caption = &quot;&lt;start&gt; &quot;<br>    for i in range(max_decoded_sentence_length):<br>        tokenized_caption = vectorization([decoded_caption])[:, :-1]<br>        mask = tf.math.not_equal(tokenized_caption, 0)<br>        predictions = caption_model.decoder(<br>            tokenized_caption, encoded_img, training=False, mask=mask<br>        )<br>        sampled_token_index = np.argmax(predictions[0, i, :])<br>        sampled_token = index_lookup[sampled_token_index]<br>        if sampled_token == &quot;&lt;end&gt;&quot;:<br>            break<br>        decoded_caption += &quot; &quot; + sampled_token<br><br>    decoded_caption = decoded_caption.replace(&quot;&lt;start&gt; &quot;, &quot;&quot;)<br>    decoded_caption = decoded_caption.replace(&quot; &lt;end&gt;&quot;, &quot;&quot;).strip().capitalize()<br>    print(&quot;PREDICTED CAPTION: &quot;, decoded_caption)<br><br><br># Check predictions for a few samples<br>generate_caption()<br>generate_caption()<br>generate_caption()</pre><p>Predicted captions for each randomly selected image. We have displayed each image with Kangas.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*YfZFgqxuwR_dF2x71cj99A.png" /><figcaption>Predicted captions for random images: Image Captioning mode.</figcaption></figure><p>Perfect!</p><h4>Visualize the Loss and Accuracy</h4><pre>plt.plot(caption_model.history.history[&#39;loss&#39;], label=&#39;loss&#39;)<br>plt.plot(caption_model.history.history[&#39;val_loss&#39;], label=&#39;val_loss&#39;)<br>plt.ylim([0, max(plt.ylim())])<br>plt.xlabel(&#39;Epochs&#39;)<br>plt.ylabel(&#39;CE/token&#39;)<br>plt.legend()</pre><p>Loss:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*PttjYUgfn6wNZOGSlZqEiQ.png" /><figcaption>Model loss</figcaption></figure><p>Accuracy:</p><pre>plt.plot(caption_model.history.history[&#39;val_acc&#39;], label=&#39;val_accuracy&#39;)<br>plt.plot(caption_model.history.history[&#39;acc&#39;], label=&#39;accuracy&#39;)<br>plt.ylim([0, max(plt.ylim())])<br>plt.xlabel(&#39;Epochs&#39;)<br>plt.ylabel(&#39;CE/token&#39;)<br>plt.legend()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*WAFVSpcZvbebOhAWyr6oPg.png" /><figcaption>Model accuracy</figcaption></figure><h3>Final Thoughts</h3><p>In this piece, we have learned to generate image captions with TensorFlow and Transformer based encoder and decoder. We have learned:</p><ul><li>How to visualize image data with Kangas and using the Kangas UI.</li><li>How to preprocess image and caption data for proper model compatibility.</li><li>How to create an image captioning model.</li></ul><p><em>Editor&#39;s Note: </em><a href="https://heartbeat.comet.ml/"><em>Heartbeat</em></a><em> is a contributor-driven online publication and community dedicated to providing premier educational resources for data science, machine learning, and deep learning practitioners. We&#39;re committed to supporting and inspiring developers and engineers from all walks of life.</em></p><p><em>Editorially independent, Heartbeat is sponsored and published by </em><a href="http://comet.com/?utm_source=heartbeat&amp;utm_medium=referral&amp;utm_campaign=AMS_US_EN_AWA_heartbeat_EN"><em>Comet,</em></a><em> an MLOps platform that enables data scientists &amp; ML teams to track, compare, explain, &amp; optimize their experiments. We pay our contributors, and we don&#39;t sell ads.</em></p><p><em>If you&#39;d like to contribute, head on over to our</em><a href="https://heartbeat.comet.ml/call-for-contributors-fee7f5b80f3e"><em> call for contributors</em></a><em>. You can also sign up to receive our weekly newsletter (</em><a href="https://www.deeplearningweekly.com/"><em>Deep Learning Weekly</em></a><em>), check out the </em><a href="https://www.comet.com/site/blog/"><em>Comet blog</em></a><em>, join us on</em><a href="https://join.slack.com/t/fritz-ai-community/shared_invite/enQtNTY5NDM2MTQwMTgwLWU4ZDEwNTAxYWE2YjIxZDllMTcxMWE4MGFhNDk5Y2QwNTcxYzEyNWZmZWEwMzE4NTFkOWY2NTM0OGQwYjM5Y2U"><em> </em></a><a href="https://join.slack.com/t/cometml/shared_invite/zt-49v4zxxz-qHcTeyrMEzqZc5lQb9hgvw"><em>Slack</em></a><em>, and follow Comet on </em><a href="https://twitter.com/Cometml"><em>Twitter</em></a><em> and </em><a href="https://www.linkedin.com/company/comet-ml/"><em>LinkedIn</em></a><em> for resources, events, and much more that will help you build better ML models, faster.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e2063b95eb8b" width="1" height="1" alt=""><hr><p><a href="https://heartbeat.comet.ml/image-captioning-model-with-tensorflow-transformers-and-kangas-for-image-visualization-e2063b95eb8b">Image Captioning Model with TensorFlow, Transformers, and Kangas for Image Visualization</a> was originally published in <a href="https://heartbeat.comet.ml/">Heartbeat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Create and visualize image data with Kangas for computer vision tasks]]></title>
            <link>https://heartbeat.comet.ml/create-and-visualize-image-data-with-kangas-for-computer-vision-tasks-83fdad794d?source=rss-1e1c1cda0acd------2</link>
            <guid isPermaLink="false">https://medium.com/p/83fdad794d</guid>
            <category><![CDATA[kanga]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[heartbeat]]></category>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Brian Mutea]]></dc:creator>
            <pubDate>Wed, 24 May 2023 16:07:29 GMT</pubDate>
            <atom:updated>2023-05-24T16:07:29.532Z</atom:updated>
            <content:encoded><![CDATA[<h4>Create DataGrids with image data using Kangas, and load and visualize image data from hugging face</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rE3bf23O8Ulf4szSw0-EMg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@genny_taylor?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Genny Dimitrakopoulou</a> on <a href="https://unsplash.com/photos/owiIxBvzjBo?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Visualizing data to carry out a detailed EDA, especially for image data, is critical. In <strong>computer vision</strong> datasets, if we can view and compare the images across different views with their relevant metadata and transformations within a single and well-designed UI, we are one step ahead in solving a CV task. We</p><p>In this article, we will explore the following:</p><ul><li>Creating DataGrid with images.</li><li>Adding image metadata.</li><li>Adding bounding boxes</li><li>Adding image masks</li><li>Loading and visualizing image data from hugging face.</li></ul><h3>Creating DataGrid with images.</h3><p>There are various ways we can construct a Kangas DataGrid. Check out this tutorial on<a href="https://heartbeat.comet.ml/constructing-and-visualizing-kangas-datagrid-on-kangas-ui-f63d4350ab61"> Constructing and visualizing Kangas DataGrid on Kangas UI</a>.</p><p>We can store images on a DataGrid using the Kangas Image() class or as Pillow images(PIL).</p><h4>Loading images with Image() class</h4><p>Here we will use an image of a cat and load it using the Image() class.</p><pre>image = kg.Image(&#39;/content/cats-dogs/cat1.jpeg&#39;)<br>print(&#39;TYPE OF IMAGE: &#39;, type(image))<br>print()<br>image.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/1*elu5dcdt5I2n5boUvm8Iow.png" /><figcaption>Store Image with Kangas Image() class</figcaption></figure><h4>Load image as Pillow (PIL) image with Kangas</h4><p>Kangas provides a to_pil() method that converts the image into a Pillow image.</p><pre>image = kg.Image(&#39;/content/cats-dogs/cat1.jpeg&#39;).to_pil()<br>print(&#39;TYPE OF IMAGE: &#39;, type(image))<br>print()<br>image.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*b8s5NyWclmO76X5YpVJdXA.png" /><figcaption>Store Image as PIL image with Kangas</figcaption></figure><p>The image data passed into the Image() class can take a variety of formats. Let’s try to load an image as a NumPy matrix and a TensorFlow Tensor. View more formats on the Creating an Image Asset</p><p><a href="https://github.com/comet-ml/kangas/wiki/Image#__init__">See other formats</a>.</p><pre># Image<br>image = kg.Image(&#39;/content/cats-dogs/cat1.jpeg&#39;).to_pil()</pre><p>Image data as NumPy matrix:</p><pre>from numpy import asarray<br><br>numpy_img_data = asarray(image)<br>print(f&quot;Type of data: {type(numpy_img_data)} data:  of Shape: {numpy_img_data.shape}&quot;)<br><br>print(&#39;TYPE OF IMAGE: &#39;, type(image))<br>print()<br>image.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*bkgsPfoN2wCcnfr1OP79SQ.png" /></figure><p>Image data as TensorFlow tensor:</p><pre>import tensorflow as tf<br><br>tensor_img_data = tf.convert_to_tensor(image)<br>print(f&quot;Type of data: {type(tensor_img_data)} data:  of Shape: {tensor_img_data.shape}&quot;)<br><br>print(&#39;TYPE OF IMAGE: &#39;, type(image))<br>print()<br>image.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*7g1RnD2-QGkgfr9pTNOIpg.png" /></figure><h4>Adding images to a DataGrid</h4><p>We can load the image to a DataGrid and visualize it with the UI.</p><pre>image = kg.Image(&#39;/content/cats-dogs/cat1.jpeg&#39;, name=&#39;cat.jpeg&#39;) # provide name to be used as filename in metadata<br>dg = kg.DataGrid(name=&#39;Cat&#39;) # provide name to be used as the datagrid name<br>dg.extend([{&#39;image&#39;: image}])<br><br># Fire up the UI<br>dg.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*oa4yxs40cvKYxnd2qHWwjQ.png" /><figcaption>Kangas UI</figcaption></figure><pre>import glob<br><br>#instantiate DataGrid<br>dg = DataGrid(name=&quot;Images&quot;, columns=[&quot;image_file_path&quot;, &quot;image&quot;, &#39;height&#39;, &#39;width&#39;, &quot;label&quot;])<br><br># append data to DataGrid<br>for file in glob.iglob(&quot;/*/*/*.jpeg&quot;):<br>  file_name = os.path.basename(file) # file_name to metadata<br>  image = Image.open(file)<br>  image_w = image.width<br>  image_h = image.height<br>  label = os.path.splitext(file_name)[0][:3]<br>  dg.append([file, kg.Image(file, name=file_name), image_w, image_h, label])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*433aiR1E04WgKHFTH4p2tA.png" /><figcaption>Kangas UI: Image data</figcaption></figure><h3>Adding Image metadata</h3><p>When we create images, we can specify more data describing various attributes of the images we are working with. By default, Kangas logs out the name(as filename) and theassetId.</p><p><em>We can visualize this on the UI. Click on the image, and a pop window will appear. Locate the “Metadata” section and toggle the dropdown.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*JlgJjoGIX8f7mBpCUPi5CA.png" /><figcaption>Image Metadata</figcaption></figure><p><strong>Note </strong>that the extension value is derived from the filename value. If the filename(name argument)is not provided, the value would be the file path.</p><p>The <a href="https://github.com/comet-ml/kangas/wiki/Image#__init__">Image() class</a> provides more optional arguments that we can provide to give more information about the image.</p><p>Image annotations like bounding boxes and masks are included as metadata.</p><h4>Adding bounding boxes to image with Kangas</h4><p>In object detection tasks, bounding boxes are essential as they help us determine the spatial location of a particular object.</p><ul><li>The bounding box can be represented by the x and y coordinates of the upper-left and lower-right corners of the rectangle.</li><li>We can also represent the bounding box with the (x, y)-axis coordinates of the bounding box center and the width and height of the box.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*WOkBhqQjEX7UMNejqXxagQ.png" /><figcaption>Two ways to represent a bounding box</figcaption></figure><p>The Kangas <a href="https://github.com/comet-ml/kangas/wiki/Image#add_bounding_boxes">add_bounding_boxes()</a> method can receive either of the above representations. We can pass the following to the method:</p><ul><li><strong>label</strong>: a label for the boxes.</li><li><strong>boxes</strong>: list or tuples of exactly 2 points (top-left, bottom-right) or 4 ints (x, y, width, height).</li><li><strong>score</strong>: the score relevant to the bounding box.</li><li><strong>layer_name</strong>: the layer for the label and bounding boxes.</li><li><strong>id</strong>: box ids.</li></ul><pre>img2_aug = kg.Image(&#39;/content/cat_dog.jpg&#39;)<br><br>box1 = [(40, 65), (210, 280)]<br>box2 = [(222, 23),(448, 281)]<br>imgs_bbox = img2_aug.add_bounding_boxes(&quot;Cat&quot;, box1, score=0.89, layer_name=&quot;Truth&quot;, id=&#39;cat01&#39;)<br>imgs_bbox = img2_aug.add_bounding_boxes(&quot;Dog&quot;, box2, score=0.89, layer_name=&quot;Truth&quot;, id=&#39;dog01&#39;)<br><br>img_dg = kg.DataGrid(name=&#39;cat_dog&#39;)<br>img_dg.extend([[imgs_bbox]])<br><br>img_dg.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H68e5aGm6V8MTVGv5J8JAg.gif" /><figcaption>Kangas UI: Adding bounding boxes</figcaption></figure><h4>Image Masking with Kangas</h4><p>Image masking lets us focus on the parts of the image we are most interested in. For instance, if we build a computer vision model to recognize faces, we can mask the rest of the image and focus on the actual faces.</p><p>While we can do image masking in various ways, for example, using OpenCV, Kangas has the add_mask() method we can use when creating an image.</p><p>With this method, a mask can be:</p><ul><li>A 2D array or np.array with int values.</li><li>An image filename.</li><li>A Pillow or kangas.Image image.</li></ul><pre>image = kg.Image(&#39;/content/cat_dog.jpg&#39;)<br>mask = PIL.Image.open(&#39;/content/mask_truth.png&#39;)<br>image.add_mask({1: &#39;Cat_vs_dog&#39;}, mask, layer_name=&#39;Truth&#39;)<br>dg = kg.DataGrid(name=&quot;model1&quot;)<br>dg.append([image])<br>dg.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rd6hOfoAnlakVT1GPQS_Iw.png" /><figcaption>Adding mask: Kangas UI</figcaption></figure><h3>Loading and visualizing image data from hugging face.</h3><p>We can visualize and apply built-in functions from Kangas UI on <a href="https://huggingface.co/datasets">hugging face datasets</a> with Kangas. Hugging Face contains a collection of datasets we can use to build various models.</p><p>In this section, we will see how to create DataGrid from hugging face datasets and visualize them on the Kangas UI. We will grab the <a href="https://huggingface.co/datasets/keremberke/license-plate-object-detection">license plate object detection dataset</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rCB1CZKXlhK3Mn9xzrh7Kg.png" /><figcaption>License plate object detection dataset from hugging face</figcaption></figure><p>First, import the datasets library to enable us to grab the dataset:</p><pre>%pip install datasets</pre><p>Next, import the following:</p><pre>import datasets<br>from datasets import load_datasets, list_datasets<br></pre><p>For the simplicity of this article, we will only split 20 examples from the training set of the dataset.</p><pre>license_dataset = load_dataset(&#39;keremberke/license-plate-object-detection&#39;, &#39;full&#39;, split=&#39;train[:20]&#39;)<br>license_dataset</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/1*vvqQdi_jiaQ6uBPBoYb6Iw.png" /></figure><p>We have 5 features, and we have split 20 examples.<br>Let’s load the datasets into a Kangas DataGrid and visualize it in the UI:</p><pre>dg_obj_detection = kg.DataGrid(licence_plate_dataset, name=&#39;Licence_plate_obj_detection&#39;)<br><br>dg_obj_detection.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*DjEXVvjaoAdtND9C8P4-Ag.png" /><figcaption>License plate object detection dataset: Kangas UI</figcaption></figure><p>The image annotations are stored as dictionary objects in the ‘<em>objects’ </em>column. Let’s see the annotations of the first image:</p><pre>annotations = license_dataset[0][&#39;objects&#39;]<br><br>annotations, type(annotations)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/1*FCZK9J_rXI6kuhH2eG5YwA.png" /><figcaption>image annotations</figcaption></figure><p>We can also view this clearly by clicking the object column of any of the images:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r64nG6KxGz5msoBbodaXkA.gif" /><figcaption>Image annotations: Kangas UI</figcaption></figure><h3>Final thoughts</h3><p>This article taught us how to create and visualize image data on the Kangas UI. We can visualize image data more efficiently, quickly, and with few lines of code without complicated functions and libraries.</p><p>While this is an overview, there is still more that we can do with Kangas. Be sure to explore more on the <a href="https://github.com/comet-ml/kangas/tree/main/examples/hugging_face/object_detection">Kangas repo</a>.</p><p><em>Editor’s Note: </em><a href="https://heartbeat.comet.ml/"><em>Heartbeat</em></a><em> is a contributor-driven online publication and community dedicated to providing premier educational resources for data science, machine learning, and deep learning practitioners. We’re committed to supporting and inspiring developers and engineers from all walks of life.</em></p><p><em>Editorially independent, Heartbeat is sponsored and published by </em><a href="http://comet.com/?utm_source=heartbeat&amp;utm_medium=referral&amp;utm_campaign=AMS_US_EN_AWA_heartbeat_EN"><em>Comet,</em></a><em> an MLOps platform that enables data scientists &amp; ML teams to track, compare, explain, &amp; optimize their experiments. We pay our contributors, and we don’t sell ads.</em></p><p><em>If you’d like to contribute, head on over to our</em><a href="https://heartbeat.comet.ml/call-for-contributors-fee7f5b80f3e"><em> call for contributors</em></a><em>. You can also sign up to receive our weekly newsletter (</em><a href="https://www.deeplearningweekly.com/"><em>Deep Learning Weekly</em></a><em>), check out the </em><a href="https://www.comet.com/site/blog/"><em>Comet blog</em></a><em>, join us on</em><a href="https://join.slack.com/t/fritz-ai-community/shared_invite/enQtNTY5NDM2MTQwMTgwLWU4ZDEwNTAxYWE2YjIxZDllMTcxMWE4MGFhNDk5Y2QwNTcxYzEyNWZmZWEwMzE4NTFkOWY2NTM0OGQwYjM5Y2U"><em> </em></a><a href="https://join.slack.com/t/cometml/shared_invite/zt-49v4zxxz-qHcTeyrMEzqZc5lQb9hgvw"><em>Slack</em></a><em>, and follow Comet on </em><a href="https://twitter.com/Cometml"><em>Twitter</em></a><em> and </em><a href="https://www.linkedin.com/company/comet-ml/"><em>LinkedIn</em></a><em> for resources, events, and much more that will help you build better ML models, faster.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83fdad794d" width="1" height="1" alt=""><hr><p><a href="https://heartbeat.comet.ml/create-and-visualize-image-data-with-kangas-for-computer-vision-tasks-83fdad794d">Create and visualize image data with Kangas for computer vision tasks</a> was originally published in <a href="https://heartbeat.comet.ml/">Heartbeat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Constructing and Visualizing Kangas DataGrid on Kangas UI]]></title>
            <link>https://heartbeat.comet.ml/constructing-and-visualizing-kangas-datagrid-on-kangas-ui-f63d4350ab61?source=rss-1e1c1cda0acd------2</link>
            <guid isPermaLink="false">https://medium.com/p/f63d4350ab61</guid>
            <category><![CDATA[kanga]]></category>
            <category><![CDATA[comet]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[datagrid]]></category>
            <category><![CDATA[heartbeat]]></category>
            <dc:creator><![CDATA[Brian Mutea]]></dc:creator>
            <pubDate>Mon, 03 Apr 2023 12:11:59 GMT</pubDate>
            <atom:updated>2023-04-03T12:11:59.251Z</atom:updated>
            <content:encoded><![CDATA[<h4>Pandas can make tasks like grouping, filtering, and sorting a total nightmare — Instead try Kangas for large, complex queries!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*dyu3e2TV1Y_X3NpYVqUQqA.png" /><figcaption>Source: <a href="https://github.com/comet-ml/kangas">Kangas</a></figcaption></figure><p>Exploring datasets becomes more cumbersome as the dataset grows. When the data gets too huge, performance and analysis of the data with tools like Pandas, which stores its DataFrames in memory, are affected. Performing tasks like <em>grouping</em>, <em>filtering</em>, and <em>sorting </em>against the huge data become a nightmare. But what if there was a way this could be made easier?</p><p>Well, this article, will give you an <strong>in-depth base knowledge</strong> to <strong>get started with </strong><a href="https://github.com/comet-ml/kangas"><strong>Kangas</strong></a>. Kangas is an open-source project geared towards redefining how we explore and visualize our data while making it extremely easy to explore these huge datasets.</p><p>While Pandas stores DataFrames in memory, Kangas uses DataGrids, which are actual SQLite databases. This means that with <a href="https://github.com/comet-ml/kangas">Kangas</a>, our data (a DataGrid) is stored in a database giving it the power to store huge amounts of data and perform complex queries quickly. An interactive <a href="https://kangas.comet.com/?datagrid=/data/coco-500.datagrid"><strong>User Interface</strong></a> also comes with it that enables visualizing these data.</p><p>So, let’s dive into how to:</p><ul><li>Create <a href="https://github.com/comet-ml/kangas">Kangas </a>DataGrid from scratch.</li><li>Add/append/extend data into them.</li><li>Perform basic Pandas-like operations on the DataGrid.</li><li>Save DataGrid in various formats.</li><li>Visualize the resulting DataGrid on <a href="https://kangas.comet.com/?datagrid=/data/coco-500.datagrid">Kangas UI</a>.</li></ul><p>You can<strong> follow along </strong>with <a href="https://colab.research.google.com/drive/1tV-r1v0333gEq6apdleBWBbVSpOOEW_J?usp=sharing">this Colab</a><strong>.</strong></p><h3>Creating an empty DataGrid</h3><p>According to the Kangas documentation, Kangas <a href="https://github.com/comet-ml/kangas/wiki/DataGrid#datagrid">DataGrid </a>can be thought of as a list of lists or a list of dictionaries.</p><p>Kangas stores the DataGrid objects in an SQLite database, allowing it to store huge amounts of data that can be accessed anytime and very quickly.</p><p>So how do we create a <a href="https://github.com/comet-ml/kangas/wiki/DataGrid#datagrid">DataGrid</a>? First, we need to import Kangas:</p><pre>import kangas as kg</pre><p>We can access the DataGrid class by importing kangas as kg as with pd in Pandas. Let’s first look at what attributes instances created from the DataGrid will have access to:</p><pre>kg.DataGrid(data=None,<br>            columns=None,<br>            name=&quot;Untitled&quot;,<br>            datetime_format=&quot;%Y/%m/%d&quot;,<br>            heuristics=False,<br>            converters=None)</pre><p>With these attributes, we can see exactly where we can begin. Since we have not created the data, we can use the columns and the name attributes.</p><p>Let’s assume we are creating data of startup companies with their expenses and profit data (we will create that later) based on their state. So, we could name our data <em>“startups.” </em>Also, we can provide the columns we expect.</p><p>Setting these columns is optional now, as we could still create them while adding data.</p><pre># name is a meaninful name<br># columns is a list<br><br>dg = DataGrid(name=&#39;Startups&#39;, columns=[&#39;R&amp;A Spending&#39;, &#39;Administration&#39;, &#39;Marketing Spend&#39;, &#39;Profit&#39;])<br><br>dg.info()</pre><p>Just like in Pandas, Kangas has a kg.info() method that displays the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/1*TsIPigyF2mHmf9u6Pf5Zrg.png" /></figure><p>We can see that our DataGrid’s name is as we specified, and we have 0 rows and 4 columns. There are 0 rows since we have yet to add any data. Also note that the DataGrid is in memory, meaning that we have not yet saved it.</p><p>Kangas recommends that we save the DataGrid after creating it by calling dg.save(), but since we have yet to add data/rows in the DataGrid, we will keep it in memory.</p><p>Also, note that the <em>DataGrid Types column</em> is all <em>None</em>. If we were to call the dg.save() method at this point, we would get the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/508/1*lGQboXQrdRGTij1EZDohSw.png" /></figure><p>Now all of our column types are <em>TEXT</em>. <strong>Do not</strong> <strong>save </strong>yet. If saved, altering it will be more complex than creating the DataGrid from scratch.</p><p>Now let’s add rows in the DataGrid.</p><h3>Appending data to the DataGrid</h3><p>Kangas provides two methods we can add data with. It is also easier to use these methods with in-memory DataGrids.</p><ul><li>DataGrid.extend()</li><li>DataGrid.append()</li></ul><h4>Appending data/rows with DataGrid.extend() method</h4><p>There are two ways we can add data with this method:</p><ul><li>If you already specified the columns while defining the DataGrid object, we can <em>extend </em>a DataGrid with <strong>multiple rows at once</strong>:</li><li>If columns are not specified while defining the DataGrid object, we can <em>extend </em>with many rows using a <strong>dictionary of keyword-oriented data, where each keyword is a column name</strong>:</li></ul><p><strong><em>But, I have defined these two ways because we are creating our DataGrid from scratch. If modifying an existing DataGrid in memory, you can add rows with either way.</em></strong></p><p><strong><em>Extending </em></strong>the<strong>DataGrid with multiple rows at once:</strong></p><pre># syntax<br>dg.extend([<br>    [value1_1, value1_2, value1_3, ...],<br>    [value2_1, value2_2, value2_3, ...],<br>])</pre><p>With extend() the rows will work as shown:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*FExNUrVh0F8hlZLf04ssSg.png" /><figcaption>Image by author</figcaption></figure><p><strong>Note: </strong>If we had not specified the columns, when we extend with multiple rows, Kangas will label the columns as A, B, C, and so on. But since we labeled ours, the column names specified should show instead.</p><p>Extending:</p><pre>dg.extend([<br>    [165349.2, 136897.8, 471784.1, &#39;New York&#39;, 192261.83]<br>    [162597.7, 151377.59, 443898.53, &#39;California&#39;, 191792.06]<br>    [153441.51, 101145.55, 407934.54, &#39;Florida&#39;, 191050.39]<br>    [144372.41, 118671.85, 383199.62, &#39;New York&#39;, 182901.99]<br>    [142107.34, 91391.77, 366168.42, &#39;Florida&#39;, 166187.94]<br>])<br><br>dg.info()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/1*HoRoiJQqSnZnDYO9i9muUg.png" /></figure><pre>dg.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/1*Z78FiIrQt5LFGkTmG-pAwg.png" /></figure><p>Extending with a <strong>dictionary of keyword-oriented data</strong>:</p><p>We can use this method if we have not decided to specify the columns. The keys will serve as the column names.</p><p>Let’s redefine our DataGrid with no columns:</p><pre>dg = DataGrid(name=&#39;startups&#39;)<br><br>dg.info()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*2AdzRr6qmL57d_kK5zRxiQ.png" /></figure><p>We have 0 rows and columns.</p><p>Extending:</p><pre># syntax<br><br>dg.extend([<br>    <br>    {&quot;column a&quot;:  value1_a, &quot;column b&quot;: value1_b, &quot;column c&quot;: value1_c, ...},<br><br>    {&quot;column a&quot;: value2_a, &quot;column b&quot;: value2_b, &quot;column c&quot;: value2_c, ...},<br>)</pre><pre>dg2.extend([<br>    {&quot;R&amp;A Spending&quot;:165349.2, &quot;Administration&quot;: 136897.8, &quot;Marketing Spend&quot;:471784.1, &quot;State&quot;:&quot;New York&quot;, &quot;Profit&quot;:192261.83},<br>    {&quot;R&amp;A Spending&quot;:162597.7, &quot;Administration&quot;: 151377.59, &quot;Marketing Spend&quot;:443898.53, &quot;State&quot;:&quot;California&quot;, &quot;Profit&quot;:191792.06},<br>    {&quot;R&amp;A Spending&quot;:153441.51, &quot;Administration&quot;: 101145.55, &quot;Marketing Spend&quot;:407934.54, &quot;State&quot;:&quot;Florida&quot;, &quot;Profit&quot;:191050.39},<br>    {&quot;R&amp;A Spending&quot;:144372.41, &quot;Administration&quot;: 118671.85, &quot;Marketing Spend&quot;:383199.62, &quot;State&quot;:&quot;New York&quot;, &quot;Profit&quot;:182901.99},<br>    {&quot;R&amp;A Spending&quot;:142107.34, &quot;Administration&quot;: 91391.77, &quot;Marketing Spend&quot;:366168.42, &quot;State&quot;:&quot;Florida&quot;, &quot;Profit&quot;:166187.94},<br>])<br><br>dg2.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/1*Z78FiIrQt5LFGkTmG-pAwg.png" /></figure><p>The extend() method can add multiple rows since it will unpack each object we pass and adds the items to the end of the DataGrid sequentially.</p><h4>Appending data/rows with DataGrid.append() method</h4><p>With append, we can only add a single row at a time. Just like with extend, we can choose to append rows in two ways:</p><ul><li>Appending with a list or position-oriented data if the columns are already defined when creating the DataGrid.</li><li>Appending a row using a dictionary of keyword-oriented data, where each keyword is a column name.</li></ul><p><strong><em>But, I have defined these two ways because we are creating our DataGrid from scratch. If modifying an existing DataGrid in memory, you can add rows with either way.</em></strong></p><p>Appending with a <strong>list or position-oriented data</strong>:</p><pre># syntax - normal python<br>dg.append([value1, value2, value3, ...])</pre><pre>dg = kg.DataGrid(name=&quot;startups&quot;, columns=[&#39;R&amp;D Spending&#39;, &#39;Administration&#39;, &#39;Marketing Spend&#39;, &#39;State&#39;, &#39;Profit&#39;])<br><br>dg.append([165349.2, 136897.8, 471784.1, &#39;New York&#39;, 192261.83])<br><br>dg.info(), dg.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*kbA4egBP5yNuPrFPl08bLw.png" /></figure><p>We successfully appended a new row to our DataGrid. If we had not specified the columns, Kangas would label them with A, B, C, etc.</p><p>If you choose to add multiple rows, you will do so one append one after the other:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*DLSXcvzop4JIruOsRO5qSg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*-X9sq-ujIkt4OeQsMpYcYA.png" /></figure><p>Appending using a <strong>dictionary of keyword-oriented data:</strong>We could use this method if we did not specify the columns. The keys will automatically be the column names.</p><pre>dg = kg.DataGrid(name=&quot;startups&quot;)</pre><pre># syntax<br>dg.append({&quot;column a&quot;: value_a, &quot;column b&quot;: value_b, &quot;column c&quot;: value_c, ...})</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*vgzcZvs0nLAqAiciCr5r7Q.png" /></figure><p>We can complete creating the data by appending one row at a time, as seen below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7qpy6p1maBqxGbUCuGzF0A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/1*xJuTiHsFzJKI4BSNJeji7g.png" /></figure><p>Now that we have created our DataGrid, we are <a href="https://github.com/comet-ml/kangas/wiki/Constructing-DataGrids">recommended</a> to save it.</p><p>Remember, our DataGrid is currently in memory. Calling the dg.save() method saves to disk. In memory, data is volatile, and we can lose it if the system crashes. Saving it on disk makes our DataGrid non-volatile (sort of backing it up).</p><p>In memory:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/506/1*4h5Zz1L4dni_6x33Q-NLKQ.png" /><figcaption>Temporary DataGrid (Kangas)</figcaption></figure><p>Saved on disk:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/1*Sen2fRew4JOXSdio2-ryeQ.png" /><figcaption>On disk DataGrid (Kangas)</figcaption></figure><p>When saving on disk, the DataGrid is saved with the name specified when creating it and a .datagrid extension.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/178/1*w8HyA0z0RwwpsVApMwdxXQ.png" /><figcaption>On disk DataGrid (Kangas)</figcaption></figure><p>We can also<strong> read </strong>the<strong> DataGrid file </strong>with read_datagrid() method:</p><pre># read a saved DataGrid file<br>dg = kg.read_datagrid(&#39;startups.datagrid&#39;)</pre><blockquote>How does the team at Uber manage to keep their data organized and their team united? Comet’s experiment tracking. <a href="https://www.comet.com/site/customers/uber/?utm_source=heartbeat&amp;utm_medium=referral&amp;utm_campaign=AMS_US_EN_AWA_heartbeat_CTA">Learn more from Uber’s Olcay Cirit</a>.</blockquote><h3>Converting DataGrid into various formats (CSV, and DataFrame)</h3><p>Kangas allows us to save our DataGrid into CSV and DataFrame formats. We will also see how Kangas reads the new saved formats.</p><h4>Saving DataGrid as CSV and reading the CSV with Kangas</h4><p>To convert our DataGrid to CSV, call the to_csv() method on the DataGrid:</p><pre>DataGrid.to_csv(filename,<br>           sep=&quot;,&quot;,<br>           header=True,<br>           quotechar=&#39;&quot;&#39;,<br>           encoding=&quot;utf8&quot;,<br>           converters=None)</pre><ul><li><strong>filename</strong>: Name of the file.</li><li><strong>sep</strong>(separator): Default is a “,” for csv files.</li><li><strong>header</strong>: Writes out the column names.</li><li><strong>quotechar</strong>: Character to surround text (default is ‘ <strong>“</strong> ‘).</li><li><strong>encoding</strong>: Default is ‘utf-8.’</li><li><a href="https://github.com/comet-ml/kangas/wiki/Reading-data#converters"><strong>converters</strong></a>: A dictionary of functions to convert items into values.</li></ul><pre>dg.to_csv(&#39;startups.csv&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/161/1*iKxHFQaLXevlzuACHRI_Ow.png" /><figcaption>DataGrid saved as CSV (Kangas)</figcaption></figure><p><strong>Reading </strong>the CSV with Kangas:</p><p>Kangas provides a DataGrid.read_csv() method to read directly from a CSV file. Let’s read the startups.csv file we just created.</p><pre>dg = kg.read_csv(&#39;startups.csv&#39;)<br><br>dg.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/579/1*Z7tclc8p9iUQW7PEMQ-RXw.png" /><figcaption>Reading CSV with Kangas</figcaption></figure><h4>Converting DataGrid to a Pandas DataFrame</h4><p>Calling the to_dataframe() method on the DataGrid converts it into a Pandas DataFrame.</p><pre>pd_dataframe = dg.to_dataframe()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*GTh9TOupGozFjAmgqpQVtQ.png" /><figcaption>Creating DataFrame from DataGrid (Kangas)</figcaption></figure><p><strong>Reading</strong> the DataFrame with Kangas.</p><pre>dg_from_df = kg.read_dataframe(pd_dataframe)<br><br>dg_from_df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*TLGg0VGn9J_18tsoHQlP3g.png" /></figure><h3>Basic operations on DataGrid</h3><p>We can perform Pandas-like operations on the DataGrid. Let’s see:</p><h4>Getting the head() and tail() of the data</h4><p>Getting the head():</p><pre># Get the first 3 items in our DataGrid<br>dg.head(3)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/1*Ks7qqmFOz-FF68Gmf2O_qg.png" /><figcaption>First 3 items in DataGrid (Kangas)</figcaption></figure><p>Getting the tail():</p><pre># Get the last 3 items in our DataGrid<br>dg.tail(3)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/1*bbbQJKHgS0VtyknPCwC6DA.png" /><figcaption>Last three items in DataGrid (Kangas)</figcaption></figure><h4>Getting rows and columns (shape) of the DataGrid</h4><p>Getting the shape() of the DataGrid:</p><pre># Display the rows and columns in the DataGrid<br>dg.shape</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/115/1*LxW137e4KODcw4zN1_4bqg.png" /><figcaption>DataGrid shape (Kangas)</figcaption></figure><p>Notice the shape returns 6 columns while we have 5. As with the current version, Kangas counts the <em>row-id</em> column too.</p><h4>Displaying information about the DataGrid</h4><p>Get info() about the DataGrid:</p><pre>dg.info()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/482/1*QKhUOX9jtBcp83mshHHAsg.png" /><figcaption>DataGrid info (Kangas)</figcaption></figure><h4>Getting the columns in the DataGrid</h4><p>get_columns() in the DataGrid:</p><pre>dg.get_columns()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*rWZdJ-Lhwd1skkw7DGTcIQ.png" /><figcaption>Columns in DataGrid (Kangas)</figcaption></figure><h4>Selecting the count of items with “where” expression in a DataGrid</h4><p>We can select_count() of items in our DataGrid based on particular criteria using the where expression. In the expression, the columns are expressed as {“column name”}.</p><p>Let’s count the number of startups in our DataGrid where the <em>R&amp;D Spending</em> is more than 150,000:</p><pre># count of startups where R&amp;D Spending is more than...<br>dg.select_count(&quot;{&#39;R&amp;D Spending&#39;} &gt; 150000&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/433/1*YxeO-nJf7HTcjPECtg_FMw.png" /><figcaption>3 startups where R&amp;D Spending &gt; 150000(Kangas)</figcaption></figure><h4>Querying and selecting rows with a sorting order</h4><p>To do this, Kangas provides the <a href="https://github.com/comet-ml/kangas/wiki/DataGrid#select">select()</a> method.</p><pre>DataGrid.select(where=&quot;1&quot;,<br>                sort_by=None,<br>                sort_desc=False,<br>                to_dicts=False,<br>                count=False,<br>                computed_columns=None,<br>                limit=None,<br>                offset=0)</pre><ul><li><strong>where</strong>: Python expression where column names are written as {“Column Name”}.</li><li><strong>sort_by</strong>: Name of the column to sort on.</li><li><strong>sort_desc</strong>: Sort in descending order. The default is ascending.</li><li><strong>to_dicts</strong>: If True, return the rows in dicts where the keys are the column names.</li></ul><p><a href="https://github.com/comet-ml/kangas/wiki/DataGrid#select">Read more</a> on the other parameters.</p><p>Example:</p><p>Let’s query rows in our DataGrid where <em>R&amp;D Spending</em> &gt; <em>Administration</em> and <em>Profit</em> &gt; 190000.</p><pre><br>dg.select(&quot;{&#39;R&amp;D Spending&#39;} &gt; {&#39;Administration&#39;} and {&#39;Profit&#39;} &gt; 190000&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/1*yQIxbPExQUVsDDr_Ejj9Yg.png" /></figure><p>Including sort_by parameter:</p><pre># Sort by State<br>dg.select(&quot;{&#39;R&amp;D Spending&#39;} &gt; {&#39;Administration&#39;} and {&#39;Profit&#39;} &gt; 190000&quot;, sort_by=&quot;State&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/543/1*vm9ptJsJ4D_a3qrYWa_Agw.png" /></figure><p>Including sort_desc :</p><pre># Display in descending order sorted by State<br>dg.select(&quot;{&#39;R&amp;D Spending&#39;} &gt; {&#39;Administration&#39;} and {&#39;Profit&#39;} &gt; 190000&quot;, sort_by=&quot;State&quot;, sort_desc=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/1*k8BWJvZbwOjqczcH4FDniw.png" /></figure><p>Including to_dicts:</p><pre>dg.select(&quot;{&#39;R&amp;D Spending&#39;} &gt; {&#39;Administration&#39;} and {&#39;Profit&#39;} &gt; 190000&quot;, sort_by=&quot;State&quot;, sort_desc=True, to_dicts=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/379/1*n9MBYV52rLayG27RjoXPeA.png" /></figure><h3>Visualizing the data with the Kangas UI</h3><p>The Kangas UI is the best thing so far. Although its main purpose is to ease visualization and exploration, mostly on huge datasets, we can explore it with the DataGrid we have just created.</p><p>The UI makes it easier to group, sort, and filter columns depending on our needs and makes an appealing UX for us.</p><p>To fire up the UI, we call the show() method on DataGrid.</p><pre>dg.show()</pre><p>The UI should display as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*RaNBwE6fl3d3FkvT5q-dqA.png" /><figcaption>Kangas UI showing DataGrid data</figcaption></figure><p>We could perform a couple of tasks on our DataGrid with this UI. We could:</p><ul><li>Group data by a particular column.</li><li>Filter data.</li><li>Check for null values.</li><li>Sort data.</li></ul><h4>GroupBy data in Kangas UI</h4><p>Creating groupings of categories can help us apply particular functions to the categories depending on our needs. Kangas allows us to visualize groups of data clearly and effortlessly.</p><p>Let’s group data by the <em>State </em>column:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*mosh71Ah3Arlh3ju6i6LXQ.png" /></figure><p>Results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7hpA3482bqLsu8og4Yqzgg.png" /><figcaption>Data grouped by “<em>State”</em> column (Kangas UI)</figcaption></figure><h4>Sorting data in Kangas UI</h4><p>While sorting data in Kangas UI, the default sorting is in ascending order.</p><p>Let’s sort the data by <em>Profits:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*f0-1_TGTIX-ybVLd83jmUw.png" /></figure><p>Results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*M6IVnjEuSHsC3ZUexRS5jQ.png" /><figcaption>Data sorted in Ascending order by “Profit” (Kangas UI)</figcaption></figure><p>We have a startup in <em>Florida</em> with the lowest profit, and a startup in <em>New York</em> has the highest profit.</p><h4>Filtering data with Kangas UI</h4><p>Kangas uses Python filter expressions for the DataGrid to select matching rows. To refer to a column in the expression, we enclose the column name in curly brackets like {“column_name”}. The Kangas UI has an input box where we can fill these filter expressions on the data.</p><p>For instance, in our DataGrid, we could select all rows where <em>R&amp;D Spending</em> &gt; <em>Administration</em> and <em>Profit</em> &gt; 190000.</p><p>The expression would be:</p><pre>({&#39;R&amp;D Spending&#39;} &gt; {&#39;Administration&#39;} and {&#39;Profit&#39;} &gt; 190000)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*V1NqfTZZYyyQRFXTUnrIlQ.png" /></figure><p>Results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*2dNEdZU_bN8gnbNtyp6Orw.png" /><figcaption>Filtering rows (Kangas UI)</figcaption></figure><p>We get three matching rows from the filtering expression.</p><h3>Final thoughts</h3><p>While <a href="https://github.com/comet-ml/kangas">Kangas</a> is in its early days, it is a great tool that will redefine how to visualize and explore data, especially huge multimedia datasets. It is an open-source project by <a href="http://comet.com/?utm_source=heartbeat&amp;utm_medium=referral&amp;utm_campaign=AMS_US_EN_AWA_heartbeat_EN">Comet</a>, and their <a href="https://github.com/comet-ml/kangas">repo</a> is free for exploring and contributions.</p><p>We have learned how to:</p><ul><li>Create DataGrids.</li><li>Add columns to the DataGrids.</li><li>Add data to the DataGrids with append and extend methods.</li><li>Perform some basic operations on the DataGrids that are identical to Pandas.</li><li>Save DataGrids into other formats like CSV.</li><li>Filter, Group, and Sort DataGrids.</li><li>Visualize and perform some operations on the Kangas&#39; awesome UI.</li></ul><p><em>Editor’s Note: </em><a href="https://heartbeat.comet.ml/"><em>Heartbeat</em></a><em> is a contributor-driven online publication and community dedicated to providing premier educational resources for data science, machine learning, and deep learning practitioners. We’re committed to supporting and inspiring developers and engineers from all walks of life.</em></p><p><em>Editorially independent, Heartbeat is sponsored and published by </em><a href="http://comet.com/?utm_source=heartbeat&amp;utm_medium=referral&amp;utm_campaign=AMS_US_EN_AWA_heartbeat_EN"><em>Comet,</em></a><em> an MLOps platform that enables data scientists &amp; ML teams to track, compare, explain, &amp; optimize their experiments. We pay our contributors, and we don’t sell ads.</em></p><p><em>If you’d like to contribute, head on over to our</em><a href="https://heartbeat.comet.ml/call-for-contributors-fee7f5b80f3e"><em> call for contributors</em></a><em>. You can also sign up to receive our weekly newsletter (</em><a href="https://www.deeplearningweekly.com/"><em>Deep Learning Weekly</em></a><em>), check out the </em><a href="https://www.comet.com/site/blog/"><em>Comet blog</em></a><em>, join us on</em><a href="https://join.slack.com/t/fritz-ai-community/shared_invite/enQtNTY5NDM2MTQwMTgwLWU4ZDEwNTAxYWE2YjIxZDllMTcxMWE4MGFhNDk5Y2QwNTcxYzEyNWZmZWEwMzE4NTFkOWY2NTM0OGQwYjM5Y2U"><em> </em></a><a href="https://join.slack.com/t/cometml/shared_invite/zt-49v4zxxz-qHcTeyrMEzqZc5lQb9hgvw"><em>Slack</em></a><em>, and follow Comet on </em><a href="https://twitter.com/Cometml"><em>Twitter</em></a><em> and </em><a href="https://www.linkedin.com/company/comet-ml/"><em>LinkedIn</em></a><em> for resources, events, and much more that will help you build better ML models, faster.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f63d4350ab61" width="1" height="1" alt=""><hr><p><a href="https://heartbeat.comet.ml/constructing-and-visualizing-kangas-datagrid-on-kangas-ui-f63d4350ab61">Constructing and Visualizing Kangas DataGrid on Kangas UI</a> was originally published in <a href="https://heartbeat.comet.ml/">Heartbeat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Natural Language Processing (NLP) Concepts With NLTK]]></title>
            <link>https://heartbeat.comet.ml/natural-language-processing-nlp-concepts-with-nltk-e1cd66e2bea1?source=rss-1e1c1cda0acd------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1cd66e2bea1</guid>
            <category><![CDATA[kanga]]></category>
            <category><![CDATA[comet]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[heartbeat]]></category>
            <dc:creator><![CDATA[Brian Mutea]]></dc:creator>
            <pubDate>Wed, 22 Mar 2023 13:41:58 GMT</pubDate>
            <atom:updated>2023-03-25T14:45:01.616Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learn NLP data processing operations with NLTK, visualize data with <a href="https://www.comet.com/site/blog/kangas-visualize-multimedia-data-at-scale/">Kangas</a>, build a spam classifier, and track it with <a href="https://www.comet.com/docs/">Comet</a> Machine Learning Platform</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-MUoHYxWr5DFi5UmJHUePQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@hostreviews?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Stephen Phillips — Hostreviews.co.uk</a> on <a href="https://unsplash.com/photos/3Mhgvrk4tjM?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>At its core, the discipline of <strong>Natural Language Processing (NLP)</strong> tries to make the human language “palatable” to computers. Many data we analyze as data scientists consist of a corpus of human-readable text. Before we can feed this data into a computer for analysis, we must preprocess it.</p><p>In this article, let’s dive deep into the <strong>Natural Language Toolkit (NLTK) </strong>data processing concepts for NLP data. Before building our model, we will also see how we can visualize this data with <a href="https://www.comet.com/site/blog/kangas-visualize-multimedia-data-at-scale/"><strong>Kangas</strong></a> as part of exploratory data analysis (EDA).</p><p>Ultimately, we will create an <strong>Email Spam Classifier with TensorFlow and the Keras API</strong> and track it with <a href="https://www.comet.com/docs/"><strong>Comet</strong></a>. In the meantime, if you need to become more familiar with Comet, check out their incredible platform <a href="https://www.comet.com/site/">here</a>.</p><h3>Getting started with the NLTK library</h3><p>NLTK offers excellent tools for developing Python programs that leverage natural language data. It has several text-processing libraries for tokenization, stemming, part-of-speech tagging, semantic reasoning, and many more tasks.</p><p>To start using NLTK, you need to install it. To do this, just run pip install nltk in your command line or !pip install nltk if using an interactive notebook.</p><h3>Tokenization</h3><p>Tokenization, also referred to as <a href="https://en.wikipedia.org/wiki/Lexical_analysis"><strong>lexical analysis</strong></a>, is common practice in NLP. It involves splitting text data into lexical tokens, which allow us to work with smaller pieces of text (phrases, sentences, words, or paragraphs) that are still relevant, even once separated from the rest of the text. A <strong>tokenizer </strong>is responsible for this process.</p><p>There are two ways we can tokenize our text data with NLTK. We can either:</p><ul><li><strong>Tokenize by word.</strong></li><li><strong>Tokenize by sentence.</strong></li></ul><h4><strong>Tokenize by word</strong></h4><p>Here, we break down the text into individual words. Among other things, this can help identify frequently occurring words for a given analysis.</p><p>NLTK provides word_tokenizer that allows us to split text into words. Let’s see how:</p><pre>import nltk<br>from nltk.tokenize import word_tokenize<br><br># Text to tokenize<br>text_example = &quot;&quot;&quot;We can do so much with Natural Language Processing ranging <br>from speech recognition, recommendation systems, spam classifiers, and so <br>many more applications.&quot;&quot;&quot;<br><br># Tokenizing<br>word_tokenized_sent = word_tokenize(text_example.casefold())<br>print(word_tokenized_sent)<br><br># RESULTS<br>&#39;&#39;&#39;<br><br>[&#39;we&#39;, &#39;can&#39;, &#39;do&#39;, &#39;so&#39;, &#39;much&#39;, &#39;with&#39;, &#39;natural&#39;, &#39;language&#39;, <br>&#39;processing&#39;, &#39;ranging&#39;, &#39;from&#39;, &#39;speech&#39;, &#39;recognition&#39;, &#39;,&#39;, <br>&#39;recommendation&#39;, &#39;systems&#39;, &#39;,&#39;, &#39;spam&#39;, &#39;classifiers&#39;, &#39;,&#39;, <br>&#39;and&#39;, &#39;so&#39;, &#39;many&#39;, &#39;more&#39;, &#39;applications&#39;, &#39;.&#39;]<br><br>&#39;&#39;&#39;</pre><p><strong>Notice </strong>that we used casefold() on the text_example. The method ignores whether the letters in the text are uppercase or lowercase and treats all as lowercase. Alternatively, we could use text_example.lower() to convert the text into lowercase.</p><p>We must convert the text data into lowercase so all words are in the same case. If this is avoided, the model could interpret words like stock, Stock, and STOCK as unique tokens.</p><h4><strong>Tokenize by sentence</strong></h4><p>When we have text with multiple sentences, we can break them into a list of individual sentences.</p><pre>from nltk.tokenize import sent_tokenize<br><br>text_example2 = &quot;&quot;&quot;We can do so much with Natural Language Processing ranging from<br>speech recognition, recommendation systems, spam classifiers, and so many more applications.<br>These applications also leverage the power of Machine Learning and Deep Learning.<br>&quot;&quot;&quot;<br><br>sentence_tokenized= sent_tokenize(sentences_example.casefold())<br>print(sentence_tokenized)<br><br># RESULTS <br>&#39;&#39;&#39;<br><br>[&#39;we can do so much with natural language processing ranging from speech recognition, recommendation systems, spam classifiers, and so many more applications.&#39;,<br> &#39;these applications also leverage the power of machine learning and deep learning.&#39;<br><br>&#39;&#39;&#39;</pre><p>With sentence tokenizing, we get a list of two sentences from the text.</p><h3>Removing stop words</h3><p>In most cases, this operation occurs after we tokenize the data. Stop words are words that only add to the fluidity of text or speech but contain no meaningful information relevant to our task or the text itself.</p><p>First, we need to <strong>download and import the stop words</strong> that are provided by NLTK. Alternatively, we can also define our own stop words, but for this example the default list is more than sufficient.</p><p>Let’s use our word tokenized sentence above:</p><pre>nltk.download(&#39;stopwords&#39;) # download the stopwords from nltk<br>from nltk.corpus import stopwords<br><br># get stopwords in english<br>eng_stopwords= stopwords.words(&#39;english&#39;)<br><br># filter stopwords with list comprehension<br>tokens_no_stopwords = [word for word in word_tokenized_sent if word not in stopwords_eng]<br>print(tokens_no_stopwords)<br><br># RESULTS<br>&#39;&#39;&#39;<br><br>[&#39;much&#39;, &#39;natural&#39;, &#39;language&#39;, &#39;processing&#39;, &#39;ranging&#39;, &#39;speech&#39;, &#39;recognition&#39;, <br>&#39;,&#39;, &#39;recommendation&#39;, &#39;systems&#39;, &#39;,&#39;, &#39;spam&#39;, &#39;classifiers&#39;, &#39;,&#39;, &#39;many&#39;, <br>&#39;applications&#39;, &#39;.&#39;]<br><br>&#39;&#39;&#39;</pre><p>Notice the disappearance of stop words like <em>‘do’, ‘so’, ‘from’</em> etc.</p><h3>Stemming</h3><p>Stemming is where we reduce words to their base form or root form, like cutting down tree branches to their stem. Stemming normalizes text and makes processing easier for Information Retrieval (IR) or text mining tasks.</p><p>A stemmer will apply <a href="https://vijinimallawaarachchi.com/2017/05/09/porter-stemming-algorithm/">rules</a> that remove suffixes and prefixes from a word and reduce it to its root. This, in turn, reduces the time complexity and space. There are various types of stemming algorithms:</p><ul><li><strong>Porter Stemmer:</strong> We mostly use this algorithm for its speed, minimal error rate, and simplicity. It is based on the fact that suffixes in the English language are composed of smaller and simpler suffixes. It is only limited to English words. <strong>Import as:</strong> from nltk.stem.porter import PorterStemmer.</li><li><strong>Snowball stemmer: </strong>This is a multilingual stemmer. Thus, it supports other languages. It is more aggressive than the Porter stemmer. <strong>Import as</strong>: <a href="https://www.nltk.org/api/nltk.stem.snowball.html">from nltk.stem import SnowballStemmer</a>.</li><li><strong>Lancaster stemmer: </strong>Despite being more aggressive and dynamic than the other stemmers, it is confusing when small words are involved. This stemmer is also less efficient. <strong>Import as</strong>: <a href="https://www.nltk.org/api/nltk.stem.lancaster.html">from nltk.stem.lancaster import LancasterStemmer</a>.</li></ul><p>We will use the Porter stemmer as the commonly preferred choice. The approach is similar if you choose to use the other stemmers, that is, based on what model you are building.</p><p>We will feed the stemmer the tokens_no_stopwords list derived from removing the stop words section.</p><pre>from nltk.stem.porter import PorterStemmer<br><br># Create a stemmer object<br>stemmer = PorterStemmer()<br><br># Stemming<br>stemmed_tokens = [stemmer.stem(token) for token in tokens_no_stopwords]<br><br>print(f&quot;===Unstemmed tokens==== {tokens_no_stopwords}&quot;)<br>print(f&quot;===Stemmed tokens==== {stemmed_tokens}&quot;)<br><br># RESULTS<br>&#39;&#39;&#39;<br><br>===Unstemmed tokens====<br> [&#39;much&#39;, &#39;natural&#39;, &#39;language&#39;, &#39;processing&#39;, &#39;ranging&#39;, &#39;speech&#39;, &#39;recognition&#39;, <br>  &#39;,&#39;, &#39;recommendation&#39;, &#39;systems&#39;, &#39;,&#39;, &#39;spam&#39;, &#39;classifiers&#39;, &#39;,&#39;, &#39;many&#39;, &#39;applications&#39;, &#39;.&#39;]<br><br>===Stemmed tokens====<br> [&#39;much&#39;, &#39;natur&#39;, &#39;languag&#39;, &#39;process&#39;, &#39;rang&#39;, &#39;speech&#39;, &#39;recognit&#39;, &#39;,&#39;, &#39;recommend&#39;, <br>  &#39;system&#39;, &#39;,&#39;, &#39;spam&#39;, &#39;classifi&#39;, &#39;,&#39;, &#39;mani&#39;, &#39;applic&#39;, &#39;.&#39;]<br><br>&#39;&#39;&#39;</pre><p>Notice how the words were cut. It is easy to notice that the resultant words are a bit confusing and appear meaningless. This poses one of the negative effects of stemming. The readability of the text is compromised, and in some cases, it may not even produce the correct root form of the word.</p><p>A stemmer can also produce inconsistent results. Let’s look at an example:</p><pre>some_txt = &quot;Discovering the wheel is among the best scientific discoveries ever made.&quot;<br><br>def stem_text(text):<br>    token_lst = word_tokenize(some_txt.casefold())<br>    token_no_stpw = [word for word in token_lst if word not in eng_stopwords]<br>    inco_stemmed_tokens = [stemmer.stem(token) for token in token_no_stpw]<br>    <br>    return inco_stemmed_tokens<br><br># RESULT<br>&#39;&#39;&#39;<br><br>[&#39;discov&#39;, &#39;wheel&#39;, &#39;among&#39;, &#39;best&#39;, &#39;scientif&#39;, &#39;discoveri&#39;, &#39;ever&#39;, &#39;made&#39;, &#39;.&#39;]<br><br>&#39;&#39;&#39;</pre><p>We notice that the words <em>‘Discovering’ </em>and <em>‘discoveries’ </em>do not end up in the same base form (<em>discover</em>). There are <strong>two main errors</strong> we could encounter while stemming:</p><ul><li><strong>Over-stemming</strong> (False positives): Occurs when the stemmer produces different base forms of two related words that should share a base form.</li><li><strong>Under-stemming</strong> (False negatives): When two unrelated words stem to the same base form while they should not.</li></ul><p>In most cases, the above errors can be caused by the following:</p><ul><li>When the stemmer is too aggressive.</li><li>When the stemmer does not consider the context of the text.</li><li>When the stemmer in use is not designed for the particular language.</li></ul><p>To get around these errors, we could use <strong>Lemmatizers</strong> instead of stemmers.</p><p>Hang on for lemmatization!</p><h3>Part of speech (PoS) Tagging</h3><p>Part of speech tagging involves labeling the words in the text data according to their part of speech. These parts of speech include:</p><ul><li>Noun (labeled with <strong>NN </strong>tag)</li><li>Pronoun (labeled with <strong>PRP </strong>tag)</li><li>Adjective (labeled with <strong>JJ </strong>tag)</li><li>Verb (labeled with <strong>VB </strong>tag)</li><li>Adverb (labeled with <strong>RB </strong>tag)</li><li>Preposition</li><li>Conjunction</li><li>Interjection</li></ul><p>To see all available tags and their meanings run:</p><pre>nltk.download(&#39;averaged_perceptron_tagger&#39;)<br>nltk.download(&#39;tagsets&#39;)<br>nltk.help.upenn_tagset()</pre><p>Let’s use the tokens_no_stopwords list derived from removing the stop words section and tag parts of speech.</p><pre>pos_tagged_tokens = nltk.pos_tag(tokens_no_stopwords)<br>print(pos_tagged_tokens)<br><br># RESULTS<br>&#39;&#39;&#39;<br><br>[(&#39;much&#39;, &#39;JJ&#39;), (&#39;natural&#39;, &#39;JJ&#39;), (&#39;language&#39;, &#39;NN&#39;), (&#39;processing&#39;, &#39;NN&#39;), <br> (&#39;ranging&#39;, &#39;VBG&#39;), (&#39;speech&#39;, &#39;NN&#39;), (&#39;recognition&#39;, &#39;NN&#39;), (&#39;,&#39;, &#39;,&#39;), <br> (&#39;recommendation&#39;, &#39;NN&#39;), (&#39;systems&#39;, &#39;NNS&#39;), (&#39;,&#39;, &#39;,&#39;), (&#39;spam&#39;, &#39;JJ&#39;), <br> (&#39;classifiers&#39;, &#39;NNS&#39;), (&#39;,&#39;, &#39;,&#39;), (&#39;many&#39;, &#39;JJ&#39;), (&#39;applications&#39;, &#39;NNS&#39;), <br> (&#39;.&#39;, &#39;.&#39;)]<br><br>&#39;&#39;&#39; </pre><p>An example of where PoS tagging would be applicable is when there is a need to know a product’s qualities in a review. In this case, we could tag the tokenized data, extract all the adjectives, and evaluate the review’s sentiment.</p><blockquote>Standardizing model management can be tricky but there is a solution. <a href="https://www.comet.com/site/blog/investing-in-ai-unlocking-profitable-machine-learning-with-experiment-management/?utm_source=heartbeat&amp;utm_medium=referral&amp;utm_campaign=AMS_US_EN_AWA_heartbeat_CTA">Learn more about experiment management from Comet’s own Nikolas Laskaris</a>.</blockquote><h3>Lemmatizing</h3><p>This is similar to stemming, but with a significant difference. Unlike stemming, lemmatizing produces words to their root form but returns a complete English word (as would appear in a dictionary) that is meaningful on its own, rather than just a fragment of a word like ‘marbl’ (from marble).</p><p>A lemmatizer also takes into consideration the <strong>context of a word</strong>! It will map words with similar meanings to one word, unlike a stemmer.</p><p>When we lemmatize a word, we generate a <strong>lemma. A lemma is a word that represents a whole group of words.</strong></p><p>Although processing text data includes stemming and lemmatizing, we prefer lemmatization over stemming in most cases.</p><p><strong>Note:</strong> To use the lemmatizer from NLTK, we need to download wordnet and Open Multilingual Wordnet (omw):</p><pre>nltk.download(&#39;omw-1.4&#39;)<br>nltk.download(&#39;wordnet&#39;)</pre><p>Then import WordNetLemmatizer from the wordnet nltk.stem.wordnet module.</p><p>We will feed the lemmatizer the tokens_no_stopwords list derived from removing the stop words section.</p><pre>from nltk.stem import WordNetLemmatizer<br><br># Create lemmatizer object<br>lemmatizer = WordNetLemmatizer()<br><br># lemmatizing<br>lemmatized_tokens = [lemmatizer.lemmatize(token) for token in tokens_no_stopwords]<br><br># RESULTS<br>&#39;&#39;&#39;<br><br>===Stemmed tokens====<br>[&#39;much&#39;, &#39;natur&#39;, &#39;languag&#39;, &#39;process&#39;, &#39;rang&#39;, &#39;speech&#39;, &#39;recognit&#39;, &#39;,&#39;, &#39;recommend&#39;, <br> &#39;system&#39;, &#39;,&#39;, &#39;spam&#39;, &#39;classifi&#39;, &#39;,&#39;, &#39;mani&#39;, &#39;applic&#39;, &#39;.&#39;]<br><br>===Lemmatized tokens==== <br>[&#39;much&#39;, &#39;natural&#39;, &#39;language&#39;, &#39;processing&#39;, &#39;ranging&#39;, &#39;speech&#39;, &#39;recognition&#39;, <br> &#39;,&#39;, &#39;recommendation&#39;, &#39;system&#39;, &#39;,&#39;, &#39;spam&#39;, &#39;classifier&#39;, &#39;,&#39;, &#39;many&#39;, &#39;application&#39;, &#39;.&#39;]<br> <br>&#39;&#39;&#39;</pre><p>Observe the difference between Stemmed and Lemmatized tokens. In a word like <em>‘</em>language<em>’</em> a Stemmer produces <em>‘languag’</em> while a Lemmatizer produces the expected root word <em>‘language.’</em></p><p>In Lemmatization, in most cases, we do not expect the lemmatized words to be very different from their lemma.</p><h3>Using Named Entity Recognition (NER)</h3><p>A text may contain noun phrases that refer to organizations, people, specific locations, etc. These phrases are called named entities, and we can use <strong>named entity recognition</strong> to determine what kind of named entities are in your text data.</p><p>The <a href="https://www.nltk.org/book/ch07.html#sec-ner">NLTK book</a> lists the common types of named entities, which include:</p><ul><li>ORGANIZATION</li><li>PERSON</li><li>LOCATION</li><li>DATE</li><li>TIME</li><li>MONEY</li><li>PERCENT</li><li>FACILITY</li><li>GPE</li></ul><p>NLTK provides a pre-trained classifier that can identify the named entities in our text data. We can access this classifier with the nltk.ne_chunk() function.</p><p>The function, when applied to a text, returns a tree.</p><p>First, we need to download the following for it to work:</p><pre>nltk.download(&#39;maxent_ne_chunker&#39;)<br>nltk.download(&#39;words&#39;)</pre><p>Next, import Tree from nltk.tree module that will help visualize the named entities once we apply the ne_chunk() function.</p><pre>from nltk.tree import Tree</pre><p>Then let’s identify the named entities:</p><pre># Let&#39;s use the follwoing text as example<br>named_ent_txt = &quot;I&#39;m trying to track down a guy named Josh Doew who worked in mining in Ouray back in the 1960&#39;s.&quot;<br>tree = nltk.ne_chunk(nltk.pos_tag(word_tokenize(named_ent_txt)))<br><br>tree</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/1*BDlyqK7ZbJZ2EdSE52pUxw.png" /></figure><p>Notice that the named entities are labeled with their types of entities. For instance, <strong>Ouray </strong>is tagged with <strong>GPE</strong>, which stands for <strong>geo-political entities.</strong></p><p>The ne_chunk() function also has a binary=True argument where, if specified, we only get the named entities labeled with <strong>NE</strong> showing that they are named entities rather than what type they are.</p><pre>named_ent_txt = &quot;I&#39;m trying to track down a guy named Josh Doew who worked in mining in Ouray back in the 1960&#39;s.&quot;<br>tree = nltk.ne_chunk(nltk.pos_tag(word_tokenize(named_ent_txt)), binary=True)<br><br>tree</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*PHm_y-NqfrS6T03SWoOATw.png" /></figure><p>Rather than displaying a tree of all the named entities, we can create a function to extract them without any repeats and store them as a list.</p><p>To do so, we will need to tokenize the text, tag the words with their respective PoS and then extract the named entities based on those PoS tags without repetition if a word exists multiple times.</p><p>In the function, we loop to find the presence of the <a href="https://www.nltk.org/api/nltk.chunk.html">chunk structure</a> where the chunk structure is of type Tree with tokens and chunks (subtree with tokens). In our case, these chunks are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*AvUbabETBjBZ19XtC2nAXQ.png" /></figure><p>So, we need to convert them back into a list of tokens with the<a href="https://www.nltk.org/api/nltk.chunk.html">leaves()</a> method and add them back into the contionus_chunk.</p><pre><br>def extract_named_entities(text):<br>  tree = nltk.ne_chunk(nltk.pos_tag(word_tokenize(text)))<br>  continuous_chunk = []<br>  current_chunk = []<br><br>  for chunk in tree:<br>    if type(chunk) == Tree:<br>      current_chunk.append(&quot; &quot;.join([token for token, pos in chunk.leaves()]))<br>      if current_chunk:<br>        named_entity = &quot; &quot;.join(current_chunk)<br>        if named_entity not in continuous_chunk:<br>          continuous_chunk.append(named_entity)<br>          current_chunk = []<br>      else:<br>        continue<br>  return continuous_chunk</pre><pre># call the function<br>extract_named_entities(named_ent_txt)<br><br># RESULTS<br><br>[&#39;Josh Doew&#39;, &#39;Ouray&#39;]</pre><p>Great!</p><h3>Word Frequency Distribution</h3><p>We can identify words frequently appearing in our text data by building a frequency distribution. To do this, we use the FreqDist module in NLTK.</p><p>Let’s use the tokens_no_stopwords list derived from removing the stop words section.</p><pre>from nltk import FreqDist<br>freq_distribution = FreqDist(tokens_no_stopwords)<br><br># extract the 10 frequent words in the text<br>freq_distribution.most_common(10)<br><br># RESULTS<br><br>&#39;&#39;&#39;<br>[(&#39;,&#39;, 3),<br> (&#39;much&#39;, 1),<br> (&#39;natural&#39;, 1),<br> (&#39;language&#39;, 1),<br> (&#39;processing&#39;, 1),<br> (&#39;ranging&#39;, 1),<br> (&#39;speech&#39;, 1),<br> (&#39;recognition&#39;, 1),<br> (&#39;recommendation&#39;, 1),<br> (&#39;systems&#39;, 1)]<br><br>&#39;&#39;&#39;</pre><p>We can also visualize the distribution:</p><pre>freq_distribution.plot(20, cumulative=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/1*wGPwl3buggszMzAu9vI6HQ.png" /></figure><p>With the knowledge we have gained on text preprocessing with NLTK, it’s time to put some of it into use. Next, we will build an Email Spam Classifier.</p><h3>Email spam classification with TensorFlow, Keras, and NLTK.</h3><p>This section will create a spam classification model with TensorFlow and Keras. The ability to classify a text/email sent to you as spam or ham (not harmful) is a crucial functional detail in many messaging applications we use today, like Gmail or Apple’s official messaging app.</p><p>The following are the steps we will take:</p><ul><li>First, initialize <a href="https://www.comet.com/b-capt#projects">Comet</a> tracking</li><li>Take care of all the necessary imports</li><li>Grab a dataset and visualize it with <a href="https://www.comet.com/site/blog/kangas-visualize-multimedia-data-at-scale/">Kangas</a>.</li><li>Preprocessing the data<br>- Removing stop words.<br>- Tokenizing the text data.<br>- Lemmatization — We will choose Lemmatization over Stemming for the best performance of our model.<br>- Vectorizing the data.</li><li>Building the model.</li><li>Evaluating the model.</li></ul><h4>Initialize Comet tracking</h4><p>To use comet, we will need a personal API key that will be used to communicate between your experiment and the comet platform. Sign up and find the API key under your Account Settings under your profile.</p><p>Also, install comet_ml with pip, if not installed in your environment:</p><pre># on Juypyter<br><br>%pip install comet_ml<br><br># on command line<br><br>pip install comet_ml</pre><pre># initialize comet<br><br>import comet_ml<br><br>experiment = comet_ml.Experiment(<br>    api_key=&quot;your_API_Key&quot;, #Use your api_key from your comet_account<br>    project_name=&quot;Email Spam Classifier&quot;, <br>    log_code=True,<br>    auto_metric_logging=True,<br>    auto_param_logging=True,<br>    auto_histogram_weight_logging=True,<br>    auto_histogram_gradient_logging=True,<br>    auto_histogram_activation_logging=True,<br>)</pre><p>We are good to go!</p><p>Let’s take care of all the necessary imports:</p><pre># for data<br>import numpy as np<br>import pandas as pd<br><br># for visuals<br>from matplotlib.pyplot import figure, plt<br>import seaborn as sns<br>from wordcloud import WordCloud<br><br># from nltk for NLP<br>import nltk<br>import string # for use in punctuation removal<br>from nltk.corpus import stopwords<br>from nltk.tokenize import word_tokenize<br>from nltk.stem import WordNetLemmatizer<br><br># text preprocessing from sklearn<br>from sklearn.feature_extraction.text import TfidfVectorizer<br>from sklearn.model_selection import train_test_split<br><br># for the NN<br>import tensorflow as tf<br>from tensorflow import keras<br>from tensorflow.keras.models import Sequential<br>from keras.layers import Dense<br><br># Evaluation<br>from sklearn.metrics import confusion_matrix, classification_report, accuracy_score, precision_score, recall_score, f1_score</pre><h3>Working on the dataset for NLP</h3><p>We will use the <a href="https://www.kaggle.com/datasets/uciml/sms-spam-collection-dataset">spam collection dataset</a> from <a href="https://www.kaggle.com/">Kaggle</a>. We will read the data using <a href="https://www.comet.com/site/blog/kangas-visualize-multimedia-data-at-scale/">Kangas </a><a href="https://github.com/comet-ml/kangas/wiki/DataGrid#read_csv">Datagrid.read_csv()</a> class method.</p><p>I know we are used to Pandas, but this time let’s do it the <a href="https://www.comet.com/site/blog/kangas-visualize-multimedia-data-at-scale/">Kangas</a> way.</p><p>Simply install <a href="https://www.comet.com/site/blog/kangas-visualize-multimedia-data-at-scale/">Kangas </a>with %pip install kangas on your notebook.</p><h4>Reading and visualizing the data with Kangas</h4><p>First, we need to import the kangas with alias kg:</p><pre>import kangas as kg</pre><pre># Read the data using Kangas<br>dg = kg.DataGrid.read_csv(&quot;spam_data.csv&quot;)<br>dg.save()</pre><p>When we read the data with Kangas, we get a DataGrid, unlike in Pandas with DataFrame. We can perform most of the functions of Pandas, like head(), tail(), etc, with the DataGrid.</p><pre># For example get the first ten rows<br><br>dg.head(10)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/449/1*KeB6f8ukt2riN-eLvw2HUg.png" /></figure><pre># Get the columns<br><br>dg.get_columns()<br><br># [&#39;label&#39;, &#39;message&#39;]</pre><p>However, this is not the goal of Kangas. <a href="https://www.comet.com/site/blog/kangas-visualize-multimedia-data-at-scale/">Kangas </a>has a user interface that we can see more clearly and beautifully visualize our data. To fire up the UI, we call the show() method on the DataGrid:</p><pre>#Fire up the Kangas UI<br><br>dg.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*QQIY7ep25xmpLlP36qmm4A.png" /><figcaption>Kangas UI</figcaption></figure><p>Clearly, we can visualize:</p><ul><li>The first ten rows of our data(<strong>dg.head(10)</strong>).</li><li>The columns (ROW-ID, LABEL, MESSAGE)(<strong>dg.get_columns()</strong>).</li></ul><p>Visualize the last rows of the data(<strong>dg.tail()</strong>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*bX7ckh0KVKpIrLd_pnkRPw.png" /><figcaption>Last rows of the data(Kangas UI)</figcaption></figure><p>We can check the value counts of the spam and ham messages. Use Group by labels on the UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*SIoV9JBsgOWzy_Rc9D85Tg.png" /><figcaption>Spam and Ham messages count(Kangas UI)</figcaption></figure><p>We have <strong>4825 messages</strong> that are <strong>ham </strong>and <strong>747 messages</strong> that are <strong>spam.</strong></p><p>We can also check if we have <strong>null values</strong> in the data. To do this on the UI, DataGrid has filter expressions that we can use.</p><p>We use is Noneto check for null values where we will combine expressions with and and use parenthesis to force evaluation.</p><p>To check if we have null values:</p><pre># paste this on the UI&#39;s Filter input box<br><br>(({&quot;message&quot;} is None) and ({&quot;label&quot;} is None))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*mkAcxCr_HEjJz3DFFl55Ew.png" /><figcaption>Checking for Null Values (Kangas UI)</figcaption></figure><p>Since we have no matching rows for our criteria it means we have no null values.</p><p><em>To quench your curiosity, here is an example where we would have null values filtered from another dataset using the criteria in the UI:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FhsRHWZHhXmIb9LshwKulQ.png" /></figure><p>It is also good that we check for empty strings. Sometimes, a database will fill with empty strings when no data is found somewhere.</p><pre>empty_strs = []<br>for iter, label, msg in df.itertuples(name=&quot;Ham_spam&quot;):<br>  if msg.isspace():<br>    empty_strs.append(1)<br><br># Result is a empty list as we have no empty strings<br>[]</pre><p>Just to be “fancy”, let’s check the percentage distribution of each label (ham and spam):</p><pre>category_count = df[&#39;label&#39;].value_counts()<br><br>fig , ax = plt.subplots(figsize=(15, 5))<br>explode = [0.1, 0.3]<br>ax.pie(category_count, labels=category_count.index, explode=explode, autopct=&#39;%1.2f%%&#39;)<br>ax.set_title(&#39;Distibution of spam and ham&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/527/1*U225_S4VYffrhKdmAr-nUw.png" /></figure><h4>Preprocessing the text data with NLTK</h4><p>Here we will:</p><ul><li>Tokenize each message in the message column.</li><li>Convert all text to lowercase and remove stop words and punctuations.</li><li>Lemmatize the message.</li></ul><p>We have chosen to use lemmatization on the dataset since it performs better and gives meaningful words.</p><p>Let’s create a function that does the above processes for us.:</p><pre>import re<br><br>def preprocess_mesg(message):<br><br>  # define list to hold all the preprocessed words<br>  preprocessed_msg = [] <br><br>  # first lets convert the messages into lower case<br>  message_lower = message.lower()<br>  message = re.sub(r&#39;[^a-zA-Z0-9\s]&#39;, &#39; &#39;, message_lower)<br><br>  # combine stopwords, and punctuations<br>  stopword_and_punctiation = set(stopwords.words(&#39;english&#39;) + list(string.punctuation))<br>  <br>  # tokenize each message with word_tokenize<br>  message = word_tokenize(message)<br><br>  # Initialize the lemmatizer<br>  lemmatizer = WordNetLemmatizer()<br><br>  # clean the message<br>  cleaned_message = [preprocessed_msg.append(lemmatizer.lemmatize(word)) for word in message if word not in stopword_and_punctiation]<br>  cleaned_message = &quot; &quot;.join(preprocessed_msg)<br>  <br>  return cleaned_message<br><br>df[&#39;message&#39;] = df[&#39;message&#39;].apply(preprocess_mesg)</pre><p>If we display the DataFrame, we get a column with cleaned messages with no stop words, lowercase, and lemmatized.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*CfY7_XjV3B-_PKH_mG84Nw.png" /></figure><p>Since our data is now clean, we need to grab the predictor and the target variables.</p><pre>messages = df[&#39;message&#39;].values # X<br>labels = df[&#39;label&#39;].values # y</pre><p>We will convert the <em>labels </em>into numerical types with Keras’ one-hot-encoding using to_categorical() method.</p><pre>from keras.utils import to_categorical<br><br>label2int = {&#39;ham&#39;: 1, &#39;spam&#39;:0}<br>labels = [label2int[label] for label in labels]<br>labels = to_categorical(labels)<br><br>labels</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/1*iIFZ38YDefWjp5Mriipd3w.png" /></figure><h4>Splitting the data into training and test sets</h4><p>We need to split and shuffle the data into training sets for training the model and test sets for evaluating the model’s performance.</p><pre>X_train, X_test, y_train, y_test = train_test_split(messages, labels, test_size=0.30, random_state=42)</pre><pre>X_train.shape, X_test.shape, y_train.shape, y_test.shape<br><br># Shapes<br>((3900,), (1672,), (3900, 2), (1672, 2))</pre><h4>Feature extraction with TfidfVectorizer</h4><p>Before training our model, we need to convert our training samples into computer-understandable numerical data.</p><p>The common way we do this is by using the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html">TfidfVectorizer </a>from Scikit-learn. We could use the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html">CounterVectorizer</a> for this, but as the CountVectorizer only returns a sparse matrix of the only integer counts of the words, TfidfVectorizer returns only floats as scores as it takes into consideration the importance of each word. This means that less important words will have very low scores, thus helping improve our model.</p><pre># initialize the vecotrizer<br><br>tfidfV = TfidfVectorizer()<br><br># transform the train and test sets and transform to array<br>X_train_tfidfV = tfidfV.fit_transform(X_train).toarray()<br>X_test_tfidfV = tfidfV.transform(X_test).toarray()</pre><h3>Building the model</h3><p>Here we will use the Keras Sequential model and set the input dimensions equal to the training data we have vectorized.</p><p>We will also use <a href="https://www.tensorflow.org/api_docs/python/tf/keras/activations/relu"><strong>rectified linear activation function(</strong>relu)</a> activation function for our hidden layers and the <a href="https://www.tensorflow.org/api_docs/python/tf/keras/activations/softmax">softmax </a>function, which will convert the vectors into probability distributions for our output layer.</p><pre># initialize the Sequential model<br>model = Sequential()<br><br>input_dim = X_train_tfidfV.shape[1]<br><br># add Dense layers to the neural network<br>model.add(Dense(8, input_dim=input_dim, activation=&#39;relu&#39;))<br>model.add(Dense(8, input_dim=input_dim, activation=&#39;relu&#39;))<br>model.add(Dense(2, activation=&#39;softmax&#39;))<br><br>model.compile(loss=&#39;categorical_crossentropy&#39;, optimizer=&#39;adam&#39;, metrics=&#39;accuracy&#39;)</pre><pre># call the summary method of the model<br><br>model.summary()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/587/1*qurXCygMFFfJk_Wa5zQQ_A.png" /></figure><p>Next, we need to train the model for the fixed number of epochs or iterations on the dataset using the fit method of the Sequential model class.</p><p>We will iterate through the training set 10 times(epochs), from which we will pick the validation data before the model is trained. The validation data will help gauge the model’s performance and losses, which can help us identify if it is overfitting or not.</p><pre>model.fit(X_train_tfidfV, y_train, epochs=10, verbose=True, validation_data=(X_test_tfidfV, y_test) ,batch_size=10)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qqRKMkcsIM-6aIp8sNkaqQ.png" /></figure><h4>Evaluating the model</h4><p>Now that our model is trained, we can create the confusion matrix and classification report. We will also view the performance in Comet.</p><pre>preds = (model.predict(X_test_tfidfV) &gt; 0.5).astype(&quot;int32&quot;)<br><br># round the testing and predictions data<br><br>y_test = np.argmax(y_test, axis=1)<br>preds = np.argmax(preds, axis=1)</pre><p>Classification report:</p><pre>print(classification_report(y_test, preds))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*tt3ILXPmyFupwEQPhWtB_A.png" /></figure><p>Confusion matrix:</p><pre>figure(figsize=(7, 5), dpi=80)<br>sns.heatmap(pd.DataFrame(cf_matrix), annot=True, cmap=&quot;YlGnBu&quot; ,fmt=&#39;g&#39;, cbar=False)<br>plt.title(&#39;Confusion matrix&#39;, y=1.1)<br>plt.ylabel(&#39;Actual label&#39;)<br>plt.xlabel(&#39;Predicted label&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*JMzEHZ7_Gb1d59U7t7iO_Q.png" /></figure><p>Looking at the confusion matrix, we can see that the model has <strong>high TP (188) </strong>and <strong>TN (1451) </strong>rates and <strong>low FP and FN rates</strong>. Thus, it is predicted remarkably.</p><p><strong>Comet</strong> time!</p><p>To view the experiment on Comet, include the following line in the last cell of your notebook or code.</p><pre>experiment.end()</pre><p>The accuracy and the loss and validation loss are logged automatically on Comet:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5JrLiQ365EvCz9yzIQrtUQ.png" /><figcaption>Comet tracking</figcaption></figure><p>Comet will also show the hyperparameters on the trained model automatically.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*sUPOIDf-UeFB9d_5JFIGaA.png" /><figcaption>Comet logged hyperparameters</figcaption></figure><p>Metrics:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_2lPgSZQJMHg9cTYMGlLVg.png" /></figure><p>We can also view visualizations for the gradients, activations, weights, and biases. For instance, we can see the biases over each epoch in the visualizations below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6OCDBvn9Qeb2Yu7Mij5DFA.png" /></figure><p>Comet is a beneficial platform for building models that entirely encourages you to focus on developing the particular model, and the rest will be logged out for you.</p><h3>Final thoughts</h3><p>Natural language processing is a broad field and is not limited to classifying spam in text. You can ideally use the knowledge you have gained on text preprocessing with NLTK to explore other NLP tasks.</p><p>Also, we have not exhausted all the capabilities of NLTK in text preprocessing. However, we have covered the most common concepts you will begin with when doing NLP.</p><p>This is an all-in-one article where you have learned.</p><ul><li>NLTK text preprocessing concepts.</li><li>Visualizing data with <a href="https://www.comet.com/site/blog/kangas-visualize-multimedia-data-at-scale/">Kangas</a>.</li><li>Vectorizing data with <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html">TFidfVectorizer</a>. Also, explore other methods like <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html">CounterVectorizer </a>for the same function.</li><li>Building an email spam classifier using TensorFlow and Keras.</li><li>Tracking your experiment with <a href="https://www.comet.com/docs/">Comet </a>and viewing the various metrics and visualizations produced by the platform.</li></ul><p><em>Editor’s Note: </em><a href="https://heartbeat.comet.ml/"><em>Heartbeat</em></a><em> is a contributor-driven online publication and community dedicated to providing premier educational resources for data science, machine learning, and deep learning practitioners. We’re committed to supporting and inspiring developers and engineers from all walks of life.</em></p><p><em>Editorially independent, Heartbeat is sponsored and published by </em><a href="http://comet.com/?utm_source=heartbeat&amp;utm_medium=referral&amp;utm_campaign=AMS_US_EN_AWA_heartbeat_EN"><em>Comet,</em></a><em> an MLOps platform that enables data scientists &amp; ML teams to track, compare, explain, &amp; optimize their experiments. We pay our contributors, and we don’t sell ads.</em></p><p><em>If you’d like to contribute, head on over to our </em><a href="https://heartbeat.comet.ml/call-for-contributors-fee7f5b80f3e"><em>call for contributors</em></a><em>. You can also sign up to receive our weekly newsletter (</em><a href="https://www.deeplearningweekly.com/"><em>Deep Learning Weekly</em></a><em>), check out the </em><a href="https://www.comet.com/site/blog/"><em>Comet blog</em></a><em>, join us on </em><a href="https://join.slack.com/t/cometml/shared_invite/zt-49v4zxxz-qHcTeyrMEzqZc5lQb9hgvw"><em>Slack</em></a><em>, and follow Comet on </em><a href="https://twitter.com/Cometml"><em>Twitter</em></a><em> and </em><a href="https://www.linkedin.com/company/comet-ml/"><em>LinkedIn</em></a><em> for resources, events, and much more that will help you build better ML models, faster.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1cd66e2bea1" width="1" height="1" alt=""><hr><p><a href="https://heartbeat.comet.ml/natural-language-processing-nlp-concepts-with-nltk-e1cd66e2bea1">Natural Language Processing (NLP) Concepts With NLTK</a> was originally published in <a href="https://heartbeat.comet.ml/">Heartbeat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Entropy, information gain, and Gini impurity]]></title>
            <link>https://medium.com/@brianmuteak/entropy-information-gain-and-gini-impurity-c50a73edaad8?source=rss-1e1c1cda0acd------2</link>
            <guid isPermaLink="false">https://medium.com/p/c50a73edaad8</guid>
            <category><![CDATA[information-gain]]></category>
            <category><![CDATA[gini-impurity]]></category>
            <category><![CDATA[entropy]]></category>
            <category><![CDATA[data-scientist]]></category>
            <category><![CDATA[decision-tree]]></category>
            <dc:creator><![CDATA[Brian Mutea]]></dc:creator>
            <pubDate>Tue, 06 Dec 2022 12:03:57 GMT</pubDate>
            <atom:updated>2023-01-04T07:03:13.632Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learn decision tree splitting criteria and how they work.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2PACFNL19wxVZdZp.png" /><figcaption>Image by author</figcaption></figure><p>Decision trees are supervised machine-learning models used to solve classification and regression problems. They help to make decisions by breaking down a problem with a bunch of if-else-then-like evaluations that result in a tree-like structure.</p><p>For quality and viable decisions to be made, a decision tree builds itself by splitting various features that give the best information about the target feature till a pure final decision is archived.</p><p>This article will discuss three common <strong>splitting criteria</strong> used in decision tree building:</p><ul><li>Entropy</li><li>Information gain</li><li>Gini impurity</li></ul><h3>Entropy</h3><p>Entropy measures data points’ degree of <strong>impurity, </strong>uncertainty, or surprise. It ranges between 0 and 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/535/0*6FxGZWVXGZkdWhve.png" /><figcaption>Image by author</figcaption></figure><p>We can see that the entropy is 0 when the probability is o or 1. We get a maximum entropy of 1 when the probability is 0.5, which means that the data is perfectly randomized.</p><p>To understand entropy, let’s familiarize ourselves with the concept of <strong>surprise</strong>. Surprise, by definition, is an unexpected event.</p><p>Suppose we had three baskets with random colored balls:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*206LQ6e29R_yFwL1.png" /><figcaption>Image by author</figcaption></figure><p>Each basket has a varying number of red and blue balls. Suppose we decided to pick a random ball from each basket. We will notice that:</p><ul><li>From Basket A, the probability of picking a blue ball is higher than that of picking a red ball. We would be very surprised if we picked a red ball.</li><li>From Basket B, the probability of picking a red ball is higher than that of picking a blue ball. We would be relatively surprised if we picked a blue ball.</li><li>From Basket C, the probability of picking either a red or blue ball is equal. Regardless of what ball we picked, we would be equally surprised.</li></ul><p>The results show that surprise is inversely related to probability( <strong>p(x)</strong>). That is, for instance, from Basket A, when the probability of picking a red ball is low, the surprise is high ( <strong>1/p(x)</strong>).</p><p>However, we can not use the inverse of probability to calculate the surprise. For instance, let’s say we decided to pick balls from basket A three times.</p><p>If we picked blue balls and never picked a red one each time we picked, then it would be relatively correct to say that the probability of picking a red ball would always be 0 and that of picking a blue ball 1 for each round we decided to pick. That means the surprise of picking a blue ball should be 0 for each round. But the inverse of the probability of picking a blue ball is 1 1/1 = 1.</p><p>Since we have found out that we can not use the inverse of probability to calculate the surprise, we introduce <strong>log</strong> to the inverse of probability:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/293/0*2cJO9Q3diP2r09v2.png" /><figcaption>The formula for Surprise: Image by author</figcaption></figure><p>Thus the surprise of picking a blue ball from basket A the given number of times would be 0, just as expected:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/219/0*BcTTGMNytDTESBRb.png" /><figcaption>Surprise of picking a blue ball from Basket A</figcaption></figure><p>Now with the knowledge of expected values, we can try to calculate the surprise of picking a blue ball and a red ball from Basket A. Since we already know that there’s a 0.1 chance that we would pick a red ball and a 0.9 chance of picking a blue ball, we would have the following:</p><p>Surprise for picking a blue ball:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/264/0*DqDfz1TxPPXvwXvL.png" /><figcaption>Image by author</figcaption></figure><p>Surprise for picking a red ball:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/264/0*9AQ_c090I-eW16JI.png" /><figcaption>Image by author</figcaption></figure><p><strong>Entropy</strong> <strong>is now the expected value of surprise</strong>. So every time we pick a ball from basket A, we expect a certain amount of surprise. That is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/431/0*muY0HKf_yjc3NbkL.png" /><figcaption>Surprise each time we pick a ball</figcaption></figure><p>Now from this, we can derive the <strong>formula for Entropy</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/0*Cfhxtc5fnM49Gdac.png" /><figcaption>Derived entropy formula: Image by author</figcaption></figure><p>From the equation above, we can get the standard equation for Entropy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/0*xWbc43I0J6mh3R-V.png" /><figcaption>Entropy formula: Image by author</figcaption></figure><p>Having understood what entropy is and its formula, let’s calculate the entropy of the balls from each basket in our example above:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/0*Yush5D4iCCRKPxE0.png" /><figcaption>Image by author</figcaption></figure><p>Entropy in basket A:</p><pre>import math<br><br>entropy_A = -(9/10) * math.log2(9/10) - (1/10) * math.log2(1/10)<br>entropy_A<br># 0.4689955935892812</pre><p>Entropy in basket B:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/0*PTA6FQsJPoK0wO-W.png" /><figcaption>Image by author</figcaption></figure><pre>entropy_B = -(19/20) * math.log2(19/20) - (1/20) * math.log2(1/20)<br>entropy_B<br># 0.28639695711595625</pre><p>Entropy in basket C:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/0*bSckpc0M83yD1ifC.png" /><figcaption>Image by author</figcaption></figure><pre>entropy_C = -(5/10) * math.log2(5/10) - (5/10) * math.log2(5/10)<br>entropy_C<br># 1.0</pre><p>The higher the entropy, the less pure the dataset or, the higher the level of disorder. Basket C has the highest entropy(1), which makes it very impure.</p><p>A dataset with high entropy(very impure) is good for learning as it is less likely to cause model overfitting.</p><h3>Information gain</h3><p>Before exploring this criterion, be sure you are comfortable with entropy. It is not possible to discuss information gain without entropy.</p><p>Information gain is the <strong>difference between before and after a split</strong> on a given attribute. It measures how much information a feature provides about a target.</p><p>Constructing a decision tree is solely about finding a feature that returns the highest information gain. The feature with the highest information gain produces the best split, classifying the training dataset better according to the target variable.</p><p>Information gain has the following formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/0*DtrvjZbGDAhJfm9E.png" /><figcaption>Information gain formula: Image by author</figcaption></figure><p>Where:</p><ul><li><em>a </em>is the specific attribute or class label.</li><li><em>Entropy(S) </em>is the entropy of dataset <em>S.</em></li><li>|Sv| / |S| is the <strong>proportion </strong>of the values in S to the number of values in the dataset S.</li></ul><p>Suppose we had the following dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/0*AoEF89NpdKxPyPEM.png" /><figcaption>Image by author</figcaption></figure><p>From the above example dataset, we are required to construct a decision tree to help us decide whether we should play volleyball based on the weather conditions.</p><p>To construct a decision tree, we need to pick the features that will best guide us to make a viable decision on whether we should play or not play volleyball.</p><p>We can’t randomly select a feature from the dataset to build the tree, so <strong>Entropy </strong>and <strong>information gain </strong>are good criteria for this problem.</p><p>To begin with, we have four features we need to consider:</p><ul><li>Outlook</li><li>Temp</li><li>Humidity</li><li>Wind</li></ul><p>A decision tree has various parts, the root node, internal nodes, and leaf nodes. Read more on the <a href="https://www.machinelearningnuggets.com/decision-trees-and-random-forests/">Decision Tree and Random Forests</a> article.</p><h3>Finding the root node feature</h3><p>Since we can not just pick one of the features to start our decision tree, we need to make calculations to get the feature with the highest information gain from which we start splitting.</p><h4>Calculate the entropy of the entire dataset(Entropy(S))</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/0*0nrcsQhK-PampgY_.png" /><figcaption>Image by author</figcaption></figure><p>We can see that we have <strong>5 Noes</strong>(or negatives) and <strong>9 Yeses</strong>(or positives). The total number of entries is <strong>14</strong>.</p><p>The entropy of the whole dataset is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*njF6rPc5m2F-p8es.png" /><figcaption>The entropy of Dataset: Image by author</figcaption></figure><pre>import math<br><br>S = -(9/14) * math.log2(9/14) - (5/14) * math.log2(5/14)<br>S<br># 0.9402859586706311</pre><h4>Calculate the information gain for the Outlook feature</h4><p>Outlook has 3 attributes:</p><ul><li>Sunny</li><li>Overcast</li><li>Rain.</li></ul><p>So we will calculate the entropy of each of these attributes( <em>Sv</em>) as follows:</p><p>We have 5 Sunny attributes for Outlook:</p><ul><li>3 negative Sunny Outlooks(When Play Volleyball is No).</li><li>2 positive Sunny Outlooks(When Play Volleyball is Yes).</li></ul><p>Let’s calculate:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/0*c-uZlD8WMlkWSt9Z.png" /><figcaption>Image by author</figcaption></figure><pre>import math<br><br>entropy_S_sunny = -(2/5) * math.log2(2/5) - (3/5) * math.log2(3/5)<br>entropy_S_sunny<br># 0.9709505944546686</pre><p>We have 4 Overcast attributes for Outlook:</p><ul><li>0 negative Overcast Outlooks.</li><li>4 positive Overcast Outlooks.</li></ul><p>Let’s calculate:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/0*r46Q9hDIu3eQmrvx.png" /><figcaption>Image by author</figcaption></figure><p>We have 5 Rain attributes for Outlook:</p><ul><li>2 negative Rain attributes.</li><li>3 positive Rain attributes.</li></ul><p>Let’s calculate:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/0*v9EpJ3awbN_c8kq2.png" /><figcaption>Image by author</figcaption></figure><pre>import math<br><br>entropy_S_rain = -(3/5) * math.log2(3/5) - (2/5) * math.log2(2/5)<br>entropy_S_rain<br># 0.9709505944546686</pre><h4>The information gain for Outlook</h4><p>We have the following Entropies:</p><ul><li>Entropy(<em>S</em>) = 0.94</li><li>Entropy(<em>SSunny</em>) = 0.97</li><li>Entropy(<em>SOvercast</em>) = 0</li><li>Entropy(<em>SRain</em>) = 0.97</li></ul><p>We use the formula for information gain to calculate the gain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/0*_sd3BHFnmdzJXbsv.png" /><figcaption>Image by author</figcaption></figure><p>So:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/0*DlcDj_7AoeC6lAkW.png" /><figcaption>Image by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/0*Vc-dq7gIYRQ_omvM.png" /><figcaption>Image by author</figcaption></figure><p><strong>Information gain for Outlook is 0.24.</strong></p><p>Similarly, we have to calculate the information gain for the other features.</p><h4>Calculate the information gain for the Temp feature</h4><p>Temp has 3 attributes:</p><ul><li>Hot</li><li>Mild</li><li>Cool</li></ul><p>Since we already have the entropy for the entire dataset( <strong>Entropy(<em>S</em>)</strong>), we will calculate the entropy of each attribute(Entropy ( <em>Sv</em>)) of Temp, just as we did with Outlook.</p><p>The entropy of Hot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/467/0*toHquV0YYzJ84oLM.png" /><figcaption>Image by author</figcaption></figure><pre>import math<br><br>entropy_S_hot = -(2/4) * math.log2(2/4) - (2/4) * math.log2(2/4)<br>entropy_S_hot<br># 1.0</pre><p>The entropy of Mild:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/0*Xai1zS5ZEgppyvyG.png" /><figcaption>Image by author</figcaption></figure><pre>import math<br><br>entropy_S_mild = -(4/6) * math.log2(4/6) - (2/6) * math.log2(2/6)<br>entropy_S_mild<br># 0.9182958340544896</pre><p>The entropy of Cool:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/0*eWNpg3pZJ82xcMOy.png" /><figcaption>Image by author</figcaption></figure><pre>import math<br><br>entropy_S_cool = -(3/4) * math.log2(3/4) - (1/4) * math.log2(1/4)<br>entropy_S_cool<br><br># 0.8112781244591328</pre><p>Calculate the information gain for the Temp feature:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/0*525F35tWFOGe6hE2.png" /><figcaption>Image by author</figcaption></figure><p><strong>Information gain for Temp is 0.03.</strong></p><p>Similarly, calculate the information gain for Humidity and Wind. All information gain values will be:</p><ul><li>Gain(<em>S, Outlook</em>) = 0.24</li><li>Gain(<em>S, Temp</em>) = 0.03</li><li>Gain(<em>S, Humidity</em>) = 0.15</li><li>Gain(<em>S, Wind</em>) = 0.04</li></ul><blockquote>💡</blockquote><blockquote>Outlook gives the highest information about our target variable from the information gain values. It will act as the <strong>root node</strong> of our tree from where the splitting will begin.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/0*edya78mWcFvrhDAb.png" /><figcaption>Image by author</figcaption></figure><p><strong>Note, </strong>for Sunny and Rain branches, we can not easily conclude a yes or a no since we have events where Play Volleyball is yes and Play volleyball is no. That means that their entropy is more than zero and hence impure. So we need to split them.</p><blockquote>💡</blockquote><blockquote>Overcast is a branch with zero entropy since it has all events as Play volleyball(Yes), so it automatically becomes a leaf node.</blockquote><h3>Finding the internal nodes</h3><p>We will calculate information gain for the rest of the features when the Outlook is <strong>Sunny </strong>and when the Outlook is <strong>Rain</strong>:</p><h4>Splitting on the Sunny attribute</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/0*kdtmZ_1OvP0nd_4Y.png" /><figcaption>Image by author</figcaption></figure><h4><strong>Calculate the information gain for Temp</strong></h4><p><em>Values</em>( <em>Temp</em>) = Hot, Mild, Cool.</p><p>The entropy for Hot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/490/0*Nfy5-Gs4HGeSIV0q.png" /><figcaption>Image by author</figcaption></figure><p>The entropy for Mild:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/0*7CUEIxW7t_BgzkcE.png" /><figcaption>Image by author</figcaption></figure><p>The entropy for Cool:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/0*eX6YXUHXbGUzGuMO.png" /><figcaption>Image by author</figcaption></figure><p><strong>The Information gain for Temp:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/0*chpOxwP8HS4KJAWV.png" /><figcaption>Image by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/0*PR5HABle9LIp_XP9.png" /><figcaption>Image by author</figcaption></figure><h4><strong>Calculate the information gain for Humidity</strong></h4><p><em>Values</em>( <em>Humidity</em>) = High, Normal.</p><p>The entropy for Sunny:</p><p>Entropy( <em>SSunny</em>) = 0.97</p><p>The entropy for High:</p><p>Entropy( <em>SHigh</em>) = 0</p><p>Then entropy for Normal:</p><p>Entropy( <em>SNormal</em>) = 0</p><p><strong>The Information gain for Humidity:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/0*nV8NEJj-x3Ld9V_Z.png" /><figcaption>Image by author</figcaption></figure><h4><strong>Calculate the information gain for Wind</strong></h4><p><em>Values</em>( <em>Wind</em>) = Strong, Weak.</p><p>The entropy for Sunny:</p><p>Entropy( <em>SSunny</em>) = 0.97</p><p>The entropy for Strong:</p><p>Entropy( <em>SStrong</em>) = 1.0</p><p>The entropy for Weak:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/489/0*PB_Od5Krdp7vTvIo.png" /><figcaption>Image by author</figcaption></figure><p><strong>The Information gain for Wind:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/0*9agV8oAQgFk2YeXs.png" /><figcaption>Image by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/0*jfGQPvQe79f9BQgR.png" /><figcaption>Image by author</figcaption></figure><p><strong>Humidity </strong>gives the <strong>highest </strong>information gain value(0.97). So far, our Tree will look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/0*J_5tkAqEi0TD9uIQ.png" /><figcaption>Image by author</figcaption></figure><h4>Splitting on the Rain attribute</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/0*sQHvGQ2A3NgziGON.png" /></figure><h4><strong>Calculate the information gain for Temp</strong></h4><p><em>Values</em>( <em>Temp</em>) = Mild, Cool</p><p>The entropy for Mild:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/0*gBNOMf82rrCRSWjm.png" /></figure><p>The entropy for Cool:</p><p>Entropy( <em>SCool</em>) = 1.0</p><p><strong>The information gain for Temp:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/0*MSyvXhBcCEjHmxh-.png" /></figure><h4><strong>Calculate the information gain for Humidity</strong></h4><p><em>Values</em>( <em>Humidity</em>) = High, Normal.</p><p>The entropy for High:</p><p>Entropy( <em>SHigh</em>) = 1.0</p><p>The Entropy for Normal:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/0*LNNd2EkzwJbrJOF6.png" /></figure><p><strong>The Information gain for Humidity:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/0*s4ddKjpzatGlpZME.png" /></figure><h4><strong>Calculate the information gain for Wind</strong></h4><p><em>Values</em>( <em>Wind</em>) = Strong, Weak.</p><p>The entropy for Strong:</p><p>Entropy( <em>SStrong</em>) = 0</p><p>The entropy for Weak:</p><p>Entropy( <em>SWeak</em>) = 0</p><p><strong>The information gain for Wind:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/0*5tEuf18wcVxp1kaE.png" /></figure><p><strong>Wind </strong>gives the <strong>highest </strong>information gain value(0.97). Now we can complete our Decision Tree.</p><h3>A complete decision tree with Entropy and Information gain criteria</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/0*Bw0c_VutmcwiXe40.png" /><figcaption>Decision tree (splits criteria: Entropy and Information gain): Image by author</figcaption></figure><h3>Gini impurity</h3><p>Gini impurity is the probability of incorrectly classifying a random data point in a dataset. It is an impurity metric since it shows how the model differs from a pure division.</p><p>Unlike Entropy, Gini impurity has a maximum value of 0.5(very impure classification) and a minimum of 0(pure classification).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/0*afgj7M164fYVv4ol.png" /><figcaption>Image by author</figcaption></figure><p>Gini impurity has the following <strong>formula</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/368/0*URP7AFG2v7K-WR6h.png" /><figcaption>Gini impurity formula: Image by author</figcaption></figure><p>Where:</p><p>If we have a dataset of classes <em>C, P</em>( <em>i</em>) is the probability of picking a data point of class <em>i, </em>and <em>1-P</em>( <em>i</em>) is the probability of not picking a data point of class <em>i.</em></p><p>When the above formula is simplified, we get the following formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/258/0*W4r-Qjg06VNBKHuB.png" /><figcaption>Gini impurity formula: Image by author</figcaption></figure><p>Let’s take the following dummy dataset, for example. We will use the Gini criteria to construct a decision tree from it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/0*90g459_fZI3sOi89.png" /></figure><p>First, we calculate the Gini for the whole dataset. We have 3 possible output variables:</p><ul><li>Hire — 6 instances.</li><li>Reject — 2 instances.</li><li>Train — 2 instances.</li></ul><p>So the Gini will be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/0*sOx7PAk0sNutu8db.png" /><figcaption>Image by author</figcaption></figure><p><strong>Gini for the entire dataset is 0.56.</strong></p><p>Next, we calculate the Gini for the features to decide on the root node.</p><h4><strong>Calculate Gini for Student Background</strong></h4><p>The “Student Background” feature has 3 possible values:</p><ul><li>Math — 3 instances.</li><li>IT — 3 instances.</li><li>CS — 4 instances.</li></ul><p>Gini for when the “Student Background” is in Math. There are 2 examples when the decision is <em>Hire</em> and 1 when the decision is <em>Reject</em>. So:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/449/0*Xj13QX3IxKCRt_Lz.png" /><figcaption>Image by author</figcaption></figure><p>We do the same for IT and CS, where all three Gini values will be:</p><ul><li><em>Gini(Math)</em> = 0.44.</li><li><em>Gini(CS)</em> = 0.38.</li><li><em>Gini(IT)</em> = 0.44.</li></ul><p>Now we calculate the weighted average for “Student Background”:</p><pre>w_avrg = 0.44*(3/10) + 0.38*(4/10) + 0.44*(3/10)<br>w_avrg<br><br># 0.41600000000000004</pre><p><strong>Gini for the Student background feature is 0.416.</strong></p><h4><strong>Calculate Gini for Online Courses</strong></h4><p>The “Online Courses” feature has two possible values:</p><ul><li>Yes — 5 instances.</li><li>No — 5 instances.</li></ul><p>Gini for when the “Online Courses” is No. There is 1 example when the “Decision” is <em>Hire</em>, 2 examples when the “Decision” is <em>Reject,</em> and 2 examples when the “Decision” is “Train”. So:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/0*aR18qVJ_Lkj35Egm.png" /><figcaption>Image by author</figcaption></figure><p>The Gini for when the “Online Courses” is Yes is 0 since there are 5 out of 5 instances when the decision is <em>Hire</em>.</p><p>Now we calculate the weighted average for “Online Courses”:</p><pre>w_avrg = 0*(5/10) + 0.64*(5/10)<br>w_avrg<br><br># 0.32</pre><p><strong>Gini for the Online Courses feature is 0.32.</strong></p><h4>Calculate Gini for Working</h4><p>The “Working” feature has 2 possible values:</p><ul><li>W(Working) — 7 instances.</li><li>NW(Not working) — 3 instances.</li></ul><p>Gini for when the “Working” is W. There are 4 examples when the “Decision” is <em>Hire</em>, 2 examples when the “Decision” is <em>Reject,</em> and 1 example when the “Decision” is <em>Train</em>. So:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/479/0*If-Np1wXyf7njGlz.png" /><figcaption>Image by author</figcaption></figure><p>We do the same for Not Working, and the Gini values will be :</p><ul><li><em>Gini(W)</em> = 0.57.</li><li><em>Gini(NW)</em> = 0.44.</li></ul><p>Calculate the weighted average for “Woking”:</p><pre>w_avrg = 0.57*(7/10) + 0.44*(3/10)<br>w_avrg<br><br># 0.5309999999999999</pre><p><strong>Gini for the Working status feature is 0.53.</strong></p><p>To decide on the root node, we check the Gini values achieved by each feature. The lower the Gini value, the better.</p><ul><li><em>Gini(Student Background)</em> = 0.416</li><li><em>Gini(Online Courses)</em> = 0.32.</li><li><em>Gini(Working Status)</em> = 0.53.</li></ul><blockquote>💡</blockquote><blockquote>The “Online Courses” feature has the least Gini value and thus provides the best information about our target variable. This qualifies it as the root node.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/0*qpy5rCRC-WIMpoeg.png" /><figcaption>Image by author</figcaption></figure><p>Since the classification when “Online Course” is Yes results in one decision, we get a leaf node as <em>Hire</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/0*0Qz5tWZ8e963hp3M.png" /><figcaption>Image by author</figcaption></figure><p>Since the right node is not pure, we will further split it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/0*F_jMRGPt9ZS164Mz.png" /><figcaption>Image by author</figcaption></figure><p>We will compute the Gini for when:</p><ul><li>“Online Courses” is No, and “Student Background”</li><li>“Online Courses” is No and “Working.”</li></ul><h4>Calculate the Gini for “Student Background”</h4><p>When “Online Course” is No, and “Student Background” is Math, we have 2 instances with similar Decision( <em>Reject</em>). So the Gini value will automatically be 0.</p><p>When “Online Course” is No, and “Student Background” is IT, we have 1 instance: “Decision” is Train. So the Gini value will be 0.</p><p>When “Online Course” is No, and “Student Background” is CS, we have 2 instances: “Decision” is Train. So the Gini value will be 0.5.</p><p>Weighted average:</p><pre>w_avrg = 0*(2/5) + 0*(1/5) + 0.5*(2/5)<br>w_avrg<br><br># 0.2</pre><p><strong>Gini For “Student Background” is 0.2.</strong></p><h4>Calculate the Gini for “Working”</h4><p>When Online Course is No, and “Working” is W. We get the Gini as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/0*6_Oso1AfGZ9wEk_q.png" /><figcaption>Image by authro</figcaption></figure><p>When “Online Course” is No, and “Working” is NW, we have 1 instance so the Gini value is 0:</p><p>Weighted average:</p><pre>w_avrg = 0.63*(4/5) + 0*(1/5)<br>w_avrg<br># 0.5</pre><p><strong>Gini For “Working” is 0.5.</strong></p><p>Since “Student Background” has the least Gini value, it will be the next internal node. So now we can complete the decision tree.</p><h3>A complete decision tree with Gini criteria</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/629/0*n6d2K4r99u3QTbME.png" /><figcaption>Image by author</figcaption></figure><h3>Final thoughts</h3><p>In this article, we have learned three splitting criteria used by decision trees. In general practice, we don’t do all these calculations manually. When building the decision tree model, an algorithm will do all these calculations in the back end.</p><p>Check out our article on decision trees and random forests to learn how to build and optimize them.</p><p><em>The </em><a href="https://www.udemy.com/course/data-science-bootcamp-in-python/?referralCode=9F6DFBC3F92C44E8C7F4"><em>Complete Data Science and Machine Learning Bootcamp on Udemy</em></a><em> is a great next step if you want to keep exploring the data science and machine learning field.</em></p><p><em>Follow us on </em><a href="https://www.linkedin.com/company/mlnuggets"><em>LinkedIn</em></a><em>, </em><a href="https://twitter.com/ml_nuggets"><em>Twitter</em></a><em>, and </em><a href="https://github.com/mlnuggets"><em>GitHub</em></a><em>, and </em><a href="https://www.machinelearningnuggets.com/#/portal">subscribe to our <em>blog</em></a>,<em> so you don’t miss a new issue.</em></p><p><em>Originally published at </em><a href="https://www.machinelearningnuggets.com/splitting-criteria-in-decision-trees/"><em>https://www.machinelearningnuggets.com</em></a><em> on December 6, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c50a73edaad8" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>