<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Farhan Ahmad on Medium]]></title>
        <description><![CDATA[Stories by Farhan Ahmad on Medium]]></description>
        <link>https://medium.com/@farhanhubble?source=rss-41546b1c4faa------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*bt4dG1iV3hEn6C-fV1h5ug.jpeg</url>
            <title>Stories by Farhan Ahmad on Medium</title>
            <link>https://medium.com/@farhanhubble?source=rss-41546b1c4faa------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 14:38:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@farhanhubble/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Sniffing a Bricked Glucose Monitor with Codex]]></title>
            <link>https://medium.com/@farhanhubble/sniffing-a-bricked-glucose-monitor-with-codex-690c74229d02?source=rss-41546b1c4faa------2</link>
            <guid isPermaLink="false">https://medium.com/p/690c74229d02</guid>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[medical-devices]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Farhan Ahmad]]></dc:creator>
            <pubDate>Sat, 10 Oct 2026 16:19:16 GMT</pubDate>
            <atom:updated>2026-10-11T00:16:42.113Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using AI to postmortem an oft-failing continuous glucose monitor</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hFgZXQNA0TyIbb9dOX1Quw.jpeg" /><figcaption>Roche CGM. Source: Roche Website</figcaption></figure><p><strong>tl;dr</strong></p><p>I got myself a Roche Accu-Chek SmartGuide continuous glucose monitor because of the fuss about blood sugar variations being the culprit behind several common conditions. The first one stopped working before it could warm up. The replacement was dead in two days. I got a refund, but I was disappointed that a $100 device could fail so often and without any kind of diagnostics about its own health. Being a software engineer and data scientist and knowing a thing or two about embedded devices, I turned to <strong>Codex</strong> and <strong>OpenAI Astra</strong> to look for clues.</p><p><strong>Some Soul Searching, Some Dumpster Diving</strong></p><p>After I wrote to Roche for a refund, I had binned the sensor. A day later, the software engineer-turned data scientist beckoned: “We should not just bury a problem like that. We have never done that!” I had known that the sensor still had 2 weeks of lifetime left. So I dove back into the bins to find the sensor and its applicator bearing the Bluetooth PIN.</p><p><strong>Salvaging Some Bits</strong></p><p>I hadn’t hoped to root into a proprietary sensor and read raw data, but I hoped to salvage some diagnostics, which, I thought, might not have been shown in their app. Since Astra has been in the news for being able to cobble up some software on-the-fly, I asked it this:</p><blockquote>I have a bricked continuous glucose monitor made by Roche. It’s lying near this laptop. It identifies itself as AC-1R00075XXXX. Can you establish a Bluetooth connection and retrieve whatever data it is sending?</blockquote><p>Within a few minutes, Codex strung together some Python code to scan Bluetooth devices. The code paired with the sensor and dumped the data still sitting on the sensor. It pulled a day’s worth of glucose readings, information about the mode the sensor was operating in, and a status bit indicating abnormal operation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IdtkQDKKUbVgfSscB4kIfQ.png" /><figcaption>Sensor readings from 1700 hours on September 15 to 0200 hours on September 16. The sensor was uncalibrated, so absolute values might not have been accurate.</figcaption></figure><p><strong>Discovering Scraps</strong></p><p>As it turned out, there was no diagnostics information in the dump. It wasn’t unexpected since the Bluetooth connection relays information to users and doesn’t provide debugging information. Or does it? I don’t know yet, but given these devices are used in therapeutic settings as well, and conform to strict regulations, I didn’t expect retrieving diagnostics to be a cakewalk.</p><p>All I can is that if I built such a system, I’d include some telemetry. Had the sensing element failed? Was there a calibration problem? Had the software decided that the measurements were no longer trustworthy?</p><p>If you have read so far, and you are also someone who does not go to sleep without finding the root cause, read on and hit me up if you would like to team up.</p><p><strong>The Tidbits</strong></p><p>We used Python and <a href="https://bleak.readthedocs.io/en/latest/">Bleak</a>, a library for communicating with Bluetooth Low Energy devices. On my Mac, it used Apple’s CoreBluetooth interface.</p><p>The sensor appeared in the scan and accepted a connection. It exposed three services:</p><ul><li>Device Information</li><li>Continuous Glucose Monitoring</li><li>Reconnection Configuration.</li></ul><p>The device information returned Roche Diabetes Care GmbH, model `303`, and firmware `FW_V7.5.1–9205bbad`. The radio and enough of the firmware to answer these requests were clearly still functioning.</p><p>The sensor implements the standard Bluetooth Continuous Glucose Monitoring service. Among its characteristics is a Record Access Control Point, which lets a client request stored measurements.</p><p>We issued two requests: count all records, then report all records. The sensor reported <strong>96</strong>, delivered <strong>96</strong>, and returned a successful completion response. The <a href="https://www.bluetooth.com/wp-content/uploads/Files/Specification/HTML/CGMS_v1.0.2/out/en/index-en.html)">Bluetooth CGM service specification</a> describes this retrieval procedure.</p><p>The records contain:</p><ul><li><strong>34 numeric glucose values</strong></li><li><strong>29 `NaN` values</strong>, indicating that no numeric measurement was available.</li><li><strong>33 `−Infinity` values</strong>, accompanied by a warning that the result was below the range the device could process.</li></ul><p>All 96 record checksums passed. That gives us confidence in the integrity of the retrieved packets; it does not validate the underlying glucose measurements.</p><p><strong>Reading the Fault Flags</strong></p><p>The sensor advertized a particular service characteristic that corresponds to CGM Status. We read it, and it was only three bytes:</p><p>09 00 00</p><p>In its first byte, `0x09` has bits 0 and 3 set: <strong>session stopped</strong> and <strong>sensor malfunction</strong>. That is a useful confirmation, but it is still a broad diagnosis. There is no accompanying explanation such as a broken filament or a particular failed internal check.</p><p>We also decoded the other flags. None of the retained records or the fresh status read reported low battery, excessive temperature, insufficient temperature, or a general device fault. The sensor’s feature information says it supports detecting those conditions. The field definitions are in the <a href="https://btprodspecificationrefs.blob.core.windows.net/gatt-specification-supplement/GATT_Specification_Supplement.pdf">Bluetooth GATT Specification Supplement</a>.</p><p><strong>The Calibration Issue</strong></p><p>All 96 records were tagged “<strong>calibration required”</strong>. In 63, “<strong>calibration not allowed”</strong> is also set.</p><p>Roche documents the calibration-required flag as indicating <a href="https://tools.accu-chek.com/documents/dms/en/smartguide_app_manual/smartguide-app-man-html-en_1-1-0/index/technicaldat/technicaldat_2/technicaldat_2.html">Trend Mode</a>, which was expected since these sensors require calibration against a prick-based monitor and I had not done any calibration. In Trend Mode, the reported readings could deviate from true readings by up to 10%.</p><p><strong>What Might Have Happened?</strong></p><p>The strongest conclusion from the capture is that the sensor repeatedly failed to produce a usable measurement, often while reporting a result below its processable range.</p><p>Several physical or software explanations could fit parts of that pattern. Pressure on a worn sensor can produce falsely depressed CGM readings, as observed in a <a href="https://pmc.ncbi.nlm.nih.gov/articles/PMC3879750/">study of sleep position and sensor performance</a>. However, that study involved other sensors and does not demonstrate what happened to this Roche unit.</p><p>Sensor displacement, damage to the sensing element, or a problem in measurement processing are also possibilities. We have no raw electrode signal or independent glucose reference in this capture with which to distinguish them.</p><p>The discovered Bluetooth services exposed no detailed fault log, raw electrode-current stream, battery-voltage history, or crash dump. A manufacturer-specific diagnostic procedure may exist, but we did not discover one. We also did not try undocumented commands, reset the sensor, change its calibration, or restart the session.</p><p><strong>An Invitation to Investigate</strong></p><p>I now have the raw Bluetooth captures, decoded records, a plot, and the Python scripts used to retrieve and inspect them. The next useful step is to have more informed eyes on the evidence.</p><p>I’d particularly like to hear from people who can help with:</p><ul><li><strong>Roche CGM behavior:</strong> What conditions produce the combination of malfunction, below-range results, and calibration-not-allowed flags? Is there a documented distinction between a transient measurement fault and a stopped session?</li><li><strong>Sensor engineering:</strong> What additional observation would help distinguish pressure or displacement from a defective sensing element</li><li><strong>Bluetooth and firmware:</strong> Is there a documented diagnostic interface we missed? Are there details in the standard fields that deserve a different interpretation?</li><li><strong>Comparable cases:</strong> Has anyone captured the same model and firmware while it was working normally, or around a similar failure?</li><li><strong>Hardware Debug Interface:</strong> If all else fails, prying open the sensor and trying to find any debugging pads on the circuit board or the chips can be a final resort. The small size of the package would make it a challenge still to solder on any debugging leads.</li></ul><p>If you have relevant experience, or work on similar devices, please leave a comment. I’d like to compare the captures and turn the hypotheses into questions we can actually test.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=690c74229d02" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is Agentic Coding Just a PR Fad?]]></title>
            <link>https://medium.com/@farhanhubble/is-agentic-coding-just-a-pr-fad-5fa661b8c9b1?source=rss-41546b1c4faa------2</link>
            <guid isPermaLink="false">https://medium.com/p/5fa661b8c9b1</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Farhan Ahmad]]></dc:creator>
            <pubDate>Fri, 13 Feb 2026 06:30:37 GMT</pubDate>
            <atom:updated>2026-02-13T06:30:37.852Z</atom:updated>
            <content:encoded><![CDATA[<p>There have been big claims from some very respected people in the software industry, like Steve Yegge and Eric Meijer, about how the newest breed of AI coding agents are no longer mere assistants, and are instead the drivers, as they design software and write the code at a super-human scale. They claim that big projects are now being built using prompts and rubrics instead of constant human intervention and review. Of course, big tech has been saying that too, but these guys are not corporate PR, and they have a legacy others look up to. However, so far, it has been mere claims. That changes with the release of https://emsh.cat/one-human-one-agent-one-browser/</p><p>For the first time, to my knowledge, someone has published both the recipe and the artefacts for such a project*. Agents built a browser, which some consider the most complex type of software. It was built using a simple agentic setup with infrequent human feedback. Sure, it’s incomplete and nowhere near being a replacement for real browsers, but it is a testament that software has changed forever.</p><p>Some might say “Oh! It is unmaintainable”. The truth is that AI writes better code than the average Joe. It is the scale and speed at which it spit outs code that make a human-in-the-loop system unviable. But the speedup is such an insane leverage that organisations cannot ignore it anymore. The challenge for software teams now is to design the scaffolding that supports the AI-in-the-driver ’s-seat approach while still allowing engineers to control the process from a higher level. The challenge is to come up with ways to cope with codebases that can only be understood, enhanced, and maintained by agents. The challenge is to rethink testing so that there is no distinction between testing and real deployment.</p><p>*Cursor did this first but their setup was more complex, required more interventions and the code is significantly more obscure.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5fa661b8c9b1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Detective Work with Data Science]]></title>
            <link>https://medium.com/100-shades-of-machine-learning/detective-work-with-data-science-1530ef34324c?source=rss-41546b1c4faa------2</link>
            <guid isPermaLink="false">https://medium.com/p/1530ef34324c</guid>
            <category><![CDATA[indexing]]></category>
            <category><![CDATA[ocr]]></category>
            <category><![CDATA[legal]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Farhan Ahmad]]></dc:creator>
            <pubDate>Wed, 16 Apr 2025 07:02:07 GMT</pubDate>
            <atom:updated>2026-05-08T10:02:27.305Z</atom:updated>
            <content:encoded><![CDATA[<h3>What’s There inside the JFK Assassination Records?</h3><h4>Releasing the JFK Assassination Dataset</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h4Vp7hT4G66zj0nwIQ-hNQ.png" /></figure><h3><strong>TL;DR</strong></h3><p>It is impossible to read about the JFK <a href="https://en.wikipedia.org/wiki/Assassination_of_John_F._Kennedy">Assassination</a> without developing a lifelong obsession — especially if you see data science as a form of detective work. We are releasing the text extracted from some 50,000 assassination records on <a href="https://huggingface.co/datasets/farhanhubble/jfk-tell">Hugging Face</a> so other data scientists can put their investigative instincts to the test.</p><h3>The Assassination Records</h3><p>The US President John F. Kennedy(JFK) was assassinated in 1963. Over the years, multiple investigations of the case have produced millions of files known as the JFK Assassination Records. These records contain pieces of evidence, intelligence reports, and reports from analyses of the evidence.</p><p>The US government started releasing these records digitally through <a href="https://www.archives.gov/research/jfk">archives.org</a>. Since 2017, more than fifty thousand files have been uploaded to the website as scanned PDFs.</p><p>While efforts like <a href="https://history-matters.com/index.htm">History Matters</a> have cataloged and indexed the records for manual research our goal is to enable large-scale computational analysis, which may offer new clues, and help substantiate or refute some of the theories.</p><h3>Download and Processing</h3><p>As of April 2025, there are 50,000 unique files published on <a href="http://archives.org/">archives.org</a>, totaling around 50GB, and is going to grow more massive as the publication of new records is ongoing.</p><p>We downloaded the current snapshot of the PDF files and version-controlled them. We then processed each file, page by page, using the Google Gemini LLM API to extract the text as Markdown. The resulting text files were version-controlled again.</p><p>The code can be found in the <a href="https://github.com/farhanhubble/jfk-tell">JFK-Tell</a> Github repository. Text from all pages from a single PDF file was concatenated into a single .md file.</p><h3>Release</h3><p>The original records downloaded as PDF files were released on Hugging Face as the <a href="https://huggingface.co/datasets/farhanhubble/jfk-archives">JFK Archives</a> dataset. Since HF has limits on the size of individual files and the number of files, the dataset was published as five Parquet files.</p><p>The extracted text was published as the <a href="https://huggingface.co/datasets/farhanhubble/jfk-tell">JFK Tell</a> dataset on HF. Here is list of all published resources:</p><ul><li>The Github repo with the code for download, version control, text extraction, and publishing: <a href="https://github.com/farhanhubble/jfk-tell">https://github.com/farhanhubble/jfk-tell</a></li><li>The PDF dataset: <a href="https://huggingface.co/datasets/farhanhubble/jfk-archives">https://huggingface.co/datasets/farhanhubble/jfk-archives</a> 🤗</li><li>The Markdown dataset: <a href="https://huggingface.co/datasets/farhanhubble/jfk-tell">https://huggingface.co/datasets/farhanhubble/jfk-tell</a> 🤗</li><li>The Markdown dataset on Kaggle: <a href="https://www.kaggle.com/datasets/farhanhubble/jfk-tell">https://www.kaggle.com/datasets/farhanhubble/jfk-tell</a></li></ul><h3>Using the Datasets</h3><p>The datasets can be used to benchmark the fidelity of PDF-to-text conversion with LLMs. The Gemini 2.0-Flash model we used was fed with snapshots of PDF pages along with really simple <a href="https://github.com/farhanhubble/jfk-tell/tree/main/prompts/extraction">prompts</a>. We checked a few dozen files and the outputs looked good. However, it will be worthwhile to evaluate the accuracy and structure preservation for a larger sample.</p><p>The idea behind PDF-to-text conversion is to index the text with a semantic search engine which will enable asking questions with RAG, generating summaries, and organizing the records more meaningfully. The index should link back to the original PDFs to make it easier to cross-check details and cite the original.</p><p>Finally, the semantic search backend can be paired with a frontend to build an application that non-technical people can use to search and analyze the information. This seems to be the most compelling use as it sets a precedent in analyzing legal and other high-accuracy documents at scale, with AI tools.</p><h3>Disclaimer</h3><p>The download and conversion of the records is a lossy process. The purpose of making the data available is to foster open source research and no warranties of any kind regarding the accuracy of the data are implied. We are aware of the following issues and many other unknown issues may be present too.</p><ul><li>Some of the PDF URLs from <a href="https://www.archives.gov/research/jfk">archives.org</a> fail to download</li><li>Many files were listed multiple times on different <a href="https://www.archives.gov/research/jfk">archives.org</a> pages and we only downloaded one copy for each such filename. Note that no content-based de-duplication was applied. We relied on <a href="http://archives.org/">archives.org</a> file names alone.</li><li>A few thousand files failed to extract any text. Several of these are reports, etc., where there is little information other than the formatting and seals/stamps, etc.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1530ef34324c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/100-shades-of-machine-learning/detective-work-with-data-science-1530ef34324c">Detective Work with Data Science</a> was originally published in <a href="https://medium.com/100-shades-of-machine-learning">100 Shades of Machine Learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Graceful API Failure in Data Science]]></title>
            <link>https://medium.com/100-shades-of-machine-learning/graceful-api-failure-in-data-science-c607c4eed021?source=rss-41546b1c4faa------2</link>
            <guid isPermaLink="false">https://medium.com/p/c607c4eed021</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Farhan Ahmad]]></dc:creator>
            <pubDate>Fri, 21 Mar 2025 17:09:36 GMT</pubDate>
            <atom:updated>2026-10-15T16:18:08.139Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gffvaHeN7AsHGgJSulRrEQ.png" /><figcaption>Data Science Does Not Have to Be A Programming Anti-Pattern</figcaption></figure><h3>Graceful API Failure 101 for Data Scientists</h3><h4>How to deal with API failures in long-running pipelines</h4><p>First of all, what is unique about handling failures in data science? Isn’t proper failure handling a crucial practice in any programming in general? Well, yes but we see too many data scientists who are oblivious or dismissive of following any software engineering practices in the name of “data science”.</p><p>Secondly, general error handling is easy, at least in Python, the lingua franca of data science. Just throw in a try-except block. Failure handling for APIs is more nuanced though, for two reasons:</p><ul><li>You’re at the mercy of the API server for details of the failure</li><li>Failure handling is more nuanced than just being a report-or-continue decision</li></ul><p>Here we focus on these aspects using a very relevant and real example — using LLM APIs.</p><p>Here is a piece of code we wrote using the google-genai Python SDK.</p><pre>class GeminiClient:<br>  def generate(self,...):<br>    ...<br>    response = self._client.models.generate_content(<br>      model=self._model.value,<br>      contents= attached_files + [prompt],<br>      config=types.GenerateContentConfig(<br>          max_output_tokens=max_tokens,<br>          system_instruction=system_prompt or None,<br>      )<br>    )<br>    ...</pre><p>The whole code can be viewed <a href="https://github.com/farhanhubble/jfk-tell/blob/bf82bffb093b291162e0ffe97b96cd29c5f2819b/ai/google/__init__.py">here</a>. It uses Google’s Gemini API to extract information from PDF files passed as attachments to a prompt. When we run this code on a few thousand files, it generally runs well, except:</p><ul><li>The Gemini server times out once in a while when it gets overloaded</li><li>The server times out reading an uploaded attachment once in a while</li><li>Some files exceed the context window of the model we were using</li></ul><p>When there is no error handling the entire, pipeline containing the code has to be resumed manually. To handle these scenarios we need sophisticated failure handling:</p><ul><li>First, we need to identify the exception type</li><li>Then, for the first two situations, we need to retry a few times before letting the failure cause an exception.</li><li>For the third situation, we may want to either skip over large files or use another strategy, like splitting up into two calls.</li></ul><p>Writing this logic in a try-except block makes the code deeply indented and difficult to follow. Instead, we can use Python’s decorator mechanism to wrap the existing function call in error-handling logic. A wrapped call will then look like this:</p><pre>class GeminiClient:<br><br>  @timeout_wrapper(retry_backoffs=[...], when: Callable[[Exception], bool])<br>  @size_exceeded_wrapper(when: Callable[[Exception], bool])<br>  def generate(self, ...):<br>    ...<br>    response = self._client.models.generate_content(<br>      model=self._model.value,<br>      contents= attached_files + [prompt],<br>      config=types.GenerateContentConfig(<br>          max_output_tokens=max_tokens,<br>          system_instruction=system_prompt or None,<br>      )<br>    )<br>    ...</pre><p>You may read <a href="https://realpython.com/primer-on-python-decorators/">this</a> primer on decorators before following along on the implementation journey below.</p><p>Our two wrappers are gonna look and feel a bit different from each other. Both the wrappers ultimately call the generatemethod. timeout_wrapper takes a list of backoff periods, for which it will wait before retrying, and a filter function to determine if an exception needs retries. size_exceeded_wrapper only takes a filer function.</p><p>The timeout wrapper needs to store its parameters somewhere. So it can either be implemented as a class or more easily as a nested function, like this:</p><pre>def retry_with_backoff(backoffs: List[int], when: Callable[[Exception], bool]):<br>    def decorator(func):<br>        @functools.wraps(func)<br>        def wrapper(self, *args, **kwargs):<br>            for backoff in backoffs:<br>                try:<br>                    return func(self, *args, **kwargs)<br>                except Exception as e:<br>                    if not when(e):<br>                        raise<br>                    time.sleep(backoff)<br>            # Re-raise, if we exhaust all backoffs without success<br>            else:<br>                raise<br><br>        return wrapper<br><br>    return decorator</pre><p>The nested functionsdecorator and wrapper are the standard way of implementing a decorator in Python. The outermost function retry_with_backoff is required to capture the external parameters we want to set.</p><p>The size exceeded wrapper can look like this:</p><pre>def skip_silently(when: Callable[[Exception], bool]):<br>    def decorator(func):<br>        def wrapper(*args, **kwargs):<br>            try:<br>                return func(*args, **kwargs)<br>            except Exception as e:<br>                if not when(e):<br>                    raise<br>                else:<br>                    return (<br>                        &quot;Observations/Remarks:\n\n&quot;<br>                        &quot;File too large to be processed.\n\n&quot;<br>                        &quot;```markdown\n\n```&quot;<br>                    )<br><br>        return wrapper<br><br>    return decorator </pre><p>This is simpler but we may add a strategy to split the oversized file and process each segment in a separate call.</p><p>After fleshing out the failure-handling logic, the filter functions are straightforward. For the timeout wrapper, we define _is_retryable as the filter, which again is composed of two more granular filters.</p><pre>def _is_server_overloaded(e: Exception):<br>    return isinstance(e, ServerError) and e.code == 503 and str(e).find(&quot;The model is overloaded&quot;) &gt;= 0<br><br><br>def _is_file_io_timeout(e: Exception):<br>    return isinstance(e, ReadTimeout)<br><br><br>def _is_retryable(e: Exception):<br>    return _is_server_overloaded(e) or _is_file_io_timeout(e)</pre><p>For the size exceeded wrapper we have a simpler filter, _is_file_size_exceeded :</p><pre>def _is_file_size_exceeded(e: Exception):<br>    return (<br>        isinstance(e, ClientError)<br>        and e.code == 400<br>        and str(e).find(<br>            &quot;The request&#39;s total referenced files bytes are too large to be read&quot;<br>        )<br>        &gt;= 0<br>    )</pre><p>Here’s how we decorate the generate method, more concretely:</p><pre>class GeminiClient:<br><br>  @retry_with_backoff([30, 60], when=_is_retryable)<br>  @skip_silently(when=_is_file_size_exceeded)<br>  def generate(self, ...):<br>    ...<br>    response = self._client.models.generate_content(<br>      model=self._model.value,<br>      contents= attached_files + [prompt],<br>      config=types.GenerateContentConfig(<br>          max_output_tokens=max_tokens,<br>          system_instruction=system_prompt or None,<br>      )<br>    )<br>    ...</pre><p>The diff can be viewed <a href="https://github.com/farhanhubble/jfk-tell/commit/7300cceb59291637b5e5afeeb8c23b3804ab6bdf">here</a>.</p><p>To summarize, here’s how this implementation addresses the two issues with API failures that we pointed out at the beginning:</p><ul><li>The server sends out HTTP error codes along with more specific “error messages”. Our filters make the target failure types more explicit.</li><li>Our decorator wrappers cleanly implement the handling logic and make it easier to control and update the failure handling.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c607c4eed021" width="1" height="1" alt=""><hr><p><a href="https://medium.com/100-shades-of-machine-learning/graceful-api-failure-in-data-science-c607c4eed021">Graceful API Failure in Data Science</a> was originally published in <a href="https://medium.com/100-shades-of-machine-learning">100 Shades of Machine Learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Particle Filters]]></title>
            <link>https://medium.com/knowchow/particle-filters-832a816baa63?source=rss-41546b1c4faa------2</link>
            <guid isPermaLink="false">https://medium.com/p/832a816baa63</guid>
            <category><![CDATA[localization]]></category>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[self-driving-cars]]></category>
            <dc:creator><![CDATA[Farhan Ahmad]]></dc:creator>
            <pubDate>Fri, 09 Dec 2022 09:08:49 GMT</pubDate>
            <atom:updated>2022-12-09T09:08:49.032Z</atom:updated>
            <content:encoded><![CDATA[<h4>Beyond Groping in The Dark for Robots</h4><p><em>A more technical treatment of this blog post was originally published on the </em><a href="https://www.udacity.com/blog/2019/07/particle-filtering-part-1-beyond-groping-in-the-dark-for-robots.html"><em>Udacity blog</em></a>.</p><p><em>Robots use a surprisingly simple but powerful algorithm to find out where they are on a map, a problem called localization by engineers. The algorithm known as particle filtering looks amazingly cool. The first part of this two-part article was published </em><a href="https://blog.udacity.com/2019/07/particle-filtering-part-1-beyond-groping-in-the-dark-for-robots.html"><em>here</em></a><em>. In this first article, we will elucidate the mathematics needed to build your own particle filters.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4L2RmikHUO6zjoMV" /><figcaption>Source: <a href="https://www.nytimes.com/interactive/2016/12/14/technology/how-self-driving-cars-work.html">The Race for Self-Driving Cars</a> — The New York Times</figcaption></figure><p><strong>S</strong>elf-driving cars are designed to behave much like human drivers. A typical self-driving car system is built up of six components. Each of the six components solves a critical problem needed for driving, autonomous or otherwise. The components can be thought of as performing the functions that mimic human drivers, which are:</p><p><strong>Localization</strong>: Where are we? <br><strong>Route planning</strong>: How do we reach the destination?<br><strong>Perception</strong>: What else is there in the surroundings?<br><strong>Prediction</strong>: How will the environment look like a little later?<br><strong>Planning</strong>: What is the course that we will follow in the next few seconds? <br><strong>Control</strong>: How are we going to follow the course we want?</p><p>Localization for humans tends to be very easy. We typically recollect from memory where we are in the world, often aided by a GPS device like a smartphone that helps us identify our location much more precisely on a map. We also use visual clues like navigation signs, lane markings, and our understanding of other vehicles, the road, and the surroundings to ensure that we are always in the right lane, on the right road, and headed in the right direction.</p><p>When it comes to localization, autonomous driving also requires the position and the direction the vehicle is moving in, also called the heading. However, unlike humans, the level of precision required by autonomous vehicles is very high. It does not suffice to just know that a vehicle is in the right lane and going parallel to the lane. High precision is needed for planning. If the localization is off by more than a few centimeters about the position or by more than a few degrees about the heading, the planning component might generate a disastrous course resulting in a collision among many other unwanted outcomes.</p><p>Like humans, autonomous vehicles get their first estimate of position and heading from a GPS. GPS receivers, even the tiny ones found on a smartphone, tend to be quite accurate. However, the precision of the localization information received from a GPS is <a href="https://www.gps.gov/systems/gps/performance/accuracy/">inadequate</a> for safe autonomous driving. For example, the position reported can be off by anywhere from a meter to several meters or in the worst case even more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/996/0*Uu-f3kaVl-NCdhqu" /><figcaption>Source: <a href="https://www.bosch-presse.de/pressportal/de/en/safe-automated-driving-from-bosch-centimeters-make-all-the-difference-176768.html">Bosch</a></figcaption></figure><p>The precision fluctuates frequently and wildly depending on factors like the number of satellites visible to the receiver, reflections of satellite beams off large objects like buildings, and absorption by atmospheric elements.</p><p>If that is not bad enough, the rate at which GPS updates the localization information is also pretty slow, ranging between 1–10Hz. Even at 10Hz, a vehicle moving at a tardy 20Kmph would have moved about half a meter between two GPS updates.</p><p>To mitigate the uncertainty in localization due to movement, autonomous vehicles use inertial measurement units(IMUs). IMUs are fast and accurate and measure a vehicle’s acceleration and rate of turn. This information can then be used to ascertain the position of the car between two successive GPS updates. IMUs tend to accumulate small approximation errors over time but that is not the problem we are dealing with here.</p><p>The core problem the localization system needs to contend with is that the GPS updates are inherently inaccurate. Instead of reporting an exact position, for example, the GPS reports a small area on the ground where a vehicle is likely to be present when the GPS updates itself. We refer to this area as a GPS spot.</p><p>The problem of performing precise localization reduces to the problem of reducing the area of the GPS spot.</p><p>One of the techniques that have been used in robotics for accurate localization, starting from a less accurate one is known as <strong>particle filtering</strong>. It is useful when the environment has a number of landmarks, that is, large, stationary objects that do not frequently change location, appearance, and size. These could be buildings, cellular phone towers, bridge piers, utility poles or traffic signals, and signages.</p><p>To understand how particle filterings we will use an analogy. Suppose you come back tired to a hotel room, turn off the lights and crash in the bed face down. You wake up in the middle of the night and want to find out where you are. Of course, you do remember crashing into the bed but you would want to know which direction you are facing, where the light switches are and where your slippers are. This calls for more accuracy than a faint recollection of being in the bed.</p><p>To quickly orient yourself, you make use of the surroundings. You first imagine yourself in different locations on the bed. Then you sense the ‘landmark’ objects in the surroundings by touching and identifying them. You might push a water bottle off a nearby table or stub your toe against a chair. These observations give you important clues about where you are likely to be in the realm of the hotel bed. Since you remember where the bottle, the table, and the chair were relative to the bed, you are able to infer your position on the bed more are more accurately. With each new observation, you can rule out certain locations on the bed and increase your chances of being in other locations.</p><p>This is exactly how particle filtering works for autonomous vehicles. Armed with an initial GPS spot the vehicle knows its rough position and heading on the map. This rough position is then divided into small imaginary cells. The vehicle also knows which landmarks are there in the vicinity of that rough location. The vehicle then probes the distance to these landmarks using an onboard lidar. The localization system then tallies these measurements against each imaginary cell. As more and more landmarks get probed, most of the cells are ruled out and only a few highly-likely ones remain as probable locations.</p><p>The name particle filtering comes from the fact that the location of each imaginary cell is represented as by particle on the map and we rule out particles by filtering them by calculating distances to landmarks and discarding particles that do not seem to be the right distance away from the landmarks.</p><p>The animation below shows the initial location of the particles spanning a sizeable area, which could be spread across several lanes, roads, and open areas. The sensing or probing of the environment is shown by Blue rays. The green dot representing the most likely position initially jumps around the map but settles very quickly as more and more particles are ruled out by sensing the environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/0*CwirCBSo-v37Xbrr" /><figcaption>Demo: The Red “particles” quickly converge after a few filtering steps.</figcaption></figure><p>Particle filtering is an incredibly simple algorithm that powers many robots and autonomous cars. Nvidia Drive Labs has released a <a href="https://youtu.be/jcKnb65wpWA">short video</a> on how localization can benefit from rich maps, with a variety of landmarks. Uber ATG has published a blog on how they improve GPS accuracy <a href="https://eng.uber.com/rethinking-gps/">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=832a816baa63" width="1" height="1" alt=""><hr><p><a href="https://medium.com/knowchow/particle-filters-832a816baa63">Particle Filters</a> was originally published in <a href="https://medium.com/knowchow">Knowchow</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why are Pytorch models invoked as functions?]]></title>
            <link>https://medium.com/knowchow/why-are-pytorch-models-invoked-as-functions-dc2a8b67ac07?source=rss-41546b1c4faa------2</link>
            <guid isPermaLink="false">https://medium.com/p/dc2a8b67ac07</guid>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Farhan Ahmad]]></dc:creator>
            <pubDate>Fri, 09 Dec 2022 08:48:34 GMT</pubDate>
            <atom:updated>2022-12-09T08:48:34.470Z</atom:updated>
            <content:encoded><![CDATA[<p>For a long time, I have wondered why we invoke a Pytorch model itself, like, model(input,targets)instead of invoking its forward(inputs, targets) method. Today I went looking for answers to this design oddity.</p><p>A model in scikit-learn is relatively straightforward to use. You create a model which is an object of a model class, then pass the data through the model by invoking its fit() method.</p><pre>from sklearn.linear_model import LinearRegression<br><br># Create a LinearRegression model<br>model = LinearRegression()<br><br># Train the model on the training data<br>model.fit(inputs, targets)</pre><p>One would expect similar behavior from a Pytorch model. Define a model and then call its forward() method to pass the data through the model. Except, in Pytorch we never call the forward() method directly, we instead invoke the model itself 😬, like in the invocation model(inputs) in the code below.</p><pre># Import PyTorch<br>import torch<br><br># Define the model architecture<br>class Net(nn.Module):<br> def __init__(self):<br> super(Net, self).__init__()<br> self.fc1 = nn.Linear(10, 5)<br> self.fc2 = nn.Linear(5, 1)<br> <br> def forward(self, x):<br> x = self.fc1(x)<br> x = self.fc2(x)<br> return x<br># Create a new instance of the Net class<br>model = Net()<br><br># Loop through the training data and make predictions<br>for inputs, targets in training_data:<br>  # Pass the input data through the network<br>  predictions = model(inputs)</pre><p>The variable model is being invoked as a function or method in model(inputs).</p><p>This is possible as torch.nn.Module implements the method __call__() and object in Python that define a __call__() method become callable. The Python documentation says:</p><blockquote>Instances of arbitrary classes can be made callable by defining a <a href="https://docs.python.org/3/reference/datamodel.html?highlight=__call__#object.__call__">__call__()</a> method in their class.</blockquote><p>But why does Pytorch attempt to make an object work like a function? If we look at the <a href="https://github.com/pytorch/pytorch/blob/6145ac07b5431a95e7d29f84ab979b332883b4ea/torch/nn/modules/module.py#L1065">code</a> for the torch.nn.Module class, instead of defining __call__ , it is actually set to a private method, appropriately named, _call_impl</p><pre>__call__ : Callable[…, Any] = _call_impl</pre><p>Inspecting <a href="https://github.com/pytorch/pytorch/blob/6145ac07b5431a95e7d29f84ab979b332883b4ea/torch/nn/modules/module.py#L1009">_call_impl</a> reveals the mystery. The method ultimately invokes the user-defined forward() method but it can carry out some additional work, for example, calling a different method, _slow_forward(), if the code is being profiled.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dc2a8b67ac07" width="1" height="1" alt=""><hr><p><a href="https://medium.com/knowchow/why-are-pytorch-models-invoked-as-functions-dc2a8b67ac07">Why are Pytorch models invoked as functions?</a> was originally published in <a href="https://medium.com/knowchow">Knowchow</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reinventing Histograms Part 4]]></title>
            <link>https://medium.com/knowchow/summary-stats-from-frequency-density-d84ed21fc4ac?source=rss-41546b1c4faa------2</link>
            <guid isPermaLink="false">https://medium.com/p/d84ed21fc4ac</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[k-12-education]]></category>
            <dc:creator><![CDATA[Farhan Ahmad]]></dc:creator>
            <pubDate>Wed, 24 Jun 2020 12:50:27 GMT</pubDate>
            <atom:updated>2020-07-04T13:05:51.442Z</atom:updated>
            <content:encoded><![CDATA[<h4>Summary statistics from frequency density histograms</h4><p><a href="https://medium.com/knowchow/reinventing-histograms-part-1-e415d62f743d"><strong><em>Part 1</em></strong></a><strong><em> of the series dealt with frequency and probability histograms as visual summaries of data, and the type of questions that can be answered with histograms. </em></strong><a href="https://medium.com/@farhanhubble/reinventing-histograms-part-2-764ad9c5e3e"><strong><em>Part 2</em></strong></a><strong><em> was focused on deriving summary statistics from histograms. </em></strong><a href="https://medium.com/knowchow/reinventing-histograms-part-3-frequency-density-224292ec5cd4"><strong><em>Part 3</em></strong></a><strong><em> was an introduction to histograms for continuous data and the need for frequency density histograms. In this part, we will look at how to compute summary statistics from frequency density histograms.</em></strong></p><h3>Recap</h3><p>In <a href="https://medium.com/knowchow/reinventing-histograms-part-3-frequency-density-224292ec5cd4">part 3</a>, we built a frequency density histogram for the area of 130 houses. The bins we used were <strong><em>620–1120sqft</em></strong>, <strong><em>1120–162-sqft</em></strong>, <strong><em>2120–2620sqft</em></strong>, <strong><em>2620–3120sqft</em></strong>, <strong><em>3120–3620sqft</em></strong>, <strong><em>3620–4120sqft</em></strong> and <strong><em>4120–4620sqft.</em></strong> The computed frequency densities were:<strong><em> </em></strong>0.026, 0.042, 0.064, 0.062, 0.034, 0.018, 0.008, 0.006.The histogram looked like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*8fGaxxmuTfvD-Eq-LeKCUw.png" /><figcaption>Frequency density histogram for 130 houses, built using 500sqft. bins</figcaption></figure><h3>Summary Statistics from Frequency Density Histograms</h3><p><a href="https://medium.com/@farhanhubble/reinventing-histograms-part-2-764ad9c5e3e">Part2</a> of the series discussed deriving summary statistics. While the definitions remain the same, the procedures change slightly as the <strong><em>x-axis </em></strong>now has bins instead of exact numbers and the <strong><em>y-axis </em></strong>has frequency densities instead of frequencies(count).</p><h4>Median</h4><p>The median is the value of the area that bisects the <strong>sorted</strong> data. The data, in this case, is the number size of the houses, which have been binned. For a frequency density histogram calculating the median is a four-step process:</p><ol><li>Starting at the left-most bin, multiply each bin’s width by its <strong>frequency density</strong> to get back the <strong>frequency</strong> of that bin.</li><li>Sum up the frequencies to get the cumulative frequency <strong>up to</strong> every bin.</li><li>Determine the bin where the cumulative count reaches half the value of the total (number of houses, 130/2 = 65)</li><li>Determining the exact point within the bin where the cumulative count reaches the halfway point.</li></ol><p>In our case, the frequency takes on the values:</p><pre>0.026*500=13, 0.042*500=21, 0.064*500=32, 0.062*500=31, 0.034*500=17, 0.018*500=9, 0.008*500=4, 0.006*500=3</pre><p>The cumulative frequency increases like:</p><p>[13,13+21,13+21+32, 13+21+32+31, 13+21+32+31+17, 13+21+32+31+17+9, 13+21+32+31+17+9+4, 13+21+32+31+17+9+4+3] <br>or [13, 34, 66, 97, 114, 123, 127, 130]</p><p>and crosses the halfway mark of 65 after we include bin 3.</p><p>So the median has to lie in bin 3. The width of the bin is 500 and the cumulative frequency changes from 34 to 66 across the bin. We want to determine where it reaches the value 65. Here’s how to do that:</p><p>The change in the cumulative frequency across the bin is 66–34, which is of course 32, the bin width. The desired change is 65–34 or 31. So the desired bin width would be 31/32*500 or 484.36. Bin 3 starts at 1620sqft. so the exact point where the cumulative frequency reaches 65 would be 1620+484.36 or 2104.36</p><h4>Mode</h4><p>Calculating the mode is a matter of looking up the tallest bar in the histogram. Bin 3 is the tallest one in our case. However, because of the continuous nature of the data on the <strong><em>x-axis</em></strong>, the mode is not a unique number. Instead, any value within the bin is considered <strong>a mode</strong>. In this case, any value in the range 1620–2120 is <strong>a mode</strong>.</p><h4>Mean</h4><p>Finding the mean (house area) requires adding up the area of all the 130 houses and dividing by the total number of houses (130). However, we do not know the area of individual houses (it was discarded during binning). Instead, the midpoint of each bin is considered as the representative area of every house in the bin. The process to estimate the mean is as follows:</p><ol><li>Convert frequency densities to frequencies(counts) by multiplying them with the bin widths.</li><li>For each bin, select the midpoint of the bin as the value (area) representative of the whole bin.</li><li>Multiply this value(area) by the corresponding frequency(count).</li><li>Sum up the products from step 3 to get the are of all the (130) houses.</li><li>Divide the sum from step 4 by the number of houses to get an estimate of the mean house size.</li></ol><p>In our case the steps would look like this:</p><ol><li>The frequencies, as computed earlier for median computation, are 13, 21, 32, 31, 17, 9, 4, 3.</li><li>The representative areas are 620+1120/2, 1120+1620/2 … 4120+4620/2 or 870, 1370, …, 4370.</li><li>Estimate the sum of areas of all house: 13*870 + 21*1370 + 32*1870 + 31*2370 + 17*2870 + 9*3370 + 4*3870 + 3*4370 = 281100.</li><li>Divide the sum by the number of houses 281100/(13+21+31…+4+3) = 28100/130 = 2162.32.</li></ol><p>So the mean area is 2162.32sqft.</p><h4>Variance and Standard Deviation</h4><p>The definition of variance and its calculation is exactly the same as discussed in <a href="https://medium.com/@farhanhubble/reinventing-histograms-part-2-764ad9c5e3e">part 2</a>. Here’s a quick summary of how to calculate variance:</p><ol><li>Find the difference between each value(area) and the mean.</li><li>Square the difference and sum up all squared differences.</li><li>Divide by the number of values.</li></ol><p>Since our data is binned, we will again use the bin mid-points as values. The differences will be: 870–2162.32, 1370–2162.32, ..., 4370–2162.32 or -1292.32, -792.32, ..., 2207.68. These can be squared and summed up and then divided by the values, 130.</p><p>The standard deviation will just be the square root of the variance.</p><p><strong><em>In this part, we learned how to calculate summary statistics from frequency density histograms. Next up, in part 5 we learn about probability density histograms which are also required when dealing with continuous data.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d84ed21fc4ac" width="1" height="1" alt=""><hr><p><a href="https://medium.com/knowchow/summary-stats-from-frequency-density-d84ed21fc4ac">Reinventing Histograms Part 4</a> was originally published in <a href="https://medium.com/knowchow">Knowchow</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reinventing Histograms Part 3]]></title>
            <link>https://medium.com/knowchow/reinventing-histograms-part-3-frequency-density-224292ec5cd4?source=rss-41546b1c4faa------2</link>
            <guid isPermaLink="false">https://medium.com/p/224292ec5cd4</guid>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[k-12-education]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Farhan Ahmad]]></dc:creator>
            <pubDate>Wed, 24 Jun 2020 10:35:55 GMT</pubDate>
            <atom:updated>2020-07-04T13:04:19.160Z</atom:updated>
            <content:encoded><![CDATA[<h4>Density histograms for continuous data.</h4><p><a href="https://medium.com/knowchow/reinventing-histograms-part-1-e415d62f743d"><strong><em>Part 1</em></strong></a><strong><em> of the series dealt with frequency and probability histograms as visual summaries of data, and the type of questions that can be answered with histograms. </em></strong><a href="https://medium.com/@farhanhubble/reinventing-histograms-part-2-764ad9c5e3e"><strong><em>Part 2</em></strong></a><strong><em> was focused on deriving summary statistics from histograms. In this post, we will see why we need frequency density histograms when dealing with continuous data.</em></strong></p><h3>Recap</h3><p>So far we have been looking at histograms of the number of rooms across houses in a neighborhood. We started by building a summary table for 4595 house, that looked like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/479/1*1UYUiJlB7gb8rxKP_cflKw.png" /><figcaption>Summary Table with Counts and Probabilities</figcaption></figure><p>We then converted the table into frequency and probability histograms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*y5cdce5eHEGL1r8SUeYsuQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*fQy4JZXRR3apzsUgwPXVvg.png" /><figcaption>Frequency and Probability Histograms Corresponding to The Table Above</figcaption></figure><p>In these histograms, the quantity on the <strong><em>x-</em>axis </strong>is the number of bedrooms. There were multiple houses, sometimes several thousand, with the same number of rooms. This repetition allowed us to summarize the data. But what happens if the data does not have any repetition?</p><h3>Dealing with Non-repeating Data</h3><p>Suppose we want to plot the histogram of the area of 130 houses in a neighborhood. When sorted, the data might look like this: [620, 650, 670, 850, 910, 920, 940, 970, 990, 1010, 1040, …,3740, 3760, 3950, 4260, 4410, 4600]. The numbers represent the living area reported in square feet, rounded to the nearest integer. There seems to be no repetition in this data. If we try to build a summary table it will look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/1*Jac4QjKezj_DMVVVMxQM3w.png" /><figcaption>A summary of the area of 130 houses. The summary is not useful as the data has no repetition.</figcaption></figure><p>The summary table is not very useful as each value of the area appears only once and, as a result, the summary has as many rows as the data itself. If we went ahead and plotted a frequency histogram from this table it would look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vkdTfzv6W22u587y.png" /><figcaption>Histogram of Living Area Across 130 Houses (Truncated after 1000sqft because of less space).</figcaption></figure><p>There is a lot of seemingly missing data on the <strong><em>x-axis</em></strong>. This histogram is far from a concise description of the data, unlike the histograms we have seen earlier. We can make it succinct by binning the data so that instead of counting the number of houses that have an exact living area, like 620sqft, we can count the number of houses whose living area falls in a range, like 620sqft-1120sqft. Here’s how the summary table looks like when we use bins with a range of 500sqft:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*BcqsPOm_GCGxPaaoFJsBvQ.png" /><figcaption>Summary Of House Counts by Living Area (binned into bins of size 500sqft.)</figcaption></figure><p>With this summary we can build a useful histogram that looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*ipNpfGV2HpH7jpVZnWLWkg.png" /><figcaption>Histogram of Living Area of 130 houses, Built Using 500sqft Bins</figcaption></figure><p>There is a catch though. We can no more look up the number of houses with an exact area. For example, the number of houses that are exactly 800sqft. in size, can only be incorrectly calculated as 13, since 800sqft. falls in the first bin (620–1120sqft.) but the number would be incorrect. Why? Because 13 is the number of all houses that have an area in that range and there could be many other house sizes in the same bin. When we bin the list, we lose information about the size of individual houses and only retain aggregate information about each bin as a whole.</p><p>Although this imprecision is inevitable, it can be controlled by changing the bin width. A wider bin width means more imprecision but a more succinct summary while a narrower bin has less imprecision but then the summary, that is, the number of rows in the summary table or the number of bars in the histogram is not as short.</p><p>The imprecision is often acceptable and useful in practice. When we want to search for houses we usually look for a range of sizes like 1000sqft-1200sqft and are seldom interested in finding houses that have an exact area, like, 1000sqft.</p><h3>Introducing Frequency Density</h3><p>Well aware of the fact that it is pointless to try to find the number of houses with an exact area, let us see if we can find the number of houses that have an area in a range, say 1800–2000sqft. This range falls entirely within the third bin (1620–2120sqft.). The histogram tells us that the number of houses for this bin is 32 but our range is narrower than the bin width and we cannot precisely find the number we are interested in. However, we can use a trick.</p><p>We first calculate the ratio of our range’s width to the bin’s width, which is (2000–1800)/(2120–1620) or 0.4. Since our range is just a fraction (0.4) of the bin width, we can <strong>estimate </strong>that the number of houses will be the same fraction of the number of houses in the bin. To get the desired number we can multiply 32 by 0.4 to get about 13 as the number of houses. The overall calculation looks like this:</p><pre>((2000–1800) / (2120-1620)) * 32</pre><p>This can be re-written as:</p><pre>(32 / (2120–1620)) * (2000–1800) </pre><p>Why do we want to do that? Because all the numbers in the first factor: 32/(2120–1620) are known to us at the time of constructing the histogram, while the numbers in the second factor will change every time, depending on the range of house sizes we want to inquire about.</p><p>The ratio 32/(2120–1620) = 0.064 is known as frequency density and can be pre-calculated for every bin by dividing the height of the bin by its width. The frequency densities, in our case, are 13/500, 21/500, 32/500, 31/500, 17/500, 9/500, 4/500, 3/500 or 0.026, 0.042, 0.064, 0.062, 0.034, 0.018, 0.008, 0.006 . When plotted, the frequency density histogram looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*8fGaxxmuTfvD-Eq-LeKCUw.png" /><figcaption>Frequency density histogram for 130 houses, built using 500sqft. bins</figcaption></figure><p>With a frequency density histogram like this, finding the number of 1800–2000sqft. houses is straightforward:</p><ol><li>Find the bin in which the range lies, that is bin 3 here.</li><li>Multiply the frequency density of the bin (0.064) by the width of the range (2000–1800) = 200 to get the estimated number of houses as 200*0.064 ~ 13</li></ol><p>What if we want to <strong>estimate </strong>the number of 1000–1200sqft.? This range does not fall inside one bin but instead straddles two bins: 620–1120sqtf. and 1120–1620sqtf. The solution is still straightforward. We split the range into two sub-ranges: 1000–1120sqft that lies entirely within the first bin and 1120–1220 that lies entirely in the second bin. We can then repeat the two steps above twice and sum up the results to get: (1120–1000)*0.026 + (1200–1120)*0.042 or roughly 6 houses.</p><p><strong><em>In this part, we were introduced to the need for frequency density histograms when dealing with continuous data. In </em></strong><a href="https://medium.com/knowchow/summary-stats-from-frequency-density-d84ed21fc4ac"><strong><em>part 4</em></strong></a><strong><em> we will learn to compute summary statistics from frequency density histograms.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=224292ec5cd4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/knowchow/reinventing-histograms-part-3-frequency-density-224292ec5cd4">Reinventing Histograms Part 3</a> was originally published in <a href="https://medium.com/knowchow">Knowchow</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reinventing Histograms Part 2]]></title>
            <link>https://medium.com/knowchow/reinventing-histograms-part-2-764ad9c5e3e?source=rss-41546b1c4faa------2</link>
            <guid isPermaLink="false">https://medium.com/p/764ad9c5e3e</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[k-12-education]]></category>
            <category><![CDATA[education-technology]]></category>
            <dc:creator><![CDATA[Farhan Ahmad]]></dc:creator>
            <pubDate>Sat, 18 Apr 2020 11:07:14 GMT</pubDate>
            <atom:updated>2020-07-04T13:02:41.887Z</atom:updated>
            <content:encoded><![CDATA[<h4>Summary statistics from histograms.</h4><p><a href="https://medium.com/knowchow/reinventing-histograms-part-1-e415d62f743d"><strong><em>Part 1</em></strong></a><strong><em> of this series showed how histograms offer a visual summary of a sequence of numbers. We saw what kind of questions can be answered with frequency and probability histograms. In this post, we will look at how the same histograms can be used to calculate the so-called summary statistics of a sequence of numbers.</em></strong></p><p>Let’s begin with what summary statistics are. In simple words, summary statistics are single number summaries. We have seen in <a href="https://medium.com/knowchow/reinventing-histograms-part-1-e415d62f743d">Part 1</a> how histograms themselves are summaries. Frequency histograms summarize the counts or frequencies of the numbers. A histogram is built from a summary table. If we summarize the frequency of houses in a locality by the number of rooms they have, the summary table may look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/253/0*7KE9AVmEDxmMWifS.png" /><figcaption>A summary of the number of rooms for 4595 houses.</figcaption></figure><p>Let’s see what summary statistics are commonly used and how they relate to histograms.</p><h4><strong>Mean</strong></h4><p>The mean of a sequence of numbers is a single value that is the most representative of the entire sequence. Let us reuse our example of surveying houses and recording the number of rooms. The list of the number of rooms looked like <strong>[2,4,3,1,2,3,3,7,4…..] </strong>and had 4595 numbers in it<strong>.</strong> Since the list has far too many 3’s and 4’s in it, and fewer numbers of 1’s, 2’s, 5’s, 6’s and 7’s, intuitively, 3 or 4 are the numbers that are more prominent and thus more representative.</p><p>The mean is calculated as the sum of all these numbers divided by the total count. We can definitely sum up the numbers manually but the summary table above can be used to cut down on the work. We know that 1 occurs 38 times so instead of summing 1, 38 times, we can multiply them to get the sum of all 1’s in the sequence, which is 1<strong>x</strong>38. Similarly, 2’s occur 566 times so their sum will be 2<strong>x</strong>566 and so on. The mean is then (1x38 + 2x566 + 3x2032 + 4x1531 + 5x353 + 6x61 + 7x14) / (38+566+2032+1531+353+61+14) = 15619/4595 = 3.399. So the mean is neither 3 nor 4, but actually lies between 3 and 4.</p><p>The frequency histogram corresponding to the summary table above looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*e6IYlFGgX10lLNa0.png" /><figcaption>A Histogram of Number of Bedrooms in 4595 Houses with counts(frequencies) on the y-axis.</figcaption></figure><p>To calculate the means from this histogram, multiply the number on the x-axis(number of rooms), by the number on the y-axis(frequency), sum up all the products, and divide the sum so obtained by the sum of frequencies.</p><p>The calculation so far uses frequencies, but we can get the same results from probabilities too. If we bring the denominator inside the expression for the mean, it becomes (1x38/4595 + 2x566/4595 + 3x2032/4595 + 4x1531/4595 + 5x353/4595 + 6x61/4595 + 7x14/4595. The fractional ratios 38/3595, 566/4595 are nothing but probabilities.</p><p>If we look at the probability histogram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*Q_IwA8CGJ4GwG_UX.png" /><figcaption>A Histogram of Number of Bedrooms in 4595 Houses with proportions(probabilities) on the y-axis.</figcaption></figure><p>the mean can be calculated by simply multiplying the number on the <strong><em>x-axis</em></strong>(number of rooms) by the number of the <strong><em>y-axis</em></strong>(probability) and adding up the products. There is no need to divide by the sum of values on the <strong><em>y-axis </em></strong>because probabilities sum up to 1 anyway.</p><h4>Median</h4><p>The median represents the center of a <strong>sorted</strong> sequence. It is literally defined to be the number at the mid-point of the sequence when it is <strong>sorted. </strong>If we sort the list in increasing order, it will look like <strong>[1, 1, 1….(38 times), 2, 2, 2, 2,…. (566times), 3, 3, 3, 3(2032 times) …. 7,7,7,7 (14 times)]</strong>.</p><p>The center point will be at the position (4595+1)/2 = 2298. Locating the 2298th element is easier if we add a <strong><em>cumulative count </em></strong>column to the summary table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/381/1*AYgHylwGzJbuuoI9BRGGfQ.png" /><figcaption>Summary Table with Cumulative Counts (Frequencies)</figcaption></figure><p>The cumulative count against a row is the sum of all counts up to and including the row. For example the cumulative count for 3 bedrooms in 2636 (38+566+2032) and tells us that there are 2626 houses with 3 or fewer rooms.</p><p>It is evident that the 2298th house will fall in the third row. Why? Because the cumulative count up to the second row is 604 but it jumps to 2636 for the third row. So the third row corresponds to houses with position 605 to 2636 in the sorted list and the position 2636 lies in between. Since all numbers in this range 3, the median is 3.</p><p>Locating the median on a frequency or probability histogram can be done by first summing up bar heights until we breach one half of the total count. The median then is the x-value of the bar at which we breach the half-count.</p><h4>Mode</h4><p>The mode of a sequence is the most frequent number in the sequence. On a histogram, the value on the x-axis corresponding to the tallest bar is the mode. We already saw this in part 1, when we declared that 3 rooms are the most common or frequent, although at that time we did not use the term Mode.</p><h4>Variance</h4><p>Variance measures the ‘deviation’ of the numbers in a sequence from their mean. The mean is supposed to be the value that is most representative of the entire sequence, and variance measures how well the mean represents the members of the sequence.</p><blockquote>The variance is defined as the sum of the square of the deviation of each number in the sequence from the mean of the sequence, divided by the total count of the sequence.</blockquote><p>In our case, the mean is 3.399 and the list of numbers has integer values between 1 and 7. For each 1 in the list, its difference from the mean is (1–3.399). To calculate the variance, we square this difference, (1–3.399)². From the histogram (or summary table), we know that 1 repeats 38 times, so we multiply this value by 38 and get 38*(1–3.399)². For 2&#39;s, the difference is (2–3.399)² and since 2 has a frequency of 566, we get 566*(2–3.399)².</p><p>The variance is the mean of these values, that is (38*(1–3.399)² + 566*(2–3.399)² + 2032*(3–3.399)² + 1531*(4–3.399)² + 353*(5–3.399)² + 61*(6–3.399)² + 14*(7–3.399)²) / 4595 = 0.81</p><p>Once again, we can move the denominator, 4595, inside each term. The expression then looks like ((38/4595)*(1–3.399)² + (566/4595)*(2–3.399)² + (2032/4595)*(3–3.399)² + (1531/4595)*(4–3.399)² + (353/4595)*(5–3.399)² + (61/4595)*(6–3.399)² + (14/4595)*(7–3.399)²). The fractional ratios (38/4595), (566/4595), ... , (14/4595) are probabilities that can be read off a probability histogram.</p><p>To understand what a variance of 0.81 tells us as a summary let us look at the next section.</p><h4>Standard Deviation</h4><p>When we calculate the variance, the individual differences are squared to make them all positive. To get an accurate picture of the deviation, we need to undo the squaring. The square root of the variance, better known as the <strong><em>standard deviation </em></strong>is the summary statistic that tells us how well our mean represents the entire sequence. In our case, √(0.81) = 0.9 is the standard deviation.</p><p>In our case, the mean number of rooms being 3.399, a standard deviation of 0.9 tells us that the number of rooms deviates, on average, by 0.9</p><p>We have seen how to compute the variance and standard deviation from a histogram. It turns out that everything else being equal, the variance or standard deviation is smaller if the histogram is narrow with a pointed peak and the variance is higher if the histogram is broad or spread out.</p><p><strong><em>We now know how to derive summary statistics from histograms. In part 3, we will learn to deal with missing data on the x-axis, like what happens if there are no 5 bedroom houses in the neighborhood.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=764ad9c5e3e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/knowchow/reinventing-histograms-part-2-764ad9c5e3e">Reinventing Histograms Part 2</a> was originally published in <a href="https://medium.com/knowchow">Knowchow</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reinventing Histograms Part 1]]></title>
            <link>https://medium.com/knowchow/reinventing-histograms-part-1-e415d62f743d?source=rss-41546b1c4faa------2</link>
            <guid isPermaLink="false">https://medium.com/p/e415d62f743d</guid>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[k-12-education]]></category>
            <category><![CDATA[education-reform]]></category>
            <dc:creator><![CDATA[Farhan Ahmad]]></dc:creator>
            <pubDate>Sun, 05 Apr 2020 14:10:43 GMT</pubDate>
            <atom:updated>2020-07-04T13:01:14.555Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why do we need histograms?</h4><p><em>A Histogram is a visual summary of a list of numbers. In this post, using little more than basic counting and our intuition of chance, we see how histograms emerge from a desire to quickly count things and measure how frequent those things are compared to other things. We will start off with counting things.</em></p><h3>Counts and Histograms</h3><p>In the simplest terms, histograms visually summarize the count of some quantity. For example, suppose you went around a neighborhood, counting the number of bedrooms in each of the 4595 houses. Initially, you would not know anything about the number of bedrooms. Suppose the first house you went to had 2 bedrooms, you noted down <strong>[2]</strong>. The next house might have 4 bedrooms and now your entry would read <strong>[2,4]</strong>. Since most houses have only a small number of bedrooms, by the time you had surveyed all houses, you would have had a long list with lots of small numbers in it: <strong>[2,4,3,1,2,3,3,7,4…..]</strong>.</p><p>This list will be very long, and, other than the contentment of having surveyed so many houses, you cannot do much with the list. You cannot even tell how many 5 bedroom houses there are without counting through the entire list. This is where the need for summarization is born.</p><p>The easiest way to summarize this list would be to count the occurrences of the numbers in the list. Suppose you do that and find that the number <strong>1 </strong>appeared 38 times in the list. This tells you that there must be 38 1-bedroom houses among the 4595 houses you surveyed. You note this fact down. Similarly, you may come across the number <strong>2</strong> 566 times, and the number 3 2032 times, and so on. You could record this summary in a table like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/253/0*Nn1402sLlCDLffHj.png" /><figcaption>A summary of the number of houses according to the number of rooms they have.</figcaption></figure><p>With a summary like this, you can tell, at a glance, that there are exactly 353 <strong>5-bedroom</strong> houses. You can also tell that the number of bedrooms ranges from 1 to 7. If you add up the numbers in the second column, you get the total number of houses too, which should be 4595. You can also tell that <strong>3-bedroom</strong> houses are more common than <strong>4-bedroom</strong> houses. That is a ton of useful information. Without summarizing the list, you would have to go through the huge list every time you wanted to answer one of those questions.</p><p>This summary, when plotted becomes a histogram!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*N2fFq9AiRZCnX5Ev.png" /><figcaption>A Histogram of Number of Bedrooms in 4595 Houses with counts(frequencies) on the y-axis.</figcaption></figure><p>The <strong><em>x-</em></strong>axis has the number of bedrooms on it. While the <strong><em>y-</em></strong>axis is the number of houses with that many bedrooms. The <strong><em>x-</em></strong>axis skips alternate numbers so that it does not look very cluttered but it is clearly visible that there are 7 bars for the numbers 1 to 7. The number of houses with 0 or 8 bedrooms is 0 and they show up on the <strong><em>x-</em></strong>axis to emphasize just that. The numbers on the x-axis are placed in the middle of the bars, just to avoid any confusion about which bar corresponds to which number(of bedrooms).</p><p>The histogram lets us answer the same questions that we did with the summary table. To report the count of 5-bedroom houses one needs to look up the height of the bar corresponding to the number 5 on the <strong><em>x-</em></strong>axis.</p><p>It is straightforward to know the range of the number of bedrooms. The <strong><em>x-</em></strong>axis tells us it is from 1 to 7.</p><p>To get the total number of houses, add up the heights of all the bars. This should once again be 4595.</p><p>The histogram clearly shows that 3-bedroom houses are more common than 4-bedroom ones. Houses with too few bedrooms or too many bedrooms become less and less common.</p><p>The counts on the <strong>y-</strong>axis are called <strong><em>frequencies </em></strong>too, as they express how frequent the corresponding quantity on the <strong><em>x-</em></strong>axis is. So 3-bedroom houses are very frequently encountered while 7-bedroom ones are very infrequent.</p><p>We can also convert our intuition about frequencies into numbers by simply taking the ratio of bar-heights. In this case, the bars have heights 2032 and 1531(can be read from the table) and the ratio, 2032/1531 is about 1.33. So we can now say that not only are 3-bedroom apartment more common than 4-bedroom ones, they are 1.33 times more frequent. Wow! That is one new bit of information gleaned from a histogram.</p><p>The histogram we just saw is a frequency histogram because it has frequencies on the <strong><em>y-</em></strong>axis.</p><blockquote>The sum of frequencies is the total number of items in our data.</blockquote><h3>Chances and Histograms</h3><p>A frequency histogram tells us how which type of houses are more common and which ones are less. This is done by taking the ratio of heights or frequencies of the bars in the histograms. There is another type of comparison we can do. We can take the ratio of the frequency of a particular bar to the sum of frequencies of all the bars.</p><p>For 3-bedroom houses, this would be 2032/4595 or about 0.44. This ratio is the proportion or share of 3-bedroom houses among all houses. Note that this number will always be between 0 and 1, inclusive of both. This is also called <strong><em>probability</em></strong>. The sum of probabilities will invariably be 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/479/1*46gSWxxdorSc_ihBC-021g.png" /><figcaption>Summary table updated with proportion or probability values.</figcaption></figure><blockquote>The sum of probabilities is always 1.</blockquote><p>We can actually build a new histogram with probabilities on the <strong><em>y-</em></strong>axis and it looks very similar to the frequency histogram, except that the height of each bar is now a fraction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*w-Eeb_Mn9anxruSE.png" /><figcaption>A Histogram of Number of Bedrooms in 4595 Houses with proportions(probabilities) on the y-axis.</figcaption></figure><p>With a probability histogram, we can answer the questions that we could with a frequency histogram and some more.</p><p>So we can still compare two bars by taking the ratio of their heights. The ratio of probabilities of 3-bedroom to 4-bedroom is 0.442/0.333 = 1.33. This should not be surprising given that while calculating probabilities each frequency was divided by the same number, 4595, and taking ratios will cancel out this number.</p><p>We can still say that the number of rooms varies between 1 and 7.</p><p>To report the frequency of a particular type of house is a matter of multiplying back by the sum of frequencies, 4595. So, the total number of 3-bedroom houses will be 0.442*4595, roughly 2031, only slightly off from the actual value due to rounding.</p><p>Okay, so what is that extra information that we get from a probability histogram but not from a frequency histogram?</p><p>Probabilities quantify our intuition of chance. Suppose you go to a random house in the neighborhood. How many bedrooms is the house most likely to have? It seems the answer should be three. 3-bedroom houses are the most common ones after all. In fact, we can say that the chances of the house having 3 bedrooms is about 0.44, while the chances of it having 4 bedrooms are about 0.33 and so on.</p><p>How does that help us? Well, if I am looking for a small 1-bedroom house, and my strategy is to randomly inquire house owners, given that the chance of finding a 1-bedroom house is just 0.008, I will immediately question my strategy of randomly asking around.</p><blockquote>Probabilities quantify chances and help make informed choices.</blockquote><p><strong><em>Now that we know a thing to two about histograms and how useful they are, in </em></strong><a href="https://medium.com/@farhanhubble/reinventing-histograms-part-2-764ad9c5e3e"><strong><em>Part 2</em></strong></a><strong><em>, we will see how to derive more succinct summaries from histograms.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e415d62f743d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/knowchow/reinventing-histograms-part-1-e415d62f743d">Reinventing Histograms Part 1</a> was originally published in <a href="https://medium.com/knowchow">Knowchow</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>