<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Muhammad Karim, Ph.D. on Medium]]></title>
        <description><![CDATA[Stories by Muhammad Karim, Ph.D. on Medium]]></description>
        <link>https://medium.com/@raju.monjurulkarim?source=rss-c3bc5b95942e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*O7j7-qynacBlEPPJexVtww.jpeg</url>
            <title>Stories by Muhammad Karim, Ph.D. on Medium</title>
            <link>https://medium.com/@raju.monjurulkarim?source=rss-c3bc5b95942e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 19:55:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@raju.monjurulkarim/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Cameras Could Measure Traffic. We Still Couldn’t Show the Video]]></title>
            <link>https://medium.com/@raju.monjurulkarim/the-cameras-could-measure-traffic-we-still-couldnt-show-the-video-4868dd7199b2?source=rss-c3bc5b95942e------2</link>
            <guid isPermaLink="false">https://medium.com/p/4868dd7199b2</guid>
            <category><![CDATA[transportation]]></category>
            <category><![CDATA[youtube]]></category>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[video-editing]]></category>
            <dc:creator><![CDATA[Muhammad Karim, Ph.D.]]></dc:creator>
            <pubDate>Tue, 22 Sep 2026 22:46:22 GMT</pubDate>
            <atom:updated>2026-09-22T22:46:22.287Z</atom:updated>
            <content:encoded><![CDATA[<p>How transportation safety research led me to rethink redaction: a review-first, offline workflow for faces, plates, and anything else you need to hide.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kbu9qTYlo-SYqXFM6Ws0og.png" /></figure><p>In transportation safety, video is not just media. It is measurement data.</p><p>I work with footage from roadside cameras, intersections, and agency vehicles to study vehicle trajectories, estimate speed, and understand conflicts between road users. The same frames that let me measure a near-miss can also contain an identifiable face or license plate.</p><p>That creates a practical boundary. In the projects I work on, raw footage is collected, stored, and analyzed under the rules of the agency that provided it. In some cases, the footage cannot leave an agency-controlled computer. It cannot be uploaded to a convenient web tool, attached to a file-sharing link, or passed through a cloud editor just because I need a short clip for a report or presentation.</p><p>The privacy requirement is reasonable. The workflow problem is real: how do you share the useful part of a video without sharing the people and vehicles in it?</p><p>That question led me to build an offline video redaction tool for my own workflow.</p><h3>The work behind the problem</h3><p>My PhD is in transportation safety, and my work is in computer vision. Over the years I have built pipelines that detect and track vehicles, calibrate roadside cameras, estimate speed and trajectories, and compute traffic-conflict measures such as time-to-collision (TTC) and post-encroachment time (PET). I have also worked on running detection and tracking at the edge, including Jetson-class devices used near the point of data collection.</p><p>That background shapes how I think about redaction. A detector can make a review practical, but it cannot turn an uncertain output into a fact. In safety research, we inspect outputs, investigate failures, and record the limits of the measurement. Redacting footage needs the same discipline.</p><p>The need usually appears late in a project. A team wants one figure for a report, a short clip for a conference talk, or a demonstration for a public meeting. The analysis is complete, but the video is not ready to leave the controlled environment. Every visible face and plate becomes part of the publication decision.</p><p>That became concrete while I was working with the Yakama Nation on a USDOT-funded SMART project. Our team installed cameras along the US 97 corridor between Toppenish and Union Gap. We collect the footage and perform computer-vision analysis on it, but the Yakama Nation does not permit us to publicly show identifiable faces or license plates from that data. That requirement determines what can appear in a presentation, a public-facing dashboard, or any other material outside the analysis environment.</p><p>This is where offline redaction becomes useful. We can keep the original footage within the project’s controlled environment, prepare a redacted version locally, and show the traffic behavior that matters without exposing the people and vehicles captured by the cameras.</p><h3>Redaction is a publication gate</h3><p>The obvious options each fail for a different reason.</p><p><strong>Manual masking</strong> gives a reviewer control, but it becomes exhausting when subjects enter and leave a long clip. Keyframing one face is manageable. Following a corridor of pedestrians and a stream of traffic is another problem.</p><p><strong>Online tools</strong> are not an option when the source file cannot be uploaded. Their convenience is built around the exact data movement the workflow prohibits.</p><p><strong>General-purpose video editors</strong> can track masks and produce polished exports, but they are still timeline tools. They do not naturally answer the question, “Which faces and plates were found, which were missed, and which decisions are still pending?”</p><p><strong>Fully automatic redaction</strong> improves throughput but can hide its own failures. Angled plates, low light, motion blur, and occlusion are normal conditions in roadside footage. A completed render is not evidence that every subject was found.</p><p>This is also a broader privacy concern, not only an internal agency rule. The UK’s Information Commissioner’s Office <a href="https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/cctv-and-video-surveillance/guidance-on-video-surveillance-including-cctv">treats faces and license plates in CCTV and ANPR footage as personal data</a>.</p><p>The requirement became clear: I did not need a button that blurred a face. I needed a controlled workflow that made the uncertain parts visible before a file was released.</p><h3>From editing problem to design specification</h3><p>At first, it is natural to think about redaction as an editing task: draw a mask, track it, and render a new video. That approach is fine for a small, known number of subjects. It does not scale well to research footage with many people, vehicles, and changing visibility.</p><p>The next instinct is automation: detect the targets, draw the masks, and let the computer finish. That improves throughput, but it introduces a more dangerous failure mode. A script can finish successfully while missing a face behind an occlusion or a plate at an angle. The absence of an error message is not the same as a safe export.</p><p>The useful design sits between those two approaches. The computer should do the repetitive search. The reviewer should decide what the result means. The export should fail closed when the decisions are incomplete.</p><p>Three principles carried over directly from my research work:</p><ol><li>The model proposes; the reviewer decides. Detections and tracks are candidates, not final truth.</li><li>Uncertainty is visible. Occlusions and interpolated spans are flagged for inspection instead of being presented as certain.</li><li>The output is checked before it is trusted. A successful render is not enough; approved redactions must be present where they are expected to be.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tfyFO3PWzX0GLfwEpx7llQ.png" /></figure><h3>The review-first workflow</h3><p>The <a href="https://www.mkarim.us/video-redaction/">Video Redactor guide</a> documents a desktop application that automatically proposes faces and license plates, while also letting you draw a box around any other object and track it through the video. For a manually drawn object, you can blur, pixelate, or black it out as it moves. It works locally, without sending the source video to a server. The practical loop is import, mark, review, export.</p><p>The core workflow looks like this:</p><pre>analyze selected frames<br>  automatically detect faces and plates<br>  link detections into tracks<br>  discard isolated detections that look like noiseor draw a box around any object<br>  track that box as the subject moves<br><br>put every track in PENDING<br>  approve it, or exempt it deliberately<br>  flag occluded or interpolated spans for review<br><br>unlock export only after every track has a decision<br>  render the selected style for approved tracks<br>  validate the expected redaction coverage<br><br>PASS: write the output file<br>FAIL: keep no partial export</pre><p>The important choice is that detection or drawing does not immediately become redaction. A reviewer can approve a detected subject, draw a box around something else, exempt a subject deliberately, or inspect only the spans that need attention. Solid black, blur, and pixelation are available per subject, along with shape, margin, and feather controls. Blur and pixelation can preserve a watchable image, but they should not be described as guaranteed anonymization. When stronger masking is required, the choice of style and the final inspection still matter.</p><p>The application also keeps the source untouched. The models are bundled with the application, there are no runtime downloads, and the product is designed without accounts or telemetry. Temporary working data is deleted when the application closes, and the log view uses source/output placeholders instead of exposing real file paths. The application makes no network connections, so the processing model is compatible with an environment where the cable can be pulled before analysis begins. Audio is retained in the export but is not analyzed; redaction applies to the picture.</p><p>If that review-first workflow matches your constraints, the <a href="https://www.mkarim.us/video-redaction/">illustrated guide</a> shows the interface and a complete demo before you decide whether it fits your process.</p><h3>The implementation is deliberately practical</h3><p>The computer-vision stack is not presented as a replacement for review. It is a way to make review possible on footage that would otherwise be too large to inspect manually:</p><ul><li>Faces: <a href="https://github.com/opencv/opencv_zoo/tree/main/models/face_detection_yunet">YuNet from the OpenCV Zoo</a>, a small detector that produces face boxes and landmarks without identifying the person.</li><li>Plates: a YOLOv11-based license-plate detector using the <a href="https://huggingface.co/morsetechlab/yolov11-license-plate-detection">morsetechlab model</a>, trained on a <a href="https://universe.roboflow.com/roboflow-universe-projects/license-plate-recognition-rxg4e">Roboflow license-plate dataset</a>.</li><li>Tracking: automatically detected faces and plates become subject tracks, while a manually drawn box can track any object, logo, tattoo, or missed subject through the video. Isolated detector false positives are discarded rather than promoted automatically into a redaction decision.</li><li>Analysis controls: the frame-analysis stride ranges from 0.05 seconds to 2 seconds, with 0.2 seconds as the default. Analysis resolution defaults to 1280 pixels and can be raised when smaller faces need more attention. These controls expose the speed-versus-thoroughness tradeoff instead of hiding it.</li><li>Rendering and packaging: <a href="https://ffmpeg.org/">ffmpeg</a> handles the export re-encode. The application is built with Tauri for macOS (Apple Silicon) and Windows (x64), with the analysis engine running as a separate process that can restart if it fails.</li></ul><p>Those settings are not a substitute for testing. A detector can only flag what it sees, and a face below its threshold is never proposed for review; manual drawing is the fallback for missed subjects or anything outside the two automatic categories. Export validation can catch a rendering or coverage failure for an approved track; it cannot prove that the detector found every subject in the original video. That is why the review queue remains central to the product.</p><h3>The same boundary appears elsewhere</h3><p>Transportation is where this constraint became formal for me, but it is not unique to transportation. Journalists handle witness and incident footage. Research teams de-identify recordings under ethics-board requirements. People share dashcam, security, and neighborhood video where faces and plates appear even though they are not the subject of the story.</p><p>In each case, the useful media has to be separated from the identifying details before it travels. The exact policy depends on the project and jurisdiction. The workflow lesson is simpler: redaction belongs before publication, not after a file has already been uploaded.</p><h3>For YouTubers and other creators</h3><p>The same problem shows up when a creator films in public. Street B-roll, event footage, interviews, and travel videos often include bystanders, children, employees, or passing vehicles. A creator may have permission to publish the story without having permission to publish every identifiable person who happened to enter the frame. Uploading the original to an online redaction service adds another party to that workflow.</p><p>The creator’s process is different from a transportation study, but the useful product boundary is similar: analyze locally, review the proposed faces and plates, draw boxes around anything else that should be hidden, choose a treatment for each subject, and inspect the exported file before uploading it to YouTube or another platform. The tool does not decide what a creator’s local rules require. It provides a controlled privacy step that can happen before the footage leaves the creator’s machine.</p><p>Transportation research is still the reason I built the product. The creator use case is an extension of the same engineering principle: sensitive video should not need to visit somebody else’s server before you can make it publishable.</p><h3>A local-first pattern</h3><p>Video Redactor is part of a broader pattern in my work. <a href="https://mkarim.us/apps/privateredact.html">PrivateRedact</a> applies the same boundary to documents. <a href="https://mkarim.us/apps/meetingmint.html">MeetingMint</a> keeps meeting notes on-device, while <a href="https://mkarim.us/apps/audiorecord.html">AudioRecord</a> handles local audio capture. <a href="https://www.mkarim.us/tidymind/">TidyMind</a> organizes files on the Mac, and <a href="https://mkarim.us/apps/self-hosted-rag.html">Self-Hosted RAG</a> lets users work with their own documents locally.</p><p>These are different tools, not one product suite. The common constraint is that the user’s data should stay under the user’s control whenever the task does not require a remote service.</p><h3>A practical privacy boundary</h3><p>The design principle is simple: privacy should not require handing the most sensitive data in the workflow to somebody else’s server. For transportation footage, that means keeping raw video inside the agency environment until a person has reviewed and approved what can leave it.</p><p>For clarity: Video Redactor began as a tool I built for my own redaction work, and I later released it as an independent product. It is not a deliverable of the funded project, and no project footage appears in the product or its demo.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FWzJkE7S12Y4%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DWzJkE7S12Y4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FWzJkE7S12Y4%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9a9b63174ddbabb179b488b7020f5ba2/href">https://medium.com/media/9a9b63174ddbabb179b488b7020f5ba2/href</a></iframe><p>You can find the complete Video Redactor package, including both platform installers, <a href="https://monjurulkarim.gumroad.com/l/video-redactor">here</a>. The <a href="https://www.mkarim.us/video-redaction/">guide page</a> includes a demo recorded end to end in the application. One click finds 26 faces in the demo clip, followed by the review step and a validated export.</p><p>If your redaction workflow has different requirements, such as different formats, required styles, or batch processing, document the workflow before choosing a tool. That is how this product will improve: around real review procedures, not a feature checklist. Questions and workflow feedback are welcome at <a href="/service/mailto:karim@mkarim.us">karim@mkarim.us</a>.</p><p>The cameras are not going away. The least the software can do is keep the people in the footage out of the picture when the picture needs to travel.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4868dd7199b2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Real-Time Traffic Conflict Estimation on an Edge Device]]></title>
            <link>https://medium.com/@raju.monjurulkarim/real-time-traffic-conflict-estimation-on-an-edge-device-4c9273e8bdef?source=rss-c3bc5b95942e------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c9273e8bdef</guid>
            <category><![CDATA[traffic-safety]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[transportation]]></category>
            <category><![CDATA[edge-computing]]></category>
            <dc:creator><![CDATA[Muhammad Karim, Ph.D.]]></dc:creator>
            <pubDate>Tue, 01 Sep 2026 18:25:00 GMT</pubDate>
            <atom:updated>2026-09-01T18:25:00.444Z</atom:updated>
            <content:encoded><![CDATA[<p>Measure near-misses, not crashes — tracked trajectories to TTC and PET, in plain numpy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c5VNV30xUy9eMYgBreSwSQ.png" /><figcaption>Image generated by AI</figcaption></figure><p>Every traffic pipeline I have shipped — including the <a href="https://github.com/monjurulkarim/YOLOv8_Object_Tracking_TensorRT">YOLOv8 + TensorRT + ByteTrack one on a Jetson</a> — hits the same wall once the <a href="https://mkarim.us/blog/2026-08-29-yolov8-tensorrt-bytetrack-jetson.html">tracker works</a> and the <a href="https://mkarim.us/blog/2026-08-30-roadside-camera-calibration-speed-estimation.html">camera is calibrated</a>: now what? Speeds are nice, counts are nice, but the question a transportation engineer actually gets funded to answer is “did this treatment make the site safer?” Crash counts need years to move. Real-time traffic conflict estimation is the practical answer: measure near-misses instead of crashes, continuously, on the edge device itself. This post closes the trilogy — tracked trajectories, calibrated to meters, and now turned into safety events.</p><h3>Why conflicts instead of crashes</h3><p>A crash at a single intersection is a rare event; waiting for enough of them to compare a before and after period is a multi-year project. The traffic conflict technique predates deep learning by decades: Perkins and Harris generalized the idea at General Motors in 1967, Hayward formalized time-to-collision in his 1972 Penn State report, and Hydén’s Swedish conflicts technique turned observer ratings into a field method in 1987. What changed since then is not the math — it’s that a small, cheap box can now run the whole observation automatically.</p><p>The modern reference for computing these measures is Gettman and Head’s Surrogate Safety Assessment Methodology (<a href="https://www.fhwa.dot.gov/publications/research/safety/03050/">FHWA-RD-03–050</a>; journal version in <a href="https://doi.org/10.3141/1840-12">TRR 1840</a>). It defines the surrogate measures and their algorithms for conflict points (crossing flows) and conflict lines (merging and following flows) — the taxonomy any serious implementation should follow.</p><h3>The two measures that carry most of the weight</h3><p><strong>Time-to-collision (TTC).</strong> Hayward’s definition: the time required for two vehicles to collide if they continue at their present speed and on the same path. For a tracked pair you compute it at every time step and keep the minimum over the interaction. Thresholds are a research choice, not physics: the SSAM algorithms use 1.5 s as their worked example of an upper limit for a valid conflict at intersection approaches — from van der Horst’s dissertation work — while higher critical values (2.6–3.5 s) appear in high-speed car-following.</p><p><strong>Post-encroachment time (PET).</strong> The time gap at the crossing point between one road user leaving the area and the other arriving. PET needs only where the paths cross and when each user passed it — no velocities, which makes it the more robust choice when detection noise is unavoidable. Its weakness: it assumes you can identify the conflict point, which is exactly where two vehicles occlude each other in the image.</p><h3>The code</h3><p>Prerequisites from the previous two posts: per-road-user tracks (id, pixel positions, timestamps) from the tracker, and a homography mapping undistorted pixels to ground meters. Everything below is plain numpy.</p><p>TTC, in closest-approach form:</p><pre>import numpy as np<br><br>def min_ttc(a, ta, b, tb, extent=1.0):<br>    &quot;&quot;&quot;Minimum time-to-collision between two ground-plane tracks.<br><br>    a, b: (N,2) tracks in meters; ta, tb: strictly increasing seconds.<br>    extent: center-to-contact distance that counts as a collision (m).<br>    &quot;&quot;&quot;<br>    bx = np.stack([np.interp(ta, tb, b[:, 0]),<br>                   np.interp(ta, tb, b[:, 1])], axis=1)<br>    dt = np.diff(ta)<br>    va = np.diff(a, axis=0) / dt[:, None]        # velocity samples<br>    vb = np.diff(bx, axis=0) / dt[:, None]<br>    r  = a[:-1] - bx[:-1]                        # relative position<br>    v  = va - vb                                  # relative velocity<br>    vv = np.einsum(&#39;ij,ij-&gt;i&#39;, v, v)<br>    t  = np.where(vv &gt; 0, -np.einsum(&#39;ij,ij-&gt;i&#39;, r, v) / vv, np.inf)<br>    ok = (t &gt; 0) &amp; (t &lt; 10)                      # plausible horizon<br>    dmin = np.linalg.norm(r + v * t[:, None], axis=1)<br>    hit = ok &amp; (dmin &lt;= extent)                  # on a collision course<br>    return float(np.min(t[hit])) if hit.any() else np.inf</pre><p>PET, in path-crossing form:</p><pre>def _time_at(track, ts, x):<br>    &quot;&quot;&quot;Timestamp at which a track passed closest to ground point x.&quot;&quot;&quot;<br>    d = np.linalg.norm(track - x, axis=1)<br>    k = int(np.argmin(d))<br>    if k == 0 or k == len(ts) - 1:<br>        return float(ts[k])<br>    lo, hi = k - 1, k + 1<br>    seg = track[hi] - track[lo]<br>    t = np.clip((x - track[lo]) @ seg / (seg @ seg), 0.0, 1.0)<br>    return float(ts[lo] + t * (ts[hi] - ts[lo]))<br><br>def pet(a, ta, b, tb):<br>    &quot;&quot;&quot;Post-encroachment time at the crossing of two paths (seconds).&quot;&quot;&quot;<br>    D = np.linalg.norm(a[:, None, :] - b[None, :, :], axis=2)<br>    i, j = np.unravel_index(np.argmin(D), D.shape)<br>    x = 0.5 * (a[i] + b[j])          # conflict point (m)<br>    return _time_at(b, tb, x) - _time_at(a, ta, x)</pre><p>For crossing conflicts, report both; they disagree in informative ways.</p><h3>What real-time conflict estimation costs on the edge</h3><p>Here is the good news: everything above is microseconds of numpy. The compute budget of this pipeline was spent before the safety stage — on the detector. That is why the TensorRT engine from the <a href="https://mkarim.us/blog/2026-08-29-yolov8-tensorrt-bytetrack-jetson.html">tracking post</a> is the right base: export once, run batch-1 inference on the Jetson, and treat conflict estimation as a callback on track updates. Two practical notes:</p><ul><li>Gate candidate pairs by spatial proximity (a few meters) before running TTC/PET. At typical intersection densities the full pairwise matrix is already trivial, but gating keeps it trivial at peak.</li><li>Run the safety stage on every track update, not on saved video. The output worth storing is the event log — timestamp, pair, measure values, conflict point — not the video itself. A day of events is kilobytes; a day of video is a privacy liability.</li></ul><h3>Where it goes wrong</h3><ul><li><strong>ID switches.</strong> A tracker that merges two vehicles into one identity erases conflicts silently. ByteTrack is sturdy, not perfect: audit switches before trusting a low count, and prefer PET (path-based) when tracks are short.</li><li><strong>Centroid vs. contact point.</strong> Vehicles are 4.5 m long. Map the bottom-center of the detection box to the ground plane, not the box center, and set extent from real vehicle dimensions — not 1.0 by habit.</li><li><strong>Far-field amplification.</strong> The homography error that was tolerable for speeds multiplies in TTC, which divides by closing speed. Measure conflicts inside your <a href="https://mkarim.us/blog/2026-08-30-roadside-camera-calibration-speed-estimation.html">ground-control region</a>, full stop.</li><li><strong>Occlusion at the worst moment.</strong> Two vehicles about to conflict overlap in the image exactly when you need them most. Predict through short gaps with a constant-velocity model, and flag events whose tracks were patched.</li><li><strong>Threshold-only reporting.</strong> “14 conflicts below 1.5 s” hides more than it shows. Report the distribution of TTC/PET by hour and location — the shape moves before the count does.</li></ul><blockquote>Roadside studies produce paperwork too — crash reports, survey sheets, screenshots, and the documents attached to a paper. That’s personal data as well. If a file needs to leave the device for a report or a paper, redact it first: <a href="https://www.mkarim.us/apps/privateredact.html">PrivateRedact</a> strips identifiers from PDFs, scans, DOCX, and images 100% offline — local LLM, no upload.</blockquote><h3>FAQ</h3><h4>Which measure should I compute first, TTC or PET?</h4><p>PET for crossing movements — it needs no velocities and survives noisy detections. TTC for following and rear-end interactions, where there is no crisp crossing point. Report both when you can.</p><h4>Does conflict estimation need special camera placement?</h4><p>The <a href="https://mkarim.us/blog/2026-08-30-roadside-camera-calibration-speed-estimation.html">calibration post’s</a> requirements still apply, plus one: the conflict zone itself must be visible. A camera that sees long approach legs but not the crossing point produces speeds, not conflicts.</p><h4>How much video do I need for stable estimates?</h4><p>More than you think — serious conflicts are rarer than vehicles. The method matters as much as the volume: compare like with like (same weekday mix, same hours), and treat the event count as a trend line, not a one-shot verdict.</p><h4>Can I run this CPU-only?</h4><p>The safety stage, trivially. The detector is the cost — and that is a TensorRT/Jetson question, covered in the <a href="https://mkarim.us/blog/2026-08-29-yolov8-tensorrt-bytetrack-jetson.html">tracking post</a>.</p><p>This closes the series this post was promised in. The pieces it builds on: <a href="https://mkarim.us/blog/2026-08-29-yolov8-tensorrt-bytetrack-jetson.html">detection and tracking on a Jetson</a> and <a href="https://mkarim.us/blog/2026-08-30-roadside-camera-calibration-speed-estimation.html">roadside camera calibration</a>; the wider research direction is in <a href="https://mkarim.us/blog/2026-08-29-llm-traffic-safety-applications.html">LLMs in traffic safety</a>.</p><p>I’m Muhammad Karim — transportation researcher; I build privacy-first, local-first tools: <a href="https://www.mkarim.us/apps/privateredact.html">PrivateRedact</a>, <a href="https://www.mkarim.us/apps/self-hosted-rag.html">Self-Hosted RAG</a>, MeetingMint, AudioRecord.</p><p><em>Originally published at </em><a href="https://www.mkarim.us/"><em>mkarim.us,</em></a><em> with the PET code, the Jetson deployment notes, and the calibration/tracking posts this builds on.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4c9273e8bdef" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why I Rebuilt My Flask AI Notetaker as a Native Mac App Using MLX]]></title>
            <link>https://aiadvances.org/why-i-rebuilt-my-flask-ai-notetaker-as-a-native-mac-app-using-mlx-cc684a533394?source=rss-c3bc5b95942e------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc684a533394</guid>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[swift]]></category>
            <category><![CDATA[notetaking]]></category>
            <category><![CDATA[macos]]></category>
            <category><![CDATA[productivity]]></category>
            <dc:creator><![CDATA[Muhammad Karim, Ph.D.]]></dc:creator>
            <pubDate>Wed, 26 Aug 2026 22:35:12 GMT</pubDate>
            <atom:updated>2026-08-26T22:35:12.060Z</atom:updated>
            <content:encoded><![CDATA[<h3>A technical deep dive into running WhisperKit and MLX on Apple Silicon unified memory — eliminating daemons, managing CoreAudio traps, and optimizing on-device inference.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jHkPN7npGiWjUykEV86DqQ.png" /><figcaption>MeetingMint: On-device AI meeting Note taking (AI-assisted image by Author)</figcaption></figure><p>Running machine learning models locally on personal hardware has unlocked a compelling promise: complete data privacy, zero recurring cloud API bills, and total data ownership. For sensitive workflows like recording meetings, client consultations, or research discussions, having audio and transcripts never leave your machine is a game-changer.</p><p>A year ago, I built an offline, self-hosted meeting transcription and summarization system. Like many local AI projects, it started with a standard and practical Python stack:</p><ul><li>A <strong>Flask</strong> backend (app.py) orchestrating audio processing in background threads.</li><li><strong>faster-whisper / PyTorch</strong> for Automatic Speech Recognition (ASR).</li><li>An external <strong>Ollama</strong> daemon managing GGUF-quantized LLMs over a local HTTP REST API (localhost:11434).</li><li>A vanilla JavaScript single-page application for the user interface.</li></ul><p>It worked, but it suffered from a fundamental flaw common to local AI tools: the “Local Server” paradigm turns the user into a system administrator.</p><p>That architecture continues to be a great solution for cross-platform environments: it runs on macOS, Windows, and Linux, can be hosted headlessly on a home server or NAS, and gives developers complete freedom to customize Python pipelines and model backends. Operating it involves a familiar developer workflow: configuring runtime dependencies (Python 3.10+ and FFmpeg), launching the Ollama background service, running the Flask server from a terminal, and interacting through a local browser tab.</p><p>However, for a dedicated desktop workflow on Apple Silicon, there is another compelling architectural path. Rather than running a client-server stack with background daemons and terminal sessions, modern macOS hardware allows us to leverage Unified Memory Architecture (UMA), the Apple Neural Engine (ANE), and the Metal compute framework to execute the entire AI pipeline in-process.</p><p>To explore this platform-specialized paradigm, I built a dedicated <strong>macOS</strong> application called <a href="https://apps.apple.com/app/meetingmint/id6801946726"><strong>MeetingMint</strong></a> using <strong>SwiftUI, SwiftData, WhisperKit</strong> (<strong>CoreML),</strong> and <strong>Apple’s MLX framework </strong>— providing a one-click desktop app with zero local servers to manage.</p><p>I asked myself a simple question: <strong>Why can’t local AI feel like a real Mac app?</strong><br>Why can’t a user simply launch an app from Spotlight, click Record, and get structured meeting minutes and transcriptions — with <strong>zero servers to start, zero terminal windows to manage, zero background daemons, and zero configuration?</strong><br>Historically, shipping a desktop UI for AI meant bundling Python via PyInstaller or wrapping a web app inside Electron. But those approaches still drag along the overhead of multiple runtimes and background processes. <br>To achieve a <strong>true one-click, zero-server native experience</strong>, the entire AI pipeline — audio capture, acoustic speech modeling, token generation, and persistence — had to run in-process within a single native Swift binary.<br>Recently, I rebuilt the entire system from the ground up as a native macOS application using <strong>SwiftUI, SwiftData, WhisperKit (CoreML on the Neural Engine), and Apple’s MLX framework</strong>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F6arG7skP1Fs%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D6arG7skP1Fs&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F6arG7skP1Fs%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/cf18c499590a6c706ea8faf61924c519/href">https://medium.com/media/cf18c499590a6c706ea8faf61924c519/href</a></iframe><p>Here is the deep dive into why moving to a serverless native architecture matters, why MLX is the key to local inference on Apple Silicon, and the low-level CoreAudio, DSP, and memory engineering required to make it work.</p><h3>1. Architectural Comparison: Multi-Process vs. In-Process</h3><p>Let’s examine the data flow and runtime boundaries of both architectures.</p><h3>The Original Architecture (Python + Ollama + Web UI)</h3><pre>┌──────────────────────────────────────────────────────────────┐<br>│ Browser (Web UI / MediaRecorder)                            │<br>└──────────────────────────────┬───────────────────────────────┘<br>                               │ HTTP POST (Audio File)<br>┌──────────────────────────────▼───────────────────────────────┐<br>│ Flask Server (Python 3.10+ Process)                          │<br>│                                                              │<br>│  1. Audio Ingestion ──► Pydub / FFmpeg (Subprocess slice)    │<br>│  2. ASR Execution   ──► faster-whisper (C++ / Python GIL)    │<br>│  3. Job Polling     ──► In-memory _JOBS dictionary           │<br>│  4. Summarization   ──► HTTP Request (JSON payload)          │<br>└──────────────────────────────┬───────────────────────────────┘<br>                               │ HTTP REST (localhost:11434)<br>┌──────────────────────────────▼───────────────────────────────┐<br>│ Ollama Daemon (C++ / llama.cpp Server Process)               │<br>│  - GGUF Model Loading                                        │<br>│  - GPU Metal Buffer Allocation &amp; Context Processing          │<br>└──────────────────────────────────────────────────────────────┘</pre><h4>Key Technical Pain Points of the Multi-Process Stack:</h4><ol><li><strong>Redundant Memory Overhead:</strong> The system ran three separate runtime environments simultaneously: the Python interpreter (with PyTorch and scientific libraries loaded in memory), the Ollama daemon holding model weights and KV caches, and the browser rendering the UI.</li><li><strong>IPC &amp; Serialization Latency:</strong> Audio and text data crossed multiple process boundaries: Browser → Flask (HTTP/Multipart) → FFmpeg (disk pipe) → Whisper → Flask → Ollama (HTTP/JSON) → Browser (Polling).</li><li><strong>Fragile Process Lifecycle:</strong> If the Ollama background daemon crashed, ran out of context memory, or was killed by macOS under memory pressure, the Python server was left polling dead sockets without clean recovery mechanisms.</li><li><strong>Browser Audio Capture Limits:</strong> Web browsers cannot natively adapt to dynamic hardware sample rate changes (e.g., switching from built-in microphones to Bluetooth HFP headsets) without re-initializing the entire MediaStream.</li></ol><h3>The In-Process Native Architecture (Swift + WhisperKit + MLX)</h3><p>In the rewritten application, the entire pipeline executes within a single sandboxed macOS process:</p><pre>┌──────────────────────────────────────────────────────────────────────────┐<br>│ Native macOS Application (Single Mach-O Binary)                          │<br>│                                                                          │<br>│  ┌────────────────────────────────────────────────────────────────────┐  │<br>│  │ SwiftUI Layer (Views, NavigationSplitView, Observable State)       │  │<br>│  └──────────────────────────────────┬─────────────────────────────────┘  │<br>│                                     │ async / await                      │<br>│  ┌──────────────────────────────────▼─────────────────────────────────┐  │<br>│  │ Pipeline Orchestrator (AppState &amp; MinutesGenerator)               │  │<br>│  │                                                                    │  │<br>│  │  1. Audio Capture  ──► AVAudioEngine (1ch Float32 @ HW Rate)       │  │<br>│  │  2. Energy VAD     ──► Accelerate.framework (vDSP RMS Analysis)    │  │<br>│  │  3. ASR Engine     ──► WhisperKit (CoreML on Apple Neural Engine) │  │<br>│  │  4. LLM Engine     ──► mlx-swift-lm (In-process Metal Shaders)    │  │<br>│  │  5. Retrieval RAG  ──► In-memory TF-IDF Keyword Chunking           │  │<br>│  │  6. Persistence    ──► SwiftData / SQLite in Container             │  │<br>│  └────────────────────────────────────────────────────────────────────┘  │<br>└──────────────────────────────────────────────────────────────────────────┘</pre><p>By executing in-process, we eliminate all intermediate serialization. Model weights and audio buffers reside directly in Apple Silicon’s <strong>Unified Memory Architecture (UMA)</strong>, where the CPU, GPU, and Apple Neural Engine (ANE) share zero-copy memory access.</p><h3>But why MLX on Apple Silicon?</h3><p>To build a standalone, serverless macOS application, we need an in-process Large Language Model engine that integrates seamlessly into a compiled Swift binary.</p><p>When evaluating on-device LLM inference engines on macOS, developers generally evaluate four approaches:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*1r8yPuIKjvvVME0Y1UX9Sg.png" /><figcaption>Image by Author</figcaption></figure><p><strong>MLX (</strong><strong>mlx-swift-lm)</strong> is built specifically for Apple Silicon&#39;s <strong>Unified Memory Architecture (UMA)</strong>. Unlike traditional frameworks that copy tensors across PCIe buses to discrete VRAM, MLX operates with zero-copy shared memory between the CPU and Metal GPU shaders. Unlike CoreML—which requires rigid, multi-minute static graph compilation that struggles with dynamic LLM sequence lengths and expanding KV caches—MLX supports dynamic, eager execution directly on quantized Hugging Face .safetensors with immediate startup times.</p><p>Pairing <strong>MLX</strong> with <strong>WhisperKit</strong> creates an optimal division of compute across Apple Silicon: WhisperKit runs ASR on the <strong>Apple Neural Engine (ANE)</strong>, while MLX runs language generation on the <strong>Metal GPU cores</strong>. This prevents the two models from competing for the same GPU compute queues or thermal headroom.</p><h3>2. Audio Pipeline &amp; CoreAudio Realities</h3><p>Writing robust audio capture in macOS requires navigating low-level hardware constraints that high-level abstractions like Python’s pydub or browser navigator.mediaDevices mask.</p><h3>2.1 The MacBook Multi-Mic Array Trap</h3><p>Modern MacBooks feature a 3- or 4-microphone beamforming array. When initialized via AVAudioEngine.inputNode, CoreAudio exposes a <strong>4-channel deinterleaved Float32 format</strong>.</p><p>If you write this 4-channel stream to a CoreAudio Format (.caf) file and later convert it to AAC .m4a using AVAssetExportSession (preset AVAssetExportPresetAppleM4A), the export session will report .completed without error — but <strong>the output file will be completely silent (zero-amplitude samples)</strong>. The default export preset fails to downmix the 4-channel deinterleaved stream properly.</p><p>To fix this, we enforce single-channel (mono) Float32 capture at the hardware sample rate directly in the input tap:</p><pre>// AudioRecorder.swift<br>let inputNode = engine.inputNode<br>let hardwareFormat = inputNode.outputFormat(forBus: 0)</pre><pre>// Must preserve hardware sample rate verbatim; forcing a rate causes CoreAudio hangs<br>let hwRate = hardwareFormat.sampleRate<br>let sampleRate = hwRate &gt; 0 ? hwRate : 44100</pre><pre>let inputFormat = AVAudioFormat(<br>    commonFormat: .pcmFormatFloat32,<br>    sampleRate: sampleRate,<br>    channels: 1,<br>    interleaved: false<br>) ?? hardwareFormat</pre><pre>inputNode.installTap(onBus: 0, bufferSize: 1024, format: inputFormat) { [weak self] buffer, _ in<br>    guard let self else { return }<br>    do {<br>        try self.file?.write(from: buffer)<br>    } catch {<br>        Task { @MainActor in self.writeError = error }<br>    }<br>}</pre><h3>2.2 Bluetooth Route Changes and Blocking engine.start()</h3><p>A critical edge case in desktop audio is the dynamic connection or disconnection of Bluetooth devices (e.g., AirPods).</p><p>When AirPods connect, the macOS audio input switches to Bluetooth Hands-Free Profile (HFP), which operates at <strong>16 kHz or 24 kHz</strong>, while built-in laptop microphones operate at <strong>48 kHz</strong>.</p><p>If an application attempts to reuse an existing AVAudioEngine across route changes or forces a 48 kHz tap onto a 16 kHz HFP device, engine.start() can <strong>block indefinitely inside CoreAudio kernel drivers</strong>. If executed on the main thread, the entire application UI freezes permanently.</p><p>We handle this by:</p><ol><li>Instantiating a clean AVAudioEngine instance on each recording take.</li><li>Wrapping engine.start() in an asynchronous task group racing against a timeout:</li></ol><pre>private nonisolated static func startEngine(_ box: EngineBox, timeout: TimeInterval) async throws {<br>    try await withThrowingTaskGroup(of: Void.self) { group in<br>        group.addTask { <br>            try box.engine.start() <br>        }<br>        group.addTask {<br>            try await Task.sleep(nanoseconds: UInt64(timeout * 1_000_000_000))<br>            throw AppError.other(&quot;Audio input timed out. Input route may have changed.&quot;)<br>        }<br>        try await group.next()!<br>        group.cancelAll()<br>    }<br>}</pre><h3>3. Speech Recognition: CoreML on the Neural Engine via WhisperKit</h3><p>For on-device speech-to-text, we use <strong>WhisperKit</strong>, which converts OpenAI’s Whisper model architecture into CoreML graphs capable of running on the Apple Neural Engine (ANE) and GPU.</p><h3>3.1 Bounded-Memory Incremental Transcription</h3><p>Meeting recordings often span 60 to 120 minutes. Attempting to decode the entire audio buffer in a single pass can cause severe memory spikes.</p><p>WhisperKit supports an .incremental loading mode with integrated Voice Activity Detection (VAD). This streams the audio in rolling windows, decoding segments as speech is detected while keeping active RAM footprint tightly bounded:</p><pre>// TranscriptionService.swift<br>let pipe = try await loadPipeline()</pre><pre>let audioOptions = AudioInputOptions(audioLoadingMode: .incremental)<br>var decodeOptions = DecodingOptions()</pre><pre>if let language = language {<br>    decodeOptions.language = language<br>} else {<br>    // Enable language identification for multilingual audio<br>    decodeOptions.detectLanguage = true<br>}</pre><pre>// Deliver real-time, monotonic progress based on time-coded segments<br>pipe.segmentDiscoveryCallback = { segments in<br>    let maxEnd = segments.map { Double($0.end) }.max() ?? 0<br>    tracker.advance(to: maxEnd)<br>    <br>    let current = tracker.latest<br>    let fraction = min(0.99, current / totalSeconds)<br>    let timeLabel = &quot;\(formatTime(current)) of \(formatTime(totalSeconds))&quot;<br>    <br>    Task { @MainActor in<br>        progressReporter?.update(fraction: fraction, detail: timeLabel, job: jobId)<br>    }<br>}</pre><pre>let results = try await pipe.transcribe(<br>    audioPath: audioURL.path,<br>    audioInputOptions: audioOptions,<br>    decodeOptions: decodeOptions<br>)</pre><h3>3.2 Pre-Inference DSP Energy Gating vs. Whisper Hallucinations</h3><p>An inherent characteristic of Whisper’s encoder-decoder architecture is its tendency to hallucinate during silent or low-energy audio segments (often producing repetitive loops like <em>“Thank you for watching!”</em>).</p><p>Rather than relying solely on post-hoc text filters, we implement a two-tier filtering strategy:</p><pre>[ Audio File ]<br>      │<br>      ▼<br>[ Accelerate vDSP Energy Gate ] ──(Voiced Duration &lt; 2.0s)──► Exit: &quot;No Speech Detected&quot;<br>      │ (Voiced Duration &gt;= 2.0s)<br>      ▼<br>[ WhisperKit CoreML Inference ]<br>      │<br>      ▼<br>[ Post-Inference Segment Filter ] ──(noSpeechProb &gt; 0.6 OR high compression)──► Drop Segment<br>      │<br>      ▼<br>[ Clean Transcript Segments ]</pre><h4>Tier 1: Fast Accelerate RMS Scan</h4><p>Before initializing or running the CoreML model, we scan the audio using Apple’s Accelerate framework (vDSP_rmsqv). If the cumulative duration of audio frames exceeding an RMS energy threshold of 0.02 is less than 2.0 seconds, we immediately abort transcription:</p><pre>private nonisolated static func voicedSeconds(_ url: URL) async -&gt; Double {<br>    guard let file = try? AVAudioFile(forReading: url) else { return .greatestFiniteMagnitude }<br>    let format = file.processingFormat<br>    let windowFrames = AVAudioFrameCount(max(1, Int(format.sampleRate * 0.1))) // 100ms window<br>    guard let buffer = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: windowFrames) else {<br>        return .greatestFiniteMagnitude<br>    }<br>    <br>    var voicedDuration = 0.0<br>    while true {<br>        do { try file.read(into: buffer) } catch { break }<br>        let frames = Int(buffer.frameLength)<br>        guard frames &gt; 0, let channel = buffer.floatChannelData?[0] else { break }<br>        <br>        var rms: Float = 0<br>        vDSP_rmsqv(channel, 1, &amp;rms, vDSP_Length(frames))<br>        if rms &gt; 0.02 {<br>            voicedDuration += Double(frames) / format.sampleRate<br>        }<br>    }<br>    return voicedDuration<br>}</pre><h4>Tier 2: Post-Inference Statistical Filter</h4><p>For segments that pass the energy gate, we filter based on Whisper’s internal metrics:</p><pre>private nonisolated static func isNotHallucinated(_ segment: TranscriptionSegment) -&gt; Bool {<br>    // 1. Strict threshold on silence probability<br>    if segment.noSpeechProb &gt; 0.6 { return false }<br>    <br>    // 2. Repetition detection: high zlib compression ratio paired with low confidence<br>    if segment.compressionRatio &gt; 2.4 &amp;&amp; segment.avgLogprob &lt; -1.0 { return false }<br>    <br>    return true<br>}</pre><h3>4. On-Device LLMs via MLX (mlx-swift-lm)</h3><p>Apple’s <strong>MLX</strong> framework provides native, hardware-accelerated tensor operations designed around Apple Silicon’s unified memory. Through mlx-swift-lm, we execute 4-bit quantized language models directly in the application process using Metal compute pipelines.</p><h3>4.1 In-Process Model Loading and Streaming</h3><p>Unlike external daemons that require HTTP sockets, MLX models are managed as in-process Swift actors:</p><pre>// LLMService.swift<br>import MLXLLM<br>import MLXLMCommon<br>import MLXHuggingFace</pre><pre>@MainActor<br>final class MLXLLMService: LLMService {<br>    private var container: ModelContainer?<br>    private var loadedId: String?</pre><pre>    func chatStream(<br>        question: String,<br>        systemPrompt: String,<br>        history: [ChatTurn],<br>        contextTokens: Int<br>    ) -&gt; AsyncThrowingStream&lt;String, Error&gt; {<br>        AsyncThrowingStream { continuation in<br>            let task = Task {<br>                do {<br>                    let container = try await loadContainer()<br>                    let params = GenerateParameters(<br>                        maxTokens: max(256, min(contextTokens, 4096)),<br>                        temperature: 0.2<br>                    )<br>                    let hist = history.map { Chat.Message(role: Self.role($0.role), content: $0.content) }<br>                    let session = ChatSession(<br>                        container,<br>                        instructions: systemPrompt,<br>                        history: hist,<br>                        generateParameters: params<br>                    )<br>                    <br>                    let stream = await session.streamResponse(to: withNoThink(question))<br>                    try await Self.cleanInto(stream: stream) { tokenChunk in<br>                        continuation.yield(tokenChunk)<br>                    }<br>                    continuation.finish()<br>                } catch {<br>                    continuation.finish(throwing: error)<br>                }<br>            }<br>            continuation.onTermination = { @Sendable _ in task.cancel() }<br>        }<br>    }<br>}</pre><h3>4.2 Handling Reasoning Model Artifacts (&lt;think&gt; Tags)</h3><p>Models in the Qwen3 family incorporate reasoning patterns that emit &lt;think&gt;…&lt;/think&gt; blocks before generating their answer. While useful for complex logic, these reasoning chains pollute downstream markdown parsers and structured JSON extraction.</p><p>To handle this cleanly during real-time token streaming, we apply a dual strategy:</p><ol><li><strong>Prompt-Level Suppression:</strong> Injecting /no_think into the prompt template when a Qwen3 model is active.</li><li><strong>Streaming Buffer Sanitization:</strong> A sliding-window parser that buffers boundary tokens and strips &lt;think&gt; blocks on the fly without delaying visible text rendering:</li></ol><pre>private static func cleanInto(<br>    stream: AsyncThrowingStream&lt;String, Error&gt;,<br>    emit: (String) -&gt; Void<br>) async throws {<br>    var buffer = &quot;&quot;<br>    for try await chunk in stream {<br>        buffer += chunk<br>        <br>        // Scan for complete think blocks<br>        while let close = buffer.range(of: &quot;&lt;/think&gt;&quot;, options: .literal) {<br>            let beforeClose = buffer.startIndex..&lt;close.lowerBound<br>            if let open = buffer.range(of: &quot;&lt;think&gt;&quot;, options: .literal, range: beforeClose) {<br>                let before = String(buffer[buffer.startIndex..&lt;open.lowerBound])<br>                if !before.isEmpty { emit(before) }<br>                buffer.removeSubrange(buffer.startIndex..&lt;close.upperBound)<br>            } else {<br>                buffer.removeSubrange(buffer.startIndex..&lt;close.upperBound)<br>            }<br>        }<br>        <br>        // Buffer potential opening tag across chunks (tag length: 7 characters)<br>        if !buffer.contains(&quot;&lt;think&gt;&quot;) &amp;&amp; buffer.count &gt; 7 {<br>            let flushEnd = buffer.index(buffer.startIndex, offsetBy: buffer.count - 7)<br>            emit(String(buffer[buffer.startIndex..&lt;flushEnd]))<br>            buffer.removeSubrange(buffer.startIndex..&lt;flushEnd)<br>        }<br>    }<br>    if !buffer.isEmpty &amp;&amp; !buffer.contains(&quot;&lt;think&gt;&quot;) {<br>        emit(buffer)<br>    }<br>}</pre><h3>5. Memory Management &amp; Hardware-Adaptive Model Selection</h3><p>A major engineering challenge of on-device LLM deployment is handling disparate unified memory sizes (ranging from 8 GB on base MacBooks to 128 GB on high-end configurations).</p><p>If an application loads a model whose weight footprint and KV cache exceed available memory headroom, macOS is forced to compress memory or page to swap, causing stutter across the OS.</p><h3>Dynamic Memory Detection</h3><p>To address this, the application inspects physical RAM via ProcessInfo.processInfo.physicalMemory and dynamically selects the recommended model pair:</p><pre>// ModelManager.swift<br>private static let gemmaDefaultRAMBytes = 12 * 1024 * 1024 * 1024 // 12 GB threshold</pre><pre>nonisolated static var recommendsGemmaDefault: Bool {<br>    ProcessInfo.processInfo.physicalMemory &gt;= Self.gemmaDefaultRAMBytes<br>}</pre><h3>Model Catalog &amp; Memory Footprint Matrix</h3><p>The catalog configures models with their specific quantization formats and on-disk footprints (all the models are multilingual):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Gf9SmxM-o1tmfawNv6Dgw.png" /><figcaption>Models associated with the app (image by author)</figcaption></figure><p><em>Note on DWQ:</em> Gemma 3 4B uses <strong>Dynamic Weight Quantization (DWQ)</strong>, which selectively quantizes sensitive layers to maintain high structured-extraction accuracy while preserving a 2.5 GB on-disk footprint.</p><h3>6. Lightweight RAG: In-Memory TF-IDF vs. Vector Databases</h3><p>For the <em>“Ask Your Meeting”</em> interactive Q&amp;A feature, the original Python implementation evaluated local vector databases (such as Chroma or FAISS with sentence-transformers).</p><p>For meeting transcripts ranging from 3,000 to 25,000 words, bundling an entire embedding model and vector database engine adds unnecessary binary bloat and runtime memory overhead.</p><p>Instead, we implemented an in-memory <strong>TF-IDF (Term Frequency × Inverse Document Frequency)</strong> chunk retriever directly in Swift:</p><pre>// MinutesGenerator.swift<br>static func retrieveContext(<br>    question: String,<br>    transcript: String,<br>    topN: Int = 3,<br>    chunkChars: Int = 1200,<br>    maxChars: Int = 2400<br>) -&gt; String {<br>    let chunks = chunkTranscript(transcript, chunkChars: chunkChars)<br>    guard chunks.count &gt; 1 else { return String(transcript.prefix(maxChars)) }</pre><pre>    let keywords = questionKeywords(question) // Tokenized and filtered against English stopwords<br>    guard !keywords.isEmpty else { return chunks.first ?? &quot;&quot; }</pre><pre>    let totalChunks = Double(chunks.count)<br>    var docFreq: [String: Int] = [:]<br>    <br>    // 1. Calculate Document Frequency<br>    for chunk in chunks {<br>        let lower = chunk.lowercased()<br>        for kw in Set(keywords) where lower.contains(kw) {<br>            docFreq[kw, default: 0] += 1<br>        }<br>    }</pre><pre>    // 2. Compute Inverse Document Frequency (smoothed)<br>    let idf = Dictionary(uniqueKeysWithValues: keywords.map { kw in<br>        (kw, log(totalChunks / Double(1 + (docFreq[kw] ?? 0))) + 0.01)<br>    })</pre><pre>    // 3. Score Chunks via TF * IDF<br>    var scored: [(index: Int, score: Double)] = []<br>    for (i, chunk) in chunks.enumerated() {<br>        let lower = chunk.lowercased()<br>        var score = 0.0<br>        for kw in keywords {<br>            score += Double(occurrences(of: kw, in: lower)) * (idf[kw] ?? 0)<br>        }<br>        if score &gt; 0 { scored.append((i, score)) }<br>    }</pre><pre>    // 4. Select Top-N chunks and restore original chronological order<br>    let selected = scored<br>        .sorted { $0.score &gt; $1.score }<br>        .prefix(topN)<br>        .sorted { $0.index &lt; $1.index }</pre><pre>    return selected.map { chunks[$0.index] }.joined(separator: &quot;\n…\n&quot;)<br>}</pre><h3>Why this works well for transcript Q&amp;A:</h3><ol><li><strong>Preserved Timestamps:</strong> Because each line in the transcript maintains timecodes (e.g., [12:34]), the retrieved excerpts carry precise time markers directly into the prompt context, allowing the LLM to generate verifiable citations.</li><li><strong>Zero Overhead:</strong> Context scoring executes in-memory over text chunks with zero additional neural network inference passes or external database processes.</li></ol><h3>7. System Architecture &amp; Operational Profile Comparison</h3><p>Comparing the fundamental system characteristics between the two implementations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6Anx3xR2kcRSX7_QEDb14A.png" /></figure><h3>8. Key Architectural Takeaways</h3><p>Rebuilding a local AI application from a Python/Ollama server into an in-process Swift/MLX binary highlighted several key principles for on-device machine learning:</p><ol><li><strong>Unified Memory Eliminates IPC:</strong> On Apple Silicon, running models in-process allows audio buffers, model parameters, and token streams to move across CPU, GPU, and Neural Engine cores without network serialization or IPC boundaries.</li><li><strong>Combine DSP with Neural Networks:</strong> Don’t use large neural networks to solve problems classical DSP can handle faster. A 10-line vDSP root-mean-square energy check prevents Whisper hallucination loops and saves battery life.</li><li><strong>Adapt to Hardware Tiers:</strong> On-device AI cannot assume uniform hardware. Dynamic model selection based on physical RAM thresholds ensures stability across the entire device ecosystem.</li><li><strong>Prefer Lightweight Heuristics Over Heavyweight Abstractions:</strong> Before reaching for a vector database and embedding models to query a single document or meeting transcript, consider whether TF-IDF or lexical search provides equal or better accuracy with a fraction of the compute and zero binary bloat.</li></ol><p>The combination of <strong>CoreML (via WhisperKit)</strong> for speech recognition and <strong>MLX</strong> for language model inference provides a modern, robust foundation for building fully local, privacy-first software on macOS.</p><h3>Want to Run It Yourself?</h3><p>This project takes the core principles of local processing from the original web-based AI note-taker and expands them into a robust, native macOS experience ready for your daily workflow.</p><p>I’ve packaged both tools and made them available on the Mac App Store. It’s a fantastic way to get a powerful, completely private AI meeting assistant up and running in minutes on your Apple Silicon Mac — with zero servers to configure or manage.</p><p><strong>Special Offer for Readers:</strong> As a thank you to the community here, I’ve set up a limited-time 50% discount on the <strong>Lifetime License</strong> for MeetingMint for a limited time. Use the custom offer code <strong>MEDIUM50</strong> directly in the Mac App Store to unlock lifetime access!</p><p>You can check out both native applications here:</p><ul><li><a href="https://apps.apple.com/app/meetingmint/id6801946726"><strong>MeetingMint</strong></a></li><li><a href="https://apps.apple.com/app/audiorecord-mic-system/id6802515117"><strong>AudioRecord</strong></a></li></ul><p>Thanks for reading! I hope this deep-dive inspires you to take control of your data and explore the incredible potential of on-device, privacy-first AI.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc684a533394" width="1" height="1" alt=""><hr><p><a href="https://aiadvances.org/why-i-rebuilt-my-flask-ai-notetaker-as-a-native-mac-app-using-mlx-cc684a533394">Why I Rebuilt My Flask AI Notetaker as a Native Mac App Using MLX</a> was originally published in <a href="https://aiadvances.org/">AI Advances</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Redacting PII 100% Offline: A Technical Deep Dive]]></title>
            <link>https://medium.com/@raju.monjurulkarim/redacting-pii-100-offline-a-technical-deep-dive-a326f9fed3b8?source=rss-c3bc5b95942e------2</link>
            <guid isPermaLink="false">https://medium.com/p/a326f9fed3b8</guid>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[ollama]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[redaction]]></category>
            <dc:creator><![CDATA[Muhammad Karim, Ph.D.]]></dc:creator>
            <pubDate>Mon, 10 Aug 2026 14:40:19 GMT</pubDate>
            <atom:updated>2026-09-01T23:03:47.476Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>How to redact sensitive documents using local LLM — without uploading them.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O35xJ50ls1hro8KSCcRq5g.png" /><figcaption>Original image by the author; modified with AI.</figcaption></figure><p>If you handle contracts, medical records, legal discovery, vendor agreements, or even just research data, redacting PII is not optional. HIPAA, GDPR, PCI DSS, FOIA, e-discovery — they all demand it. And yet the tools most people reach for are stuck in one of two failure modes.</p><p>The first is the cloud route. You upload a document full of the exact personal data you’re trying to protect to a third-party server, and it sends back a “redacted” copy. This has always struck me as a paradox: you’re solving a privacy problem by first exposing the document to a server you don’t control. For anything sensitive — patient records, a contract under negotiation, an HR complaint — that’s a non-starter.</p><p>The second is the naive desktop route. Open the PDF, draw black rectangles by hand over every Social Security number, every name, every dollar figure. This is slow, it’s error-prone (did you catch the SSN on page 47?), and because most viewers black the <em>rendering</em> rather than the <em>text</em>, the “redacted” layer can often be selected, copied, or removed to reveal what’s underneath. Or worse: the tool auto-blacks entire text lines, turning a readable document into a wall of censor bars.</p><p>Neither is good enough. I wanted the third option: accurate, automatic, value-only redaction that runs entirely on my own machine. So I built one. (If you want the tool and not the story, it’s here: <a href="https://github.com/monjurulkarim/privateredact">github.com/monjurulkarim/privateredact</a>.) In this article I’ll walk through the architecture of <strong>PrivateRedact</strong>, a desktop app that finds PII with a hybrid regex + local-LLM pipeline and exports a securely redacted document — no upload, no account, no cloud.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FXtSbOfy0b18%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DXtSbOfy0b18&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FXtSbOfy0b18%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/75e63b6e097a776991eb6a993060db9e/href">https://medium.com/media/75e63b6e097a776991eb6a993060db9e/href</a></iframe><h3>The “Why”: Defining Accurate, Local Redaction</h3><p>The goal wasn’t just to scribble black over a PDF. It was to build something I’d actually trust with a real medical record or a real contract. That meant:</p><ul><li><strong>Read anything:</strong> text-based PDFs, scanned/image PDFs, DOCX, TXT, and plain images. Redaction is useless if it can’t read the document.</li><li><strong>Find PII semantically, not just by format:</strong> regex catches a phone number. It does not catch “patient presented with Type 2 diabetes” or a project codename. I needed an AI layer for the things patterns can’t express.</li><li><strong>Redact the value, not the label:</strong> black “555–1234”, not “Phone: 555–1234”. And never black the whole line.</li><li><strong>Preserve the document:</strong> DOCX formatting, page layout, fonts — all intact. A redacted contract that lost its formatting is useless.</li><li><strong>Prove it worked:</strong> a post-export scan that re-OCRs the output and reports exactly what (if anything) survived.</li><li><strong>Be 100% local:</strong> no file, hash, or byte of content ever leaves the machine.</li><li><strong>Be compliance-aware:</strong> presets for HIPAA, GDPR, PCI DSS, HR, Legal, and Financial that toggle the right categories on.</li></ul><h3>The Tech Stack: Local-First Powerhouses</h3><p>Everything below runs on the user’s machine. No backend.</p><ul><li><strong>Electron:</strong> a cross-platform desktop shell that bundles the whole pipeline — renderer, the PDF/OCR engines, and the LLM — into one installable app. No “install Python, then Node, then…” for the end user.</li><li><strong>pdf.js (Mozilla):</strong> reads born-digital PDFs. Critically, it exposes not just text but each text item’s bounding box — which you need to know <em>where</em> to draw the redaction.</li><li><strong>Tesseract (v6):</strong> OCR for scanned PDFs and images. v6 is important: it can return word-level boxes, which you need for proportional redaction.</li><li><strong>Ollama:</strong> the local LLM runtime. This is the “brain” for semantic detection. Models run on-device; nothing is sent anywhere.</li><li><strong>pdf-lib (encrypted-aware fork):</strong> for assembling the output PDF and for reading/writing password-protected documents.</li><li><strong>sharp + node-canvas:</strong> image processing — rasterizing pages and painting the actual redaction boxes.</li><li><strong>jszip:</strong> DOCX files are zip archives of XML. jszip lets you edit the document’s XML in place, which is how you get format-preserving redaction.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*klnthL-BsiXEOfC_KYRegg.png" /><figcaption>The main window (Image by Author)</figcaption></figure><h3>Building the Backend</h3><p>The pipeline has four stages: <strong>extract → detect → redact → prove</strong>. Each has its own subtlety.</p><h3>1. Extraction: Born-Digital vs. Scanned</h3><p>A PDF is not a PDF. Some have a real text layer (exported from Word, say). Others are just wrapped images of scanned paper. You have to handle both, because redacting text you never read is how leaks happen.</p><p>The approach is a fork based on whether the page actually has a text layer:</p><pre># Conceptual — two extraction paths<br>if page.has_text_layer:<br>    # born-digital: pdf.js gives each text item + its bounding box<br>    words = [{&#39;text&#39;: item.text, &#39;box&#39;: item.bbox} for item in page.items]<br>else:<br>    # scanned / image: OCR it, and we need WORD-level boxes, not just text<br>    ocr = tesseract.recognize(page.render_to_image(), blocks=True)<br>    words = [{&#39;text&#39;: w.text, &#39;box&#39;: w.bbox} for w in ocr.words]</pre><p>The non-obvious part: when you OCR, you must ask for word-level boxes (Tesseract’s { text: true, blocks: true }). If you don&#39;t, the engine hands you whole paragraphs as one box — and you&#39;re back to blacking entire lines.</p><p>For low-quality scans there’s an optional vision model that double-checks the OCR. But it only adopts the vision result when it agrees with Tesseract on at least 80% of the text — you don’t want a hallucinated box drawn over real content.</p><h3>2. Detection: Regex First, LLM Second</h3><p>Detection runs in two layers, because neither is sufficient alone.</p><p>The <strong>regex layer</strong> is fast and precise for well-formatted PII — SSNs, phone numbers, credit cards, IBANs, dates of birth. The trick is value-only matching. You don’t want to black “SSN:”, you want to black the number after it:</p><pre># Redact the VALUE (capture group 2), not the label (group 1)<br>pattern = r&#39;(SSN|Social Security)[:\s]+([0-9X\-]{9,})&#39;<br>for m in re.finditer(pattern, text):<br>    value_start, value_end = m.span(2)   # the number&#39;s offsets only<br>    boxes = offsets_to_boxes(value_start, value_end, page)</pre><p>A real gotcha here: ID patterns need a digit lookahead. Without one, they false-positive on ordinary prose — “passport control”, “case study”, “member number two”. Requiring at least one digit in the candidate kills almost all of those.</p><p>The <strong>LLM layer</strong> catches what patterns structurally cannot: a diagnosis, a project codename, someone’s address mentioned in a sentence, a casual “you can reach me at…”. The page text is chunked and sent to the local Ollama model with a strict JSON schema, and it returns { type, text, start, end, confidence }:</p><pre>#Two layers — regex for precision, LLM for semantics<br>detections = regex_scan(page.text)                  # fast, known formats<br>detections += llm_scan(page.text, model=ollama)     # context-only PII<br># LLM gives text + offsets; locate that text on the page and box it<br>for d in detections:<br>    d.boxes = resolve_span(d.text, page)            # tolerant of case/whitespace</pre><p>The hard part of LLM detection isn’t calling the model — it’s <em>trusting</em> its output enough to draw a box. The model returns a string and approximate offsets; you have to actually find that string in the page text (case-insensitive, whitespace-tolerant) and map the match to a real bounding box. If the match rate is low, you drop the detection rather than risk a misplaced box.</p><h3>3. The Accuracy Problem: Why “Black the Line” Fails</h3><p>This is where most redaction tools fall apart, and it’s the part I spent the most time on.</p><p>pdf.js doesn’t return one box per word. It returns <em>runs</em> of text, and a single run can span an entire line — sometimes 80% of the page width. If you naively black the run’s box for any entity inside it, you black the label, the surrounding words, everything. The document becomes unreadable, and you’ve over-redacted.</p><p>The fix is <strong>proportional sub-boxes</strong>. Inside a run, you know the character offset where the entity starts and ends. Assuming roughly uniform character width within that run, you can compute the horizontal slice the entity actually occupies and black only that:</p><pre><br># Don&#39;t black the whole run. Black the SLICE the entity occupies.<br>def proportional_box(run_box, ent_start_char, ent_end_char, run_len):<br>    width = run_box.x1 - run_box.x0<br>    pad   = 0.4 * (width / run_len)          # a little breathing room<br>    x0 = run_box.x0 + (ent_start_char / run_len) * width - pad<br>    x1 = run_box.x0 + (ent_end_char   / run_len) * width + pad<br>    return Box(x0, run_box.y0, x1, run_box.y1)</pre><p>So “Phone: <strong>555–1234</strong>” redacts just the bold part. The label survives. The document stays readable.</p><p>Two more correctness rules, learned the hard way:</p><ul><li><strong>One union function, used twice.</strong> The boxes you show in the review overlay must be <em>exactly</em> the boxes you paint on export. If they differ, what the user approves is not what they get. So both paths call the same union_boxes routine — true WYSIWYG.</li><li><strong>Reject pathological boxes.</strong> If a computed box covers 95%+ of the page, something went wrong (a regex ate a whole block). Rather than black the page, you reject that detection outright.</li></ul><h3>4. Export and Proof</h3><p>For PDF output, each page is rasterized, the unioned redaction boxes are painted solid black on top, and the result is embedded into a fresh PDF. Rasterized output is the point: there is no recoverable text layer underneath. You cannot “select and delete” the black box to reveal the original. It’s gone.</p><pre># Rasterize -&gt; paint -&gt; embed. No text layer survives.<br>for page in document:<br>    raster = render(page)<br>    raster = paint_black(raster, page.unioned_boxes)<br>    output_pdf.embed(raster)</pre><p>DOCX is different — and trickier — because you want to keep the formatting. A DOCX is a zip of XML, so redaction is in-place surgery on the XML text nodes, not a re-render. You find the text runs, replace the PII characters inside them, and leave every other node (styling, tables, headers) untouched.</p><p>The last stage is the one I care most about: the <strong>leak scan</strong>. After the file is written, the output is OCR’d again and re-scanned for PII. It runs on the exact bytes that were written, and it never claims “clean” if the scan was skipped or OCR failed:</p><pre><br># Re-OCR the OUTPUT and scan it. No false &quot;clean&quot; on failure.<br>output_text = ocr(exported_pdf_bytes)<br>remaining   = scan_for_pii(output_text)<br>if scan_failed_or_skipped:<br>    report(&quot;could not verify — do not ship&quot;)<br>elif remaining:<br>    report(remaining)          # exactly what survived<br>else:<br>    report(&quot;0 PII remaining&quot;)</pre><p>This is the proof. “0 PII remaining” isn’t a button label — it’s the result of actually re-reading the redacted file and checking.</p><h3>The Frontend: Approve What You Export</h3><p>A backend this autonomous is also dangerous — you don’t want to hand a model the keys to black out a contract unsupervised. So the frontend is a review surface: a React overlay on the page image showing every proposed redaction as a box, with a toggle between the image view and the raw text. You can remove a box the model got wrong, or draw one it missed.</p><p>Because the overlay uses the same union_boxes logic as the exporter, what you see in review is byte-for-byte what gets painted. No surprises at export time.</p><h3>System Requirements</h3><p>Because the LLM runs locally, this isn’t a featherweight script:</p><ul><li><strong>OS:</strong> macOS (Apple Silicon) and Windows.</li><li><strong>RAM:</strong> ~8 GB minimum for the smallest useful model; more is better. The app picks the model size automatically based on available memory.</li><li><strong>The AI engine (Ollama):</strong> auto-installs on first use, but only when you click the button — never silently. If you skip it entirely, the app still works in regex-only mode.</li></ul><h3>Want to Try It Yourself?</h3><p>I packaged the whole thing — the extraction pipeline, the regex + local-LLM detection, the review UI, and the leak-scan proof — into a desktop app so you don’t have to wire up Electron, pdf.js, Tesseract, and Ollama yourself. It runs entirely on your machine: no account, no upload, nothing leaving your disk.</p><p>The fastest way to see it work on your own documents is to grab the free build from GitHub. You can load, detect, and review a real file end to end — the only thing behind a license is exporting the final redacted output.</p><p>→ <strong>Try it:</strong> <a href="https://github.com/monjurulkarim/privateredact/releases">github.com/monjurulkarim/privateredact/releases</a><br>→ <strong>Or grab a license to export:</strong> <a href="http://monjurulkarim.gumroad.com/l/wqnyp">in this link</a></p><p>Use the code <strong>MEDIUM25 </strong>to get a 25% discount for a limited time.</p><p>See it on my site — full feature list, FAQ, and my other local-first tools: <a href="https://www.mkarim.us/apps/privateredact.html">PrivateRedact on mkarim.us</a>.</p><p>Thanks for reading — and if you’re still redacting documents by drawing rectangles by hand, there is a better way.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a326f9fed3b8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Local LLMs to Organize Messy Files: A Technical Deep Dive]]></title>
            <link>https://medium.com/data-science-collective/using-local-llms-to-organize-messy-files-a-technical-deep-dive-79433165f4fb?source=rss-c3bc5b95942e------2</link>
            <guid isPermaLink="false">https://medium.com/p/79433165f4fb</guid>
            <category><![CDATA[ollama]]></category>
            <category><![CDATA[fileorganizer]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[productivity]]></category>
            <dc:creator><![CDATA[Muhammad Karim, Ph.D.]]></dc:creator>
            <pubDate>Wed, 22 Oct 2025 15:56:34 GMT</pubDate>
            <atom:updated>2025-10-25T19:51:36.799Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>A deep dive into reading PDFs, analyzing images, and categorizing files automatically — all 100% offline.</strong></h4><p>Digital clutter is a classic entropy problem. No matter how organized we try to be, the ~/Downloads folder eventually descends into chaos—a mix of invoices, code snippets, vacation photos, and research papers.</p><p>Most “file sorter” scripts are naive. They just look at file extensions, lumping all .pdf files into a single &quot;Documents&quot; folder. This is useless. A bank statement and a Dungeons &amp; Dragons character sheet are both PDFs, but they have completely different contexts.</p><p>Cloud-based organizers are smarter, but they come with the same privacy trade-off as AI note-takers: you must upload your private, often confidential, files to a third-party server. For my personal finances, contracts, or work-in-progress, that’s a non-starter.</p><p>I wanted a robust, local-first solution. It needed the intelligence of a cloud service but with the privacy of a local script. So, I built one. In this article, I’ll walk through the architecture of a Python application that reads the <em>actual content</em> of your files to sort them intelligently on your own machine.</p><h3>The “Why”: Defining an Intelligent Organizer</h3><p>The goal wasn’t just to move files. It was to build a “librarian” that could:</p><ol><li><strong>Read Anything:</strong> Go beyond filenames. It had to extract text from PDFs, Word documents, Excel sheets, and even PowerPoint presentations.</li><li><strong>See Images:</strong> Use vision models to understand what a photo <em>is</em> — a screenshot, a family picture, a diagram.</li><li><strong>Be 100% Private:</strong> All processing must happen locally. No file hashes, metadata, or content should ever leave my machine.</li><li><strong>Be Reversible:</strong> A script organizing thousands of files is dangerous. It <em>must</em> have a robust undo feature.</li><li><strong>Be Accessible:</strong> A command-line tool is great for developers, but for this to be truly useful, it needed a simple, cross-platform GUI that <em>anyone</em> could use.</li></ol><h3>The Tech Stack: Local-First Powerhouses</h3><p>This project is built on a stack of powerful, local-first tools:</p><ol><li><strong>Ollama:</strong> For the AI “brain.” Ollama makes it trivial to run powerful models locally. I chose gemma3:4b as the default, as it&#39;s a strong, lightweight multimodal model. This means it can &quot;see&quot; images just as well as it can &quot;read&quot; text, all from one model.</li><li><strong>Document Parsing Libraries:</strong> A suite of Python libraries to read proprietary formats: PyPDF2 for PDFs, python-docx for Word, openpyxl for Excel, and python-pptx for PowerPoint.</li><li><strong>Tkinter:</strong> For an optional (but crucial for non-technical users) cross-platform GUI that acts as a simple controller for the backend.</li><li><strong>JSON:</strong> For the undo log. A simple, human-readable database of every file operation.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/1*DjyRjrJlzTOOgXtfOPLyIA.png" /><figcaption>GUI (Image by Author)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/1*unJXwo2JvcRw7CQWWQMPaw.png" /><figcaption>GUI (Image by Author)</figcaption></figure><h3>Building the Backend: The Python Core</h3><p>The application is built on three core components: the Content Extractor, the AI Classifier, and the Undo Manager.</p><h4>1. The Content Extractor: Reading Everything</h4><p>Before the AI can classify a file, it needs context. For a .py file, we can just read the first 150 lines. But for a .docx or .pdf, we need to extract the text. I built a _get_comprehensive_preview function that tries multiple methods.</p><p>This function is essentially a giant try/except block that checks the file extension and uses the appropriate library to extract a text preview.</p><pre># In file_organizer.py<br>try:<br>    from PyPDF2 import PdfReader<br>    PYPDF2_AVAILABLE = True<br>except ImportError:<br>    PYPDF2_AVAILABLE = False<br># ... (similar checks for docx, pptx, openpyxl) ...<br><br>class FileOrganizer:<br>    # ...<br>    def _read_pdf_content(self, file_path: Path, max_pages: int = 10) -&gt; Optional[str]:<br>        &quot;&quot;&quot;Extract text content from PDF files&quot;&quot;&quot;<br>        if not PYPDF2_AVAILABLE:<br>            return None<br>        <br>        try:<br>            reader = PdfReader(str(file_path))<br>            text_parts = []<br>            num_pages = min(len(reader.pages), max_pages)<br>            for i in range(num_pages):<br>                page = reader.pages[i]<br>                text = page.extract_text()<br>                if text:<br>                    text_parts.append(text)<br>            <br>            full_text = &#39;\n&#39;.join(text_parts)<br>            return full_text if full_text.strip() else None<br>        except Exception as e:<br>            print(f&quot;      ⚠️  Could not read PDF: {str(e)}&quot;)<br>            return None<br>    # ... (similar _read_docx_content, _read_pptx_content, etc.) ...</pre><p>This gives the AI a rich, content-based preview (up to 3000 chars) instead of just a filename.</p><h4>2. The AI Classifier (and Vision)</h4><p>With the file content extracted, we query our local Ollama model. The key is the system prompt. We need to instruct the AI to act as an expert file organizer and return a structured JSON response.</p><p>For image files, we use a multimodal model and send the Base64-encoded image along with the prompt. For documents, we send the text preview.</p><pre># In ollama_handler.py<br>class OllamaHandler:<br>    # ...<br>    def _build_intelligent_prompt(<br>        self,<br>        filename: str,<br>        file_extension: str,<br>        custom_categories: Optional[List[str]] = None,<br>        file_content_preview: Optional[str] = None<br>    ) -&gt; str:<br>        <br>        # ... (logic to build category_instruction) ...<br>        <br>        prompt = f&quot;&quot;&quot;You are an expert file organization AI. Analyze this file deeply based on its ACTUAL CONTENT (not just filename) and categorize it intelligently.<br>FILENAME: {filename}<br>EXTENSION: {file_extension}<br>&quot;&quot;&quot;<br>        <br>        if file_content_preview:<br>            prompt += f&quot;\n\n=== FILE CONTENT (MOST IMPORTANT) ===\n{file_content_preview[:2000]}\n=== END CONTENT ===\n&quot;<br>            <br>        prompt += f&quot;&quot;&quot;<br>{category_instruction}<br>CRITICAL ANALYSIS GUIDELINES:<br>1. READ THE CONTENT CAREFULLY - this is the most important factor<br>2. Look for: invoice numbers, contract terms, project names, personal vs business language<br>3. Use ONLY ONE subcategory level (e.g., &quot;Work/Invoices&quot; NOT &quot;Work/Finance/Invoices&quot;)<br>Respond in JSON format:<br>{{<br>  &quot;category&quot;: &quot;Main category (e.g., Work, Personal, Projects)&quot;,<br>  &quot;subcategory&quot;: &quot;ONE specific subcategory (e.g., Invoices, Reports, Photos)&quot;,<br>  &quot;reasoning&quot;: &quot;Brief explanation based on CONTENT analysis&quot;<br>}}<br>Response:&quot;&quot;&quot;<br>        return prompt</pre><p>This prompt engineering is what allows the AI to distinguish an invoice from a contract, even if both are .pdf files.</p><h4>3. The Undo Manager: A Critical Safety Net</h4><p>Moving files programmatically is terrifying. A single logic error could mangle your file system. To solve this, I built an UndoManager that logs every single file operation to a central JSON file <em>before</em> it happens.</p><p>The manager creates a “session” for each organization task. Every shutil.move is preceded by a call to record_move.</p><pre># In undo_manager.py<br>class UndoManager:<br>    # ...<br>    def start_session(self, input_folder: str, output_folder: str) -&gt; str:<br>        operation_id = datetime.now().strftime(&#39;%Y%m%d_%H%M%S&#39;)<br>        session = {<br>            &#39;id&#39;: operation_id,<br>            &#39;timestamp&#39;: datetime.now().isoformat(),<br>            &#39;input_folder&#39;: str(input_folder),<br>            &#39;output_folder&#39;: str(output_folder),<br>            &#39;moves&#39;: [],<br>            &#39;status&#39;: &#39;in_progress&#39;<br>        }<br>        self.history[&#39;sessions&#39;].append(session)<br>        self.current_session_id = operation_id<br>        self._save_history()<br>        return operation_id<br><br>def record_move(self, original_path: str, new_path: str, category: str):<br>        # ... (finds current session) ...<br>        move_record = {<br>            &#39;original&#39;: str(original_path),<br>            &#39;new&#39;: str(new_path),<br>            &#39;category&#39;: category,<br>            &#39;timestamp&#39;: datetime.now().isoformat()<br>        }<br>        session[&#39;moves&#39;].append(move_record)<br>        self._save_history()</pre><p>A separate undo.py script can then read this JSON file, find the most recent session, and reverse every move, putting all files back in their original locations.</p><h3>Building the Frontend: A GUI for Everyone</h3><p>This backend is great, but unusable for a non-technical audience. The final piece of the puzzle is a simple Tkinter GUI.</p><p>I chose Tkinter because it’s built into Python, meaning no extra dependencies for the user to install. The GUI is a simple “controller” that calls the same FileOrganizer class the main.py script uses.</p><p>A conceptual look at the code for the “Organize” button shows how it wires everything together:</p><pre># Conceptual GUI code (e.g., in a new gui.py)<br>import tkinter as tk<br>from tkinter import filedialog, messagebox<br>from file_organizer import FileOrganizer<br>from ollama_handler import OllamaHandler<br>class App:<br>    def __init__(self, root):<br>        # ... (code to build buttons, labels, etc.) ...<br>        self.organize_button = tk.Button(root, text=&quot;Start Organizing&quot;, command=self.start_organization)<br>        self.dry_run_var = tk.BooleanVar(value=True)<br>        self.dry_run_check = tk.Checkbutton(root, text=&quot;Dry Run (Preview Only)&quot;, variable=self.dry_run_var)<br>def start_organization(self):<br>        input_dir = filedialog.askdirectory(title=&quot;Select Folder to Organize&quot;)<br>        if not input_dir:<br>            return<br><br># 1. Initialize the same backend classes<br>        ollama_handler = OllamaHandler(model=&quot;gemma3:4b&quot;) #<br>        if not ollama_handler.check_model_available():<br>            messagebox.showerror(&quot;Error&quot;, &quot;Ollama model &#39;gemma3:4b&#39; not found. Please run: ollama pull gemma3:4b&quot;)<br>            return<br>        <br>        organizer = FileOrganizer(<br>            input_folder=Path(input_dir),<br>            output_folder=Path(input_dir) / &quot;organized&quot;,<br>            ollama_handler=ollama_handler,<br>            dry_run=self.dry_run_var.get(),<br>            enable_undo=True<br>        )<br>        <br>        # 2. Run the organization<br>        # (This should be in a separate thread to avoid freezing the GUI)<br>        try:<br>            results = organizer.organize_files()<br>            messagebox.showinfo(&quot;Success&quot;, f&quot;Organization complete! Processed {results[&#39;success&#39;]} files.&quot;)<br>        except Exception as e:<br>            messagebox.showerror(&quot;Error&quot;, f&quot;An error occurred: {e}&quot;)<br><br># ... (main loop to run the app) ...<br>This approach keeps the core logic separate from the UI, allowing both a main.py and a gui.py to use the same powerful backend.</pre><h3>System Requirements</h3><p>A build like this isn’t free in terms of resources. Because the AI models run locally, the system requirements are more than a simple script:</p><ul><li><strong>Ollama:</strong> Needs to be installed and running.</li><li><strong>AI Model:</strong> The gemma3:4b model requires about 3GB of disk space (a one-time download).</li><li><strong>Python:</strong> Python 3.8+</li><li><strong>RAM:</strong> You need at least 8GB of RAM, as loading the AI model is memory-intensive</li></ul><h3>The Result: A “Before &amp; After”</h3><p>After putting it all together and building a simple Tkinter GUI on top, the results are exactly what I wanted.</p><p><strong>Before:</strong> ~/Downloads/</p><pre>├── app.py<br>├── invoice_2024_03.pdf<br>├── IMG_3045.jpg<br>├── Screen Shot 2024-10-20.png<br>├── project_contract.docx<br>├── invoice_2024_04.docx<br>├── Client_NDA.pdf<br>├── IMG_3049.png<br>├── quarterly_report.pdf<br>├── bank_statement.txt<br>├── 2024-tax-summary.xlsx<br>└── ui_mockup_v3.jpg</pre><p><strong>After:</strong> ~/Downloads/organized/</p><pre>├── Work/<br>│   ├── Invoices/<br>│   │   ├── invoice_2024_03.pdf<br>│   │   └── invoice_2024_04.docx<br>│   ├── Legal/<br>│   │   ├── project_contract.docx<br>│   │   └── Client_NDA.pdf<br>│   └── Reports/<br>│       └── quarterly_report.pdf<br>├── Personal/<br>│   ├── Finance/<br>│   │   ├── bank_statement.txt<br>│   │   └── 2024-tax-summary.xlsx<br>│   └── Photos/<br>│       ├── IMG_3045.jpg<br>│       └── IMG_3049.png<br>├── Development/<br>│   └── Python/<br>│       └── app.py<br>└── Screenshots/<br>    ├── Screen Shot 2024-10-20.png<br>    └── ui_mockup_v3.jpg</pre><p>The AI created a logical, nested structure based on <em>content</em>, and I have peace of mind knowing I can undo it at any time.</p><h3>Want to Run It Yourself?</h3><p>If you’re interested in running this project, I’ve packaged the entire source code — including the GUI, the backend logic, and detailed documentation — and made it available for a one-time purchase. It includes setup scripts and all the code discussed in this article, ready to run out of the box.</p><p>It’s a great way to get a powerful, private AI tool up and running in minutes and support independent development.</p><p>You can get the complete project here: <a href="https://monjurulkarim.gumroad.com/l/ai-file-organizer">in this link</a></p><p>Thanks for reading, and I hope this inspires you to explore the exciting world of local-first AI!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79433165f4fb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/using-local-llms-to-organize-messy-files-a-technical-deep-dive-79433165f4fb">Using Local LLMs to Organize Messy Files: A Technical Deep Dive</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Built a Private, Multi-User “Chat with Your Documents” App That Runs 100% Offline]]></title>
            <link>https://medium.com/data-science-collective/how-i-built-a-private-multi-user-chat-with-your-documents-app-that-runs-100-offline-713e10da573a?source=rss-c3bc5b95942e------2</link>
            <guid isPermaLink="false">https://medium.com/p/713e10da573a</guid>
            <category><![CDATA[ollama]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[self-hosted]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[private]]></category>
            <dc:creator><![CDATA[Muhammad Karim, Ph.D.]]></dc:creator>
            <pubDate>Tue, 30 Sep 2025 15:39:11 GMT</pubDate>
            <atom:updated>2025-10-08T23:10:47.801Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A technical deep-dive into a self-hosted RAG solution using Python, Ollama, and ChromaDB that puts you back in control</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a3z6knPy-VcLj1MMgRyYEw.png" /><figcaption>Generated by AI</figcaption></figure><p>Last month, I wrote an article on <a href="https://medium.com/data-science-collective/i-built-an-self-hosted-ai-meeting-note-taker-that-runs-100-offline-heres-how-you-can-too-d110b7ef0b95">how I built a self-hosted AI Meeting Note Taker</a>. The response was fantastic and showed a clear demand for private, offline AI tools that put users back in control of their data.</p><p>Whether you’re a small business uploading sensitive client data, a developer working on proprietary code, or just someone who wants to organize personal files without sending them to a third-party server, the problem is the same. You’re forced to choose between powerful AI tools and data privacy.</p><p>Building on that momentum, I decided to create the solution I wanted to see: a full-featured, multi-user “Chat with Your Documents” application that runs entirely on your own hardware. This is the second part of my self-hosted AI series — a technical deep-dive into a production-ready RAG application for everyone.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FAsCBroOevGA%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DAsCBroOevGA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FAsCBroOevGA%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c0591a394dcb715fbb6ec82d54d1ba06/href">https://medium.com/media/c0591a394dcb715fbb6ec82d54d1ba06/href</a></iframe><h3>The Mission: A Private AI for Everyone</h3><p>The core technical goals for this project were:</p><ol><li><strong>100% Local Processing</strong>: All data, from your company’s strategy documents to your personal financial records, is processed locally. Nothing ever leaves your server.</li><li><strong>Team &amp; Family Ready</strong>: The system is designed for multiple users, with isolated knowledge bases and a central admin dashboard to manage everything.</li><li><strong>Efficient &amp; Smart</strong>: The RAG pipeline intelligently syncs only new or modified documents, saving time and computational resources.</li><li><strong>Incredibly Simple to Deploy</strong>: The entire application is packaged into a single executable with easy-to-use installer scripts. No complex setup required.</li></ol><h3>The Tech Stack: The Powerhouses</h3><p>This application is built on a foundation of powerful open-source tools:</p><ul><li><strong>Backend</strong>: <strong>Flask</strong> serves as the lightweight web server, with Gunicorn for threaded performance to handle concurrent users.</li><li><strong>AI &amp; Embeddings</strong>: <strong>Ollama</strong> runs open-source LLMs (like Llama 3, Gemma) and embedding models locally.</li><li><strong>Vector Store</strong>: <strong>ChromaDB</strong> provides a persistent, on-disk vector database for storing document embeddings efficiently.</li><li><strong>RAG Orchestration</strong>: <strong>LangChain</strong> glues the components together, managing the flow from document loading to question-answering.</li><li><strong>User Management</strong>: A custom authentication layer using <strong>SQLite</strong> for persistence and <strong>PyJWT</strong> for secure sessions.</li></ul><h4>Hardware Requirements &amp; Performance</h4><p>A common question with self-hosted AI is about the hardware required. The application is designed to be flexible, and performance will scale with your machine’s capabilities. The main requirement is sufficient RAM to load the language models.</p><p>To give you a real-world idea, here are a few setups I’ve tested:</p><ul><li><strong>Good (Accessible Start):</strong> For users with more modest hardware (e.g., a laptop with 8–16GB of RAM), the application runs well with a smaller, efficient model like <strong>gemma3:4b</strong>. The results are surprisingly good for most document Q&amp;A tasks, making this a great starting point.</li><li><strong>Better (Smooth Personal Use):</strong> On my MacBook with 24GB of RAM, it handles a 12-billion parameter model (eg.<strong>gemma3:12b)</strong> very smoothly for all my personal documents and projects. This offers a noticeable boost in the quality of the AI’s responses.</li><li><strong>Best (Team Performance):</strong> At my office, we have it on a server with an NVIDIA A6000 GPU. Our whole team of 6 uses it with the much larger <strong>gemma3:27b</strong> model without any issues.</li></ul><p>This demonstrates that the application can effectively scale from a standard laptop for individual use to a dedicated server for team collaboration, depending on your needs.</p><h3>Deep Dive: The System Architecture</h3><p>The application is broken down into three main Python components: the web server (main_app.py), the authentication system (auth.py), and the RAG core (local_rag_chroma.py).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*26DUAUUNyRiUFVVvSD2-cg.png" /><figcaption>The application includes a secure login portal and a dashboard for user administration. These features allow for multi-user support, making the app suitable for shared use by a project team, a small business, or a family, with each user having their own secure access. (image by author)</figcaption></figure><h4>1. The RAG Core: local_rag_chroma.py</h4><p>This is the heart of the application. I designed a KnowledgeBaseManager class to handle the entire lifecycle of document processing and retrieval.</p><p><strong>Intelligent Document Synchronization</strong></p><p>To avoid redundant processing, I implemented a synchronization function that compares the state of files on disk with the metadata stored in ChromaDB.</p><p>The _synchronize_documents method works in a few steps:</p><ol><li>It scans the document directories and creates a dictionary of all current files and their last-modified timestamps.</li><li>It queries the ChromaDB collection to get a list of already-indexed files and their modification times stored in the metadata.</li><li>By comparing these two lists, it identifies:</li></ol><ul><li><strong>New files</strong> to be added.</li><li><strong>Modified files</strong> that need to be deleted and re-indexed.</li><li><strong>Deleted files</strong> whose chunks must be removed from the database.</li></ul><p>This ensures that only necessary changes are processed, making startup and resyncing incredibly fast.</p><p><strong>Document Loading and Splitting</strong></p><p>The system uses LangChain’s document loaders to handle various file types (.pdf, .docx, .md, .txt, .csv,etc.). Each document is then split into manageable chunks using the RecursiveCharacterTextSplitter.</p><pre># A snippet from _load_and_split_document in local_rag_chroma.py<br>def _load_and_split_document(self, file_path):<br>    # ... logic to select the correct loader based on file extension ...<br>    loader = PyPDFLoader(file_path) # Example for PDF<br>    documents = loader.load()    # Add file metadata for synchronization<br>    file_mod_time = os.path.getmtime(file_path)<br>    for doc in documents:<br>        doc.metadata[&#39;source&#39;] = file_path<br>        doc.metadata[&#39;last_modified&#39;] = file_mod_time<br>        text_splitter = RecursiveCharacterTextSplitter(<br>                chunk_size=CHUNK_SIZE, <br>                chunk_overlap=CHUNK_OVERLAP<br>            )<br>    split_docs = text_splitter.split_documents(documents)<br>    return split_docs</pre><p><strong>Embedding and Storage</strong></p><p>The chunks are then passed to a locally running embedding model via OllamaEmbeddings. Each resulting vector is stored in a persistent ChromaDB collection, creating a searchable index for that specific knowledge base. The application creates separate, isolated collections for each team&#39;s knowledge base and each user&#39;s personal documents, ensuring data segregation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/1*5ZDyibiH5CcT3t88nAanCA.png" /><figcaption>Here, the RAG system is demonstrated on a private, personal note. The model is able to extract and synthesize information directly from the user’s content. Because all processing is handled locally, it’s possible to run queries on sensitive information without the data ever leaving the machine. (image by author)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/1*pnY05MXrT_C1Nt7Bh5yEvQ.png" /><figcaption>This example demonstrates the system’s ability to query complex, structured documents. The model extracts specific data points from a financial report in response to a direct question. This showcases its utility for analyzing dense information, whether it’s technical documentation, financial data, or academic papers. (image by author)</figcaption></figure><h4>2. The Multi-User System: auth.py</h4><p>A robust application needs a solid authentication and authorization layer. I built one from scratch using a simple SQLite database to store user and session information.</p><p><strong>Database and Authentication</strong></p><p>The DatabaseManager class sets up tables for users, sessions, and analytics. User passwords are never stored in plain text; instead, I use werkzeug.security to store salted and hashed passwords.</p><p>When a user logs in, the system verifies their credentials and generates a JSON Web Token (JWT) that is used to authenticate subsequent API requests.</p><p><strong>Securing Endpoints with Decorators</strong></p><p>To protect the API, I implemented custom decorators. The @require_auth decorator checks for a valid session or JWT, while @require_role(&#39;admin&#39;) restricts access to admin-only endpoints, like the analytics dashboard.</p><pre># A snippet from auth.py<br>def require_role(required_role):<br>    &quot;&quot;&quot;Decorator to require specific user role.&quot;&quot;&quot;<br>    def decorator(f):<br>        @wraps(f)<br>        @require_auth<br>        def decorated_function(*args, **kwargs):<br>            user_role = request.current_user.get(&#39;role&#39;, &#39;user&#39;)<br>            # ... logic to check if user has sufficient privileges ...<br>            return f(*args, **kwargs)<br>        return decorated_function<br>    return decorator</pre><p>This design makes it easy to secure new API routes as the application grows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*5hY0KKDPR8oSjEqdSAgUqw.png" /><figcaption>Admin Dashboard for analytics, knowledge base management, and user management.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*OC4BNtWWTJQ9cIWmZ40eXg.png" /><figcaption>User management using the admin dashboard</figcaption></figure><h4>3. The Web Server: main_app.py</h4><p>The Flask application ties everything together. It defines the API endpoints for the frontend to interact with. The most important one is /api/ask.</p><pre># A simplified view of the /api/ask endpoint in main_app.py<br>@app.route(&#39;/api/ask&#39;, methods=[&#39;POST&#39;])<br>@require_auth<br>def ask_question_api():<br>    session_id = get_or_create_session_id()<br>    data = request.get_json()<br>    question = data[&#39;question&#39;]<br>    kb_id = data.get(&#39;knowledge_base_id&#39;, &#39;user_personal&#39;)<br>    # Get or create a RAG instance for this session and knowledge base<br>    rag_chain, retriever, memory = get_or_create_rag_instance(session_id, kb_id)<br>    <br>    if not rag_chain:<br>        return jsonify({&quot;error&quot;: &quot;RAG service not ready.&quot;}), 503<br>    # Invoke the RAG chain and return the response<br>    response = rag_chain.invoke({&quot;query&quot;: question})<br>    <br>    # ... format and return the answer and sources ...</pre><p>The application maintains a thread-safe dictionary (rag_instances) to manage separate RAG chains and conversation histories for each user session and selected knowledge base, preventing memory leaks and ensuring that conversations are isolated.</p><h3>The Result: Your Private AI Knowledge Base for Work and Life</h3><p>The final result is a powerful, production-ready application that turns any collection of documents into an interactive AI assistant. It’s the perfect tool for:</p><ul><li><strong>Small Businesses</strong> wanting a secure, internal knowledge base for their team without the high cost and privacy risks of SaaS products.</li><li><strong>Developers &amp; Freelancers</strong> who need a powerful, private RAG system for their own documents or as a foundation for client projects.</li><li><strong>Personal Users &amp; Families</strong> who want to securely organize and ask questions about their private files, from financial records to research papers.</li></ul><p>It’s the best of both worlds: the power of modern AI without sacrificing control over your data.</p><h3>Want to Run It Yourself?</h3><p>This project takes the core principles of local processing from the AI Meeting Note Taker and expands them into a robust, multi-user platform ready for any use case.</p><p>I’ve packaged the entire source code and made it available for a <strong>one-time purchase</strong>. It’s a fantastic way to get a powerful, private AI tool up and running in minutes and serves as a solid foundation for your own customizations.</p><p>You can get the complete <strong>RAG Application</strong> project here: <a href="https://monjurulkarim.gumroad.com/l/self-hosted-rag"><strong>Self-hosted-rag</strong></a></p><p>And if your primary need is transcribing and summarizing audio, be sure to check out the first project in this series!</p><p>You can get the complete <strong>AI Meeting Note Taker</strong> project here: <a href="https://monjurulkarim.gumroad.com/l/ai-meeting-notes"><strong>AI-meeting-note</strong></a></p><p>Thanks for reading! I hope this deep-dive inspires you to take control of your data and explore the incredible potential of self-hosted AI.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=713e10da573a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/how-i-built-a-private-multi-user-chat-with-your-documents-app-that-runs-100-offline-713e10da573a">How I Built a Private, Multi-User “Chat with Your Documents” App That Runs 100% Offline</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Built an self-hosted AI Meeting Note Taker That Runs 100% Offline — Here’s How You Can Too]]></title>
            <link>https://medium.com/data-science-collective/i-built-an-self-hosted-ai-meeting-note-taker-that-runs-100-offline-heres-how-you-can-too-d110b7ef0b95?source=rss-c3bc5b95942e------2</link>
            <guid isPermaLink="false">https://medium.com/p/d110b7ef0b95</guid>
            <category><![CDATA[meeting-notes]]></category>
            <category><![CDATA[self-hosted]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ollama]]></category>
            <category><![CDATA[productivity]]></category>
            <dc:creator><![CDATA[Muhammad Karim, Ph.D.]]></dc:creator>
            <pubDate>Thu, 21 Aug 2025 20:24:51 GMT</pubDate>
            <atom:updated>2026-08-20T04:34:57.833Z</atom:updated>
            <content:encoded><![CDATA[<h3>I Built a self-hosted AI Meeting Note Taker That Runs 100% Offline</h3><h4><strong>A private, unlimited, and self-hosted solution with Python, Whisper, and Ollama.</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ega-bRi_wYVKVyorEO29sA.png" /><figcaption>Generated by AI</figcaption></figure><p><strong>Update (Aug 2026):</strong> I rebuilt MeetingMint as a native Mac app — Whisper + on-device LLMs, no server at all. If you want the supported version, it’s on the Mac App Store: <a href="https://apps.apple.com/app/meetingmint/id6801946726">MeetingMint</a>. This article remains the deep-dive on the Flask architecture.</p><p>Taking notes during meetings is a chore.</p><p>Trying to capture every detail while also participating is nearly impossible. Naturally, many of us have turned to AI-powered transcription services. They’re magical, but they come with a catch: you have to upload your private, often confidential, meeting recordings to a third-party server.</p><p>What if your company discusses unannounced products? What if you’re a therapist, lawyer, or journalist handling sensitive client information? The privacy trade-off can be a deal-breaker. On top of that, the free tiers are often restrictive, with short recording limits that are quickly exhausted.</p><p>I wanted a better solution: a tool with the power of AI but the privacy of a local-first application. So, I built one. In this article, I’ll walk you through how I created a web application that transcribes and summarizes audio files of any length, right on your own machine.</p><h4>The “Why”: Taking Back Control of Your Data</h4><p>The goal was simple: create a tool that was:</p><ol><li><strong>100% Private:</strong> All processing, from transcription to summarization, happens locally. No data ever leaves the machine.</li><li><strong>Unlimited:</strong> No more watching the clock. Transcribe a 3-hour lecture or a 10-minute huddle without hitting a paywall.</li><li><strong>Easy to Use:</strong> A simple web interface that anyone can use without needing to be a command-line wizard.</li></ol><h4>The Tech Stack: Our Open-Source Powerhouses</h4><p>This project stands on the shoulders of some incredible open-source tools:</p><ul><li><strong>OpenAI’s Whisper:</strong> For transcription. Whisper is a state-of-the-art speech-to-text model that provides incredibly accurate transcriptions across a wide range of accents and languages. We’ll be running it locally for free.</li><li><strong>Ollama:</strong> For summarization. Ollama is a fantastic tool that makes it incredibly easy to download and run powerful Large Language Models (LLMs) like Gemma, Phi-4, or any other model on your own hardware.</li><li><strong>Flask &amp; Pydub:</strong> For the backend. Flask is a lightweight Python web framework that will handle file uploads and serve our application. Pydub is a simple and powerful library for audio manipulation, which we’ll use to split large audio files into manageable chunks for Whisper.</li></ul><h4>Building the Backend: The Python Core</h4><p>The heart of our application is a Flask server (app.py) that orchestrates the entire process. Let&#39;s break down the key components.</p><h4>1. Handling the File Upload</h4><p>First, we need a simple Flask route to accept an audio file. We’ll secure the filename, save it to an uploads folder, and then kick off the processing in a background thread so the user interface doesn&#39;t freeze.</p><pre># In app.py<br>@app.route(&#39;/upload&#39;, methods=[&#39;POST&#39;])<br>def upload_file():<br>    if &#39;file&#39; not in request.files:<br>        return jsonify({&#39;error&#39;: &#39;No file part&#39;}), 400<br>    <br>    file = request.files[&#39;file&#39;]<br>    if file and allowed_file(file.filename):<br>        filename = secure_filename(file.filename)<br>        file_path = os.path.join(app.config[&#39;UPLOAD_FOLDER&#39;], filename)<br>        file.save(file_path)<br>        <br>        job_id = str(uuid.uuid4())<br>        <br>        # Start background processing<br>        thread = threading.Thread(target=process_audio_file, args=(file_path, job_id))<br>        thread.daemon = True<br>        thread.start()<br>        <br>        return jsonify({&#39;job_id&#39;: job_id, &#39;status&#39;: &#39;started&#39;})</pre><h4>2. The Audio Processing Pipeline</h4><p>This is where the magic happens. The process_audio_file function is a multi-step pipeline:</p><p><strong>Step A: Split the Audio:</strong> Whisper works best on smaller audio segments. We use pydub to split the source audio into 10-minute chunks.</p><pre># In app.py<br>from pydub import AudioSegment<br>import math<br><br>def split_audio(audio_path, temp_dir, segment_minutes=10):<br>    segment_ms = segment_minutes * 60 * 1000<br>    audio = AudioSegment.from_file(audio_path)<br>    # ... logic to slice audio and export segments ...<br>    return segment_paths</pre><p><strong>Step B: Transcribe Each Chunk:</strong> We loop through our audio chunks and feed each one to our locally loaded Whisper model.</p><pre># In app.py<br>import whisper<br>WHISPER_MODEL = whisper.load_model(&quot;large-v3&quot;) # Or &quot;base.en&quot; for speed<br><br>def transcribe_segment(segment_path):<br>    result = WHISPER_MODEL.transcribe(segment_path, language=&quot;en&quot;)<br>    return result.get(&quot;text&quot;, &quot;&quot;).strip()</pre><p><strong>Step C: Summarize with Ollama:</strong> Once we have the full transcript, we send it to our local LLM via Ollama&#39;s API with a clear prompt.</p><pre># In app.py<br>import requests<br><br>def query_ollama(prompt):<br>    full_url = f&quot;{OLLAMA_BASE_URL}/api/generate&quot;<br>    payload = {<br>        &quot;model&quot;: &quot;gemma3:27b&quot;, # Or any model you have pulled<br>        &quot;prompt&quot;: prompt,<br>        &quot;stream&quot;: False<br>    }<br>    response = requests.post(full_url, json=payload)<br>    # ... error handling and response parsing ...<br>    return result.get(&quot;response&quot;)<br><br># Inside the processing pipeline...<br>prompt = f&quot;Based on the transcripts below, make a concise professional meeting minutes\n\n---\n\n{combined_transcript}&quot;<br>meeting_minutes = query_ollama(prompt)</pre><h3>Creating the Frontend: A Simple UI</h3><p>The frontend is a single index.html file with some vanilla JavaScript. The core logic is:</p><ol><li><strong>Upload:</strong> Use the fetch API to send the file to our Flask backend.</li><li><strong>Poll for Status:</strong> After the upload starts, the JavaScript periodically hits a /status/&lt;job_id&gt; endpoint on our server every few seconds.</li><li><strong>Display Progress:</strong> The status endpoint returns the current stage (e.g., &quot;Transcribing segment 2/5...&quot;), which we display to the user.</li><li><strong>Render Results:</strong> Once the status is &quot;completed,&quot; the final transcript and minutes are returned and rendered on the page.</li></ol><h3>The Result: Your Personal AI Note Taker</h3><p>After putting it all together, we have a fully functional web application that can take any audio file and turn it into a neatly summarized set of meeting minutes, complete with a full transcript.</p><p>The best part? It all happens on your terms, on your machine, with no one else seeing your data. It&#39;s the perfect blend of modern AI power and old-school privacy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hwaqxF9fUz77OwsaUZN62w.png" /><figcaption>The simple workflow: (1) Drag and drop any audio file, (2) watch the real-time processing status, (3) review the AI-generated meeting minutes, and (4) access the full transcript.</figcaption></figure><h3>Want to Run It Yourself?</h3><p>If you&#39;re interested in running this project, I&#39;ve packaged the entire source code and made it available for a one-time purchase. It includes detailed setup instructions and all the code discussed in this article, ready to run out of the box.</p><p>It&#39;s a great way to get a powerful, private AI tool up and running in minutes and support independent development.</p><p>The project has since become a native <a href="https://apps.apple.com/app/meetingmint/id6801946726">Mac app</a> (Mac App Store, on-device AI, no Python server to run). This Flask version is still available below if you prefer self-hosting<strong>: </strong><a href="https://monjurulkarim.gumroad.com/l/ai-meeting-notes/MEDIUM25"><strong>in this link</strong></a></p><p>Thanks for reading, and I hope this inspires you to explore the exciting world of local-first AI!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d110b7ef0b95" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/i-built-an-self-hosted-ai-meeting-note-taker-that-runs-100-offline-heres-how-you-can-too-d110b7ef0b95">I Built an self-hosted AI Meeting Note Taker That Runs 100% Offline — Here’s How You Can Too</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tutorial on Surface Crack Classification with Visual Explanation (Part 2)]]></title>
            <link>https://medium.com/data-science/tutorial-on-surface-crack-classification-with-visual-explanation-part-2-f8638960d0d7?source=rss-c3bc5b95942e------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8638960d0d7</guid>
            <category><![CDATA[xai]]></category>
            <category><![CDATA[explainable-ai]]></category>
            <category><![CDATA[crack]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[pytorch]]></category>
            <dc:creator><![CDATA[Muhammad Karim, Ph.D.]]></dc:creator>
            <pubDate>Sun, 05 Dec 2021 18:18:15 GMT</pubDate>
            <atom:updated>2021-12-06T16:34:58.143Z</atom:updated>
            <content:encoded><![CDATA[<p>A step-by-step guide to develop explainable AI to classify surface crack images with Pytorch. This is the second part of a 2 part series.</p><p>In the last part (<a href="https://towardsdatascience.com/tutorial-on-surface-crack-classification-with-visual-explanation-part-1-14542d2ea7ac">Part 1</a>) you have seen how to build a CNN model using transfer learning and how to train the network. You saw that our network can perform well in classifying crack images. But you may have questions - how is our network making decisions? Did our network look at the right location of an image to take its decision? We are not sure about that. Our network is a black box to us. We do not know what our convolutional neural network sees and understands when it makes a decision.</p><p>In this blog, we are going to produce visual explanation heat-maps that will help us to understand how our deep convolutional neural network algorithm makes it’s decision of having a crack or no-crack in an image.</p><p>In the previous part (<a href="https://towardsdatascience.com/tutorial-on-surface-crack-classification-with-visual-explanation-part-1-14542d2ea7ac">Part 1</a>), you did not see how to visualize the prediction results. Therefore, in this part I am going to do the following:</p><ol><li>Predict crack class on different images using our trained network (Inferencing)</li><li>Use Grad-CAM [1] to generate visual explanation heatmap that can explain the decision of our network</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/735/1*R6qKQhE8Wd1SxNQYtmDRBQ.png" /><figcaption>Image by Author.</figcaption></figure><h3><strong>1. Inferencing</strong></h3><p>We have already trained our network in <a href="https://towardsdatascience.com/tutorial-on-surface-crack-classification-with-visual-explanation-part-1-14542d2ea7ac">Part 1</a>. Now we only have to see what it looks like to predict an image with our trained network. To predict an image crack class, we need to create a python file called <em>inference.py</em> and keep it in the working directory.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b0190cb16226429767c685ebb5baaa1e/href">https://medium.com/media/b0190cb16226429767c685ebb5baaa1e/href</a></iframe><p>If you look at the code above, you may find that this part is very similar to the training file. Here we imported the same libraries that we imported for training. After importing the libraries, we will create our helper functions that will help us with the inference.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/93c85ea45919b76dcd47fbca7e8fb354/href">https://medium.com/media/93c85ea45919b76dcd47fbca7e8fb354/href</a></iframe><p>We will randomly select some samples from the validation data directory for prediction. In line #7, we created a function called <em>random_image()</em>. In line #17, we also created another function called <em>save_image()</em> to save our output results in the disk.</p><p>So, we are ready with our helper functions. Now we need a function that takes the input image and then gives us the prediction result through our trained network. To do this, we will create a function called <em>inference()</em>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/10af8092ff0ff5ebc9d60fde413fb8ae/href">https://medium.com/media/10af8092ff0ff5ebc9d60fde413fb8ae/href</a></iframe><p>In the above code, line # 2 is our test dataset’s directory. In our test directory, there are two subfolders: <em>Positive</em> and <em>Negative. </em>We can take our test samples either from <em>Positive</em> or from <em>Negative</em> folder. In line #3 we define from where we want to choose our test sample images. Notice that, in line #3, there is an argument <em>p.type</em> that passes this information of <em>Positive</em> or <em>Negative</em> folder. At this point look at line #25, we called an <em>ArgumentParser()</em>. This function will pass an argument while the bash command is given. In line 26, we added an argument <em>type</em>. This means that we want to pass on the information <em>type</em> to the bash. By default, the information is <em>Positive</em>. Then we named this parser argument as <em>p</em> in line # 28. That means that <em>p</em> holds the information <em>type</em>. Inside the <em>type</em>, it has the information <em>Positive</em>. When you run a command in the command line prompt/bash, it will be clearer for you. Okay, now let’s get back to the function <em>inference(). </em>The rest of this is simple, line #7–9, we compiled our model and loaded the trained weight. Then, in line 11, we created a list of random images with the helper function <em>random_image()</em> that we created earlier. In line #14, we transformed our test images and stacked together to pass through our model. With line #16 we pass the images through our model and get our prediction logit. The next line creates a <em>softmax</em> probability for test images. Now we have our prediction probability in hand. Then we are going to save the results with line #20.</p><p>We are done with the coding to visualize the prediction result. To run the code, enter the following command in your terminal. It will take 9 samples from <em>Positive</em> folder, produce results, and save it in your disk.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/035e7018d662c10244679249403a2ad5/href">https://medium.com/media/035e7018d662c10244679249403a2ad5/href</a></iframe><p>Note that, we entered <em>type </em>as <em>Positive. </em>As a result, it takes samples from the folder Positive. If we have entered <em>type </em>as <em>Negative</em>, it would have taken samples from the folder <em>Negative</em>.</p><p>When you run the command, you will get something like the image below. It is a grid of 9 images that have passed through our network. The prediction probabilities are on top of each image. The results are OK, but you can easily improve performance by adopting transfer learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*_j9ePmZXsHmn5FNmAjjEiQ.png" /><figcaption>Visualizing the prediction results. Image by Author</figcaption></figure><h3><strong>2. Visual Explanation</strong></h3><p>Now, we are going to generate visual explanation heat maps. To generate the heat-maps we are going to use Grad-CAM[1] algorithm. The heat-maps identify the image regions that influence the network’s decision. If we look at the heat map, we can easily understand which image pixels contribute to the network’s decision.</p><p>To work with grad-cam, type the following in your terminal to install grad-cam in your local machine:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/be9d078dd893bd7e72c3c74e5921da49/href">https://medium.com/media/be9d078dd893bd7e72c3c74e5921da49/href</a></iframe><p>Now, from this github <a href="https://github.com/monjurulkarim/crack_classification">repository</a>, download the folder <em>pytorch_grad_cam</em> and put it in your working directory. You can see that, there are source code for different variants of grad-cam inside that folder. Once you have kept this folder in your directory, you are ready to use grad-cam and some of its variant algorithms.</p><p>Now you are all set to use grad-cam to generate heat-maps. At this point, you need to create a python file <em>xai.py</em> in the main working directory.</p><p>Inside the <em>xai.py</em>, first let’s import the required libraries and our model:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e8870143cb745f8b73d0ced707a2c082/href">https://medium.com/media/e8870143cb745f8b73d0ced707a2c082/href</a></iframe><p>Then, we need to import the grad-cam libraries:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8e14eda727b52a5e90b7b8217cc3d089/href">https://medium.com/media/8e14eda727b52a5e90b7b8217cc3d089/href</a></iframe><p>In the above code, notice that from the folder <em>pytorch_grad_cam</em> we imported Grad-CAM and different variants of it. We also imported some of the packages for image visualization.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4b1b9f0237d0dd68aa87dc94dde675b5/href">https://medium.com/media/4b1b9f0237d0dd68aa87dc94dde675b5/href</a></iframe><p>The above part is pretty straightforward. In line # 6, we just saved all the imported methods name inside a dictionary. We can call any of the method from here later to produce heat maps. Line # 19 is our helper function for random image selection.</p><p>Then, we need to create our main wrapper function <em>xai()</em> for generating our heat maps.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/637e533302b8b1661c1857d8a94f29b5/href">https://medium.com/media/637e533302b8b1661c1857d8a94f29b5/href</a></iframe><p>In line #2–4, we compiled our model and loaded the trained weight. We need to select a target layer from where gradients will be used to produce heat maps. With line # 6, we set the target layer of our convolutional neural network which is the last layer of the ResNet in our architecture.</p><p>We also need a target category for heat map generation. We have only two classes. For negative class our <em>target_category</em> is 0 and for positive class it is 1. We defined these in line #13–16.</p><p>Then we need to select a grad-cam method that we want to use. We do this in line #18 and wrap it up in line #19. Then in line #22, we loop through our images to create heat maps using our chosen method.</p><p>To run the code, you need to type the following in your terminal:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2ade0144dfe4af33f641ddb4a1c82975/href">https://medium.com/media/2ade0144dfe4af33f641ddb4a1c82975/href</a></iframe><p>Here, notice that we gave two arguments, one is <em>type </em>and the other one is <em>method</em>. With the <em>type </em>we are selecting positive samples from our test dataset and with <em>method </em>we are setting a method for our heat map generation. Here, we are setting gradcam as our method. You can select any other method, eg. Xgrad-cam, Eigen-cam, etc. as well.</p><p>If you run the command you will get something like the below image. It is a grid of 9 images with the heat-maps. Here, the hot zone indicates that they are responsible for the network’s decision of Positive class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*WoIPiTM0axjU3rH3PpQOrw.png" /><figcaption>Heat maps for positive class. Image by Author</figcaption></figure><p>If you look at these images, you can see that hot zones overlap with cracks in the image. That means our network is correctly looking at the right location which is the location of cracks to take its decision.</p><h3>Conclusion</h3><p>In this part, you have seen how to infer using your trained model to predict the class of cracks on different images. You have also seen how to use a grad-cam tool to generate heatmaps that can explain the networks’ decisions. Looking at the heatmaps, you can easily understand how your deep learning algorithm makes decisions. You can easily extend this example and use it in any image classification problem to explain your network’s decision.</p><p>You can find the complete code in <a href="https://github.com/monjurulkarim/crack_classification">Github</a>. If you have not checked out the previous part yet, find it <a href="https://towardsdatascience.com/tutorial-on-surface-crack-classification-with-visual-explanation-part-1-14542d2ea7ac">here.</a></p><p>References:</p><p>[1] Ramprasaath R. Selvaraju et al., 2019, “Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization”, <a href="https://arxiv.org/pdf/1610.02391.pdf">https://arxiv.org/pdf/1610.02391.pdf</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8638960d0d7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/tutorial-on-surface-crack-classification-with-visual-explanation-part-2-f8638960d0d7">Tutorial on Surface Crack Classification with Visual Explanation (Part 2)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tutorial on Surface Crack Classification with Visual Explanation (Part 1)]]></title>
            <link>https://medium.com/data-science/tutorial-on-surface-crack-classification-with-visual-explanation-part-1-14542d2ea7ac?source=rss-c3bc5b95942e------2</link>
            <guid isPermaLink="false">https://medium.com/p/14542d2ea7ac</guid>
            <category><![CDATA[image-classification]]></category>
            <category><![CDATA[explainable-ai]]></category>
            <category><![CDATA[transfer-learning]]></category>
            <category><![CDATA[crack]]></category>
            <category><![CDATA[pytorch]]></category>
            <dc:creator><![CDATA[Muhammad Karim, Ph.D.]]></dc:creator>
            <pubDate>Thu, 07 Oct 2021 19:25:15 GMT</pubDate>
            <atom:updated>2021-12-06T18:19:27.010Z</atom:updated>
            <content:encoded><![CDATA[<h4>A step-by-step guide to develop explainable AI to classify surface crack images with Pytorch. This is part 1 of 2 part series.</h4><h3><strong>Introduction</strong></h3><p>We are in the era of artificial intelligence. Many technologies are evolving to make life easier for an engineer. However, the civil engineering field seems still struggling to be at the pace of this evolution.</p><p>Infrastructure deteriorates rapidly and is approaching their designed life spans. Because infrastructure often operates in harsh environments. For example, there are constant loads and vibrations on the roads and bridges. Besides, there are multiple hazards such as earthquake, flood, hurricane, tornado, or tsunami takes place all around the year. If we only consider bridges in the United States, there are more than 600,000 highway bridges in the National Bridge Inventory. 39% have exceeded their design life of 50 years. National Bridge Inspection Standards require bridges to be inspected every two years to ensure that there are no cracks, rusting, or other damage. That means, every day hundreds of bridges need to be inspected. Current infrastructure inspection systems have many issues. For example, the data collection is time-consuming, it may require dangerous field activities with many heavy equipments. Sometimes accessibility is an issue for elevated bridge structures. Also, there could be biasness of human visual inspection. Thus, results of the visual inspection can vary largely among different inspectors.</p><p>One possible solution to this is the use of a robotic platform to collect inspection video data. Collecting inspection video data using aerial platforms reduces or eliminates the labor-intensive onsite inspection process and allows inspectors to assess bridges from a safer location.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LW4hobo4_t65eJYV" /><figcaption>Photo by <a href="https://unsplash.com/@jakobowens1?utm_source=medium&amp;utm_medium=referral">Jakob Owens</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Yet, the use of robotic inspection platforms has solved just part of the above-discussed issues, efficient, reliable analysis of inspection video is another important task. Letting inspectors watch the collected videos for hours and days to find cracks are inefficient. It is desired that a tool can be developed to assist inspectors in extracting different types of cracks from the videos or images.</p><p>To automate this inspection process, this article will use artificial intelligence to classify images to find cracks in the image. More specifically, we will build and train a deep neural network using Pytorch to find cracks in an image to replace the human in the tedious task of cracks classification in a structure.</p><p>Ok, enough motivation for today. Now, let’s get straight to the point. We are going to do the following three things:</p><ol><li>Create a deep neural network from scratch using PyTorch</li><li>Implement an Explainable AI tool to explain the decision made by the deep neural network</li></ol><p>To not make the blog very long, I split the above two contents into a 2 part series. This is the first part of the series. The link for the other part is given at the end of this blog.</p><h3><strong>1. Deep Neural Network with Pytorch</strong></h3><p>Now, let’s create a deep neural network using PyTorch. I will show you all the steps to create a deep neural network for image classification. As an application, I am going to classify surface cracks in an image. After reading the blog you will be able to do image classification for any different application domain of your choice using the same procedure as this article. So please bear with me.</p><p>To create the classifier we are going to follow 4 steps:</p><ol><li>Data preparation</li><li>Building the model</li><li>Training</li><li>Testing and inferencing</li></ol><h3>Data preparation</h3><p>At first, we are going to prepare a dataset for our classification. As I already mentioned we are going to classify surface cracks from images, we are going to use a crack dataset. We are going to use a subset of Mendeley [1] (License: CC BY 4.0) dataset. You can download the dataset from this <a href="https://drive.google.com/drive/folders/1HMhZR9erch7OHL3X-HCrdU3-JCxxPDcx?usp=sharing">link</a>. If you want you can directly use the Mendeley dataset by downloading from their website.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/709/1*6uEEdR69fdHlV57UYkez7Q.png" /><figcaption>Positive Mendeley[1] data. Image by Author.</figcaption></figure><p>Anyway, we are going to use the small subset of this dataset as our primary aim is to build a neural network. The dataset has a total of 1200 (600 positives + 600 negatives) images of size 227 x 227. Which is then split into 80% train and 20% test data. Please, download the dataset and put it inside your working directory. The initial folder structure will look like this, blue ones are folders and white ones are files (we will add more files and folders later).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/210/1*lPo90ADz5fa-QZQQvQcjMA.png" /><figcaption>Folder structure. Image by Author.</figcaption></figure><p>For loading the data we are going to create a python file named ‘dataloader.py’ as you can see inside the folder ‘src’.</p><p>At first, we need to import some required packages:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4feb815c79feffd533b308f7f4e89f0c/href">https://medium.com/media/4feb815c79feffd533b308f7f4e89f0c/href</a></iframe><p><strong>torch.utils.data</strong> imports the required functions we need to create and use Dataset and DataLoader.</p><p><strong>PIL</strong> is a popular computer vision library to load images in our python program.</p><p>Now let’s create a custom dataset class, that we named the class CrackDataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bee72a173b5c46028910909a0b37ec03/href">https://medium.com/media/bee72a173b5c46028910909a0b37ec03/href</a></iframe><p><strong>class CrackDataset(Dataset):<em> </em></strong>Create a class called ‘CrackDataset’. We can call this class from anywhere we want. It will be passed to the ‘Dataset’ module that we imported earlier.</p><p><strong>def __init__(self, data_path, phase, transform): </strong>After initializing the class we need to pass three variables. data_path is the directory of our dataset. We will pass phase=train or test. Transform to perform input data transformation. We will talk about this more when we pass this information to our class CrackDataset. We create a list named self.data containing the pair of image path and their respective labels. We also want to map all the Negative classes as 0 and the positive classes as 1.</p><p><strong>def __len__(self): </strong>This function returns the length of the labels.</p><p><strong><em>def __getitem__(self, idx): </em></strong>This function is used by Pytorch’s Dataset module to get a sample and construct the dataset. When initialized, it will loop through this function creating a sample from each instance in the dataset. It defines the (x,y) or (image, label) pair. Index (idx) is used internally within pytorch to keep track of all the data and create batches. It keeps track of all the batches as well. Simply saying, it takes care of all the bookkeeping of our custom dataset. Here, self.data is the list containing the image path and it’s class name. For each index we will have a pair of img_path and class_name. Then using PIL library we load the image and convert it to RGB format. After that we use self.transform to convert the image to tensor and a few more things (You will see what transformation we are going to use later). One of the requirement of the PyTorch loss function is that the input must be tensor form before feeding to a CNN network, so we converted it to float and tensor.</p><p>Now, we have created our custom dataloader. Compile all the code in a file ‘dataloader.py’ and put the file inside the folder ‘src’.</p><h3>Building the model</h3><p>Now let’s create our model in the model.py file and put it inside the ‘src’ folder.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a8db45e124fe0016632f993b953e2938/href">https://medium.com/media/a8db45e124fe0016632f993b953e2938/href</a></iframe><p>We are going to use transfer learning. You probably know about transfer learning. It is a technique where we use a pre-trained model that were already been trained on a large dataset. We will use the trained weights to initiate our training. The only thing we will have to do is that we need to modify the last few layers of our model and update the weights only for those layers.</p><p>Torchvision has many built-in CNN models like VGG16, ResNet, Inception, and so on. We just need to import them. Here we are going to use ResNet architecture.</p><p>We are going to create a class named CrackClassifier. It will use nn.Module from PyTorch. self.resnet will download the pre-trained resnet50 models from torchvision. From the code you can see that, we modified self.resnet.fc to sequentially add two linear layers a ReLU and a LogSoftmax. Here you can see, in the last linear layer we map the input features to num_classes. That means our output node will be equal to our number of classes. If you want to use your own CNN architecture just replace the self.resnet with it.</p><p>The def forward function defines how our model will run after taking the input. Here, in our case it will take the input feature (x) and will pass through the self.resnet that we already defined.</p><h3>Training the network</h3><p>We are going to create a file named train.py and keep it in the main directory as shown in the Figure folder structure. At first, let’s import the required libraries:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d91556f0f29e645e3e93b065104a5916/href">https://medium.com/media/d91556f0f29e645e3e93b065104a5916/href</a></iframe><p>If you look at the code, from line 1–4 we imported some libraries from torch we have seen before. In line 6 and 7, notice that we used “from src.model” and “src.dataloader”. Remember that inside the “src” folder we already created two python files model and dataloader. We are going to import class CrackClassifier and class CrackDataset from those two files respectively. In line 8 we imported tqdm which will be used to show training progress. os module will be used for defining directory. We are going to use SummaryWriter from tensorboardX to visualize our training and testing performance. Line 12 ensures if we have GPU then it will use the GPU otherwise will use a CPU for computation.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/11219ff5acbf142bd80273df3ccd0394/href">https://medium.com/media/11219ff5acbf142bd80273df3ccd0394/href</a></iframe><p>torchvision.transform library helps us doing a lot of processing of our input images. For example, here Resize option makes sure that all of our images in a batch are of size 64 x 64. Then we convert the images to tensor using ToTensor option. Finally, we normalize the images. Here, in the Normalize option, the first tuple is the mean and the second tuple is the standard deviation of three channels of RGB across all batches. You can also use different data augmentation techniques here.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3cf4b5f6899c8cd718264b3f4198ddcc/href">https://medium.com/media/3cf4b5f6899c8cd718264b3f4198ddcc/href</a></iframe><p>Next, we need to define the hyperparameters for our training. For example, number of epochs, learning rate, batch size. After that, we need to set up our dataset and dataloader. For train and test we set two datasets separately using our CrackDataset class. After setting up these two datasets we use pytorch’s Dataloader to create our train and test dataloader named train_dataloader and test_dataloader respectively.</p><p>Now, before we jump into the training loop at first create a function to check the accuracy of our model. We will call the function from our training loop to check the accuracy on our test dataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9a8dda695c575c37a7e6ae9e738ca9f7/href">https://medium.com/media/9a8dda695c575c37a7e6ae9e738ca9f7/href</a></iframe><p>Using our model we will get the output, which will be compared with the ground truth label of that image to calculate the accuracy. It is very simple which you can easily understand looking at the code, so I am skipping the detailed explanation.</p><p>Now we will create two more simple function to use summary writer of tensorboard. So that we can visualize the loss and accuracy at each epoch in a graph form.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/89ce8d7f66fe7c9845edd361b75f0397/href">https://medium.com/media/89ce8d7f66fe7c9845edd361b75f0397/href</a></iframe><p>Now, let’s create a training loop:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7a52bd9f35df4bfe703e28e73e1fa3c1/href">https://medium.com/media/7a52bd9f35df4bfe703e28e73e1fa3c1/href</a></iframe><p>From line 2–8, creates our directory for saving the trained model and our logs. Line 9 creates a summary writer in our logs directory.</p><p>Line 11, creates our model using the class that we created named CrackClassifier. Criterion is the loss function. We will use CrossEntropyLoss function from pytorch module nn. Then we define our optimizer to optimize the network using gradient descent. We use the popular Adam optimizer from torch.optim.</p><p>Recall that we have two fully connected layers in our model. We only want to update the weights of those two layers. We can freeze the weights of the rest of the layers. Line 16–22 will do that for us. Our fc.0 and fc.2 are two fully connected layers. We want to update the weights and bias terms of these two layers. Therefore, for these two layers we set param.requires_grad= True. For the rest of the layers it is false. That is we froze the rest of the layers.</p><p>Line 24 initiate the training. Then we loop through the number of epochs. For each epoch we iterate through all batches of images and labels in the train loader. After every two epochs we use our test_loader to check accuracy on the test dataset and write the accuracy on the logger using write_test_scalars() function that we created earlier. We use our model’s output and calculate the loss using our predefined loss function. Before we do back-propagation to calculate gradients we must perform the optimizer.zero_grad() operation- this empties the gradient tensors from the previous batch so that the gradients for the new batch are calculated newly. Now to perform back-propagation we use loss.backward(). Notice that we have used tqdm. This is used to show the training progress with a progress bar. Then with the write_scalars() function we write the losses of each epoch. We save the model when the accuracy on the test dataset is the highest. Finally, we close our summary writer by using logger.close().</p><p>Now, we have all our code ready. We need to run the code. To run the training code, on your terminal type: “python train.py”. It will train the network, save the trained model inside “snapshot” and logs inside “logs” folder.</p><p>To visualize your tensorboard, go to your terminal and type the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9923036626ce3eca209aa6b576ee75f1/href">https://medium.com/media/9923036626ce3eca209aa6b576ee75f1/href</a></iframe><p>After that, go to your browser, and in the address bar type: “localhost:6006”. Then you can see your training performance in the tensorboard as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/1*03s6Xa6wB-tEKBopNkKa0Q.png" /><figcaption>Image by Author</figcaption></figure><p>See, our model&#39;s performance gradually increases.</p><h3>Conclusion</h3><p>In this part, you have seen how to create a custom dataloader, how to build a CNN model using transfer learning, how to train the network, and how to log the training performance using tensorboard. In the next part, we will see how to predict cracks class on different images using our trained model. That means we will do inferencing with our trained model. And finally, we will use the grad-cam tool to explain the decision of our network. Please, check out the next part of this blog in <a href="https://towardsdatascience.com/tutorial-on-surface-crack-classification-with-visual-explanation-part-2-f8638960d0d7">here</a>.</p><p>You can find the complete code in <a href="https://github.com/monjurulkarim/crack_classification">Github</a>. Download the <a href="https://drive.google.com/drive/folders/1HMhZR9erch7OHL3X-HCrdU3-JCxxPDcx?usp=sharing">Dataset</a>.</p><p>References:</p><ol><li>Özgenel, Çağlar Fırat (2019), <a href="https://data.mendeley.com/datasets/5y9wdsg2zt/2">“Concrete Crack Images for Classification”</a>, Mendeley Data, V2, doi: 10.17632/5y9wdsg2zt.2. (License : CC BY 4.0)</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=14542d2ea7ac" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/tutorial-on-surface-crack-classification-with-visual-explanation-part-1-14542d2ea7ac">Tutorial on Surface Crack Classification with Visual Explanation (Part 1)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>