<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aqib on Medium]]></title>
        <description><![CDATA[Stories by Aqib on Medium]]></description>
        <link>https://medium.com/@aqibnawab1100?source=rss-6ea1c739483------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*3txE31J3kLVJkPUmkHG9PQ.jpeg</url>
            <title>Stories by Aqib on Medium</title>
            <link>https://medium.com/@aqibnawab1100?source=rss-6ea1c739483------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 17:04:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aqibnawab1100/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Week 10: Cleaning up config and teaching articles to carry real metadata]]></title>
            <link>https://medium.com/@aqibnawab1100/week-10-cleaning-up-config-and-teaching-articles-to-carry-real-metadata-fe52e9a66569?source=rss-6ea1c739483------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe52e9a66569</guid>
            <category><![CDATA[b0bot]]></category>
            <category><![CDATA[google-summer-of-code]]></category>
            <category><![CDATA[gsoc]]></category>
            <category><![CDATA[c2si]]></category>
            <dc:creator><![CDATA[Aqib]]></dc:creator>
            <pubDate>Sat, 22 Aug 2026 14:47:56 GMT</pubDate>
            <atom:updated>2026-08-22T14:47:56.153Z</atom:updated>
            <content:encoded><![CDATA[<p>Last week was about trust — fixing a dead feed, putting unsubscribe in digest emails, and making logs readable when something went wrong. Week 10 was a different kind of work. The pipeline was running, but the data coming out of it was still quite plain. An article was basically a title, a URL, a snippet, and an embedding vector sitting in Postgres. That is fine for storage, but not very exciting for anyone opening a dashboard or trying to filter news by severity or topic.</p><p>So this week I did two things that sound separate but actually fit together nicely. First, I pulled scattered settings into proper config modules in ingestion-service and notification-service. Second, I started enriching every article at ingest time with CVE IDs, severity levels, affected systems, and topic tags. One makes the codebase easier to maintain. The other makes the product feel like it understands cybersecurity news rather than just collecting links.</p><p><strong>Why config modules mattered</strong></p><p>When you are building three Docker services with Vishak on api-service and yourself on ingestion and notification, small things pile up fast. Redis URL here, polling interval there, SMTP timeout copied in another file. It works until someone changes a default in one place and forgets the other. Then you spend an hour wondering why digest checks run every hour on your laptop but every ten minutes in Docker.</p><p>The fix was simple in idea, a bit tedious in practice. Each backend service got its own config.py that reads from environment variables and holds shared constants — queue names, database URLs, embedding model name, digest window sizes, SMTP settings, that sort of thing. Poller, worker, handler, db, email — they all import from one place now instead of defining their own defaults.</p><p>For ingestion-service this also meant putting the article.discovered event contract in one spot: event type string, schema version, idempotency key helper, known topic tag list. For notification-service it meant FRONTEND_URL for unsubscribe links, digest template path, delivery status constants. Nothing flashy, but when you are debugging at midnight during GSoC, knowing exactly where to look for a timeout value is a small mercy.</p><p>Along with that refactor we documented the three-service architecture properly in the repo — how api-service talks to Postgres and Redis, how ingestion pushes jobs and writes vectors, how notification reads subscribers and sends mail without importing anything from the other services. I will not draw it again here; the point is that new contributors should not have to reverse-engineer the layout from scattered files.</p><p><strong>The metadata problem</strong></p><p>Raw RSS gives you a title and maybe a short description. Sometimes the CVE number is right there in the headline. Sometimes severity is buried in CVSS score text. Sometimes the feed says nothing useful except “critical vulnerability” and you have to guess from context.</p><p>Vishak was building dashboard and news views on his side. If every row in articles looked the same, those pages would stay empty or generic. We needed structured fields in Postgres: cve_id, severity, affected_system, and topic_tags as a text array. The schema already had placeholders for these columns from earlier weeks. What was missing was the logic to fill them reliably when a new article lands.</p><p><strong>Regex first, LLM second — and never guess</strong></p><p>I split metadata extraction into two layers on purpose.</p><p>The first layer is pure Python in metadata.py. It strips HTML from snippets, hunts for CVE patterns like CVE-2024-1234, reads CVSS scores when they appear in text, maps those scores to CRITICAL, HIGH, MEDIUM, or LOW using standard cutoffs, and falls back to keyword phrases when no score is there. For affected system it checks against a curated list of common products and vendors — Windows, Apache, Chrome, Cisco, that kind of thing. Topic tags come from a fixed vocabulary: malware, ransomware, cve, data breach, vulnerability. If the title mentions ransomware, you get the ransomware tag. If a CVE is found or the word “exploit” shows up, vulnerability and cve tags can both apply.</p><p>This layer never calls an API. It is fast, deterministic, and easy to unit test. When CISA or Krebs publishes something with a clear CVE in the title, regex alone does the job.</p><p>The second layer is optional LLM enrichment in llm_metadata.py. If you set HUGGINGFACE_TOKEN in the environment, the handler asks a small model through HuggingFace inference — we used Cohere’s tiny-aya-global — to return strict JSON with severity and affected system. The prompt is very explicit: return null when unknown, do not guess, only allowed severity values. We parse the JSON carefully because LLMs love to wrap answers in extra text.</p><p>Here is the important part. If the LLM fails, times out, or returns rubbish, we do not crash the ingest job. We fall back to the regex layer. CVE ID always comes from regex because that format is too strict to leave to a model. Topic tags also stay on the keyword side so we do not get random tags the dashboard cannot filter on. The LLM is a helper for severity and affected system when the snippet is vague, not the source of truth for everything.</p><p><strong>Where it plugs into the pipeline</strong></p><p>Nothing changes about the overall flow from Week 9. Poller fetches RSS, deduplicates by url hash, enqueues article.discovered. Worker picks up the job, checks idempotency in processed_jobs, generates the MiniLM embedding, then calls enrich_metadata_with_llm before upserting into Postgres.</p><p>Those four metadata fields get written in the same transaction as the embedding. On duplicate url hash we merge sensibly — newer non-null severity wins, topic tags update only when the new set is non-empty. So re-polling the same article does not wipe good data with blanks.</p><p>On the api-service side, Vishak’s news model can now query by severity, filter articles that have a cve_id, and eventually surface topic tags on the dashboard. My job ended at the database row. His job was making it look good in the UI. That split kept us from stepping on each other’s PRs.</p><p><strong>How I tested it</strong></p><p>Unit tests were the backbone here. test_metadata.py covers CVE extraction, CVSS-to-severity mapping, ransomware and breach tag detection, LLM JSON parsing, and the fallback path when the model returns invalid severity. Integration tests in test_ingestion_service.py mock the LLM and assert the handler passes cve_id and severity through to the upsert.</p><p>Locally I ran ingestion with Docker, let a few feeds poll, then queried Postgres directly:</p><blockquote>SELECT title, cve_id, severity, affected_system, topic_tags</blockquote><blockquote>FROM articles</blockquote><blockquote>ORDER BY ingested_at DESC</blockquote><blockquote>LIMIT 10;</blockquote><p>Articles from advisory-heavy feeds showed CVE IDs and severity. General news pieces might only have topic tags or nothing at all — and that is correct. We are not fabricating CVE numbers for stories that do not have them.</p><p>If you want to see the LLM path, set HUGGINGFACE_TOKEN in your .env and watch the worker logs on the next job. Without a token, the whole service still runs on regex only, which is how CI and most dev setups behave.</p><p><strong>What this week taught me</strong></p><p>Config refactor weeks rarely make exciting demo videos, but they save you later. Metadata weeks do make demos better — suddenly the dashboard has red CRITICAL badges and CVE links instead of a flat list of headlines.</p><p>The design lesson I took away is to treat LLM enrichment as optional seasoning, not the main dish. Ingest must stay up even when HuggingFace is down or you forgot to export a token. Regex and keywords feel old-fashioned next to “AI-powered tagging,” but for CVE IDs and controlled vocabularies they are actually more trustworthy.</p><p>Next week I moved the RSS source list itself into Postgres so feeds could be managed without redeploying code. That is the Week 11 story — dynamic sources and a few chat-side improvements on Vishak’s end.</p><p><strong>Closing line you can use</strong></p><p>The config work landed in <a href="https://github.com/c2siorg/b0bot/pull/225">PR #225</a> and metadata at ingest in <a href="https://github.com/c2siorg/b0bot/pull/234">PR #234</a>. Full repo: <a href="https://github.com/c2siorg/b0bot">c2siorg/b0bot</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe52e9a66569" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Week 9: Digest emails, feed fixes, and make ingestion observable]]></title>
            <link>https://medium.com/@aqibnawab1100/week-9-digest-emails-feed-fixes-and-make-ingestion-observable-ed892fe47e6b?source=rss-6ea1c739483------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed892fe47e6b</guid>
            <category><![CDATA[gsoc]]></category>
            <category><![CDATA[google-summer-of-code]]></category>
            <category><![CDATA[c2si]]></category>
            <category><![CDATA[b0bot]]></category>
            <dc:creator><![CDATA[Aqib]]></dc:creator>
            <pubDate>Sat, 22 Aug 2026 14:44:31 GMT</pubDate>
            <atom:updated>2026-08-22T14:44:31.898Z</atom:updated>
            <content:encoded><![CDATA[<p>By Week 9, the three-service setup was mostly in place. Docker Compose came up, ingestion was pulling RSS and writing to Postgres, and notification-service was sending digests. On paper it looked done. In practice, a few things were still broken or hard to see. That is what this week was about — not big new features, but making the pipeline trustworthy enough that someone could actually subscribe, get an email, and unsubscribe without guessing what went wrong.</p><p><strong>The CISA feed problem</strong></p><p>One of the feeds we relied on was CISA advisories. It had been in the list from the start, but the URL we were using stopped working. The old link returned access denied, so we were not getting official US government advisories at all. For a cybersecurity news project that is a real gap.</p><p>The fix was straightforward once we found it: replace the dead URL with the current CISA XML feed. After that, the poller started picking up advisories again on the normal 15-minute cycle. Nothing fancy — just the kind of thing that breaks quietly in production if nobody is watching the logs.</p><p><strong>Unsubscribe in digest emails</strong></p><p>Notification-service was already finding active subscribers, pulling recent articles, and sending SMTP digests. What was missing was a proper way to opt out from the email itself.</p><p>We added an unsubscribe link in every digest. It points to /unsubscribe/&lt;subscriber_id&gt; on the api-service, which sets the subscriber’s status to unsubscribed in Postgres. The notification worker also records each send in digest_deliveries with an idempotency key so the same person does not get duplicate digests for the same window.</p><p>This matters because without it, email subscriptions feel one-way. Someone signs up, gets mail, and has no clean exit except ignoring it. That is bad for users and bad for a project we want mentors and evaluators to take seriously.</p><p>Vishak had been working on the subscribe and unsubscribe pages on the frontend around the same time. My part was making sure the backend and the email actually connected — the link in the email had to land on a route that really updates the database.</p><p><strong>Observability — knowing what ingestion is doing</strong></p><p>Before this week, when something failed during RSS polling or embedding, it was not always obvious from the outside. You would run docker compose up and hope articles appeared in Postgres.</p><p>We improved logging across ingestion and notification so the worker output is easier to follow. When a feed times out, when a job is skipped because of duplicate url_hash, when a digest goes out or fails — there is a log line for it. That sounds small, but during GSoC most of our debugging happened on a laptop with Docker, not with a fancy dashboard. Good logs saved a lot of time.</p><p><strong>How I tested it</strong></p><p>I ran the full stack with docker compose up, let ingestion poll for a while, and checked Postgres for new rows. I subscribed with a test email, waited for the digest worker to run, and confirmed the message arrived with the unsubscribe link. Clicking that link showed the unsubscribe page and changed the row in subscribers.</p><p>For the CISA feed I ran the ingestion service alone and watched the logs on the next poll cycle. Entries started showing up from that source again. Screenshots of the digest email and the log output went into the PR — mentor requirement, but also useful for this blog.</p><p><strong>What this week taught me</strong></p><p>Week 9 was less about adding a new service and more about closing gaps between “it works on my machine” and “it works as a product.” Broken RSS URLs, missing unsubscribe, and silent failures are the sort of issues that do not show up in an architecture diagram but show up immediately when someone tries to use the thing for real.</p><p>Next week the focus shifted to enriching articles at ingest time — CVE IDs, severity, topic tags — so the dashboard and filters had something meaningful to show. That story is for the Week 10 post.</p><p><strong>Closing line you can use</strong></p><p>If you are following this series, the repo is <a href="https://github.com/c2siorg/b0bot">c2siorg/b0bot</a>. The work from this week landed in <a href="https://github.com/c2siorg/b0bot/pull/231">PR #231</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed892fe47e6b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[B0Bot GSoC Week 7–8: Digest Emails, Config Cleanup, and Learning to Stay in My Lane]]></title>
            <link>https://medium.com/@aqibnawab1100/b0bot-gsoc-week-7-8-digest-emails-config-cleanup-and-learning-to-stay-in-my-lane-abce12f331d0?source=rss-6ea1c739483------2</link>
            <guid isPermaLink="false">https://medium.com/p/abce12f331d0</guid>
            <category><![CDATA[gsoc-2026]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[gsoc]]></category>
            <dc:creator><![CDATA[Aqib]]></dc:creator>
            <pubDate>Mon, 03 Aug 2026 16:47:28 GMT</pubDate>
            <atom:updated>2026-08-03T16:47:28.279Z</atom:updated>
            <content:encoded><![CDATA[<p>Weeks 5–6 got the pipeline running. Weeks 7–8 were about making it feel like a real product — nicer emails, cleaner code, and not stepping on Vishak’s toes.</p><p>Also hit midterm around this time, so a lot of focus went into making sure what we had actually worked reliably.</p><h3>What I worked on</h3><h3>1. Digest email redesign</h3><p>The first email template looked like a generic dark SaaS newsletter. Rebuilt it to match Khushal’s Figma design:</p><ul><li>Light cream background (#FBF4EA)</li><li>Orange accent (#FF6817)</li><li>Simple numbered article list — clean and readable</li><li>Links go through {FRONTEND_URL}/news?url=... so users land on our UI</li></ul><p>Tested with Gmail SMTP locally — digests actually arrived in my inbox.</p><h3>2. More tests + edge cases</h3><p>Added tests for:</p><ul><li>Empty digest (no new articles in window)</li><li>Idempotency — same subscriber + window doesn’t send twice</li><li>Missing article fields (no title, no source name)</li></ul><p>Small thing, but it saved us from silent failures later.</p><h3>3. Config refactor (PR #225)</h3><p>Env vars and constants were scattered across files. Pulled them into proper config.py modules in both services:</p><ul><li>ingestion-service/config.py — queue names, polling interval, embedding model</li><li>notification-service/config.py — SMTP settings, digest window, FRONTEND_URL</li></ul><p>Also updated the architecture diagram in the README so it matches what’s actually built now (not the old “notification-service is scaffold” version).</p><h3>4. Coordination with Vishak</h3><p>This was the messy but important part.</p><p>Vishak flagged that I was building subscribe/OTP stuff in api-service — but that’s his area (chat UI + subscriber flow via SubscriberModel). Hardik had already dropped OTP in community bonding.</p><p>So I reverted my OTP/subscribe work from api-service and kept my side focused on:</p><ul><li>ingestion-service</li><li>notification-service (digest sending only)</li><li>Postgres/pgvector</li></ul><p>Lesson: read the ownership table in the proposal before coding. Saves duplicate work.</p><h3>5. Unsubscribe link in digest emails</h3><p>Vishak finished /unsubscribe on the frontend. I wired the digest email footer to link there:</p><p>{FRONTEND_URL}/unsubscribe/{subscriber_id}</p><p>Simple change, but users can now opt out directly from the email.</p><h3>Challenges</h3><p>Who owns what?<br>Easy to overlap when two people work on the same repo. Fixed by checking open PRs before starting and aligning in group chat.</p><p>SMTP env vars lost on rebuild<br>After docker compose up --build, SMTP settings disappeared because they weren&#39;t in .env.example. Documented them properly.</p><p>CISA RSS feed broke<br>Old URL (us-cert.cisa.gov/mlist.xml) returns Access Denied. Swapped to cisa.gov/cybersecurity-advisories/all.xml. Found this while testing feeds live.</p><h3>PRs / issues from this period</h3><p>WhatLink</p><p>Config refactor + architecture diagram</p><h3>What’s next (Week 9+)</h3><ul><li>CVE/severity/tags extraction during ingestion (Vishak needs this for dashboard)</li><li>Interest-aware digest filtering</li><li>Sources backend (DB-driven RSS instead of hardcoded list)</li><li>CI/CD with GitHub Actions</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=abce12f331d0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[B0Bot GSoC Week 5–6: Finishing the Ingestion Pipeline + First Digest Emails]]></title>
            <link>https://medium.com/@aqibnawab1100/b0bot-gsoc-week-5-6-finishing-the-ingestion-pipeline-first-digest-emails-f14571135344?source=rss-6ea1c739483------2</link>
            <guid isPermaLink="false">https://medium.com/p/f14571135344</guid>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[gsoc-2026]]></category>
            <category><![CDATA[b0bot]]></category>
            <dc:creator><![CDATA[Aqib]]></dc:creator>
            <pubDate>Mon, 03 Aug 2026 16:45:30 GMT</pubDate>
            <atom:updated>2026-08-03T16:45:30.658Z</atom:updated>
            <content:encoded><![CDATA[<p>Weeks 3–4 got the queue and RSS basics working. Weeks 5–6 were about making the pipeline actually run end-to-end and not fall apart when something breaks.</p><p>Week 5 was mostly buffer time — fixing bugs, testing things together, and making sure Week 4 work was stable before building more.</p><p>Week 6 was the big push: finish ingestion-service and get notification-service sending real emails.</p><h3>What I worked on</h3><h3>1. Full ingestion pipeline</h3><p>The flow now looks like this:</p><p>RSS feeds → poller → article.discovered queue → worker → embeddings → Postgres/pgvector</p><p>RSS polling</p><ul><li>Polls curated cybersecurity feeds every 15 minutes</li><li>Dedupes articles using SHA-256 url_hash — no more delete-and-rebuild like the old Pinecone flow</li><li>Skips bad feeds and retries next cycle instead of crashing</li></ul><p>BullMQ worker</p><ul><li>Consumes article.discovered jobs from Redis</li><li>Checks processed_jobs first — same job never runs twice (idempotency)</li><li>Inserts into articles with embedding_status=pending</li></ul><p>Embeddings + pgvector</p><ul><li>Uses all-MiniLM-L6-v2 (384 dimensions)</li><li>Upserts vectors into Postgres with HNSW index</li><li>api-service can now query fresh articles from the DB</li></ul><h3>2. notification-service — first real digests</h3><p>Before this, notification-service just idled on startup. Now it actually works:</p><ul><li>Polls Postgres for active subscribers (daily / weekly)</li><li>Picks recent articles in the time window</li><li>Sends digest via SMTP</li><li>Logs every send in digest_deliveries so we don&#39;t spam the same person twice</li></ul><h3>3. Tests</h3><p>Added pytest for ingestion-service:</p><ul><li>Payload validation</li><li>Idempotency skips</li><li>Missing optional fields</li><li>Embedding text prep</li></ul><p>This caught a few edge cases early instead of finding them in production.</p><h3>Challenges</h3><p>Redis port conflicts locally<br>Docker Redis was clashing with a local Redis on 6379. Fixed by mapping host port to 6380 and updating REDIS_URL.</p><p>Embedding is slow on CPU<br>~100–200ms per article. Fine for GSoC scale, but batch inference is on the backlog.</p><p>Service boundaries<br>Had to keep reminding myself: ingestion and notification talk through Postgres and queues only — no cross-imports between services.</p><h3>How to test (what I ran)</h3><p>docker compose up ingestion-service notification-service</p><p><em># Check articles landed</em></p><pre><br>docker compose exec postgres psql -U b0bot -d b0bot \<br>-c &quot;SELECT title, embedding_status FROM articles LIMIT 5;&quot;<br># Check digest deliveries<br>docker compose exec postgres psql -U b0bot -d b0bot \<br>-c &quot;SELECT * FROM digest_deliveries;&quot;</pre><p>What’s next</p><ul><li>Better email template design</li><li>Config cleanup across services</li><li>Coordinate with Vishak on subscribe/unsubscribe (he owns the UI side)</li><li>CVE/severity extraction for the dashboard (Week 8+)</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f14571135344" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Week 3 & Week 4 Progress: Building the Queue and Embedding Backbone for Cybersecurity News…]]></title>
            <link>https://medium.com/@aqibnawab1100/week-3-week-4-progress-building-the-queue-and-embedding-backbone-for-cybersecurity-news-d71f97515de3?source=rss-6ea1c739483------2</link>
            <guid isPermaLink="false">https://medium.com/p/d71f97515de3</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[embedding]]></category>
            <category><![CDATA[gsoc]]></category>
            <category><![CDATA[redis]]></category>
            <dc:creator><![CDATA[Aqib]]></dc:creator>
            <pubDate>Mon, 29 Jun 2026 11:28:38 GMT</pubDate>
            <atom:updated>2026-06-29T11:28:38.388Z</atom:updated>
            <content:encoded><![CDATA[<h3>Week 3 &amp; Week 4 Progress: Building the Queue and Embedding Backbone for Cybersecurity News Intelligence</h3><p>Over the last two weeks, I focused on turning our ingestion flow from a basic fetch pipeline into a reliable, scalable data backbone.</p><p>Week 3 was about setting up the queue-based infrastructure.<br>Week 4 focused on adding RSS polling and embeddings, and validating the full pipeline end-to-end.</p><p>Together, these two weeks established the core ingestion architecture we’ll build on next.</p><h3>Why this phase mattered</h3><p>For a cybersecurity intelligence platform, two things are critical:</p><ol><li>New content must be collected reliably and continuously.</li><li>Content must be stored in a way that supports semantic search and downstream analysis.</li></ol><p>So instead of handling everything in one synchronous flow, I moved toward an event-driven architecture with clear separation of responsibilities.</p><h3>Week 3: Queue Infrastructure (Redis + BullMQ)</h3><h3>What I implemented</h3><ul><li>Set up Redis as the message broker for ingestion events.</li><li>Integrated BullMQ for reliable queue-based processing.</li><li>Defined event flow for article discovery and worker consumption.</li><li>Added idempotency handling to reduce duplicate processing.</li></ul><h3>Why this was important</h3><p>This decoupled the “collect” step from the “process” step:</p><ul><li>Polling can continue even if processing slows down.</li><li>Workers can scale independently.</li><li>Failures are easier to isolate and retry.</li></ul><h3>Outcome</h3><p>By the end of Week 3, queue-driven ingestion infrastructure was in place and ready to support the embedding pipeline.</p><h3>Week 4: RSS Polling + Embedding Pipeline</h3><h3>What I implemented</h3><ul><li>Added scheduled RSS polling across cybersecurity sources.</li><li>Implemented URL normalization and hash-based deduplication.</li><li>Enqueued discovered articles into BullMQ.</li><li>Built worker-side embedding generation using sentence-transformers/all-MiniLM-L6-v2.</li><li>Upserted article records + vectors into Postgres with pgvector support.</li><li>Added embedding status tracking (pending, indexed, failed).</li></ul><h3>Final flow</h3><p>RSS feeds → dedup/normalize → queue event → embedding worker → Postgres (pgvector)</p><h3>Debugging and reliability lessons</h3><p>One major issue was that behavior didn’t match logs.<br>Root cause: stale container/image state (old code running in Docker).</p><p>Fixing that required a clean rebuild/restart and then validating each layer:</p><ul><li>poller logs,</li><li>queue activity,</li><li>worker consumption,</li><li>final database writes.</li></ul><p>Big lesson: in containerized systems, correctness depends on both code and runtime state.</p><h3>Benchmarking (mentor-requested experiment)</h3><p>I ran benchmarks to compare embedding latency across token lengths.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*DGq_xMMYrEJGmdJQohUiLA.png" /><figcaption>benchmarking</figcaption></figure><h3>Interpretation</h3><ul><li>No single token length wins for every input shape.</li><li>For practical production defaults, keeping MAX_TOKEN_LENGTH = 512 is reasonable.</li><li>Embedding dimension remains fixed at 384 for this model.</li></ul><h3>What these two weeks achieved</h3><ul><li>Queue-based infrastructure is now operational.</li><li>RSS ingestion is active and repeatable.</li><li>Deduplication and idempotency protections are in place.</li><li>Embeddings are generated and indexed in Postgres.</li><li>The full end-to-end path is now validated.</li></ul><h3>Next steps</h3><ul><li>Improve observability (queue lag, processing times, failure rates).</li><li>Strengthen retry/error handling policies.</li><li>Expand tests for ingestion + embedding behavior.</li><li>Start improving retrieval quality on top of indexed vectors.</li></ul><h3>Closing note</h3><p>Week 3 built the backbone.<br>Week 4 made it intelligent.</p><p>With queueing and embeddings now working together, the system is in a strong position to improve relevance ranking, semantic search, and alerting in the coming weeks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d71f97515de3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GSoC 2026 Week 1–2: Laying the Foundation for B0Bot — Folder Structure, Docker, and PostgreSQL]]></title>
            <link>https://medium.com/@aqibnawab1100/gsoc-2026-week-1-2-laying-the-foundation-for-b0bot-folder-structure-docker-and-postgresql-41342081fae7?source=rss-6ea1c739483------2</link>
            <guid isPermaLink="false">https://medium.com/p/41342081fae7</guid>
            <dc:creator><![CDATA[Aqib]]></dc:creator>
            <pubDate>Tue, 09 Jun 2026 07:03:15 GMT</pubDate>
            <atom:updated>2026-06-09T07:03:15.416Z</atom:updated>
            <content:encoded><![CDATA[<p>This summer, I’m working on B0Bot — a cybersecurity news API built with Flask, LangGraph, and vector search — as part of Google Summer of Code 2026 under C2SI. I’m contributing alongside Vishak Baddur Sadanand, with mentorship from Hardik Jindal and Nipuna.</p><p>Weeks 1–2 (May 25 — June 7) were about one thing: foundation. Not flashy features. Not UI polish. Just the unglamorous work that everything else depends on — folder structure, Docker Compose, and PostgreSQL with pgvector.</p><p>Here’s how that week actually went — including the PR conflicts, the waiting, and what we finally shipped.</p><h3>What we walked into</h3><p>B0Bot already existed as a Flask monolith. It worked, but the combined proposal Vishak and I wrote identified serious problems:</p><ul><li>Pinecone as the vector store — external dependency, slow, and the legacy Update.py script deleted the entire index before every rebuild. A crash mid-run meant losing everything.</li><li>No service separation — ingestion and API serving lived in one codebase. You couldn’t scale them independently.</li><li>Hardcoded scrapers — CSS selectors in JSON files that broke silently when news sites changed their HTML.</li><li>No conversation history, no subscriptions, no agents — every request was stateless and linear.</li></ul><p>Our agreed architecture for GSoC: three services (api-service, ingestion-service, notification-service), PostgreSQL + pgvector replacing Pinecone, Redis for caching and queues, and Docker Compose so the whole stack runs locally with one command. No cloud dependencies.</p><p>Vishak owns the LangGraph agent framework. I own the data layer — Postgres, Docker, ingestion-service, notification-service, and BullMQ.</p><h3>The PR conflict problem (and why we waited)</h3><p>Coding period started, and both of us had work to do. Vishak was already ahead on his side — Flask skeleton, LangGraph agents wired up, /chat and /subscribe route stubs. My side was PostgreSQL schema, pgvector setup, and Docker Compose.</p><p>But we hit a wall almost immediately: we couldn’t both open PRs at the same time without creating chaos.</p><p>The legacy codebase was still a flat monolith at the repo root — app.py, services/, models/, all mixed together. Vishak had LangGraph work ready to land. I had a Docker stack ready to land. If we both pushed without agreeing on structure first, we’d get merge conflicts across half the repo and our mentor Hardik would be merging PRs that fought each other.</p><p>So we stopped and coordinated.</p><p>In our contributor group chat, Vishak asked the right question:</p><blockquote>“Where are you on Postgres and Docker? We need to post Week 2 goals tomorrow.”</blockquote><p>I shared my draft schema — five tables (articles, subscribers, subscriber_interests, digest_deliveries, processed_jobs), pgvector HNSW index on 384-dimensional embeddings, and a BullMQ article.discovered payload contract.</p><p>Then came the folder structure debate.</p><p>I initially suggested nesting everything under a services/ folder:</p><p>app/</p><p>services/</p><p>ingestion-service/</p><p>frontend/</p><p>server/</p><p>docker-compose.yaml</p><p>Vishak pushed back — politely, but firmly. His proposal matched the GSoC architecture doc:</p><p>b0bot/</p><p>api-service/</p><p>ingestion-service/</p><p>notification-service/</p><p>docker-compose.yml</p><p>His reasoning: Docker Compose works cleaner with flat service directories at the repo root. It’s the standard layout for multi-service projects, and it gives each service a clear boundary.</p><p>I agreed. We also settled on local Docker only — no cloud Postgres, no managed Redis. Everything runs on your machine.</p><p>The sequencing we agreed on:</p><ol><li>Me first — folder scaffold + docker-compose.yml + Postgres init SQL</li><li>Vishak second — move his LangGraph work into api-service/</li><li>Me third — wire models to PostgreSQL, replace Pinecone</li></ol><p>That waiting felt slow in the moment. In hindsight, it saved us days of merge conflict resolution.</p><h3>What I built in Week 1–2</h3><p>Once we had alignment, I raised PR #1 — the infrastructure foundation.</p><h3>1. Three-service folder scaffold</h3><p>b0bot/</p><p>├── api-service/ # Vishak’s territory (Flask + LangGraph)</p><p>├── ingestion-service/ # My territory (RSS + BullMQ + embeddings)</p><p>├── notification-service/ # My territory (digests + SMTP)</p><p>└── docker-compose.yml</p><p>Each service got a minimal README. No over-documentation — just enough so a new contributor knows who owns what.</p><h3>2. Docker Compose stack</h3><p>One command to start everything:</p><p>docker compose up — build -d</p><p>The stack brings up:</p><p>ServiceRole</p><p>postgres</p><p>PostgreSQL 16 + pgvector extension</p><p>redis</p><p>Queues, cache, session memory</p><p>api-service</p><p>Flask API (gunicorn)</p><p>ingestion-service</p><p>Worker scaffold (idles until Week 6+)</p><p>notification-service</p><p>Worker scaffold (idles until Week 8+)</p><p>Services wait for Postgres and Redis healthchecks before starting — no race conditions on startup.</p><h3>3. PostgreSQL schema (docker/postgres/init.sql)</h3><p>This was the core of my Week 1–2 deliverable. Five tables:</p><ul><li>articles — structured news data + vector(384) column with HNSW cosine index. Dedup via SHA-256 url_hash — no more delete-and-rebuild.</li><li>subscribers — email, OTP verification, digest frequency</li><li>subscriber_interests — interest tags (Malware, Ransomware, CVE, etc.)</li><li>digest_deliveries — delivery log with idempotency keys</li><li>processed_jobs — BullMQ idempotency markers</li></ul><p>I also seeded two sample articles so the API has data to render before the ingestion pipeline exists.</p><h3>4. Pinecone → Postgres migration</h3><p>config/Database.py became a psycopg connection helper over DATABASE_URL. models/NewsModel.py reads from the articles table and returns the same dict shape the controllers and LangGraph scraper agent already expect — so Vishak’s code didn’t need changes downstream.</p><h3>Things that broke (and how we fixed them)</h3><p>No infrastructure PR is complete without battle stories.</p><h3>Docker build pulling 1.5 GB of PyTorch</h3><p>First docker compose up --build failed mid-download with a BrokenPipeError. The culprit: sentence-transformers in api-service/requirements.txt, which pulled in torch, NVIDIA CUDA libraries, and triton — over 1.5 GB of dependencies.</p><p>But sentence-transformers was only used by the legacy db_update/Update.py Pinecone script. The API runtime doesn’t need it. Embeddings belong in ingestion-service, not api-service.</p><p>Fix: Removed it from api-service requirements. Build went from ~20 minutes of downloading CUDA packages to under 4 minutes.</p><h3>Port conflicts with another project</h3><p>Postgres defaulted to port 5432. Redis to 6379. Both were already taken on my machine by another Docker project (CodeArena). Containers failed with port is already allocated.</p><p>Fix: Defaulted B0Bot to 5433 (Postgres) and 6380 (Redis) on the host. Inside the Docker network, services still talk on standard ports — only the host mapping changes.</p><h3>Stale Postgres volumes skipping seed data</h3><p>After fixing ports, /health returned {&quot;status&quot;:&quot;ok&quot;} but articles were empty. The Postgres volume from an earlier failed attempt had the schema but not the seed data — init.sql only runs on first container start.</p><p>Fix: docker compose down -v to wipe volumes, then docker compose up -d fresh. Two seed articles appeared. /raw/news rendered them in the browser.</p><h3>How we tested it</h3><p>Our mentor Hardik emphasized PR hygiene after merging Vishak’s earlier PR — elaborate descriptions, screenshots, README updates. We took that seriously.</p><p>Verification checklist:</p><p>docker compose ps <em># all 5 services up</em></p><p>curl <a href="http://localhost:5000/health">http://localhost:5000/health</a> <em># {“status”:”ok”}</em></p><p>curl <a href="http://localhost:5000/raw/news">http://localhost:5000/raw/news</a> <em># 2 seeded articles</em></p><p>docker compose exec postgres psql -U b0bot -d b0bot -c “\dt” <em># 5 tables</em></p><p>docker compose exec postgres psql -U b0bot -d b0bot -c “SELECT title FROM articles;”</p><p>Screenshots we attached to the PR:</p><ol><li>docker compose ps — all services healthy</li><li>/health curl response</li><li>/raw/news in the browser showing seed articles</li><li>Postgres \dt + seed query output</li></ol><h3>What Week 1–2 taught me</h3><p>1. Coordination is infrastructure work too.<br>The folder structure conversation with Vishak took an evening. It felt like delay. It was actually the most important decision of the week. Two contributors on one repo without agreed boundaries will always produce merge conflicts.</p><p>2. Foundation PRs are unglamorous but load-bearing.<br>Nobody tweets about docker/postgres/init.sql. But without it, Vishak can’t wire ScraperAgent to pgvector, ingestion-service has nowhere to upsert, and notification-service has no subscriber table. Everything in Weeks 3–12 builds on this.</p><p>3. Docker teaches you things about your own dependencies.<br>I didn’t know sentence-transformers was in api-service requirements until the build tried to download half a gigabyte of CUDA libraries. Trimming dependencies to what each service actually needs at runtime is a design decision, not just a build optimization.</p><p>4. Port conflicts are real on developer machines.<br>Default ports are defaults for a reason — everyone uses them. Documenting alternate ports in .env.example saves the next contributor 30 minutes of confusion.</p><h3>Closing thought</h3><p>GSoC Week 1–2 isn’t about shipping features users can see. It’s about making features <em>possible</em>. The folder structure, the Docker stack, the Postgres schema with pgvector — that’s the plumbing. Vishak’s agents and my ingestion pipeline are the rooms. You can’t furnish rooms without plumbing.</p><p>If you’re contributing to an open-source project with another developer, my advice: agree on structure before you agree on code. Wait if you have to. One evening of coordination saves a week of conflict resolution.</p><p>More updates coming as we hit the midterm milestone.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41342081fae7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[C2SI (Google Summer of Code 2026): Choosing B0Bot’s Architecture — Three Services, BullMQ, and…]]></title>
            <link>https://medium.com/@aqibnawab1100/c2si-google-summer-of-code-2026-choosing-b0bots-architecture-three-services-bullmq-and-121180dd5f6a?source=rss-6ea1c739483------2</link>
            <guid isPermaLink="false">https://medium.com/p/121180dd5f6a</guid>
            <category><![CDATA[gsoc]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[langgraph]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[postgres]]></category>
            <dc:creator><![CDATA[Aqib]]></dc:creator>
            <pubDate>Sun, 31 May 2026 05:02:51 GMT</pubDate>
            <atom:updated>2026-05-31T05:02:51.235Z</atom:updated>
            <content:encoded><![CDATA[<h3>C2SI (Google Summer of Code 2026): Choosing B0Bot’s Architecture — Three Services, BullMQ, and pgvector</h3><p>Getting selected for GSoC with C2SI still feels unreal — especially after finishing my degree the same week coding started. Vishak and I were both selected for B0Bot, which meant we didn’t just start coding. We had to merge two different visions into one proposal Hardik could approve, without over-promising in a 12-week window.</p><p>Week 1 for me wasn’t mostly about commits. It was about making hard engineering trade-offs explicit — with pros, cons, and a clear reason for every “yes” and every “not this summer.”</p><p>This post is about those decisions.</p><h3>Why We Had to Decide Before We Coded</h3><p>B0Bot today is a Flask app that talks to Pinecone, scrapes news with brittle CSS selectors, and runs a single LangChain call per request. It works as a demo. It doesn’t work as a platform.</p><p>When Vishak and I compared proposals, we agreed on the problems quickly:</p><ul><li>Manual Update.py runs that wipe and rebuild the entire Pinecone index</li><li>A scraper that breaks silently when a site changes HTML</li><li>No conversation memory, no subscriptions, no automated ingestion</li><li>A toJSON() parser that splits LLM strings by comma — fragile by design</li><li>Tests that don’t even hit the real routes</li></ul><p>What we disagreed on was how much to rebuild this summer. That’s where the pros/cons work mattered.</p><h3>Decision 1: Three Services, Not a Monolith — and Not Eight Microservices</h3><h3>Option A — Keep the monolith</h3><p>Pros: Fastest to ship small features; one deploy; no cross-service contracts.<br>Cons: Ingestion and API serving stay coupled. A heavy embedding run can slow user requests. Hard to scale or reason about failures.</p><h3>Option B — Full microservices (8+ services)</h3><p>Pros: Clean boundaries; independent scaling; matches how I’ve worked in production with K8s.<br>Cons: Too much coordination for 12 weeks — queue schemas, deployment, observability, and integration testing across many repos/services.</p><h3>What we chose — Three services</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/1*mFqjWTLhiAMftqGahPj2vw.png" /><figcaption>Service Responsibility</figcaption></figure><p>Pros: Separates the workloads that actually need isolation (ingestion vs API vs email), without Kafka, K8s, or eight Dockerfiles on day one.<br>Cons: Still requires queue contracts and shared DB schema — more upfront design than a monolith.</p><p>Why it stuck: We get fault isolation and independent scaling where it matters, with one docker-compose up for GSoC. Service boundaries stay compatible with a future K8s split.</p><h3>Decision 2: BullMQ on Redis, Not Kafka</h3><p>Hardik had earlier discussions in #b0bot mentioning Kafka for workflows. When we moved to BullMQ, the natural question was: <em>why switch?</em></p><h3>Option A — Kafka</h3><p>Pros: Industry standard for high-throughput event streaming; great at replay and multi-consumer fan-out; scales to very large ingestion volume.<br>Cons: Extra broker infrastructure (Kafka + KRaft/ZooKeeper); heavier ops for a student project; overkill for RSS-scale volume.</p><h3>Option B — BullMQ on Redis</h3><p>Pros: Redis is already required for chat session memory and LLM response caching. BullMQ adds job queues with retries, dead-letter queues, and scheduling — zero new infrastructure.<br>Cons: Not ideal if B0Bot later ingests millions of events per day from dozens of social APIs; Redis is not a long-term event log like Kafka.</p><h3>What we chose — BullMQ</h3><p>Example job flow:</p><p>RSS poll → article.discovered (queue) → embed worker → pgvector upsert</p><p>digest scheduler → digest.send (queue) → notification-service → SMTP</p><p>Why it stuck: B0Bot’s volume is RSS feeds and email digests, not a real-time telemetry pipeline. BullMQ gives us at-least-once delivery and retries with the stack we already planned to run.</p><h3>Decision 3: PostgreSQL + pgvector, Not Pinecone</h3><p>This was the decision I felt strongest about — backed by benchmarks, not preference.</p><h3>Option A — Keep Pinecone</h3><p>Pros: Managed vector search; quick to prototype; no DB admin.<br>Cons: External API key required; can’t run fully local; structured user/subscription data lives elsewhere; latency and cost at scale.</p><h3>Option B — PostgreSQL + pgvector</h3><p>Pros: One database for articles, users, subscriptions, and vectors; fully local in Docker; hybrid search (vector + metadata filters) in a single query.<br>Cons: We own index tuning, migrations, and ops — not a managed SaaS.</p><h3>Benchmark snapshot (1,000 vectors, local-dev run)</h3><p>BackendHybrid search P95</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*3vZS5UXHqbHL-istus4__g.png" /><figcaption>Benchmark reports</figcaption></figure><p>Ingestion throughput favored local Mongo in that same run, but among options that fit our structured + vector requirement, pgvector won on hybrid query latency — the path B0Bot uses when filtering by severity, source, or CVE tags.</p><p>Why it stuck: B0Bot needs metadata-filtered retrieval for the ScraperAgent, not pure nearest-neighbor search. Co-locating relational and vector data removes an entire external dependency from the dev loop.</p><p><em>(We’ll re-run benchmarks at 10k+ vectors with real embeddings before calling it production-grade.)</em></p><h3>Decision 4: Flask, Not FastAPI — Pragmatism Over Preference</h3><p>I originally leaned FastAPI for async and OpenAPI. Vishak pushed back correctly.</p><h3>FastAPI</h3><p>Pros: Modern async stack; auto-generated OpenAPI; nice for greenfield APIs.<br>Cons: Migration work that doesn’t map to any GSoC spec requirement; existing routes, templates, and mentor familiarity are Flask.</p><h3>Flask</h3><p>Pros: Listed as a GSoC prerequisite; entire upstream codebase is Flask; Jinja2 UI (/chat, /subscribe, dashboard) fits naturally.<br>Cons: Less ergonomic for async-heavy workloads — acceptable because heavy work moves to BullMQ workers anyway.</p><p>What we agreed: Flask in api-service only. Ingestion and notification are plain Python workers. Service separation gives us concurrency where it counts, without a framework migration tax.</p><h3>Decision 5: RSS Ingestion Instead of CSS Scraping</h3><p>The current cybernews/extractor.py depends on news_types.json selectors. One HTML redesign and ingestion silently stops.</p><h3>RSS pipeline (in scope)</h3><p>Pros: Stable URLs; standard fields; adding a source = DB insert, not code deploy.<br>Cons: Not every site exposes full content in RSS; social connectors (Reddit, Mastodon, YouTube) deferred to Phase 2.</p><p>Why it stuck: Reliability beats coverage for a GSoC MVP. Three solid RSS sources plus custom URLs cover the core use case.</p><h3>Closing Thought</h3><p>The hardest part of Week 1 wasn’t picking technologies — it was ** agreeing on what not to build**. Three services instead of eight. BullMQ instead of Kafka. Flask instead of FastAPI. RSS instead of every social API. Subscriptions without a full dashboard.</p><p>Every “no” bought us a “yes” we can actually finish by August.</p><p>If you’re following along, Vishak’s Week 1 post covers the LangGraph agent pipeline from the AI side. I’ll suggest you to check that too</p><p>Muhammad Aqib · GSoC 2026 · C2SI · B0Bot<br>GitHub: <a href="https://github.com/AQIB-NAWAB">github.com/AQIB-NAWAB</a><br>Co-contributor: <a href="https://github.com/VishakBaddur">Vishak Baddur</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=121180dd5f6a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>