<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Niraj Adhikary on Medium]]></title>
        <description><![CDATA[Stories by Niraj Adhikary on Medium]]></description>
        <link>https://medium.com/@nirajmind?source=rss-9a422ea56694------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*uxIlIRFgQE1HiEBL0FlAgw@2x.jpeg</url>
            <title>Stories by Niraj Adhikary on Medium</title>
            <link>https://medium.com/@nirajmind?source=rss-9a422ea56694------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 08:37:28 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nirajmind/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Cheapest Way to Load USDC into MetaMask (UAE Edition)]]></title>
            <link>https://medium.com/@nirajmind/the-cheapest-way-to-load-usdc-into-metamask-uae-edition-1ba729aafbf8?source=rss-9a422ea56694------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ba729aafbf8</guid>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[binance]]></category>
            <category><![CDATA[cryptocurrency-investment]]></category>
            <category><![CDATA[metamask-wallet]]></category>
            <category><![CDATA[abu-dhabi-commercial-bank]]></category>
            <dc:creator><![CDATA[Niraj Adhikary]]></dc:creator>
            <pubDate>Mon, 13 Apr 2026 06:09:54 GMT</pubDate>
            <atom:updated>2026-04-13T06:09:54.012Z</atom:updated>
            <content:encoded><![CDATA[<h3>By Niraj</h3><p><strong>Introduction</strong></p><p>Anyone trying to load USDC into a self‑custody wallet like MetaMask from the UAE quickly discovers that there are multiple routes — each with different costs, risks, and levels of convenience.</p><p>I wanted to understand these differences clearly, so I ran three real‑world tests:</p><p>1. Onramp → MetaMask<br>2. Binance Direct USDC Purchase (Bank Transfer)<br>3. Binance P2P → USDT → Convert → USDC → MetaMask</p><p>To keep things simple for readers, I’m using rounded, “magical” numbers in this blog. <br>The real results were very close to these values.</p><p><strong>The Three Routes Compared</strong></p><ol><li>Onramp → MetaMask</li></ol><p>This is the simplest route. <br>You pay with your local bank card, and USDC lands directly in MetaMask.</p><ul><li>Convenience: ⭐⭐⭐⭐⭐ <br>- Speed: Fast <br>- Fees: Platform fee + network fee + payment gateway fee <br>- Effective rate: ~3.82 AED per USDC <br>- Best for: Beginners who want a one‑click experience</li></ul><p>2. Binance Direct USDC Purchase (Bank Transfer)</p><p>This is Binance’s built‑in “Buy Crypto” option using AED.</p><ul><li>Convenience: ⭐⭐⭐⭐ <br>- Speed: Fast <br>- Fees: Hidden inside the FX rate <br>- Effective rate: ~3.78 AED per USDC <br>- Best for: Users who want simplicity but lower cost than Onramp</li></ul><p>3. Binance P2P → USDT → Convert → USDC → MetaMask</p><p>This is the most hands‑on route, but also the cheapest.</p><p>- Convenience: ⭐⭐⭐ <br>- Speed: Fast <br>- Fees: Almost zero <br>- Effective rate: ~3.70 AED per USDC <br>- Best for: Cost‑conscious users and traders</p><p>📊 <strong>Effective Rate Comparison (Line Chart)</strong></p><p>The following chart visualizes the effective AED → USDC rate for each route:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MAXQgOvccKG_UQOhp-sf_w.png" /><figcaption>Effective AED → USDC Rate Comparison</figcaption></figure><p>This makes the cost difference obvious:</p><ul><li>Onramp is the most expensive <br>- Binance Direct USDC is slightly cheaper <br>- Binance P2P flow is the cheapest</li></ul><p><strong>Why the P2P Route Wins</strong></p><p>The P2P marketplace is competitive. <br>Sellers compete on price, which naturally pushes the rate down.</p><p>The steps look like this:</p><p>1. Buy USDT via Binance P2P <br>2. Convert USDT → USDC (almost zero cost) <br>3. Withdraw USDC → MetaMask on Polygon (tiny network fee)</p><p>This route requires a bit more attention, but the savings are real.</p><p>📐 <strong>Cost Calculation Formula</strong></p><p>To help readers compute their own effective rate, here’s the formula I used:</p><pre>Effective Rate (AED → USDC)<br><br>Effective Rate = Total AED Spent/Final USDC Received in MetaMask</pre><p>Where:</p><p>- <strong>Total AED Spent</strong> = amount paid in AED <br>- <strong>Final USDC Received</strong> = USDC after conversion + withdrawal fee deduction</p><p><strong>Example (using magical numbers)</strong>:</p><ul><li>AED spent: 5,600 AED <br>- USDC received in MetaMask: 1,514 USDC</li></ul><pre><br>Effective Rate = 5600/1514 approx 3.70 AED per USDC<br></pre><p>Readers can plug in their own numbers to compare routes.</p><p><strong>Final Recommendation</strong></p><p>If convenience is the priority → **Onramp** <br>If simplicity with lower cost is the priority → **Binance Direct USDC** <br>If lowest cost is the priority → **Binance P2P → USDT → USDC → MetaMask**</p><p>The P2P route consistently delivers the best rate, even after accounting for conversion and withdrawal fees.</p><p><strong>Closing Thoughts</strong></p><p>This comparison helped me understand the real cost of loading USDC into MetaMask from the UAE. <br>The differences may look small on paper, but they add up quickly — especially for traders or anyone moving larger amounts.</p><p>I hope this breakdown helps others choose the route that fits their needs, budget, and comfort level.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ba729aafbf8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ When Gemini Blocked My Singapore VM: A Real‑World Journey Through K3s, Cloud Run, Proxies & AI…]]></title>
            <link>https://medium.com/@nirajmind/when-gemini-blocked-my-singapore-vm-a-real-world-journey-through-k3s-cloud-run-proxies-ai-f95846b93566?source=rss-9a422ea56694------2</link>
            <guid isPermaLink="false">https://medium.com/p/f95846b93566</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[google-gemini]]></category>
            <dc:creator><![CDATA[Niraj Adhikary]]></dc:creator>
            <pubDate>Wed, 28 Jan 2026 18:43:07 GMT</pubDate>
            <atom:updated>2026-01-28T18:43:07.809Z</atom:updated>
            <content:encoded><![CDATA[<h3>🔥 When Gemini Blocked My Singapore VM: A Real‑World Journey Through K3s, Cloud Run, Proxies &amp; AI Architecture</h3><p>### *A deep dive into debugging, geolocation failures, Kubernetes pitfalls, and the architecture that finally worked.*</p><p>— -</p><p>## **Introduction**</p><p>Some engineering stories are neat and clean. <br>This is not one of them.</p><p>This is the story of how a **Singapore VM**, a **K3s cluster**, a **Cloud Run proxy**, and **Google Gemini’s geofencing rules** forced me into a last‑minute architectural redesign — and how I eventually built a stable, production‑ready AI pipeline.</p><p>If you’re building AI systems in the real world, this case study will save you days of pain.</p><p>— -</p><p># **1. The Setup: K3s on a Singapore VM + Gemini API**</p><p>CareerPilot’s backend runs on:</p><p>- A **K3s cluster** <br>- Hosted on a **Singapore VM** (Atlantic.net) <br>- Public IP: **43.228.212.212** <br>- All outbound traffic originates from this IP <br>- No CI/CD pipeline <br>- No container registry <br>- Images built locally and manually loaded into containerd</p><p>Everything worked locally. <br>Everything broke in production.</p><p>— -</p><p># **2. The First Major Failure: Gemini Rejects All Calls from My VM**</p><p>### **Symptom**<br>Every Gemini API call from the VM failed. <br>Same code worked perfectly from my laptop.</p><p>### **Deep RCA: The IP Was Blocked**</p><p>When I analyzed the VM’s IP:</p><p>```<br>43.228.212.212<br>```</p><p>I discovered:</p><p>| Attribute | Value |<br>| — — — — — | — — — — |<br>| Country | Singapore |<br>| Provider | Atlantic.net |<br>| ASN | AS25820 |<br>| Category | Datacenter / VPS |<br>| Risk Flags | High |<br>| Gemini Behavior | Blocked / silently dropped |</p><p>Gemini applies **geolocation + ASN risk scoring**:</p><p>- Consumer ISP → allowed <br>- Datacenter IP → often blocked <br>- Certain regions → restricted <br>- Unverified ASNs → blocked</p><p>My VM hit **all three**.</p><p>### **Result**<br>Gemini returned:</p><p>- Empty bodies <br>- 404s <br>- Timeouts <br>- Silent failures</p><p>### **Fix**<br>Introduce a **Cloud Run proxy** in a supported region:</p><p>```<br>Singapore VM → Cloud Run Proxy (us-central1) → Gemini → Proxy → VM<br>```</p><p>### **Diagram**</p><p>```<br>+ — — — — — — — — — — — — — — +<br>| K3s Cluster (Singapore) |<br>| Outbound IP: 43.228.212.212|<br>+ — — — — — — -+ — — — — — — — +<br> |<br> | (Blocked by Gemini)<br> v<br> [ ❌ Gemini API ]<br>```</p><p>After adding the proxy:</p><p>```<br>+ — — — — — — — — — — — — — — +<br>| K3s Cluster (Singapore) |<br>+ — — — — — — -+ — — — — — — — +<br> |<br> | HTTPS (x-careerpilot)<br> v<br>+ — — — — — — — — — — — — — — +<br>| Cloud Run Proxy (US) |<br>+ — — — — — — -+ — — — — — — — +<br> |<br> | Google internal network<br> v<br>+ — — — — — — — — — — — — — — +<br>| Gemini API (v1/v1beta) |<br>+ — — — — — — — — — — — — — — +<br>```</p><p>— -</p><p># **3. The Second Failure: K3s Running Stale Images**</p><p>Because we had:</p><p>- No registry <br>- No CI/CD <br>- Local images without tags</p><p>K3s often ran **old images** even after rebuilds.</p><p>### **Fix**<br>Tag images explicitly:</p><p>```bash<br>docker build -t careerpilot-api:local .<br>docker save careerpilot-api:local &gt; api.tar<br>ctr -n k8s.io images import api.tar<br>```</p><p>Update deployment:</p><p>```yaml<br>image: careerpilot-api:local<br>```</p><p>Verify:</p><p>```bash<br>kubectl describe pod api-xxxx | grep Image<br>```</p><p>— -</p><p># **4. The Third Failure: Cloud Run Proxy Failing Health Checks**</p><p>Cloud Run kept throwing:</p><p>```<br>Container failed to start and listen on port 8080<br>```</p><p>### **Root Cause**<br>Cloud Run health check hits `/`.</p><p>Our proxy forwarded `/` to Gemini → Gemini returned empty body → <br>Proxy tried `resp.json()` → crashed → Cloud Run marked unhealthy.</p><p>### **Fix**<br>Add a health endpoint:</p><p>```python<br><a href="http://twitter.com/app">@app</a>.get(“/”)<br>async def health():<br> return {“status”: “ok”}<br>```</p><p>Stop parsing JSON:</p><p>```python<br>return Response(<br> content=resp.content,<br> status_code=resp.status_code,<br> media_type=resp.headers.get(“Content-Type”, “application/json”)<br>)<br>```</p><p>— -</p><p># **5. The Fourth Failure: Gemini API Versioning Hell (v1 vs v1beta)**</p><p>Gemini’s model catalog is split:</p><p>| Model | API Version |<br>| — — — -| — — — — — — -|<br>| text-embedding-004 | v1 |<br>| gemini-1.5-pro | v1 |<br>| gemini-3-pro-preview | **v1beta only** |<br>| vision models | v1beta |</p><p>### **Symptoms**<br>Embedding failed:</p><p>```<br>models/text-embedding-004 is not found for API version v1beta<br>```</p><p>Chat failed:</p><p>```<br>models/gemini-3-pro-preview is not found for API version v1<br>```</p><p>### **Fix**<br>Use correct version per model:</p><p>```<br>v1/models/text-embedding-004:embedContent<br>v1beta/models/gemini-3-pro-preview:generateContent<br>```</p><p>— -</p><p># **6. The Fifth Failure: Double “models/models” Prefix**</p><p>We accidentally produced:</p><p>```<br>models/models/text-embedding-004:embedContent<br>```</p><p>### **Fix**<br>Standardize model names:</p><p>```python<br>embedding_model = “text-embedding-004”<br>chat_model = “gemini-3-pro-preview”<br>```</p><p>Build full path in client:</p><p>```python<br>path = “v1/models/text-embedding-004:embedContent”<br>```</p><p>— -</p><p># **7. The Sixth Failure: JSONDecodeError Everywhere**</p><p>Every time Gemini returned an empty body, the client crashed:</p><p>```<br>JSONDecodeError: Expecting value<br>```</p><p>### **Fix**<br>Once:</p><p>- API version was correct <br>- Model path was correct <br>- Proxy forwarded raw bytes</p><p>Gemini returned valid JSON and the errors disappeared.</p><p>— -</p><p># **8. The Final Architecture: Static Proxy, Smart Client**</p><p>Originally, the proxy hard‑coded:</p><p>```<br>v1<br>```</p><p>or:</p><p>```<br>v1beta<br>```</p><p>This meant **every model change required redeploying Cloud Run**.</p><p>### **Final Fix**<br>Make proxy version‑agnostic:</p><p>```python<br>GEMINI_BASE = “<a href="https://generativelanguage.googleapis.com/">https://generativelanguage.googleapis.com</a>&quot;<br>url = f”{GEMINI_BASE}/{path}”<br>```</p><p>Client sends full path:</p><p>```python<br>v1/models/text-embedding-004:embedContent<br>v1beta/models/gemini-3-pro-preview:generateContent<br>```</p><p>### **Final Architecture Diagram**</p><p>```<br>+ — — — — — — — — — — — — -+<br>| CareerPilot API (k3s) |<br>| Singapore VM |<br>+ — — — — — -+ — — — — — — -+<br> |<br> | HTTPS (x-careerpilot)<br> v<br>+ — — — — — — — — — — — — — — -+<br>| Cloud Run Proxy (us-central1) |<br>| — Validates secret |<br>| — Forwards raw payload |<br>| — No version logic |<br>+ — — — — — — -+ — — — — — — — -+<br> |<br> | HTTPS (Google internal)<br> v<br>+ — — — — — — — — — — — — — — -+<br>| Gemini API (v1 / v1beta) |<br>+ — — — — — — — — — — — — — — -+<br>```</p><p>— -</p><p># **9. Final Proxy Code (Production‑Ready)**</p><p>```python<br>from fastapi import FastAPI, Request, HTTPException<br>from fastapi.responses import Response<br>import httpx<br>import os</p><p>app = FastAPI()</p><p>GEMINI_API_KEY = os.environ[“GEMINI_API_KEY”]<br>PROXY_SECRET = os.environ[“PROXY_SECRET”]<br>GEMINI_BASE = “<a href="https://generativelanguage.googleapis.com/">https://generativelanguage.googleapis.com</a>&quot;</p><p><a href="http://twitter.com/app">@app</a>.get(“/”)<br>async def health():<br> return {“status”: “ok”}</p><p><a href="http://twitter.com/app">@app</a>.post(“/{path:path}”)<br>async def proxy(path: str, request: Request):<br> if request.headers.get(“x-careerpilot”) != PROXY_SECRET:<br> raise HTTPException(status_code=401)</p><p>url = f”{GEMINI_BASE}/{path}”<br> payload = await request.body()</p><p>async with httpx.AsyncClient(timeout=60) as client:<br> resp = await client.post(url, content=payload, headers={<br> “Content-Type”: “application/json”,<br> “x-goog-api-key”: GEMINI_API_KEY,<br> })</p><p>return Response(<br> content=resp.content,<br> status_code=resp.status_code,<br> media_type=resp.headers.get(“Content-Type”, “application/json”)<br> )<br>```</p><p>— -</p><p># **10. Kubernetes Deployment Snippet**</p><p>```yaml<br>containers:<br> — name: api<br> image: careerpilot-api:local<br> env:<br> — name: GEMINI_PROXY_URL<br> value: “<a href="https://gemini-proxy-xxxx.a.run.app/">https://gemini-proxy-xxxx.a.run.app</a>&quot;<br> — name: PROXY_SECRET<br> valueFrom:<br> secretKeyRef:<br> name: gemini-proxy-secret<br> key: PROXY_SECRET<br>```</p><p>— -</p><p># **11. Docker Build &amp; Load Commands**</p><p>```bash<br>docker build -t careerpilot-api:local .<br>docker save careerpilot-api:local &gt; api.tar<br>ctr -n k8s.io images import api.tar<br>kubectl rollout restart deployment api<br>```</p><p>— -</p><p># **12. Final Thoughts**</p><p>This journey taught me:</p><p>- Cloud AI APIs behave differently based on **geolocation** and **ASN reputation** <br>- Kubernetes without a registry requires **strict discipline** <br>- Cloud Run health checks can break your container in unexpected ways <br>- Gemini’s **v1 vs v1beta** split is easy to get wrong <br>- A proxy should be **dumb**, not “smart” <br>- Architecture evolves under pressure — and that’s a good thing</p><p>If you’re building AI systems in production, I hope this case study saves you the pain I went through.</p><p>— -</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f95846b93566" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From k3s Deployment Failures to Production HTTPS: A Full-Day Journey with FreeDNS + Caddy]]></title>
            <link>https://medium.com/@nirajmind/from-k3s-deployment-failures-to-production-https-a-full-day-journey-with-freedns-caddy-28fb5bde7234?source=rss-9a422ea56694------2</link>
            <guid isPermaLink="false">https://medium.com/p/28fb5bde7234</guid>
            <category><![CDATA[caddy-server]]></category>
            <category><![CDATA[kubernetes-cluster]]></category>
            <category><![CDATA[dns-over-https]]></category>
            <category><![CDATA[docker-image]]></category>
            <category><![CDATA[freedns]]></category>
            <dc:creator><![CDATA[Niraj Adhikary]]></dc:creator>
            <pubDate>Wed, 21 Jan 2026 09:31:42 GMT</pubDate>
            <atom:updated>2026-01-21T09:31:42.461Z</atom:updated>
            <content:encoded><![CDATA[<p>If you’ve ever spent an entire day wrestling with a homelab deployment, you’ll recognize this story. <br>It starts with a simple goal — “deploy two apps on k3s and expose them over HTTPS” — and ends with a working, production‑grade setup using FreeDNS and Caddy.</p><p>Between those two points? <br>A maze of container runtimes, DNS providers, ACME challenges, and a lot of stubborn debugging.</p><p>This is the full journey.</p><p>— -</p><p>## **1. The Day Begins: k3s Won’t Pull Local Images**</p><p>The first roadblock appeared immediately: <br>k3s refused to pull locally built Docker images.</p><p>The culprit? <br>k3s uses **containerd**, not Docker.</p><p>The fix:</p><p>```bash<br>sudo k3s ctr images import myapp.tar<br>```</p><p>Once imported, the pods finally started.</p><p>— -</p><p>## **2. Reverse Proxy with Caddy**</p><p>With the apps running on NodePorts, Caddy was configured as the public entrypoint.</p><p>The initial plan was to use DNS‑01 with DuckDNS.</p><p>That didn’t last long.</p><p>— -</p><p>## **3. DNS‑01: The Rabbit Hole**</p><p>### **DuckDNS**<br>- No TXT record support <br>- ACME DNS‑01 fails <br>- Caddy errors everywhere</p><p>### **Dynu**<br>- Better API <br>- But DDNS hostnames aren’t full DNS zones <br>- TXT record creation blocked <br>- Caddy throws: <br> ```<br> missing own domain<br> ```</p><p>After hours of debugging, it became clear:</p><p>&gt; DNS‑01 is powerful, but fragile — and free DNS providers rarely support it properly.</p><p>Time for a new strategy.</p><p>— -</p><p>## **4. The Pivot: Use HTTP‑01 Instead**</p><p>HTTP‑01 requires only:</p><p>- Port 80 open <br>- Port 443 open <br>- A domain pointing to your server</p><p>No DNS API. <br>No TXT records. <br>No plugins.</p><p>Perfect for FreeDNS.</p><p>— -</p><p>## **5. FreeDNS + Caddy = Instant Success**</p><p>FreeDNS provides free subdomains like:</p><p>- `careerpilot.chickenkiller.com`<br>- `smarttreasurer.chickenkiller.com`</p><p>Point them to your server’s IP, and Caddy does the rest.</p><p>The final Caddyfile:</p><p>```caddyfile<br>careerpilot.chickenkiller.com {<br> reverse_proxy 127.0.0.1:30001<br>}</p><p>smarttreasurer.chickenkiller.com {<br> reverse_proxy /api/* localhost:8000<br> reverse_proxy /token localhost:8000<br> reverse_proxy localhost:5173<br>}<br>```</p><p>Restart Caddy, watch the logs, and suddenly:</p><p>- “authorization finalized”<br>- “certificate obtained successfully”<br>- “renewal info received”</p><p>Both sites live. <br>Both HTTPS. <br>Both auto‑renewing.</p><p>— -</p><p>## **6. The Final Architecture**</p><p>```<br>Internet<br> ↓<br>Caddy (HTTPS termination)<br> ↓<br>k3s NodePort<br> ↓<br>Application Pods<br>```</p><p>Simple. Clean. Reliable.</p><p>— -</p><p>## **7. Lessons Learned**</p><p>- **DNS‑01 is overkill** unless you control the full DNS zone <br>- **HTTP‑01 is perfect** for homelabs and free DNS providers <br>- **Caddy is a powerhouse** — it handles everything automatically <br>- **k3s uses containerd**, so load images accordingly <br>- **Debugging is a skill**, and persistence pays off</p><p>— -</p><p>## **8. Final Thoughts**</p><p>What started as a frustrating deployment turned into a rock‑solid, production‑ready setup. <br>FreeDNS + Caddy + HTTP‑01 is now my go‑to pattern for quick, secure deployments without DNS complexity.</p><p>If you’re building a homelab or self‑hosting apps, this workflow is worth adopting.</p><p>— -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ei08gnVlUiIjA2R3X2C1QQ.png" /><figcaption>Design diagram</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28fb5bde7234" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building and Deploying a Streamlit RAG App on Google Cloud: A Journey of Learning]]></title>
            <link>https://medium.com/analytics-vidhya/building-and-deploying-a-streamlit-rag-app-on-google-cloud-a-journey-of-learning-e2089929b55f?source=rss-9a422ea56694------2</link>
            <guid isPermaLink="false">https://medium.com/p/e2089929b55f</guid>
            <category><![CDATA[streamlit]]></category>
            <category><![CDATA[containerization]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Niraj Adhikary]]></dc:creator>
            <pubDate>Thu, 06 Mar 2025 05:54:31 GMT</pubDate>
            <atom:updated>2025-03-06T05:54:31.212Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i7owYfdwv88SU1ff3cVB4w.jpeg" /><figcaption><strong>Building and Deploying a Streamlit RAG App on Google Cloud</strong></figcaption></figure><p><strong>Building and Deploying a Streamlit RAG App on Google Cloud: A Journey of Learning</strong></p><p>In this post, I’ll share my experience building a Retrieval-Augmented Generation (<strong>RAG</strong>) application using Streamlit and deploying it on Google Cloud Run. This journey involved integrating various technologies — <strong>Streamlit</strong>, <strong>FastAPI</strong>, <strong>SQLAlchemy</strong>, <strong>ChromaDB</strong>, <strong>OpenAI</strong>, and <strong>Google Cloud services</strong> (Cloud Build, Cloud Run, Artifact Registry, Cloud Storage). I’ll walk you through the architecture, key code snippets, the challenges I faced (and there were <em>many</em>!), and the solutions that finally led to a working, deployable application. My goal is to provide a practical guide and share hard-won lessons that can save you time and frustration.</p><ol><li><strong>The Vision: A Streamlit RAG Application</strong></li></ol><p>My goal was to create a web application that could answer questions based on a set of documents. The user should be able to:</p><ul><li>Register and log in.</li><li>Upload PDF documents.</li><li>Ask a question related to the content of those documents.</li><li>Receive an answer generated by a Large Language Model (LLM), along with citations of the source documents.</li><li>View a history of their conversation.</li></ul><p>This is a classic <strong>RAG </strong>(Retrieval-Augmented Generation) use case.</p><p><strong>2. The Architecture: Assembling the Pieces</strong></p><p>The application is built using a combination of technologies:</p><ul><li><strong>Frontend:</strong> Streamlit (for the user interface).</li><li><strong>Backend (initially):</strong> FastAPI (for user management, initial API design). This was later integrated <em>directly</em> into the Streamlit app for simplicity.</li><li><strong>Database:</strong> SQLite (for user accounts and conversation history). (Note: For production, PostgreSQL on Cloud SQL would be a better choice.)</li><li><strong>Vector Database:</strong> ChromaDB (for storing document embeddings and enabling semantic search).</li><li><strong>Embeddings:</strong> OpenAI’s text-embedding-ada-002 (for creating vector representations of text).</li><li><strong>LLM:</strong> OpenAI’s gpt-4o (for generating answers).</li><li><strong>Deployment:</strong> Google Cloud Run (for hosting the containerized application).</li><li><strong>Build Automation:</strong> Google Cloud Build (for building the Docker image).</li><li><strong>Container Registry:</strong> Google Container Registry (gcr.io) (for storing the Docker image. Later switched to using digests, which is compatible with Artifact Registry too).</li><li><strong>File Storage:</strong> Google Cloud Storage (for storing uploaded PDF documents).</li></ul><p><strong>Here’s the Mermaid diagram of the final architecture</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1oGZh9m8rOTKt8AdAv91NA.png" /><figcaption><strong>Multi-User RAG pipeline using Streamlit</strong></figcaption></figure><p><strong>3. Key Code Snippets (and Lessons Learned)</strong></p><p><strong>3.1. </strong><strong>Dockerfile (for Streamlit):</strong></p><pre># Use an official Python runtime as a parent image<br>FROM python:3.11-slim<br><br># Set the working directory to /app<br>WORKDIR /app<br><br># Copy the current directory contents into the container at /app<br>COPY . /app<br><br># Install any needed packages specified in requirements.txt<br>RUN pip install --no-cache-dir -r requirements.txt<br><br># Make port 8501 available to the world outside this container<br>EXPOSE 8501<br><br># Define environment variables (for production, use secrets!)<br>ENV OPENAI_API_KEY=&quot;&quot;<br>ENV DATABASE_PATH=&quot;/app/data/vector_db&quot;<br>ENV COLLECTION_NAME=&quot;my_documents&quot;<br>ENV ALLOW_RESET=True<br>ENV PYTHONPATH=&quot;/app&quot;<br>ENV HUGGINGFACEHUB_API_TOKEN=&quot;&quot;<br>ENV TOKENIZER_MODEL=&quot;sentence-transformers/all-mpnet-base-v2&quot;<br>ENV COHERE_API_KEY=&quot;&quot;<br>ENV PORT=8501<br># Create the /app/data directory<br>RUN mkdir -p /app/data<br><br># Run uvicorn when the container launches<br>CMD streamlit run app.py --server.port=$PORT --server.address=0.0.0.0</pre><p><strong>Lesson 1: The </strong><strong>CMD Instruction and Environment Variables:</strong> I learned that the CMD instruction in its <em>exec form</em> (using square brackets) <em>doesn&#39;t</em> automatically perform environment variable substitution. Switching to the <em>shell form</em> (CMD streamlit run ...) solved this, allowing Cloud Run&#39;s PORT variable to be correctly passed to Streamlit.</p><p><strong>Lesson 2: Setting the correct port</strong> Set the port to be default 8501 and create the shell command to use the environment variable.</p><p><strong>3.2. </strong><strong>deployment/gcloud_deploy.py (Deployment Script):</strong></p><pre># ... (imports: subprocess, os, time, re) ...<br><br>def run_command(command, capture_output=True, shell=False):<br>    # ... (Implementation from previous responses - handles synchronous and asynchronous execution) ...<br><br>def build_and_deploy(project_id, service_account_email, image_name, service_name, region):<br>     # ... (Build and deployment logic - as in previous responses)<br>     # Key parts:<br>      build_command = [<br>        &quot;gcloud&quot;, &quot;builds&quot;, &quot;submit&quot;,<br>        &quot;--tag&quot;, f&quot;gcr.io/{project_id}/{image_name}&quot;,<br>        f&quot;--region={region}&quot;,<br>        &quot;--timeout=3600s&quot;,<br>        &quot;--machine-type=e2-highcpu-8&quot;,<br>        f&quot;--service-account={full_service_account_name}&quot;,<br>        f&quot;--project={project_id}&quot;,<br>        &quot;--async&quot;,  # Submit asynchronously<br>        &quot;.&quot;<br>    ]<br><br>    build_id_output = run_command(build_command)<br>     # ... (Extract build ID using regex) ...<br>      match = re.search(r&quot;started \[([^\]]+)\]&quot;, build_id_output)<br>      if match:<br>          build_id = match.group(1)<br>          print(f&quot;Build ID: {build_id}&quot;)<br><br>    # --- Wait for Build Completion (Synchronously) ---<br>    status_command = [<br>        &quot;gcloud&quot;, &quot;builds&quot;, &quot;describe&quot;, build_id,<br>        f&quot;--project={project_id}&quot;,<br>        f&quot;--region={region}&quot;,<br>        &quot;--format=value(status)&quot;<br>    ]<br>    while True:<br>      status = run_command(status_command)<br>      # ... (wait loop) ...<br>       # --- Get Image Digest (After Successful Build) ---<br>    build_describe_output = run_command([<br>        &quot;gcloud&quot;, &quot;builds&quot;, &quot;describe&quot;, build_id,<br>        f&quot;--project={project_id}&quot;,<br>        f&quot;--region={region}&quot;,<br>        &quot;--format=value(results.images[0].digest)&quot;  # Get the DIGEST directly<br>    ])<br><br>    # Construct the full image name with the digest<br>    full_image_name = f&quot;gcr.io/{project_id}/{image_name}@{image_digest}&quot;<br>     # ... (Deployment command) ...<br>      deploy_command = [<br>        &quot;gcloud&quot;, &quot;run&quot;, &quot;deploy&quot;, service_name,<br>        &quot;--image&quot;, full_image_name,  # Use the full image name with digest<br>        &quot;--platform&quot;, &quot;managed&quot;,<br>        &quot;--region&quot;, region,<br>        &quot;--allow-unauthenticated&quot;,<br>        f&quot;--service-account={service_account_email}&quot;,<br>        f&quot;--project={project_id}&quot;,<br>        &quot;--set-env-vars&quot;, env_vars_string,<br>        &quot;--memory&quot;, &quot;1Gi&quot;<br>    ]<br># ... (rest of the script) ...<br><br>def setup_and_deploy():<br> # ... (Get project ID, service account, image/service names) ...<br>   project_id = os.environ.get(&quot;GOOGLE_CLOUD_PROJECT&quot;)  # First, try environment variable<br>    if not project_id:<br>        try:<br>            project_id = subprocess.check_output(<br>                [&quot;gcloud&quot;, &quot;config&quot;, &quot;get-value&quot;, &quot;project&quot;], text=True<br>            ).strip()<br>        except subprocess.CalledProcessError:<br>            raise RuntimeError(<br>                &quot;Could not determine Google Cloud Project ID.  Set the GOOGLE_CLOUD_PROJECT environment variable or configure gcloud.&quot;<br>            )<br>    print(f&quot;Using project ID: {project_id}&quot;)<br><br>    # --- Service Account Setup ---<br>    service_account_name = &quot;cloud-build-sa&quot;<br>    service_account_email = (<br>        f&quot;{service_account_name}@{project_id}.iam.gserviceaccount.com&quot;<br>    )<br>    # --- Image and Service Names (with Defaults and Environment Variables) ---<br>    image_name = os.environ.get(&quot;IMAGE_NAME&quot;, &quot;streamlit-app&quot;)<br>    service_name = os.environ.get(&quot;SERVICE_NAME&quot;, &quot;streamlit-rag-app&quot;)<br>    region = os.environ.get(&quot;REGION&quot;, &quot;us-central1&quot;)<br>    build_and_deploy(project_id, service_account_email, image_name, service_name, region)<br><br>if __name__ == &quot;__main__&quot;:<br>   # main()<br>    setup_and_deploy()</pre><p><strong>Lesson 3: Synchronous vs. Asynchronous </strong><strong>gcloud Execution:</strong> I initially struggled with getting the build ID correctly because I was mixing synchronous and asynchronous execution. The final solution uses gcloud builds submit --async to <em>start</em> the build and get an ID immediately, then uses gcloud builds describe in a loop to <em>wait</em> for the build to finish. This two-stage approach is crucial for reliability.</p><p><strong>Lesson 4: Getting the project ID:</strong> I learned to first check for GOOGLE_CLOUD_PROJECT environment variable, before falling back to using gcloud config.</p><p><strong>Lesson 5: Image digest:</strong> I learned to use results.images[0].digest to get the complete image digest.</p><p><strong>3.3. </strong><strong>app.py (Streamlit App):</strong></p><pre># --- Google Cloud Storage Imports ---<br>from google.cloud import storage<br><br># Configure logging<br>logging.basicConfig(level=logging.INFO)<br>logger = logging.getLogger(__name__)<br><br># --- GCS Configuration ---<br># Get the GCS bucket name from an environment variable<br>GCS_BUCKET_NAME = os.environ.get(&quot;GCS_BUCKET_NAME&quot;, &quot;your-default-bucket-name&quot;) #REPLACE<br># Initialize the GCS client (outside any function - for reuse)<br>storage_client = storage.Client()<br># --- Authentication (Adapted for Streamlit) ---<br>def authenticate_user(username, password, db: Session):<br>    user = user_crud.get_user_by_username(db, username=username)<br>    if not user or not HashPassword.verify_password(password, user.hashed_password):<br>        return None<br>    return user<br><br># ---  RAG Pipeline Components (as functions) ---<br>def initialize_embedder(provider=&quot;openai&quot;, api_key=None, model_name=&quot;text-embedding-ada-002&quot;):<br>    &quot;&quot;&quot;Initializes the embedding model.&quot;&quot;&quot;<br>    if provider == &quot;openai&quot; and api_key is None:<br>        raise ValueError(&quot;OpenAI API key is required when using OpenAI embeddings.&quot;)<br>    return Embedder(provider=provider, api_key=api_key, model_name=model_name)<br><br>def initialize_retriever(chunks, embedder, db_path, collection_name):<br>    &quot;&quot;&quot;Initializes the vector database and retriever.&quot;&quot;&quot;<br>    vector_db = create_vector_database(chunks, embedder, db_path=db_path, collection_name=collection_name)<br>    return Retriever(vector_db, embedder, collection_name)<br><br>def initialize_rag_pipeline(retriever, llm_provider, api_key, llm_model):<br>    return RAGPipeline(retriever=retriever, llm_provider=llm_provider, api_key=api_key, llm_model=llm_model)<br># --- Streamlit App ---<br>def main():<br><br>    # --- Database Initialization (at app startup) ---<br>    inspector = inspect(engine)<br>    if not inspector.has_table(&quot;users&quot;):<br>      user_models.Base.metadata.create_all(bind=engine)<br>    else:<br>      logger.info(&quot;users table exists&quot;)<br><br>    st.title(&quot;My Streamlit RAG App&quot;)<br><br>    # --- Sidebar: Login/Registration ---<br>    menu = [&quot;Login&quot;, &quot;Register&quot;]<br>    choice = st.sidebar.selectbox(&quot;Menu&quot;, menu)<br><br>    if choice == &quot;Register&quot;:<br>        with st.sidebar:<br>          st.subheader(&quot;Create New Account&quot;)<br>          new_username = st.text_input(&quot;Username&quot;)<br>          new_email = st.text_input(&quot;Email&quot;)<br>          new_password = st.text_input(&quot;Password&quot;, type=&#39;password&#39;)<br>          confirm_password = st.text_input(&quot;Confirm Password&quot;, type=&#39;password&#39;)<br><br>          if st.button(&quot;Signup&quot;):<br>              if new_password == confirm_password:<br>                  hashed_password = HashPassword(new_password).get_hash_password()<br>                  new_user = user_schemas.UserCreate(username=new_username, email=new_email, password=new_password) # Plain text password<br>                  db = next(get_db()) # Create db session.<br>                  try:<br>                      created_user = user_crud.create_user(db=db, user=new_user)<br>                      st.success(&quot;Account created successfully! Please login.&quot;)<br>                  except Exception as e:<br>                      st.error(f&quot;Error during registration: {e}&quot;)<br>                  finally:<br>                    db.close() # Close the session.<br>              else:<br>                  st.warning(&quot;Passwords do not match&quot;)<br><br>    elif choice == &quot;Login&quot;:<br>        with st.sidebar:<br>          st.subheader(&quot;Login Section&quot;)<br>          username = st.text_input(&quot;User Name&quot;)<br>          password = st.text_input(&quot;Password&quot;, type=&#39;password&#39;)<br>          if st.button(&quot;Login&quot;):<br>              db = next(get_db()) # Create db session.<br>              user = authenticate_user(username, password, db)<br>              db.close() # Close the session.<br>              if user:<br>                  st.success(f&quot;Logged in as {username}&quot;)<br>                  st.session_state[&#39;user&#39;] = user  # Store user in session state<br>              else:<br>                st.sidebar.warning(&quot;Incorrect Username/Password&quot;)<br><br><br>    # --- Main Content (Only Display if Logged In) ---<br>    if &#39;user&#39; in st.session_state:<br>        user = st.session_state[&#39;user&#39;]<br><br>        # --- Get Environment Variables ---<br>        openai_api_key = os.environ.get(&quot;OPENAI_API_KEY&quot;, &quot;&quot;)<br>        huggingface_api_token = os.environ.get(&quot;HF_TOKEN&quot;, &quot;&quot;)<br>        model_name = os.environ.get(&quot;TOKENIZER_MODEL&quot;, &quot;all-MiniLM-L6-v2&quot;)<br><br>        # --- User-Specific Data ---</pre><p><strong>Next Steps and Further Improvements:</strong></p><p><strong>Further Enhancements:</strong></p><ul><li><strong>RAG Pipeline Improvements:</strong></li><li><strong>Experiment with LLMs and Embedding Models:</strong> I could try with different LLMs (e.g., models from Hugging Face, Cohere) and embedding models to see which gives me the best results.</li><li><strong>Fine-tune Prompting:</strong> I could experiment with different prompts to the LLM to improve the quality and relevance of the answers.</li><li><strong>Advanced Retrieval:</strong> I could explore more advanced retrieval techniques (e.g., hybrid search, re-ranking).</li><li><strong>Metadata Filtering:</strong> I might add the ability to filter retrieved documents based on metadata (e.g., date, source, tags).</li><li><strong>Database:</strong></li><li><strong>PostgreSQL (for Production):</strong> I might migrate to PostgreSQL (or another production-ready database) for scalability and reliability. Cloud SQL is a good managed option on Google Cloud.</li><li><strong>Database Migrations (Alembic):</strong> I could implement database migrations using Alembic to manage schema changes over time.</li><li><strong>Testing:</strong> I need to write unit tests and integration tests for my code. This will help me to catch bugs early and ensure that my application continues to work as expected as I make changes, using farmeworks like pytest.</li><li><strong>CI/CD:</strong> I need to set up a Continuous Integration/Continuous Deployment (CI/CD) pipeline to automate the build, test, and deployment process. I can use Cloud Build triggers for this, combined with my already written codegcloud_deploy.py</li><li><strong>User Interface:</strong></li><li><strong>Multi-Page App: </strong>Need to make the app a multi-page Streamlit app. I could have separate pages for login/registration, document management, and querying.</li><li><strong>Deployment Optimization:</strong></li><li><strong>Resource optimization</strong>: Resources could have been optimized, such as --memory for cloud run deployment.</li></ul><p><strong>Longer-Term Considerations I didn’t have time to put into the development:</strong></p><ul><li><strong>Scalability:</strong> Consider how to scale the app to handle more users and data. This might involve using a distributed vector database, load balancing, and other techniques.</li><li><strong>Monitoring:</strong> Needed to implement monitoring and alerting to track the performance and health of my application in production.</li><li><strong>Security:</strong> I needed to add continuous review of the security of my application.</li><li><strong>Cost Optimization:</strong> Monitoring my cloud resource usage and costs, and look for ways to optimize them.</li></ul><p>Happy learning to all!</p><p>If you build something using these techniques, I’d love to hear about it! Share your projects in the comments below.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e2089929b55f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/building-and-deploying-a-streamlit-rag-app-on-google-cloud-a-journey-of-learning-e2089929b55f">Building and Deploying a Streamlit RAG App on Google Cloud: A Journey of Learning</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Frustration to Denoising Success: A Deep Dive into Building an Image Denoising Autoencoder…]]></title>
            <link>https://medium.com/analytics-vidhya/from-frustration-to-denoising-success-a-deep-dive-into-building-an-image-denoising-autoencoder-f9660a049491?source=rss-9a422ea56694------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9660a049491</guid>
            <category><![CDATA[image-denoising]]></category>
            <category><![CDATA[unet]]></category>
            <category><![CDATA[autoencoder]]></category>
            <category><![CDATA[image-processing]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Niraj Adhikary]]></dc:creator>
            <pubDate>Tue, 04 Feb 2025 09:53:30 GMT</pubDate>
            <atom:updated>2025-02-08T07:21:22.637Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gDn9cFALoyv-vM6W.png" /><figcaption>A Typical U-Net architecture</figcaption></figure><h3>From Frustration to Denoising Success: A Deep Dive into Building an Image Denoising Autoencoder with PyTorch</h3><h3>Introduction:</h3><p>Image denoising is a classic problem in computer vision: how do we take a noisy image and recover the original, clean image? Deep learning, particularly convolutional neural networks (CNNs), has shown great promise in this area. In this blog post, I’ll share my journey of building an image denoising autoencoder using PyTorch, highlighting the challenges I faced, the mistakes I made, and the key lessons I learned along the way. This post is for anyone interested in image processing, deep learning, or troubleshooting neural networks that just don’t seem to learn.</p><h3>The Challenge: A Stubbornly Stagnant Loss</h3><p>My initial goal was to build a U-Net-like autoencoder to denoise images. I used a dataset of clean images and added synthetic noise to create noisy input images. I started with a standard U-Net architecture, a reasonable learning rate, and the Adam optimizer. However, no matter what I tried, the training loss refused to decrease! It was stuck at a high value, and the model wasn’t learning anything meaningful.</p><h3>The Debugging Journey: A Systematic Approach</h3><p>This is where the real learning began. I embarked on a systematic debugging process, which involved:</p><ol><li><strong>Verifying Data Loading and Preprocessing:</strong> I carefully checked that my images were being loaded correctly and that the normalization was being applied as intended. I even visualized sample images to make sure everything looked right.</li><li><strong>Simplifying the Model:</strong> To isolate the problem, I temporarily replaced the U-Net with a much simpler autoencoder, even a basic linear model, to see if it could learn anything at all.</li><li><strong>Experimenting with Hyperparameters:</strong> I tried different learning rates, batch sizes, and even different optimizers, but the loss remained stubbornly high.</li><li><strong>Visualizing Activations:</strong> I used torchview to generate a helpful diagram of my model&#39;s architecture. I also visualized the activations of intermediate layers to understand what features the model was learning (or not learning).</li></ol><p><strong>The Key Insight: Noise Matters!</strong></p><p>After much investigation, the culprit turned out to be the way I was generating and applying noise to the clean images. Here’s what I discovered:</p><ul><li><strong>Noise Magnitude:</strong> The magnitude of the noise I was adding was too small relative to the pixel values of the normalized images. The model simply couldn’t distinguish the noise from the actual image content.</li><li><strong>Applying Noise to Clean Images:</strong> Applying noise on top of the clean images is important, as if we use the pre-made noisy images, it will be difficult for the model to learn, it was a mistake on my part to choose the dataset which had noise in it already.</li><li><strong>Applying Noise After Converting to Tensor:</strong> In my initial code, I was adding noise to the PIL Image before converting it to a tensor. This meant that the noise was being added in the [0, 255] integer range and then scaled down, effectively reducing its magnitude. The correct way is to convert the image to a tensor first and then add noise.</li></ul><p><strong>The Solution: More Realistic and Varied Noise</strong></p><p>To fix this, I made the following changes to my ImageDataset class:</p><ol><li><strong>Increased Noise Magnitude:</strong> I significantly increased the standard deviation range for Gaussian noise and the probability range for salt-and-pepper noise.</li><li><strong>Converted to Tensor Before Noise:</strong> I ensured that the noise was added to the tensor representation of the image, not the PIL Image.</li><li><strong>Combined Noise Types:</strong> I applied both Gaussian and salt-and-pepper noise to each image to create more complex noise patterns.</li><li><strong>Applied noise on clean images:</strong> The noise was applied on top of the clean images rather than using the pre-noisy images.</li></ol><p>Here’s a snippet of the corrected code:</p><h3>Import the libraries(pytorch)</h3><pre>import os<br>os.environ[&#39;PYTORCH_CUDA_ALLOC_CONF&#39;] = &#39;expandable_segments:True&#39;<br><br>import gc<br>import torch<br>import torch.nn as nn<br>import torch.nn.functional as F<br>import torch.optim as optim<br>import torchvision.transforms as transforms<br>from torch.utils.data import Dataset, DataLoader, TensorDataset<br>from PIL import Image<br>import os<br>import numpy as np<br>import matplotlib.pyplot as plt</pre><h3>Load the Data</h3><pre><br>test_transform = transforms.Compose([<br>    transforms.ToTensor(),<br>    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))<br>])<br><br>class TestImageDataset(Dataset):<br>    def __init__(self, test_dir, transform=None):<br>        self.test_dir = test_dir<br>        self.transform = transform<br>        self.image_paths = sorted([os.path.join(test_dir, f) for f in os.listdir(test_dir) if os.path.isfile(os.path.join(test_dir, f))])<br><br>    def __len__(self):<br>        return len(self.image_paths)<br><br>    def __getitem__(self, idx):<br>        image_path = self.image_paths[idx]<br>        image = Image.open(image_path).convert(&quot;RGB&quot;)<br><br>        if self.transform:<br>            image = self.transform(image)<br><br>        return image  # Only return the image (no clean/noisy pair)<br><br><br>from torch.utils.data import Dataset, DataLoader<br>from PIL import Image<br>import os<br>import random<br>import torchvision.transforms.functional as TF<br><br>class ImageDataset(Dataset):<br>    def __init__(self, clean_dir, noisy_dir, transform=None):<br>        self.clean_dir = clean_dir<br>        self.noisy_dir = noisy_dir<br>        self.transform = transform<br>        self.clean_image_paths = sorted([os.path.join(clean_dir, f) for f in os.listdir(clean_dir) if os.path.isfile(os.path.join(clean_dir, f))])<br>        self.noisy_image_paths = sorted([os.path.join(noisy_dir, f) for f in os.listdir(noisy_dir) if os.path.isfile(os.path.join(noisy_dir, f))])<br><br>    def __len__(self):<br>        return len(self.clean_image_paths)<br><br>    def __getitem__(self, idx):<br>        clean_image_path = self.clean_image_paths[idx]<br>        noisy_image_path = self.noisy_image_paths[idx]<br><br>        clean_image = Image.open(clean_image_path).convert(&quot;RGB&quot;)<br>        noisy_image = Image.open(noisy_image_path).convert(&quot;RGB&quot;)<br><br>        # Apply noise to the clean image before transform<br>        clean_image = self.transform(clean_image)<br>        noisy_image = self.apply_random_noise(clean_image)<br>        &quot;&quot;&quot;<br>          if self.transform:<br>            clean_image = self.transform(clean_image)<br>            noisy_image = self.transform(noisy_image)<br>        &quot;&quot;&quot;<br>        return noisy_image, clean_image<br><br>    def apply_random_noise(self, image):<br>        noise_type = random.choice([&quot;gaussian&quot;, &quot;salt_pepper&quot;])<br><br>        if noise_type == &quot;gaussian&quot;:<br>            # Apply Gaussian noise with varying standard deviation<br>            std = random.uniform(0.1, 0.5)  # Increased std range<br>            noise = torch.randn_like(image) * std<br>            noisy_image = image + noise<br>        elif noise_type == &quot;salt_pepper&quot;:<br>            # Apply salt-and-pepper noise with varying probability<br>            prob = random.uniform(0.05, 0.4)  # Increased prob range<br>            rnd = torch.rand_like(image)<br>            noisy_image = image.clone()  # Create a copy to modify<br>            noisy_image[rnd &lt; prob / 2] = -1.0  # Pepper<br>            noisy_image[rnd &gt; 1 - prob / 2] = 1.0  # Salt<br>        else:<br>            noisy_image = image<br><br>        # Clip the values to be within the range [-1, 1]<br>        noisy_image = torch.clamp(noisy_image, -1, 1)<br>        return noisy_image<br><br><br># Data Loaders<br>train_dataset = ImageDataset(&#39;/content/train/clean&#39;, &#39;/content/train/noisy&#39;, transform=test_transform)<br>trainloader = DataLoader(train_dataset, batch_size=32, shuffle=True)<br># Create a DataLoader for your ImageDataset<br>test_dataset = TestImageDataset(&#39;/content/train/noisy&#39;, transform=test_transform)<br>testloader = DataLoader(test_dataset, batch_size=32, shuffle=False)</pre><p><strong>Model Definition (Convolutional Autoencoder)</strong></p><pre>batch_size = 32<br><br>trainloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)<br><br>device = torch.device(&quot;cuda:0&quot; if torch.cuda.is_available() else &quot;cpu&quot;)<br><br>from torchview import draw_graph<br><br>class UNetAutoencoder(nn.Module):<br>    def __init__(self):<br>        super(UNetAutoencoder, self).__init__()<br><br>        # Encoder<br>        self.enc1 = self.conv_block(3, 32)<br>        self.enc2 = self.conv_block(32, 64)<br>        self.enc3 = self.conv_block(64, 128)<br>        self.enc4 = self.conv_block(128, 256)<br><br>        self.pool = nn.MaxPool2d(2, 2)<br><br>        # Decoder<br>        self.dec1 = self.upconv_block(256, 128)<br>        self.dec2 = self.upconv_block(256, 64)<br>        self.dec3 = self.upconv_block(128, 32)<br>        self.dec4 = nn.Conv2d(64, 3, kernel_size=1) # Final 1x1 convolution<br><br>    def conv_block(self, in_channels, out_channels):<br>        return nn.Sequential(<br>            nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1),<br>            nn.BatchNorm2d(out_channels),<br>            nn.ReLU(inplace=True),<br>            nn.Conv2d(out_channels, out_channels, kernel_size=3, padding=1),<br>            nn.BatchNorm2d(out_channels),<br>            nn.ReLU(inplace=True)<br>        )<br><br>    def upconv_block(self, in_channels, out_channels):<br>        return nn.Sequential(<br>            nn.ConvTranspose2d(in_channels, out_channels, kernel_size=2, stride=2),<br>            nn.BatchNorm2d(out_channels),<br>            nn.ReLU(inplace=True)<br>        )<br><br>    def forward(self, x):<br>        # Encoder<br>        e1 = self.enc1(x)<br>        e2 = self.enc2(self.pool(e1))<br>        e3 = self.enc3(self.pool(e2))<br>        e4 = self.enc4(self.pool(e3))<br><br>        # Decoder<br>        d1 = self.dec1(e4)<br>        d1 = torch.cat([e3, d1], dim=1)  # Skip connection<br>        d2 = self.dec2(d1)<br>        d2 = torch.cat([e2, d2], dim=1)  # Skip connection<br>        d3 = self.dec3(d2)<br>        d3 = torch.cat([e1, d3], dim=1)  # Skip connection<br>        d4 = self.dec4(d3)  # No activation in the final layer<br><br>        return torch.tanh(d4) # Tanh activation for output<br><br># Instantiate the model<br>model = UNetAutoencoder()</pre><h3>The model architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*DHb-ioVa81KS4WOGw_b3Dg.png" /></figure><h3>To visualize an ctivation layer from the model in runtime</h3><pre>import matplotlib.pyplot as plt<br>activation = {}<br>def get_activation(name):<br>    def hook(model, input, output):<br>        activation[name] = output.detach()<br>    return hook<br><br>model.enc1.register_forward_hook(get_activation(&#39;enc1&#39;))</pre><h3>Define the training loop</h3><pre>learning_rate = 0.001  # Initial learning rate<br>n_epochs = 50 # Train for more epochs<br><br># Move model to device<br>model.to(device)<br><br>criterion = nn.MSELoss()<br>optimizer = optim.Adam(model.parameters(), lr=learning_rate)<br># Learning rate scheduler (ReduceLROnPlateau)<br>scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, &#39;min&#39;, patience=7, factor=0.5, verbose=True, threshold=1e-7, threshold_mode=&#39;abs&#39;)<br><br><br># Function to train the model (to avoid code repetition)<br># Function to train the model<br>def train_model(model, trainloader, device, n_epochs, optimizer, criterion, scheduler):<br>    print(&quot;Training with Adam optimizer...&quot;)<br><br>    losses = []<br><br>    torch.cuda.empty_cache()<br>    gc.collect()<br><br>    for epoch in range(n_epochs):<br>        running_loss = 0.0<br>        for i, (noisy_images, clean_images) in enumerate(trainloader):<br>            noisy_images, clean_images = noisy_images.to(device), clean_images.to(device)<br><br>            outputs = model(noisy_images)<br>            loss = criterion(outputs, clean_images)<br>            loss.backward()<br>            optimizer.step()<br>            optimizer.zero_grad()<br><br>            running_loss += loss.item()<br><br>            del noisy_images, clean_images, outputs, loss<br>            torch.cuda.empty_cache()<br><br>        epoch_loss = running_loss / len(trainloader)<br>        losses.append(epoch_loss)<br>        print(f&#39;Epoch {epoch + 1}, Loss: {epoch_loss}, Learning Rate: {optimizer.param_groups[0][&quot;lr&quot;]}&#39;)<br><br>        scheduler.step(epoch_loss)<br>        print(f&quot;Scheduler state: {scheduler.state_dict()}&quot;)<br><br>        torch.cuda.empty_cache()<br>        gc.collect()<br><br>    print(&#39;Finished Training&#39;)<br>    return losses</pre><h3>Call the train function to start training</h3><pre>model = model.to(device)<br><br># Train the model<br>losses = train_model(model, trainloader, device, n_epochs, optimizer, criterion, scheduler)</pre><h3>Start evaluation of the model with some noisy images</h3><pre># Get some test images<br>dataiter = iter(testloader)<br>images = next(dataiter)<br>images = images.to(device)<br><br>with torch.no_grad():  # Don&#39;t calculate gradients during evaluation<br>    denoised_images = model(images)<br>    # Access the activations from the &#39;activation&#39; dictionary<br>    enc1_activations = activation[&#39;enc1&#39;]<br><br># Visualize activations of the first image in the batch<br>activations = enc1_activations[0]  # First image in batch<br>num_channels = activations.shape[0]<br><br># Create a grid for visualization<br>ncols = 8  # Adjust as needed<br>nrows = num_channels // ncols<br><br>fig, axes = plt.subplots(nrows, ncols, figsize=(20, 10))<br>for i, ax in enumerate(axes.flat):<br>    if i &lt; num_channels:<br>        ax.imshow(activations[i].cpu().numpy(), cmap=&#39;viridis&#39;)  # Use a colormap<br>        ax.axis(&#39;off&#39;)<br><br>plt.suptitle(&quot;Activations of &#39;enc1&#39; for a Single Image&quot;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r8VAQVIX9N1Osey3kCnn5g.png" /><figcaption><strong>This image shows the learning by our activation layer enc1</strong></figcaption></figure><h3>Plot some of the denoised images</h3><pre>denoised_images = model(images)<br><br># Get denoised outputs<br>denoised_images = model(images)<br><br># Denormalize images for display<br>def denormalize(images):<br>    images = images * 0.5 + 0.5<br>    return images<br><br>denoised_images = denormalize(denoised_images.cpu())<br>images = denormalize(images.cpu())<br><br>n = 5<br>plt.figure(figsize=(20, 4))<br>plt.suptitle(&quot;Denoised Images with U-Net&quot;)<br>for i in range(n):<br>    # Original Image<br>    ax = plt.subplot(2, n, i + 1)<br>    plt.imshow(np.transpose(images[i], (1, 2, 0)))<br>    plt.gray()<br>    ax.get_xaxis().set_visible(False)<br>    ax.get_yaxis().set_visible(False)<br><br>    # Denoised Image<br>    ax = plt.subplot(2, n, i + 1 + n)<br>    plt.imshow(np.transpose(denoised_images[i].detach(), (1, 2, 0)))<br>    plt.gray()<br>    ax.get_xaxis().set_visible(False)<br>    ax.get_yaxis().set_visible(False)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hFFFGbs4bPJ_BEHoIx6GWg.png" /><figcaption><strong>This is how the denoised images finally look like.</strong></figcaption></figure><p><strong>Let’s plot our training loss curve</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/1*mrlkbFwotaaTuWWrMaspug.png" /><figcaption><strong>This is the training loss curve</strong></figcaption></figure><p><strong>Results and Further Improvements</strong></p><p>With these changes, the model finally started to learn! The loss decreased significantly, and the denoised images produced by the model showed a clear improvement in quality.</p><p>Here are some further improvements that could be considered:</p><ul><li><strong>Experiment with more noise types:</strong> Explore other noise distributions like Poisson or speckle noise.</li><li><strong>Data augmentation:</strong> Apply random rotations, flips, and crops to increase the diversity of the training data.</li><li><strong>Perceptual loss:</strong> Incorporate a perceptual loss term (e.g., using a pre-trained VGG network) to improve the perceptual quality of the denoised images.</li><li><strong>Model architecture:</strong> Fine-tune the U-Net architecture by adding more layers, changing the number of channels, or experimenting with different types of skip connections.</li></ul><p><strong>Conclusion:</strong></p><p>Building a deep learning model is often an iterative process that involves careful experimentation and debugging. This project taught me the crucial importance of:</p><ul><li><strong>Understanding your data:</strong> Visualizing and analyzing your data, including the noise characteristics, is essential.</li><li><strong>Systematic debugging:</strong> When things go wrong, take a step-by-step approach to isolate the problem. Simplify your model, visualize intermediate outputs, and test your assumptions.</li><li><strong>The importance of noise:</strong> In image denoising, the way you generate and apply noise during training has a huge impact on the model’s ability to learn and generalize.</li></ul><p>I hope this blog post has been helpful to others who are working on image denoising or similar deep learning projects. If you have any questions or suggestions, please feel free to share them in the comments below! I’m always eager to learn more and improve my understanding of these fascinating topics.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9660a049491" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/from-frustration-to-denoising-success-a-deep-dive-into-building-an-image-denoising-autoencoder-f9660a049491">From Frustration to Denoising Success: A Deep Dive into Building an Image Denoising Autoencoder…</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How can DevOps improve your apps?]]></title>
            <link>https://medium.com/@nirajmind/how-can-devops-improve-your-apps-fb816fef8770?source=rss-9a422ea56694------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb816fef8770</guid>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[ansible]]></category>
            <dc:creator><![CDATA[Niraj Adhikary]]></dc:creator>
            <pubDate>Sun, 30 Jul 2017 14:05:34 GMT</pubDate>
            <atom:updated>2017-07-30T14:05:34.208Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Introduction</strong>: The DevOps practices are very common nowadays in software organizations. Even though the concept is new to software the practice is not. Lots<br> of companies used to practice DevOps when software was not an industry at all. DevOps is not a funcationality rather it is an approach to solve daily software<br> development problems like packaging, configuring, testing, and deployment etc. As DevOps is a truly cross-functional mode of working there is no single<br> DevOps tool. It is rather a set if multiple tools.</p><p><strong>Goals</strong>: The goals of DevOps span the entire delivery pipeline. They include:</p><p>Improved deployment frequency;<br>Lower failure rate of new releases;<br>Shortened lead time between fixes;</p><p>Delivery process becomes incredibly simple with DevOps.</p><p><strong>Use Ansible to launch a MongoDB cluster in Atlas using API call</strong>: Ansible is the simplest way to automate apps and IT infrastructure. This tool is written on top<br>of python. So you need to have python installed and setup in the machine where you want to use Ansible. There are certain criterias you need to fulfill before<br> a cluster of MongoDB can be launched. <br> <br> Prerequisites</p><p>MongoDB Atlas Account<br>IP Whitelist configured for API access<br>API Key<br>Group ID<br>Server to launch from<br>Ansible (sudo pip install ansible)<br>Configure Ansible</p><p>You can create an atlas account freely by following the steps in the website <a href="https://cloud.mongodb.com/">https://cloud.mongodb.com</a> . Once you created an account successfully, log in to<br> the account. You need to whitelist the IP address you want to run Ansible from in the atlas account page. Go to your user account page and click public API<br> access(tab) <a href="https://cloud.mongodb.com/v2/5936600996e82136aac2eb8a#account/publicApi">https://cloud.mongodb.com/v2/XXXXXXXXX#account/publicApi</a>. Click on add button to add new IP address to whitelist. Generate a new <br>API key by clicking generate button. Copy this API key value to a safe location. Go to the home page by clicking mongoDB Atlas link. Click on Settings link <br>and copy the group ID. Assuming you have python and pip installed already, now install Ansible by “sudo pip install ansible” command. Once done downlaod the<br> git repository <a href="https://github.com/jaydestro/atlas-ansible">https://github.com/jaydestro/atlas-ansible</a>. Once clonned the repository edit the create-atlas.yml file like the below:</p><p>user: <a href="/service/mailto:niraj.adhikari@accionlabs.com">niraj.adhikari@accionlabs.com</a><br>apikey: YYYYYYYY-YYYY-hhhhh-00000<br>groupid: 5hhhhhhhhhhhhhhhhhhhhhh</p><p>all those above values will be diifferent for your case.</p><p>Edit the hosts file in the same directory like</p><p>hosts: 127.0.0.1</p><p>You can edit atlas.json also like below</p><p>{<br> “name” : “DataStore1”,<br> “numShards” : 1,<br> “mongoDBMajorVersion” : “3.4”,<br> “replicationFactor” : 3,<br> “providerSettings” : {<br> “providerName” : “AWS”,<br> “regionName” : “US_EAST_1”,<br> “instanceSizeName” : “M10”,<br> “diskIOPS” : 50,<br> “encryptEBSVolume” : false,<br> },<br> “diskSizeGB” : 10,<br> “backupEnabled” : false<br>}</p><p>The above edit will create a free MongoDB cluster in the US_EAST_1 AWS zone. If you need to create a cluster with chargeable server capacity you need to add <br>payment method in your atlas account first.</p><p>Now run the below command</p><p>ansible-playbook -v create-atlas.yml — diff</p><p>If everything goes well you will have a screen like below in your atlas account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EgTwtZEoJvJkY-83PsXOfQ.jpeg" /></figure><p>Enjoy. <br> <br> <br><strong>USe Case Where we need to apply DevOps Practice</strong>: There are lot of other scenarios where you practice DevOps. You need to determine in which cases you need to<br> use DevOps practice. If you see the above process for MongoDB cluster it looks quite simple in terms of configurations and running ansible. The above use case<br> is ideal if we want to make a huge numbers of clusters.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb816fef8770" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MongoDB Stitch — A declarative backend service]]></title>
            <link>https://medium.com/@nirajmind/mongodb-stitch-a-declarative-backend-service-1ccd3f688720?source=rss-9a422ea56694------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ccd3f688720</guid>
            <category><![CDATA[parse]]></category>
            <category><![CDATA[mongodb]]></category>
            <dc:creator><![CDATA[Niraj Adhikary]]></dc:creator>
            <pubDate>Thu, 29 Jun 2017 08:40:33 GMT</pubDate>
            <atom:updated>2017-06-29T08:40:33.960Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>MongoDB Stitch — A declarative backend service</strong></p><p><strong>Introduction</strong>: Backend-as-a-Service(BaaS), comparatively a new concept in market. There are a couple of options available right now such as Parse (<a href="http://parse.com/">http://parse.com</a>) by Facebook, GoInstant (<a href="http://goinstant.com/">http://goinstant.com</a>) by SalesForce. It seems like most of the BaaS in your list are focused on mobile. Most BaaS offer unique features not specific to just storing data. MongoDB Stitch(<a href="https://mongodb.com/cloud/stitch">https://mongodb.com/cloud/stitch</a>) is a new entrant into this arena. Developed and supported by MongoDB, Inc(<a href="https://www.mongodb.com/">https://www.mongodb.com/</a>) in beta stage now. As the concept name suggests it leverage your backend service APIs. If you opt for BaaS no need to write code for backend APIs anymore. No need to think of managing app servers for deploying APIs. No need to manage the security of the app servers. No need to provision accessibility between app server and database servers. Scaling the backend will be a few clicks. On top of those features Stitch offers declarative way of creating backend service. So, no need to write a lot of code and no need to think about code review, refactoring and updating the language installer in the server.</p><p><strong>Major components of Stitch</strong>: Pipelines, Services and Rules are few major components of Stitch. Let’s see an example of Pipeline creation. As Stitch is a part of MongoDB Atlas(<a href="https://www.mongodb.com/cloud">https://www.mongodb.com/cloud</a>) you need to have an account in Atlas. If you don’t have an account you can create one freely from the above URL. Once you created an account you can login to the Atlas admin UI and follow</p><p>*) Build a new cluster (such as *Stitch-Demo*) if you don’t have one yet.</p><p>*) Select __Stitch Apps__ in the left navigation menu.</p><p>*) Press the __Create New Application__ button and name your application `helloStitch` (or any other name you wish). In the __Link to Cluster__ section, select the cluster name you just selected.</p><p>*) Once your Stitch app has been created, you should be redirected to the Stitch Console at [https://stitch.mongodb.com] (<a href="https://stitch.mongodb.com/">https://stitch.mongodb.com</a>).</p><p>To get our Stitch backend-as-a-service application fully set up, we must go through the following configuration steps:</p><p>1. Set up the Stitch pipelines</p><p>2. Set up the Stitch namespace rules (The name can only contain ASCII letters, numbers, underscores, and hyphens)</p><p>3. Set up authentication (if any)</p><p>4. Set up additional services (Amazon S3 and HTTP services)</p><p>Next pipeline is from Stitch example created by MongoDB team and freely available from github(<a href="https://github.com/mongodb/platespace">https://github.com/mongodb/platespace</a> )</p><p>#### geoNear pipeline</p><p>Paginated geoLocation pipeline * in the app we are paginating from the nearest restaurant (nearest to the CURRENT_LOCATION value in src/mongodb-manager/mongodb-manager.js).</p><p>Create a new pipeline with the following parameters:</p><p>* __Name__: `geoNear`</p><p>* __Private__: Disabled</p><p>* __Skip Rules__: Enabled</p><p>* __Can Evaluate__: Leave set to `{}`</p><p>* __Parameters__:</p><p>* `longitude`: Required</p><p>* `latitude`: Required</p><p>* `query`: NOT Required (leave disabled)</p><p>* `limit`: Required</p><p>* `minDistance`: Required</p><p>* __Output Type__: Array</p><p>* __Service__: mongodb-atlas</p><p>* __Action__: aggregate</p><p>* __Value__:</p><p>```json</p><p>{</p><p>“database”: “platespace”,</p><p>“collection”: “restaurants”,</p><p>“pipeline”: [</p><p>{</p><p>“$geoNear”: {</p><p>“near”: {</p><p>“coordinates”: [</p><p>“%%vars.longitude”,</p><p>“%%vars.latitude”</p><p>],</p><p>“type”: “Point”</p><p>},</p><p>“query”: “%%vars.query”,</p><p>“limit”: “%%vars.limit”,</p><p>“minDistance”: “%%vars.minDistance”,</p><p>“distanceField”: “dist”,</p><p>“spherical”: true</p><p>}</p><p>}</p><p>]</p><p>}</p><p>```</p><p>* __Bind data to %%vars__:</p><p>```json</p><p>{</p><p>“longitude”: “%%args.longitude”,</p><p>“latitude”: “%%args.latitude”,</p><p>“query”: “%%args.query”,</p><p>“limit”: “%%args.limit”,</p><p>“minDistance”: “%%args.minDistance”</p><p>}</p><p>```</p><p>* Press the `Done` button.</p><p>* Scroll up and at the top press the `Save` button.</p><p>If you download the source code and build it properly you can see the above pipeline works and it opens a page in your local machine as follows</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MXmYj7T5IpsecjXwXvqSPg.jpeg" /></figure><p>Now you can imagine the power of backend as a service. It will take you at max 2 hours to fully setup the app and Stitch console and you are good to create your content as you want.</p><p><strong>Advantages of not writing code</strong>: As a developer, you know how difficult it is to make a project running and setup the environment. Then you need to create multiple test environments and the production. And every you will see the same code is behaving rudely in production as it used to run smoothly in test environments. Next you need to choose the language for development. Even though every language claim that it is the best it terms of features easy to code and robust to deployment, only you know the real fun. Last but not the least, you don’t need to install and manage database also as Atlas(<a href="https://www.mongodb.com/cloud">https://www.mongodb.com/cloud</a>) does it for you.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ccd3f688720" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why the document model?]]></title>
            <link>https://medium.com/@nirajmind/why-the-document-model-67ba8a6cd084?source=rss-9a422ea56694------2</link>
            <guid isPermaLink="false">https://medium.com/p/67ba8a6cd084</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Niraj Adhikary]]></dc:creator>
            <pubDate>Thu, 13 Apr 2017 11:13:46 GMT</pubDate>
            <atom:updated>2017-04-13T11:13:46.184Z</atom:updated>
            <content:encoded><![CDATA[<p>Data structure of structures is here to avoid transaction<br>========================================</p><p>Applications good fit for document data structure : With the advent of powerful mobile computing, more data needed to be exchanged between server and client. <br>As more data flows between client server, more data structures needed to be designed to solve the problem. Solving this type of problem using relational <br>databases was almost impossible. Due to the design principle of relational databases it was very tough to manage application designed for mobile platforms. <br> Keeping the data in plain text format was not efficient when you need the data to be presented as it is. Think of converting data type in a mobile application, <br> it is like a night mare. So document data structure solved all these problems. Nowadys it is very rare to find a mobile application with relational database. <br> Even though the last database could be relational for an existing project, mobile application never interacts with them. As document databases are making effort<br> to make the interaction as smooth as possible, I can see all sorts of applications can shift to document based databases very soon. I don’t find any reason to <br>tell that you need to use relational databases for your application.</p><p>Ex: My present application is based on MongoDB. Which is used by mobile applications to read/write data. One write example using Java code,</p><p>MongoCollection&lt;Document&gt; orderCollRef = ConnectionUtil.getInstance().createMongoClientInstance().getCollection(“XXXXXXXXXXXXX”);<br>orderCollRef.replaceOne(new Document(“_id”, orderDetDoc.get(“_id”)), orderBean.createOrder());</p><p>This example replaces one existing document with a new one. It is just one line of code to finish.</p><p>Now let’s look at the data structure :</p><p>{<br> “shippedThrough” : “www”,<br> “deviceType” : “”,<br> “accountId” : “sssssss”,<br> “subscriptionId” : [],<br> “paymentGateway” : “Zuora”,<br> “orderDate” : “Mar 29 2016, 16:55:40.517”,<br> “firstName” : “test”,<br> “lastName” : “03290950”,<br> “workEmail” : “<a href="/service/mailto:test+test@gmail.com">test+test@gmail.com</a>”,<br> “monthlySubscription” : “Y”,<br> “trackingNumber” : “abababa123naana”,<br> “invoiceNumber” : “INV00016945”,<br> “shipmentStatus” : “Y”,<br> “shippingCarrier” : “USPS”,<br> “shippmentDate” : “Dec 05 2016, 10:52:23.074”,<br> “productRatePlanId”:””,<br> “productRatePlanChargeId”:[“”],<br> “invoiceItems” : [ <br> {<br> “chargeAmount” : 199.0,<br> “serialNumber” : “dddddddddddddddddd”,<br> “editQuantity” : “N”,<br> “shippedThrough” : “dddddd”,<br> “chargeDescription” : “Null”,<br> “productName” : “dddddd”,<br> “chargeName” : “dddddd”,<br> “quantity” : “1”,<br> “serviceStartDate” : null,<br> “serviceEndDate” : null,<br> “taxAmount” : 17.4200000762939<br> }<br> ],<br> “serialNumbers” : [ <br> “ffffffffffff”<br> ]<br>}</p><p>The above one line Java code is enough to write. In case of a SQL write operation we need to use minimum 2 tables to save the data. Which is avoided by document <br>based database.</p><p>Document as a choice of data structure for development : Document is very similar to a Java/Python/Scala/C#/C++ objetcs in many ways. Converting Object data<br> type to relational form was a night mare for all application developers. There are many helpers like hibernate/JPA frameworks available in market due to the<br> above problem. Even though they were not perfect to solve the above problem they helped a lot. But solution was not up to the standard of object-oriented<br> programmers. A database of the same data structure as application is a dream comes true for many developers like me. Creating an array of objects or embed an<br> array of ints inside an object is amazing with document based data structure. No need to think how to convert this whole thing while saving or retrieving from<br> database. No need to take help of any frameworks to manage the conversion. Omitting an extra layer of coding which reduces lot of code error.</p><p>Ex: Let’s look at one read operation in MongoDB. Again this one is also used by mobile applications.</p><p>Collection&lt;UserAccessToken&gt; result = new ArrayList&lt;&gt;(); <br>MongoCursor&lt;Document&gt; cursor = ConnectionUtil.getInstance().createMongoClientInstance().getCollection(“XXXXXXXX”).find(new Document(“zuoraId”, zuoraId))<br> .iterator();<br> <br>From the above 2 lines once you get the cursor, you can convert to any type of your wish. Let’s convert to string and then to UserAccessToken and add to the<br> list(result). <br> <br>while (cursor.hasNext()) {<br> result.add(new UserAccessToken((String) cursor.next().get(“cortexId”)));<br> }</p><p>From the above code it is evident that an application is simple and robust to create in document based database(MongoDB). Overall no need to think of <br> transaction, even though embedded objects are very common. Finally time has come to think in terms of OOPs concept everything we can develop.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=67ba8a6cd084" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Indexing and collations application on queries]]></title>
            <link>https://medium.com/@nirajmind/indexing-and-collations-application-on-queries-8621f659f4f8?source=rss-9a422ea56694------2</link>
            <guid isPermaLink="false">https://medium.com/p/8621f659f4f8</guid>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Niraj Adhikary]]></dc:creator>
            <pubDate>Thu, 06 Apr 2017 18:37:59 GMT</pubDate>
            <atom:updated>2017-04-06T18:37:59.014Z</atom:updated>
            <content:encoded><![CDATA[<p>Indexing :</p><p>a) It is always fun to work with index. Creating new indexes on existing column or adding new column with pre-existing index criteria. Internally every index<br> is a series ascending or descending.</p><p>b) Different types of indexes : Other than numeric indexes, Mongo has text index for text based search. Text indexes follow alphabetical<br> order. A collection can have at most one text index. Mongo also has compound indexes where the index is comprised of multiple columns of same collection.<br> The order of the fields are important in compound indexes. To index an array MongoDB creates an index key for each element in the array which is called<br> multikey index. Multikey indexes can be constructed over arrays that hold both scalar values (e.g. strings, numbers) and nested documents.</p><p>c) Index Prefix : Index prefixes are the beginning subsets of indexed fields. For example, consider the following compound index: <br>{ “item”: 1, “location”: 1, “stock”: 1 }</p><p>The index has the following index prefixes:</p><p>{ item: 1 }<br>{ item: 1, location: 1 }</p><p>For a compound index, MongoDB can use the index to support queries on the index prefixes. If any field is not present in prefix of an index then querying <br>on that field will result in inefficient output. Ex: query on “item” and “stock” fields; “stock” is not present in prefix of the above index will result in<br> inefficiency.</p><p>d) Index Intersecion : When multiple indexes exist, MongoDB can intersect them to get result from queries. Instead off multiple indexes on multiple fields we<br> can create one compound index with all those fields. Field order for querying has to be made as it is in the compound index. But intersection can work<br> without maintaining order of fields.</p><p>e) Case Insensitivity of a text index : All text indexes with version 3 is case insesitive. The output of a full text search will not take care of sorting of<br> the documents. What does it mean is you cannot sort the documents based on alphabetical order due to case insesitivity. Ex. if the result contains small letter,<br> they will appear after capital letters initials, [‘Apple’, ‘Cat’ ,’banana’]. A collection can have at most one text index. Text index in queries Ex: <br> <br>db.XXXXX.aggregate([{“$match”: {“$and”: [{“$or”: [{“$text”: {“$search”: “kr”}}]}]}},{“$project”: {“aa”: 1,”bb”: 1,”_id”: 1,<br> ccc:1,”XXXX P Name”: 1,”Source.XXXInstanceId”: 1}},{“$sort”: {“XXXX P Name”: 1,”Date”: -1}},{“$skip”: 0},{“$limit”: 5}])<br> <br>f) Storage requirements of text index : Text index contains one index entry for each unique word in each indexed field for each document. So it could take a<br> large amount of space. Hence the write operation will be slow in database.</p><p>g) Collation : Collation allows users to specify language-specific rules for string comparison, such as rules for lettercase and accent marks. You can specify<br> collation for a collection(without any text index) or a view, an index, or specific operations that support collation. <br> <br> The above query with collation :</p><p>db.XXXXX.aggregate([{“$match”: {“$and”: [{“$or”: [{“$text”: {“$search”: “kr”}}]}]}},{“$project”: {“aa”: 1,”bb”: 1,”_id”: 1, ccc:1,<br>“XXXX P Name”: 1,”Source.XXXInstanceId”: 1}},{“$sort”: {“XXXX P Name”: 1,”Date”: -1}},{“$skip”: 0},{“$limit”: 5}],<br>{explain:true,collation:{locale:’en’, strength:2}})</p><p>Both these queries will return the same result set, the later query will sort the result set based on the language(here it is english, <br>it will ignore the cases also).</p><p>Any collection can have multiple collation indexes. The limitation of a collation is, it can act within the language only.</p><p>A collation can be created as : db.XXXX.createIndex({ “bb”: 1},{collation:{locale:’en’,strength:2}})</p><p>And apply as below : db.XXXX.aggregate([{ $match:{Brand:’eSSENTIAL EVERYDAY’}}], { explain:true, collation:{locale:’en’,strength:2}} )</p><p>As you can see from the above query, collation cannot be part of aggregate pipeline. Collation doesn’t work on regular expression based search queries also.</p><p>Well, when I’m writing this blog there are many questions remained unanswered. I hope Mongo will update the docs with all those infos and features in future<br> releases.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8621f659f4f8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Easy ways to get started with MongoDB]]></title>
            <link>https://medium.com/@nirajmind/5-easy-ways-to-get-started-with-mongodb-1246ad0228e1?source=rss-9a422ea56694------2</link>
            <guid isPermaLink="false">https://medium.com/p/1246ad0228e1</guid>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Niraj Adhikary]]></dc:creator>
            <pubDate>Mon, 19 Sep 2016 10:34:00 GMT</pubDate>
            <atom:updated>2016-09-19T10:34:00.232Z</atom:updated>
            <content:encoded><![CDATA[<p>1) Simple use case implementation<br>===============================<br> I started developing a simple use case of our project using MongoDB and Java. Previously we used Java and MySQL for the same. It was all running fine until I found write lock<br> problem in the database while the app writes lot of data at one shot to database. Some of the times write failed without any notification. Some of the times the app used to write<br> some of the contents. Some of the times app stops responding. Some times even read from database used to retrieve data not related to the queries at all. As the volume of data is <br> increasing exponentially, so was the disruptions in servicing by the app. A simple app like this caused us lot pain. Even debugging didn’t result to any good solution to the<br> problem. As we were searching for solutions to this very problem we went to try with a diff database. And MongoDB came to us as natural rescuer. Now the app is perfect with all the <br> functionalities are working seamlessly. All problems are gone with a simple try with MongoDB. <br> <br>2) Learned by online courses<br>=============================<br> I learnt a lot from MongoDB university online courses. The free certifications are very good. The course materials and quality of those courses are excellent. I finished 5 of them. <br> In future if there is more of them, I’m sure I will do them again. These courses have added tremendous value to my career in terms of depth of understanding and practical senses. <br> Now I can confidently configure replica set of MongoDB from scratch and connect to any replica set from any Java app. I can figure out what are the problems, what are the places <br> problems can come from. I can debug and fix any problem within very short span of time.<br> <br> <br>3) New tools and technologies at the rescue<br>================================<br> Its nice to see new developments and energy around this ecosystem. The new tools and developments help developers to work enthusiastically. Its an inspiration for us to get involved<br> with MongoDB in and around. Tools like compass is of great example. I used these tools a lot. They are very helpful to design and develop new modules and check the peroformances <br> of the existing developments. <br> <br> <br>4) Great help contents<br>===================================<br> The online help contents are great. They are well organised and indexed. One thing I will ask for is the example of every single help content. It will be of great help to every <br> developer. As we follow these contents for developments, we need their usage with an example to visualize and formulate our needs. <br> <br>5) Practice more to get your hand dirty<br>=====================================<br> This is indeed the most important thing to learn anything in depth. If you want to go a long distance with MongoDB with your career, then there is no shortcut. You have to <br> practice a lot.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1246ad0228e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>