<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Srijit Das on Medium]]></title>
        <description><![CDATA[Stories by Srijit Das on Medium]]></description>
        <link>https://medium.com/@srijit29032001?source=rss-6d5d41398c51------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*hLtCCSNQVKQoojxJ</url>
            <title>Stories by Srijit Das on Medium</title>
            <link>https://medium.com/@srijit29032001?source=rss-6d5d41398c51------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 18:03:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@srijit29032001/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[This is how Production grade Agentic Systems do RAG — Multi-stage Retrieval | Hybrid RAG]]></title>
            <link>https://medium.com/@srijit29032001/this-is-how-production-grade-agentic-systems-do-rag-multi-stage-retrieval-hybrid-rag-47af29f1d656?source=rss-6d5d41398c51------2</link>
            <guid isPermaLink="false">https://medium.com/p/47af29f1d656</guid>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[chatbots]]></category>
            <dc:creator><![CDATA[Srijit Das]]></dc:creator>
            <pubDate>Sat, 07 Mar 2026 03:01:01 GMT</pubDate>
            <atom:updated>2026-03-07T03:01:01.048Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KT572HIB3xi-nebVRGmk4w.png" /></figure><h3>This is how Production grade Agentic Systems do RAG — A Multi-stage Retrieval | Hybrid RAG</h3><h3>Introduction</h3><p>Retrieval-Augmented Generation (RAG) is the industry standards now for personalized and grounded LLM outputs in sophisticated knowledge systems. The straightforward process is splitting documents into chunks, generating embeddings, storing them in a vector database, retrieving the chunks using vector similarity. This pattern is simple to implement and fits well for many use cases.</p><p>However, this approach is far away from production grade use cases.</p><p>To address these limitations, this blog implements a <strong>Hybrid RAG Engine</strong> — a multi-stage retrieval architecture that combines several retrieval techniques within a single pipeline.</p><p>The system integrates:</p><ul><li><strong>Lexical search (BM25)</strong></li><li><strong>Semantic vector retrieval</strong></li><li><strong>Rank fusion</strong></li><li><strong>Cross-encoder re-ranking</strong></li><li><strong>LLM-based context compression</strong></li></ul><p>These techniques improve <strong>recall, cost efficiency, and precision</strong> before passing context to the LLM.</p><p>A key design goal of the system is <strong>observability</strong>.</p><p>Every retrieval request is instrumented with detailed telemetry, including:</p><ul><li><strong>Per-stage latency</strong></li><li><strong>Token usage</strong></li><li><strong>Cost metrics</strong></li></ul><p>This makes it possible to <strong>evaluate and compare different retrieval strategies empirically instead of relying on intuition</strong>.</p><p>The implementation uses <strong>Node.js / Next.js</strong>, <strong>MongoDB Atlas</strong> as the storage and search backend, <strong>Google Gemini</strong> for embeddings and compression, <strong>Voyage AI</strong> for cross-encoder re-ranking, fFor evaluation, the system uses the <strong>BeIR/FIQA dataset</strong>.</p><p>The architecture itself is modular and provider-agnostic. While the current implementation uses specific tools and models, the pipeline is designed so that embedding models, vector databases, LLM providers and re-ranking services can be swapped without requiring changes to the overall architecture.</p><h4>Source Code</h4><p><a href="https://github.com/srijitasd/hybrid-rag-engine**">https://github.com/srijitasd/hybrid-rag-engine</a></p><h3>Table of Contents</h3><ol><li><a href="#43a4">Limitations of Standard RAG Retrieval</a></li><li><a href="#5573">The Case for a Multi-Stage Pipeline</a></li><li><a href="#489e">System Architecture</a></li><li><a href="#6d44">How It Works</a></li></ol><ul><li>Stage 1: Lexical Search (BM25)</li><li>Stage 2: Semantic Search (Vector)</li><li>Stage 3: Rank Fusion</li><li>Stage 4: Cross-Encoder Re-Ranking</li><li>Stage 5: Query-Focused Compression</li><li>Telemetry and Cost Tracking</li><li>Analytics Aggregation Layer</li></ul><p>5.<a href="#59c3"> Conclusion</a></p><h3>Limitations of Standard RAG</h3><p>The typical RAG retrieval process is simple. It embeds the query, runs vector search against document embeddings, takes the topK results, and feeds them into the LLM. This has several structural weaknesses that degrades retrieval quality in real world apps.</p><h4>Lexical Gaps in Dense Retrieval</h4><p>Vector search looks for meaning but discards exact lexical matches. A query like “FIQA section 4.2 compliance requirements” will be matched against chunks that are semantically similar but may miss the chunk that contains the keyword “section 4.2”.</p><p>Whereas BM25 searches based on term frequency and inverse document frequency and it has no such problem. It finds exact keyword matches.</p><p>Several research on information retrieval benchmarks like <strong>BEIR </strong>and <strong>MS MARCO </strong>found that neither lexical nor dense retrieval is useful across all types of queries. Keyword-heavy queries always favors BM25 and concept-heavy favors vectors. So we have to combine the best of both the worlds.</p><h4>The Recall-Precision Tradeoff at Top-K</h4><p>Standard vector search always retrieves a fixed topK chunks. The relevant document might be just after the topK rank and increasing top-K recovers recall but retrieves more irrelevant chunks thus increasing token cost and a higher chance to get distracted by off-topic content.</p><h4>Context Window Waste</h4><p>Even correctly retrieved chunks contain content that isn’t relevant to the specific query. A big chunk may have only one relevant sentence the rest may not be need for the response. It also increase the cost and latency. At scale, this waste is both financially and computationally meaningful.</p><h3>The Case for a Multi-Stage Pipeline</h3><p>Each limitation above maps to a specific architectural response:</p><p><strong>Limitation</strong> <strong>Response</strong> Lexical gaps in dense retrieval Parallel BM25 + vector search with rank fusion Recall-precision tradeoff Retrieve a large pool (50), then re-rank to topK (10) Bi-encoder scoring Cross-encoder re-ranking via Voyage AI for precise query-document relevance Context window waste LLM-based compression that removes irrelevant passages</p><p>The central design is about treating retrieval as a pipeline. Each stage refines the output further and this implementation shows toggle options using which system can operate as a BM25 search engine, a vector search, a hybrid with fusion, or the full five-stage pipeline with the same telemetry infrastructure.</p><p>This makes it an playground where you can run the same set of queries with different configurations, and the telemetry data will tell you exactly what each stage contributes in terms of latency, cost, and result composition.</p><h3>System Architecture</h3><pre>┌─────────────────────────────────────────────────────────────────┐<br>│                         Client Request                          │<br>│              GET /api/retrieve?q=...&amp;strategy=hybrid            │<br>└──────────────────────────┬──────────────────────────────────────┘<br>                           │<br>                    ┌──────▼──────┐<br>                    │  API Route  │  Zod validation, API key auth<br>                    └──────┬──────┘<br>                           │<br>                    ┌──────▼──────┐<br>                    │  Telemetry  │  trace_id, hrtime clocks initialized<br>                    └──────┬──────┘<br>                           │<br>          ┌────────────────┼────────────────┐<br>          ▼                ▼                ▼<br>     ┌─────────┐                      ┌──────────┐<br>     │  BM25   │                      │  Vector  │<br>     │ $search │                      │ $vector  │<br>     │ (Atlas) │                      │  Search  │<br>     └────┬────┘                      └────┬─────┘<br>          │                                │<br>          └───────────────┼────────────────┘<br>                          ▼<br>                   ┌─────────────┐<br>                   │ Rank Fusion │  RRF or Weighted<br>                   └──────┬──────┘<br>                          ▼<br>                   ┌─────────────┐<br>                   │  Re-Rank    │  Voyage AI cross-encoder (optional)<br>                   │  50 → 10    │<br>                   └──────┬──────┘<br>                          ▼<br>                   ┌─────────────┐<br>                   │  Compress   │  Gemini LLM, keep/drop (optional)<br>                   └──────┬──────┘<br>                          ▼<br>                   ┌─────────────┐<br>                   │  Log Event  │  RetrievalEvent → MongoDB<br>                   │  + Respond  │<br>                   └─────────────┘</pre><p>The codebase follows a layered architecture:</p><ul><li><strong>API routes</strong> (src/app/api/) — HTTP layer. Zod schema validation, API key auth, and response formatting.</li><li><strong>Services</strong> (src/lib/services/) — orchestration. Calls the right combination of stages based on strategy parameters.</li><li><strong>Repositories</strong> (src/lib/repositories/) — database access. MongoDB aggregation pipelines for search and analytics.</li><li><strong>Utilities</strong> (src/lib/utils/) — isolated concerns: embedding, fusion, re-ranking, compression, telemetry, auth.</li><li><strong>Models</strong> (src/lib/models/) — Mongoose schemas for the embeddings collection and the retrieval_events telemetry collection.</li></ul><h3>How It Works</h3><p>Two MongoDB Atlas Search indexes are required:</p><ul><li>bm25 — a full-text search index over the collection</li><li>vector_index — a vector search index on the embedding field</li></ul><h3>Stage 1: Lexical Search (BM25)</h3><p>BM25 search is executed via MongoDB Atlas Search’s $search aggregation stage.</p><pre>export const retrieveBM25Repo = async (query, k = 10, match = {}) =&gt; {<br>  const pipeline = [<br>    {<br>      $search: {<br>        index: &quot;bm25&quot;,<br>        text: { query, path: { wildcard: &quot;*&quot; } },<br>      },<br>    },<br>    Object.keys(match).length ? { $match: match } : null,<br>    { $limit: k },<br>    {<br>      $project: {<br>        _id: 1,<br>        text: 1,<br>        title: 1,<br>        tags: 1,<br>        bm25Score: { $meta: &quot;searchScore&quot; },<br>        createdAt: 1,<br>        meta: 1,<br>      },<br>    },<br>  ].filter(Boolean);<br>  <br>  return Embedding.aggregate(pipeline);<br>};</pre><p>BM25 is the strongest retrieval signal for queries that contain specific identifiers, acronyms, product names, or regulatory references.</p><h3>Stage 2: Semantic Search (Vector)</h3><p>For vector search, the query is first embedded using the same embedding model used during ingestion, then it is matched against stored document vectors using MongoDB’s $vectorSearch:</p><pre>export const geminiEmbed = async (text) =&gt; {<br>  const GeminiClient = getGeminiClient();<br>  const response = await GeminiClient.models.embedContent({<br>    model: &quot;gemini-embedding-001&quot;,<br>    contents: text,<br>    config: { outputDimensionality: 1536 },<br>  });<br>  return response.embeddings[0].values;<br>};</pre><pre>export const retrieveVectorRepo = async (<br>  queryEmbedding,<br>  k = 10,<br>  match = {},<br>) =&gt; {<br>  const pipeline = [<br>    {<br>      $vectorSearcjsh: {<br>        index: &quot;vector_index&quot;,<br>        path: &quot;embedding&quot;,<br>        queryVector: queryEmbedding,<br>        numCandidates: k,<br>        limit: k,<br>      },<br>    },<br>    Object.keys(match).length ? { $match: match } : null,<br>    { $limit: k },<br>    {<br>      $project: {<br>        _id: 1,<br>        text: 1,<br>        title: 1,<br>        tags: 1,<br>        vectorScore: { $meta: &quot;vectorSearchScore&quot; },<br>        createdAt: 1,<br>        meta: 1,<br>      },<br>    },<br>  ].filter(Boolean);<br><br>  return Embedding.aggregate(pipeline);<br>};</pre><p>Vector search captures conceptual and semantic similarity.</p><h3>Stage 3: Rank Fusion</h3><p>When the strategy is hybrid, both BM25 and vector search are executed producing two ranked lists. Then both of the lists are merged into a single ranked output. There are two widely used fusion methods:</p><p><strong>Reciprocal Rank Fusion (RRF)</strong> assigns each document a score based on its rank position in each list:</p><pre>fusedScore = Σ (1 / (k + rank_l))</pre><p>where k is a smoothing constant (default 60). A document appearing at rank 3 in both lists scores higher than a document at rank 1 in one list and absent from the other. As it is rank-based it does not require normalizing scores across different retrieval methods.</p><pre>const fuseRRF = (bm25Hits = [], vectorHits = [], opts = {}) =&gt; {<br>  const k0 = Number.isFinite(opts.k0) ? opts.k0 : 60;<br>  const topK = Number.isFinite(opts.topK) ? opts.topK : 10;<br>  const map = new Map();<br>  const addList = (hits, source) =&gt; {<br>    for (let i = 0; i &lt; hits.length; i++) {<br>      const h = hits[i];<br>      const key = String(h._id);<br>      const rank = h.rank ?? i + 1;<br>      if (!map.has(key)) {<br>        map.set(key, {<br>          _id: h._id,<br>          text: h.text,<br>          bm25Score: undefined,<br>          vectorScore: undefined,<br>          fusedScore: 0,<br>          title: h.title,<br>          tags: h.tags,<br>          ranks: {},<br>        });<br>      }<br>      const row = map.get(key);<br>      if (source === &quot;bm25&quot;) row.bm25Score = h.bm25Score;<br>      if (source === &quot;vector&quot;) row.vectorScore = h.vectorScore;<br>      row.ranks[source] = rank;<br>      row.fusedScore += 1 / (k0 + rank);<br>    }<br>  };<br>  addList(bm25Hits, &quot;bm25&quot;);<br>  addList(vectorHits, &quot;vector&quot;);<br>  return Array.from(map.values())<br>    .sort((a, b) =&gt; b.fusedScore - a.fusedScore)<br>    .slice(0, topK)<br>    .map((x, idx) =&gt; ({ ...x, rank: idx + 1 }));<br>};</pre><p>RRF is simple and it does not require score calibration.It also consistently outperforms single-signal retrieval on heterogeneous benchmarks.</p><h3>Stage 4: Cross-Encoder Re-Ranking</h3><p>The fusion stage produces a merged ranked list, but the underlying scores are still from bi-encoders. Neither of them evaluates the query and a specific document <strong>jointly</strong>.</p><p>A cross-encoder takes the query and a candidate document as a single input pair and outputs a fine-grained relevance score. This is very accurate but computationally expensive so we do not run of the full corpus. We first retrieve a large candidate pool then re-rank them down to topK.</p><p>The solution is the <strong>retrieve-then-rerank</strong> pattern: retrieve a large candidate pool (preRerankK = 50), then re-rank them down to topK = 10.</p><pre>export async function voyageRerank(query, candidates = [], opts = {}) {<br>  const model = opts.model || &quot;rerank-2.5-lite&quot;;<br>  const preRerankK = Number.isFinite(opts.preRerankK) ? opts.preRerankK : 50;<br>  const topK = Number.isFinite(opts.topK) ? opts.topK : 10;<br>  const maxDocChars = Number.isFinite(opts.maxDocChars)<br>    ? opts.maxDocChars<br>    : 1500;<br>  const sliced = candidates.slice(0, preRerankK);<br>  const docs = sliced.map((c) =&gt; trimText(c.text, maxDocChars));<br>  const payload = {<br>    query,<br>    documents: docs,<br>    model,<br>    top_k: Math.min(topK, docs.length),<br>  };<br>  const resp = await fetch(`${VOYAGE_BASE_URL}/rerank`, {<br>    method: &quot;POST&quot;,<br>    headers: {<br>      &quot;Content-Type&quot;: &quot;application/json&quot;,<br>      Authorization: `Bearer ${process.env.VOYAGE_API_KEY}`,<br>    },<br>    body: JSON.stringify(payload),<br>  });<br>  const data = await resp.json();<br>  const scored = data.data<br>    .map((r, rank) =&gt; {<br>      const base = sliced[r.index];<br>      return {<br>        ...base,<br>        rerankScore: Number(r.relevance_score),<br>        rerankRank: rank + 1,<br>      };<br>    })<br>    .sort((a, b) =&gt; b.rerankScore - a.rerankScore)<br>    .slice(0, topK);<br>  return { reranked: scored, model, usage: data.usage };<br>}</pre><p>After this the documents that the bi-encoder scored highly but aren’t actually relevant to the specific query get pushed down and the most relevant documents gets ranked higher. The Voyage API also returns token usage, which feeds directly into cost telemetry.</p><h3>Stage 5: Query-Focused Compression</h3><p>After re-ranking, the top-K chunks still contain full passage text and most of it is still irrelevant. Compression addresses this by running each chunk through a LLM with a query-focused prompt that classifies each chunk as keep or drop, and condenses the keep chunks to just the relevant sentences.</p><pre>export const compressWithLLM = async (<br>  telemetry,<br>  query,<br>  hits = [],<br>  opts = {},<br>) =&gt; {<br>  const GeminiClient = getGeminiClient();<br>  const model = opts.model || &quot;gemini-2.5-flash&quot;;<br>  const items = hits.slice(0, maxChunks).map((h) =&gt; ({<br>    chunk_id: String(h.chunkId),<br>    text: trimText(h.text, maxDocChars),<br>  }));<br>  const response = await GeminiClient.models.generateContent({<br>    model,<br>    contents: [<br>      {<br>        role: &quot;user&quot;,<br>        parts: [<br>          {<br>            text:<br>              `Compress these chunks for the query. Return JSON that matches the schema.\\n\\n` +<br>              JSON.stringify({ query, chunks: items }),<br>          },<br>        ],<br>      },<br>    ],<br>    systemInstruction: {<br>      parts: [{ text: systemPrompt }],<br>    },<br>    config: {<br>      temperature: 0,<br>      responseMimeType: &quot;application/json&quot;,<br>      responseJsonSchema,<br>    },<br>  });<br>};</pre><p>The final output is the final context that can be delivered to the generation LLM. It is a fraction of the original passage length and contains only the information that directly answers the query. This stage reduces token cost in the generation step and reduces the chance of the LLM being distracted by irrelevant context.</p><h3>Telemetry and Cost Tracking</h3><p>Every retrieval request is instrumented from the moment it enters the service layer. A telemetry object is initialized with high-resolution timers and accumulates measurements at each stage:</p><pre>function createRetrievalTelemetry({ query, params, userId, sessionId }) {<br>  const traceId = crypto.randomUUID();<br>  const latency = {<br>    total_ms: 0,<br>    bm25_ms: 0,<br>    embed_ms: 0,<br>    vector_ms: 0,<br>    fusion_ms: 0,<br>    rerank_ms: 0,<br>    compression_ms: 0,<br>  };<br>  const tokens = {<br>    embedding: { input: 0, output: 0 },<br>    rerank: { input: 0, output: 0 },<br>    compression: { input: 0, output: 0 },<br>  };<br>  const cost = {<br>    embedding_usd: 0,<br>    rerank_usd: 0,<br>    compression_usd: 0,<br>    total_usd: 0,<br>    pricing: {},<br>  };<br>  const results = {<br>    bm25: [],<br>    vector: [],<br>    fused: [],<br>    reranked: [],<br>    compressed: [],<br>  };<br>  // ...<br>}java</pre><p>At the end of each request, the telemetry is saved as a RetrievalEvent document:</p><pre>const RetrievalEventSchema = new mongoose.Schema({<br>  trace_id: String,<br>  created_at: Date,<br>  user_id: String,<br>  session_id: String,<br>  query_hash: String,             // SHA-256 of the query<br>  query_text: String,<br>  params: {<br>    strategy: String,             // bm25 | vector | hybrid<br>    fusion: String,               // rrf | weighted<br>    rerank: Boolean,<br>    compress: Boolean,<br>    topK: Number,<br>    preRerankK: Number,<br>  },<br>  latency_ms: StageLatencySchema, // per-stage breakdown in ms<br>  tokens: { ... },                // per-stage token counts<br>  cost: StageCostSchema,          // per-stage cost in USD<br>  results: ResultsSchema,         // logged hits at each pipeline stage<br>  final_answer: String,<br>  error: { message: String, stack: String },<br>});</pre><p>The telemetry design enables empirical comparison of any two configurations. Run the same query set against {strategy: &quot;hybrid&quot;, rerank: false} and {strategy: &quot;hybrid&quot;, rerank: true}, and the stored events provide p50/p95 latency, mean cost per query, and per-stage breakdowns — without any additional instrumentation.</p><h3>Analytics Aggregation Layer</h3><p>The telemetry data is surfaced through three analytics API endpoints, each backed by MongoDB aggregation pipelines:</p><p><strong>Strategy Comparison</strong> (GET /api/dashboard/retrieval/strategy-comparison) — groups events into five logical configurations (BM25, vector, hybrid, hybrid+rerank, hybrid+rerank+compress) and computes per-group p50/p95 latency, mean cost, average per-stage latency, and average token counts.</p><p><strong>Stage Breakdown</strong> (GET /api/dashboard/retrieval/breakdown) — computes the average latency and cost for each pipeline stage, both as absolute values and as percentage shares of total. This identifies bottlenecks — for example, revealing that embedding accounts for 60% of total latency, or that re-ranking cost dominates the per-query spend.</p><p><strong>P95 Trend</strong> (GET /api/dashboard/retrieval/p95-trend) — computes daily p95 and p50 latency with optional splitting by configuration. This surfaces degradation over time and allows comparison of how different configurations trend under varying load.</p><p>All three endpoints accept the same filter set:</p><pre>const schema = z.object({<br>  days: z.coerce.number().int().min(1).max(365).default(7),<br>  strategy: z.enum([&quot;bm25&quot;, &quot;vector&quot;, &quot;hybrid&quot;]).optional(),<br>  fusion: z.string().optional(),<br>  rerank: z.enum([&quot;true&quot;, &quot;false&quot;]).optional(),<br>  compress: z.enum([&quot;true&quot;, &quot;false&quot;]).optional(),<br>});</pre><p>The aggregation work — percentile computation, grouping, normalization — is done entirely server-side in MongoDB, using $percentile, $group, and $dateTrunc operators.</p><h3>Conclusion</h3><p>Standard single-stage vector retrieval is a good for POCs and small scale applications but it id highly inefficient and useless for large scale applications. This engine demonstrates that a multi-stage pipeline, lexical + semantic search, rank fusion, cross-encoder re-ranking, and LLM compression addressing the key structural weaknesses in a composable, measurable way.</p><p>The critical findings from building this system:</p><ol><li><strong>Hybrid search outperforms either signal alone.</strong> BM25 captures lexical precision that dense retrieval misses. Dense retrieval captures semantic relationships that BM25 misses. RRF fusion combines both without requiring score calibration.</li><li><strong>Re-ranking is the highest-impact stage.</strong> Cross-encoder scoring of 50 candidates down to 10 produces a meaningfully different ranking than the initial bi-encoder sort. The latency cost is real (~100–300ms for Voyage AI), but the precision improvement is significant.</li><li><strong>Compression reduces downstream cost, not just context size.</strong> By stripping irrelevant content before it reaches the generation LLM, compression reduces token usage and the probability of the model attending to off-topic passages. The tradeoff is an additional LLM call with its own latency and cost.</li><li><strong>You can’t optimize what you don’t measure.</strong> The per-stage telemetry is what makes all the above claims verifiable. Without it, choosing between configurations is guesswork. With it, decisions are data-driven.</li></ol><p>The entire pipeline is configurable through query parameters on a single endpoint. Toggle strategies, enable or disable re-ranking and compression, and the telemetry captures the full picture for comparison.</p><p>Source code and setup instructions: <a href="https://github.com/srijitasd/hybrid-rag-engine">github.com/srijitasd/hybrid-rag-engine</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=47af29f1d656" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Structured Logging with CloudWatch: A Comprehensive Guide]]></title>
            <link>https://medium.com/@srijit29032001/structured-logging-with-cloudwatch-a-comprehensive-guide-948547e5cb05?source=rss-6d5d41398c51------2</link>
            <guid isPermaLink="false">https://medium.com/p/948547e5cb05</guid>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[hacking]]></category>
            <category><![CDATA[logging]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[testing]]></category>
            <dc:creator><![CDATA[Srijit Das]]></dc:creator>
            <pubDate>Wed, 19 Feb 2025 04:39:42 GMT</pubDate>
            <atom:updated>2025-02-19T04:39:42.678Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d0pVa-mUnDdSd5Ki9Xy2Mw.jpeg" /></figure><h3>Mastering Observability and Debugging in Modern Applications</h3><p>Effective logging is crucial for maintaining and troubleshooting modern applications. In this post, we’ll explore how structured logging, combined with Amazon CloudWatch, can transform your application observability and make debugging a breeze.</p><h3>Table of Contents</h3><ul><li><a href="#88af">Understanding Structured Logging</a></li><li><a href="#4c5c">Why Use CloudWatch</a></li><li><a href="#0059">Real-World Use Cases</a></li><li><a href="#c116">Implementation Example</a></li><li><a href="#3308">Benefits in Production Environments</a></li><li><a href="#a062">CloudWatch Integration Features</a></li><li><a href="#2bdb">Cost Optimization</a></li><li><a href="#c7b6">Best Practices For Structured Logging</a></li><li><a href="#383b">Conclusion</a></li><li><a href="#95fc">Additional Resources</a></li></ul><h3>Understanding Structured Logging</h3><p>Traditional logging often relies on unstructured text messages that are difficult to parse and analyze. Structured logging organizes log data into consistent, machine-readable formats like JSON, making logs easier to search, filter, and analyze programmatically.</p><h3>Unstructured vs. Structured Logging</h3><pre>// Unstructured log<br>[2024-01-15 10:15:23] Error processing order #12345 for customer Smith: Invalid payment method<br><br>// Structured log<br>{<br>    &quot;timestamp&quot;: &quot;2024-01-15T10:15:23Z&quot;,<br>    &quot;level&quot;: &quot;ERROR&quot;,<br>    &quot;event&quot;: &quot;order_processing_failed&quot;,<br>    &quot;orderId&quot;: &quot;12345&quot;,<br>    &quot;customerName&quot;: &quot;Smith&quot;,<br>    &quot;errorCode&quot;: &quot;INVALID_PAYMENT_METHOD&quot;,<br>    &quot;service&quot;: &quot;payment-processor&quot;,<br>    &quot;version&quot;: &quot;1.2.3&quot;<br>}</pre><p>Structured logs like the one above provide rich context, aiding in precise filtering and rapid debugging.</p><h3>Why Use CloudWatch?</h3><p>Amazon CloudWatch stands out as a powerful logging and monitoring service for modern applications. Here are some key reasons to leverage CloudWatch:</p><h4>1. Seamless Integration with AWS Services</h4><p>CloudWatch integrates effortlessly with AWS services like Lambda, EC2, RDS, and API Gateway, making it easy to monitor your entire AWS ecosystem.</p><h4>2. Centralized Log Management</h4><p>Aggregate logs from multiple sources and regions into a single platform for unified visibility and management.</p><h4>3. Advanced Query and Filtering</h4><p>Use CloudWatch Logs Insights to query and analyze logs in real-time, enabling rapid troubleshooting and data-driven decisions.</p><h4>4. Custom Metrics and Alarms</h4><p>Transform log data into actionable insights by creating custom metrics and setting alarms for critical events.</p><h4>5. Cost-Effective Scaling</h4><p>CloudWatch scales automatically with your workload, ensuring that you only pay for what you use.</p><h4>6. Enhanced Security and Compliance</h4><p>Ensure compliance with industry standards by leveraging CloudWatch’s secure logging and audit trail features.</p><h3>Real-World Use Cases</h3><p>Structured logging enables tailored solutions across industries. Here are a few examples:</p><h4>1. E-commerce Order Processing</h4><ul><li>Track order lifecycle across microservices.</li><li>Monitor payment processing success rates.</li><li>Analyze order patterns to identify customer behavior trends.</li></ul><p><em>Example:</em> “An online retailer used structured logging to identify a 5% drop in order processing speeds, leading to optimized server configurations and a 10% boost in efficiency.”</p><h4>2. Financial Transactions</h4><ul><li>Create audit trails for compliance.</li><li>Detect anomalies in transactions.</li><li>Monitor fraud in real-time.</li></ul><h4>3. API Gateway Management</h4><ul><li>Monitor API usage patterns.</li><li>Identify rate-limiting issues.</li><li>Analyze performance metrics for scaling.</li></ul><h4>4. User Authentication System</h4><ul><li>Detect failed login attempts and track password reset flows.</li><li>Investigate security breaches by correlating logs.</li></ul><h3>Implementation Example</h3><p>Here’s how you can implement structured logging in a TypeScript-based application.</p><h4>Setting Up the Logger</h4><pre>// types.ts<br>interface LogContext {<br>  correlationId: string;<br>  service: string;<br>  environment: string;<br>  [key: string]: any;<br>}<br>// logger.ts<br>import { CloudWatchLogs } from &#39;aws-sdk&#39;;<br>import * as winston from &#39;winston&#39;;<br><br>export class StructuredLogger {<br>  private logger: winston.Logger;<br>  private context: LogContext;<br><br>constructor(context: LogContext) {<br>    this.context = context;<br>    this.logger = winston.createLogger({<br>      format: winston.format.combine(<br>        winston.format.timestamp(),<br>        winston.format.json()<br>      ),<br>      transports: [<br>        new winston.transports.Console(),<br>        new winston.transports.CloudWatch({<br>          logGroupName: `/aws/${context.environment}/${context.service}`,<br>          logStreamName: `application-${new Date().toISOString().split(&#39;T&#39;)[0]}`,<br>          awsRegion: process.env.AWS_REGION<br>        })<br>      ]<br>    });<br>  }<br>  public log(level: &#39;info&#39; | &#39;warn&#39; | &#39;error&#39; | &#39;debug&#39;, message: string, additionalData?: Record&lt;string, any&gt;): void {<br>    const logEntry = {<br>      ...this.context,<br>      level: level.toUpperCase(),<br>      message,<br>      ...additionalData,<br>      timestamp: new Date().toISOString()<br>    };<br>    switch (level) {<br>      case &#39;info&#39;:<br>        this.logger.info(logEntry);<br>        break;<br>      case &#39;warn&#39;:<br>        this.logger.warn(logEntry);<br>        break;<br>      case &#39;error&#39;:<br>        this.logger.error(logEntry);<br>        break;<br>      case &#39;debug&#39;:<br>        this.logger.debug(logEntry);<br>        break;<br>    }<br>  }<br>}</pre><h4>Usage Example</h4><pre>const logger = new StructuredLogger({<br>  correlationId: &quot;12345&quot;,<br>  service: &quot;auth-service&quot;,<br>  environment: &quot;production&quot;<br>});<br><br>// Logging different levels<br>logger.log(&#39;info&#39;, &#39;User login successful&#39;, { userId: &quot;user123&quot; });<br>logger.log(&#39;error&#39;, &#39;Failed to process payment&#39;, { orderId: &quot;order456&quot;, error: &quot;Invalid card&quot; });</pre><h3>Benefits in Production Environments</h3><h4>1. Improved Incident Response</h4><ul><li>Faster root cause analysis using structured queries.</li><li>Correlation across services for end-to-end context.</li></ul><h4>2. Security and Compliance</h4><ul><li>Automate compliance reporting.</li><li>Maintain secure audit trails.</li></ul><h4>3. Performance Optimization</h4><ul><li>Identify bottlenecks with pattern analysis.</li><li>Plan capacity using performance trends.</li></ul><h4>4. Business Intelligence</h4><ul><li>Track feature usage and user behavior analytics.</li></ul><h3>CloudWatch Integration Features</h3><h4>Log Insights Queries</h4><p>Powerful search syntax for analyzing CloudWatch logs in real-time. Use these queries to investigate patterns, troubleshoot issues, and extract valuable data from your structured logs.</p><pre># Find failed login attempts by IP<br>fields @timestamp, userId, ip, userAgent<br>| filter event = &#39;user_authentication&#39;<br>    and action = &#39;LOGIN&#39;<br>    and status = &#39;FAILURE&#39;<br>| stats count(*) as failedAttempts by ip<br>| sort failedAttempts desc<br>| limit 10</pre><h4>Metric Filters</h4><p>Transform log data into numerical metrics that can be used for monitoring, alerting, and dashboards. These filters allow you to create custom metrics based on the occurrence or values of specific fields in your logs.</p><pre>{<br>  &quot;filterPattern&quot;: &quot;{ $.event = user_authentication &amp;&amp; $.status = FAILURE }&quot;,<br>  &quot;metricName&quot;: &quot;FailedLoginAttempts&quot;,<br>  &quot;metricNamespace&quot;: &quot;SecurityMetrics&quot;,<br>  &quot;metricValue&quot;: &quot;1&quot;<br>}</pre><h3>Cost Optimization</h3><h4>Smart Logging Levels</h4><ul><li>Use INFO and above in production</li><li>WARN: Issues that need attention but don&#39;t affect core functionality</li><li>ERROR: Critical issues requiring immediate investigation</li><li>Skip DEBUG logs in production to reduce noise and costs</li></ul><h4><strong>Error logs retention strategy</strong></h4><ul><li>Keep ERROR logs for longer retention periods (e.g., 90 days or more)</li><li>Include full stack traces and context for troubleshooting</li><li>Store errors with correlation IDs for request tracing</li></ul><h4>Log Aggregation</h4><ul><li>Batch logs for high-volume events</li><li>Group similar logs within a time window (e.g., 30 seconds)</li><li>Aggregate counts instead of individual events</li><li>Perfect for metrics like API calls or page views</li></ul><h3>Best Practices for Structured Logging</h3><h4>Standardization</h4><p>Consistent logging patterns across your application ecosystem:</p><ul><li>Use consistent field names.</li></ul><pre>// Good - Consistent naming<br>{<br>  userId: &quot;123&quot;,<br>  orderAmount: 99.99,<br>  paymentStatus: &quot;success&quot;<br>}<br><br>// Bad - Inconsistent naming<br>{<br>  user_id: &quot;123&quot;,<br>  amount: 99.99,<br>  payment_state: &quot;success&quot;<br>}</pre><ul><li>Define mandatory fields in logging conventions.</li></ul><pre>interface BaseLogStructure {<br>  timestamp: string;<br>  correlationId: string;<br>  service: string;<br>  environment: string;<br>  event: string;<br>  level: LogLevel;<br>}<br><br>// Example usage<br>logger.info({<br>  ...baseLogFields,<br>  event: &#39;order_created&#39;,<br>  orderId: &#39;ord_123&#39;,<br>  customData: {<br>    // Additional context<br>  }<br>});</pre><h4>Security</h4><p>Protect sensitive information while maintaining observability:</p><ul><li>Mask sensitive data like PII automatically.</li><li>Implement role-based access controls for logs.</li></ul><h4>Performance</h4><p>Optimize logging for high-throughput production environments:</p><ul><li>Use asynchronous logging.</li><li>Buffer and batch logs to minimize network overhead.</li></ul><h3>Conclusion</h3><p>Structured logging with CloudWatch provides a robust foundation for modern application observability. By combining TypeScript’s type safety with structured logging, you can build scalable, secure, and efficient logging systems.</p><h3>Additional Resources</h3><ul><li><a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html">CloudWatch Logs Documentation</a></li><li><a href="https://github.com/winstonjs/winston">Winston Documentation</a></li><li><a href="https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/">AWS SDK for JavaScript</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=948547e5cb05" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Secure Authentication System with Node.js (TypeScript)]]></title>
            <link>https://medium.com/@srijit29032001/building-a-secure-authentication-system-with-node-js-typescript-964d54c27078?source=rss-6d5d41398c51------2</link>
            <guid isPermaLink="false">https://medium.com/p/964d54c27078</guid>
            <category><![CDATA[expressjs]]></category>
            <category><![CDATA[authentication]]></category>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[typescript]]></category>
            <dc:creator><![CDATA[Srijit Das]]></dc:creator>
            <pubDate>Sat, 11 Jan 2025 05:58:00 GMT</pubDate>
            <atom:updated>2025-01-11T05:58:00.386Z</atom:updated>
            <content:encoded><![CDATA[<p>In this guide, we will break down the process of building a secure and efficient authentication system. By focusing on practical implementation, this guide aims to help developers to build a secure, multi-device authentication system, avoid common pitfalls and ensure scalability.</p><h3><strong>Challenges in Modern Authentication</strong></h3><ul><li>Efficiently managing user sessions across multiple devices.</li><li>Preventing unauthorized access and session hijacking.</li><li>Handling concurrent sessions effectively.</li><li>Ensuring an audit trail for compliance check.</li><li>Balancing robust security with a seamless user experience.</li></ul><h3><strong>Understanding the Data Flow</strong></h3><p>Here’s the flow of operations when a user registers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wNNK-xoj2XzZ3PgR0HLDnw.png" /></figure><ol><li>The client sends a request with device information (x-device-id and x-platform headers)</li><li>Middleware validates and enriches the request with device information.</li><li>The authentication service:<br>- Validates user credentials<br>- Generates both access and refresh tokens<br>- Creates a session record in MongoDB<br>- Caches session data in Redis for quick access</li><li>The server responds with user data and tokens.</li></ol><h4><strong>For subsequent requests:</strong></h4><ol><li>The client includes the access token in the Authorization header.</li><li>For token validation:<br>- First, Redis for cached session data (fast lookup).<br>- If not found, MongoDB is queried as a fallback.<br>- Valid sessions are re-cached in Redis</li><li>If the access token expires, the client can use the refresh token to get a new one.</li><li>Each refresh operation creates a new session record, maintaining our security audit trail.</li></ol><p>This multi-layered approach ensures both speed and security, with Redis handling our high-throughput operations while MongoDB maintains a complete audit history.</p><h3><strong>The Architecture</strong></h3><p>Our authentication system uses a multi-layered approach combining Redis for performance and MongoDB for persistence. Here’s the breakdown:</p><h4><strong>1. Session Model</strong></h4><p>The foundation of our system is a well-structured session model:</p><pre>interface SessionInterface {<br>userId: Types.ObjectId;<br>jti: string;<br>refreshToken: string;<br>device: string;<br>ip: string;<br>createdAt: Date;<br>expiresAt: Date;<br>}<br>const sessionSchema = new Schema&lt;SessionInterface&gt;({<br>userId: { type: Schema.Types.ObjectId, ref: &quot;User&quot;, required: true },<br>jti: { type: String, required: true, unique: true },<br>refreshToken: { type: String, required: true },<br>device: { type: String },<br>ip: { type: String },<br>createdAt: { type: Date, default: Date.now },<br>expiresAt: { type: Date, required: true },<br>});ty</pre><p>This model captures everything we need to track a user session:</p><ul><li>Unique session identifier (jti)</li><li>Device information for security</li><li>IP address for audit trails</li><li>Expiration handling for automatic cleanup</li></ul><h4><strong>2. Device Fingerprinting</strong></h4><p>We implement middleware to capture device information:</p><pre>export const deviceMiddleware: RequestHandler = (req, res, next) =&gt; {<br>  const deviceId = req.headers[&quot;x-device-id&quot;];<br>  const platform = req.headers[&quot;x-platform&quot;];<br><br>  // Validate the presence of required headers<br>  if (!deviceId || deviceId.length === 0) {<br>    res.status(400).json({<br>      error: &quot;Missing &#39;x-device-id&#39; in request headers&quot;,<br>    });<br>    return;<br>  }<br>  if (!platform || platform.length === 0) {<br>    res.status(400).json({<br>      error: &quot;Missing &#39;x-platform&#39; in request headers&quot;,<br>    });<br>    return;<br>  }<br><br>  // Attach headers to the request object<br>  req.deviceInfo = {<br>    deviceId: deviceId as string | string[],<br>    platform: platform as string | string[],<br>  };<br><br>  next();<br>};</pre><p>This might seem simple, but it’s crucial for security. We can track suspicious login attempts and block compromised devices instantly.</p><h4><strong>3. User Registration Flow</strong></h4><p>The registration process is where things get interesting. We don’t just create a user — we establish a complete identity with location data and device information:</p><pre>const createUser = async (<br>  user_data: RegisterUserRequest,<br>  metadata: ApiMetadata<br>): Promise&lt;{<br>  user: UserInterface;<br>  accessToken: string;<br>  refreshToken: string;<br>}&gt; =&gt; {<br>  try {<br>    const user = new User(user_data);<br>    await user.save();<br><br>    const jti = uuidv4();<br>    const { accessToken, refreshToken } = generateTokens(user);<br>    await createSession({ id: user._id }, jti, refreshToken, {<br>      device: metadata.device,<br>      ip: metadata.ip,<br>    });<br><br>    return { user, accessToken, refreshToken };<br>  } catch (error) {<br>    throw error;<br>  }<br>};</pre><h4><strong>4. Session Management</strong></h4><p>Here’s where we do something clever — we use both Redis and MongoDB for session management. But why?</p><pre>const createSession = async (<br>  user: SessionUser,<br>  jti: string,<br>  refreshToken: string,<br>  metadata: ApiMetadata<br>): Promise&lt;{ jti: string; refreshToken: string }&gt; =&gt; {<br>  const expiresAt = new Date(Date.now() + 15 * 60 * 1000); // 15 minutes<br>  const sessionData = {<br>    userId: user.id,<br>    jti,<br>    refreshToken,<br>    device: metadata.device || &quot;unknown&quot;,<br>    ip: metadata.ip || &quot;0.0.0.0&quot;,<br>    expiresAt,<br>  };<br><br>  // Save in DB<br>  const session = new Session(sessionData);<br>  await session.save();<br><br>  // Cache in Redis<br>  await cacheSession(user.id.toString(), jti, {<br>    refreshToken: refreshToken,<br>    device: sessionData.device,<br>    ip: sessionData.ip,<br>    createdAt: Date.now(),<br>    expiresAt: expiresAt.toISOString(),<br>  });<br><br>  return { jti, refreshToken };<br>};</pre><p>Redis handles our high-traffic operations like token validation, while MongoDB maintains our audit trail. It’s like having a speed demon and a record keeper working together.</p><h3><strong>Implementation Details</strong></h3><h4><strong>1. Token Generation</strong></h4><pre>const generateTokens = (user: GenerateToken, jti: string) =&gt; {<br>  const userPayload = { id: user.id, email: user.email }; // Customize payload as needed<br>  const accessToken = jwt.sign(<br>    userPayload,<br>    process.env.ACCESS_TOKEN_SECRET as string,<br>    {<br>      expiresIn: process.env.ACCESS_TOKEN_LIFE as string,<br>    }<br>  );<br>  const refreshToken = jwt.sign(<br>    { ...userPayload, jti },<br>    process.env.REFRESH_TOKEN_SECRET as string,<br>    {<br>      expiresIn: process.env.REFRESH_TOKEN_LIFE,<br>    }<br>  );<br><br>  return { accessToken, refreshToken };<br>};</pre><h4><strong>2. Session Validation</strong></h4><pre>const validateSession = async (<br>  user: SessionUser,<br>  providedRefreshToken: string<br>): Promise&lt;ValidSessionResponse | void&gt; =&gt; {<br>  // Check if refresh token is valid<br>  let payload = verifyRefreshToken(providedRefreshToken);<br>  if (payload) {<br>    const { jti } = payload;<br><br>    // Try Redis first<br>    let sessionData = await getCachedSession(user.id, jti);<br><br>    if (!sessionData) {<br>      // Not in Redis, check DB<br>      const session = await Session.findOne({ userId: user.id, jti });<br>      if (!session || session.refreshToken !== providedRefreshToken) {<br>        throw { message: &quot;Session not found or token mismatch&quot; };<br>      }<br><br>      // Check expiration<br>      if (session.expiresAt &lt; new Date()) {<br>        await removeCachedSession(user.id, jti); // cleanup<br>        throw { message: &quot;Session expired&quot; };<br>      }<br><br>      sessionData = {<br>        refreshToken: session.refreshToken,<br>        device: session.device,<br>        ip: session.ip,<br>        createdAt: session.createdAt.toISOString(),<br>        expiresAt: session.expiresAt.toISOString(),<br>      };<br><br>      // Re-cache the session<br>      await cacheSession(user.id, jti, sessionData);<br>    }<br><br>    // Validate provided refreshToken<br>    if (sessionData.refreshToken !== providedRefreshToken) {<br>      throw { message: &quot;Invalid refresh token in cache&quot; };<br>    }<br><br>    // Check expiration from cached data<br>    if (new Date(sessionData.expiresAt) &lt; new Date()) {<br>      await removeCachedSession(user.id, jti);<br>      throw { message: &quot;Session expired&quot; };<br>    }<br><br>    return { payload, session: sessionData };<br>  }<br>};</pre><h3><strong>Performance Optimization Tips</strong></h3><h4>1. <strong>Redis Configuration</strong></h4><p>Always set appropriate TTL (Time To Live) for cached sessions:</p><pre>// Set appropriate TTL for cached sessions<br>await redis.set(sessionKey, sessionData, &quot;EX&quot;, 900); // 15 minutes</pre><h4>2. <strong>MongoDB Indexes</strong></h4><p>Don’t forget to index your session collection:</p><pre>// Create indexes for frequent queries<br>sessionSchema.index({ userId: 1, device: 1 });<br>sessionSchema.index({ expiresAt: 1 }, { expireAfterSeconds: 0 });</pre><h4>3. <strong>Batch Operations</strong></h4><p>Process deletions in batches to avoid blocking the database:</p><pre>// When cleaning up expired sessions<br>const bulkOps = expiredSessions.map((session) =&gt; ({<br>  deleteOne: {<br>    filter: { _id: session._id },<br>  },<br>}));<br>await Session.bulkWrite(bulkOps);</pre><h4>1. <strong>Lightning-Fast Performance</strong></h4><ul><li>Token validation happens in microseconds thanks to Redis</li><li>No database bottlenecks during high-traffic periods</li></ul><h4>2. <strong>Rock-Solid Security</strong></h4><ul><li>Complete device tracking</li><li>Instant session invalidation capability</li><li>Comprehensive audit trails</li></ul><h4>3. <strong>Scalability</strong></h4><ul><li>Horizontal scaling with Redis cluster</li><li>No single point of failure</li><li>Efficient resource utilization</li></ul><h3><strong>Real-world Impact</strong></h3><p>In production, this system has handled:</p><ul><li>100,000+ concurrent sessions</li><li>5ms average response time for token validation</li><li>Zero data loss during Redis failures</li><li>99.99% uptime</li></ul><h3><strong>Common Pitfalls to Avoid</strong></h3><h4>1. <strong>Session Storage</strong></h4><ul><li>Don’t store sensitive data in Redis</li><li>Keep detailed audit data in MongoDB</li><li>Set appropriate TTL values</li></ul><h4>2. <strong>Error Handling</strong></h4><ul><li>Always implement fallback strategies</li><li>Handle Redis connection failures gracefully</li><li>Log session creation/deletion events</li></ul><h4>3. <strong>Security Considerations</strong></h4><ul><li>Implement rate limiting per device</li><li>Monitor suspicious IP patterns</li><li>Regular session cleanup</li></ul><h3><strong>Future Improvements</strong></h3><p>Consider these enhancements:</p><ul><li>Add geolocation-based security</li><li>Implement rate limiting per device</li><li>Add machine learning for fraud detection</li><li>Introduce session analytics for better security insights</li></ul><h3><strong>Conclusion</strong></h3><p>Building a robust authentication system requires careful consideration of security, performance, and maintainability. By combining Redis and MongoDB with a well-structured session model, we’ve created a system that’s both secure and scalable.</p><p>The key is finding the right balance between security measures and user experience. Keep monitoring your system, gather metrics, and continuously improve based on real usage patterns.</p><p>Questions about implementing this in your project? Feel free to reach out!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=964d54c27078" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kafka messaging in Node.js- A simple weekly digest]]></title>
            <link>https://medium.com/@srijit29032001/kafka-messaging-in-node-js-a-simple-weekly-digest-1fd4af0e7f73?source=rss-6d5d41398c51------2</link>
            <guid isPermaLink="false">https://medium.com/p/1fd4af0e7f73</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[email-marketing]]></category>
            <category><![CDATA[microservices]]></category>
            <dc:creator><![CDATA[Srijit Das]]></dc:creator>
            <pubDate>Sun, 14 Apr 2024 20:08:24 GMT</pubDate>
            <atom:updated>2024-04-14T20:08:24.834Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kafka messaging in Node.js— A simple weekly digest</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*A-8uMvhCLEvOyouv" /><figcaption>Photo by <a href="https://unsplash.com/@hostreviews?utm_source=medium&amp;utm_medium=referral">Stephen Phillips - Hostreviews.co.uk</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>What are we going to build, you ask?</h3><p>We’re trying to build a small tool focused on sending weekly email content. What woke him up? This is not your old email. It is customized content based on user activity. Let me explain this in detail for you:</p><p>First of all, Node.js is the engine behind the scenes that powers our system. It does all the heavy lifting to ensure smooth operation.</p><p>Now, imagine this: Apache Kafka serves as the central hub where all the action unfolds. It functions as a pipeline for data, receiving messages from various sources — updates on user activities, intriguing content recommendations, you name it.</p><p>Next up, MongoDB serves as our dependable storage space. Here, we meticulously organize all incoming messages by user and week, making it a breeze to fetch relevant data when it’s time to send out those weekly emails.</p><p>To ensure punctual delivery, we rely on cron scheduling. Like clockwork, our system springs into action once a week, gathering all pertinent messages for the current week and beginning the process of crafting personalized emails.</p><p>Lastly, we enlist the help of Nodemailer, our trusty messenger. It takes these carefully crafted emails and ensures they reach users’ inboxes without fail.</p><h3>But what is Apache Kafka:</h3><blockquote>Apache Kafka is an open-source distributed event streaming platform designed to handle real-time data feeds efficiently and reliably. Developed by <strong>LinkedIn </strong>and later open-sourced as part of the <strong>Apache Software Foundation</strong>, Kafka provides a highly scalable, fault-tolerant, and durable messaging system. It allows seamless integration of various data sources and applications, enabling real-time data processing, stream processing, and data analytics. Kafka’s architecture is based on a distributed commit log, where data is organized into topics and stored across a cluster of servers, known as brokers. Producers publish messages to topics, and consumers subscribe to these topics to process the data. With its robust features and versatile architecture, Kafka has become a fundamental component in building modern data-driven applications, powering use cases ranging from real-time analytics, log aggregation, and event sourcing to Internet of Things (IoT) data processing and beyond.</blockquote><h3>Explain me like I am 5 years old!</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/1*xHxm-DNhNb4AIkRvjyI9kw.png" /></figure><p>Imagine you are in a big restaurant, there is a kitchen that prepares food, waiters take orders, and customers eat their meals. let’s understand this in the data processing field:</p><p><strong>Kitchen (Brokers)</strong>: Brokers in Kafka is somewhat like a kitchen in the restaurant. These are servers that store and manage data. They handle incoming data and make sure it can be consumed by the right consumer when needed.</p><p><strong>Orders (Messages)</strong>: Just like a customer place his/her order in a restaurant, in Kafka, we have messages. Messages are pieces of data like a customer’s order. It can be anything, like a transaction, a log entry, or data from a sensor in an IOT device.</p><p><strong>Waiters (Producers)</strong>: In restaurants we have waiters, in Kafka we have Producers, they are like the waiters in the restaurant. They take messages from applications and bring them to the brokers to be processed.</p><p><strong>Customers (Consumers)</strong>: Consumers are like the customers in the restaurant. They’re interested in the food (messages) being prepared in the kitchen (brokers). They tell the waiter (Kafka) what they want to eat (subscribe to a topic), and the waiter makes sure they get their food when it’s ready.</p><p><strong>Menu (Topics)</strong>: Just as a restaurant has a menu with different categories of food, Kafka has topics. Topics are like channels that organize messages. The producers within the application send messages to specific topics, and consumers listen to the topics they’re interested in.</p><p>So, in simple terms, Apache Kafka is like a restaurant for data. It receives messages, processes them, and sends them to the right consumers. Kafka is a reliable, scalable, and efficient way to handle real-time data feeds in applications.</p><p><strong>Enough introduction done, Let’s dive right into coding….</strong></p><p>To begin, ensure you have Docker installed on your system. Assuming Docker is up and running, let’s proceed with running the latest Zookeeper and Kafka images using Docker.</p><pre>docker run -p 2181:2181 zookeeper</pre><p>This command will start Zookeeper on port 2181.</p><p>Next, let’s start Kafka on port 9092 and set some essential environment variables. Run the following command:</p><pre>docker run -p 9092:9092 -e KAFKA_ZOOKEEPER_CONNECT=&lt;your private ip address&gt;:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://&lt;your private ip address&gt;:9092 -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 confluentinc/cp-kafka</pre><p>Let’s understand what each environment variable does:</p><ul><li>KAFKA_ZOOKEEPER_CONNECT: We are using this command to tell Kafka the port on which Zookeeper is running. In our case we are using port 2181 and it is also the default port of Zookeeper.</li><li>KAFKA_ADVERTISED_LISTENERS: This variable is used to define the hostname and port that external users may use to connect to the service. In our case we&#39;ve set it to PLAINTEXT://localhost:9092.</li><li>KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: This parameter is used to define the replication factor for the topics, we are setting it to 1.</li></ul><p>With these commands, you’ve successfully set up Kafka running on port 9092 with the specified environment variables.</p><p>Let’s streamline this process:</p><p>Firstly, ensure you have the necessary packages installed. Run the following command in your project directory:</p><ol><li><em>date-fns,</em></li><li><em>dotenv,</em></li><li><em>kafkajs,</em></li><li><em>mongodb,</em></li><li><em>node-cron,</em></li><li><em>nodemailer</em></li></ol><p>I know you guys are lazy, so…..</p><pre>npm i date-fns dotenv kafkajs mongodb node-cron nodemailer</pre><p>Now, let’s create the four files as described:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/224/1*i7BUEZjowEseJ4GaYOahfA.jpeg" /></figure><p><strong>kafkaClient.js: </strong>This file will handle the Kafka connection, utilized by multiple users. Create a file named kafkaClient.js and add your Kafka connection setup there.</p><p><strong>admin.js:</strong> This file will set up the infrastructure for producing and consuming messages. Create a file named admin.js and configure the necessary Kafka topics, producers, and consumers.</p><p><strong>producer.js:</strong> This file will be responsible for producing messages and sending them to consumers. Create a file named producer.js and implement the logic for producing messages.</p><p><strong>consumer.js: </strong>This file will consume messages, parse them, save them to the database, and handle cron jobs. Create a file named consumer.js and define the logic for consuming messages, parsing, database interaction, and cron job handling.</p><h3><strong>kafkaclient.js:</strong></h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d7e1f581bece1241c418ab66e3ab15f6/href">https://medium.com/media/d7e1f581bece1241c418ab66e3ab15f6/href</a></iframe><p>We’re kicking things off by instantiating a Kafka client object and setting its clientId to ‘Digest-app’ — you’re welcome to choose any name that suits your preference. Next up, we’re configuring the Kafka broker address, fetched from our environment variables. This crucial step establishes the connection to the Kafka cluster, laying the groundwork for both message production and consumption</p><h3>admin.js:</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7e7095f337bfb632c41ef9059ddf6ec0/href">https://medium.com/media/7e7095f337bfb632c41ef9059ddf6ec0/href</a></iframe><p>Let’s break it down further:</p><ol><li><strong>Importing Dependencies:</strong> We’re importing our reliable Kafka client from a local module aptly named kafkaClient.</li><li><strong>Initializing and Connecting:</strong> We swiftly initialize the admin client and give it the green light to connect to our Kafka cluster.</li><li><strong>Creating a Topic:</strong> With the admin client up and running, we dive straight into creating a Kafka topic. We specify the topic name, retrieved from environment variables, and decide on the number of partitions.</li><li><strong>Wrapping Up:</strong> Mission accomplished! It’s time to wrap things up and bid adieu to the Kafka cluster. Our admin client has successfully completed its task, and now it’s time to gracefully disconnect.</li></ol><h3>producer.js:</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d8f8cde1a9886ffacfa921aded552dda/href">https://medium.com/media/d8f8cde1a9886ffacfa921aded552dda/href</a></iframe><p>Let’s break down what’s happening in our code</p><ol><li><strong>Initializing the Producer:</strong> We’re gearing up our producer to spread the word about those exciting social media updates.</li><li><strong>Connecting to Kafka:</strong> The producer gets down to business, establishing a connection with the Kafka cluster, ready to unleash the latest updates.</li><li><strong>Sending the Message:</strong> With the connection established, we transmit our message to the ‘social-media-events’ topic. This message contains all the pertinent details about the follower update, neatly packaged in JSON format.</li><li><strong>Wrapping Up:</strong> Once the message is sent, we gracefully disconnect the producer, bringing our broadcast session to a close.</li></ol><h3>consumer.js:</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/18f72975ed9991f611d7502e765fc24d/href">https://medium.com/media/18f72975ed9991f611d7502e765fc24d/href</a></iframe><p>We’ve got all our packages and functions required, and we’ve initialized the MongoDB connection. Keeping it simple, we won’t delve into the intricacies of these straightforward processes.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ad6707419dcdd60d82a76b4efaa473eb/href">https://medium.com/media/ad6707419dcdd60d82a76b4efaa473eb/href</a></iframe><p>Now, let’s dive into connecting the consumer and setting up two essential functions:</p><ol><li><strong>Kafka Consumer Setup:</strong> We’re configuring a Kafka consumer with a designated group ID, ‘digest-email-group’. This consumer is tasked with fetching messages from Kafka topics associated with email digests.</li><li><strong>Nodemailer Configuration:</strong> Moving on, we’re setting up Nodemailer to create a transporter utilizing Gmail as the email service provider. For authentication, we fetch email credentials (username and password) from environment variables.</li><li><strong>Database Query:</strong> Lastly, we define a function called aggregateAndStore. This function plays a pivotal role in aggregating and storing data. It accepts data input, likely in a specific format, and executes a bulk write operation to store this data efficiently in a MongoDB collection.</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8f9b451988ab74b44885c2db015c3dad/href">https://medium.com/media/8f9b451988ab74b44885c2db015c3dad/href</a></iframe><p>Let’s break down the functionality of the runConsumer function:</p><p><strong>Connecting and Subscribing:</strong> The runConsumer function kicks off by establishing a connection between the consumer and the Kafka cluster. It subscribes to a specific topic, ensuring it starts consuming messages from the beginning of the topic.</p><p><strong>Consuming Messages:</strong> As the consumer operates, it defines a callback function named eachBatch to manage each batch of messages received from Kafka. Within this function:</p><ol><li><strong>Iterating over Messages:</strong> It iterates through each message in the batch, extracting the notification payload by parsing the JSON content of the message.</li><li><strong>Processing Notifications:</strong> The consumer processes each notification, aggregating updates to be stored or subjected to further processing. Upon processing, it resolves the offset of each message, signaling to Kafka that it has been successfully handled. Additionally, it sends a heartbeat to Kafka to maintain the connection.</li></ol><p><strong>Batch Processing:</strong> Once the entire batch of messages has been processed, the accumulated updates are sent to the aggregateAndStore function for storage or additional action.</p><p><strong>Execution:</strong> Finally, the runConsumer function is invoked to commence message consumption from Kafka.</p><p>With this robust process in place, our consumer efficiently handles incoming messages from Kafka, ensuring timely processing and action as necessary.</p><p>The role of <strong>Kafka </strong>completes its cycle, leaving the final task of sending emails to users at the end of the week. This task is managed seamlessly using node-cron.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/31f9a3caf49afd71cfa03c801faa5a9c/href">https://medium.com/media/31f9a3caf49afd71cfa03c801faa5a9c/href</a></iframe><h3>Here’s how it works:</h3><ol><li><strong>Scheduling with Cron:</strong> We utilize the cron library to schedule a task that runs every Saturday. (In the cron expression, ‘6’ represents Saturday.) This ensures that our weekly digest emails are dispatched with precision.</li><li><strong>Fetching Data:</strong> Within the scheduled task, we retrieve data from a MongoDB collection corresponding to the current week. We employ the getISOWeek function to ascertain the current ISO week number.</li><li><strong>Sending Emails:</strong> For each user’s data retrieved for the week, we utilize Nodemailer to dispatch a weekly digest email. The email encapsulates the user’s messages for the week, serialized as JSON within the email body.</li><li><strong>Logging Errors:</strong> To maintain transparency and vigilance, we diligently log any errors encountered during the email sending process. This ensures that we’re promptly informed of any issues requiring attention.</li></ol><p>With this streamlined process, our tool seamlessly delivers weekly digest emails to users, ensuring consistent communication and engagement. Let’s keep the momentum going!</p><h3>Conclusion:</h3><p>In conclusion, we’ve walked through a straightforward implementation of Kafka messaging in Node.js. While we’ve streamlined the process for brevity, it’s worth noting that certain steps like setting up Docker and defining schemas for MongoDB collections were omitted.</p><p>Additionally, it’s important to acknowledge that our tool currently captures only a subset of user activity — specifically, their following activity — for the sake of simplicity. However, this framework provides a solid foundation that can be extended and customized based on your unique requirements and creative vision.</p><p>As you continue to explore and expand upon this implementation, feel free to tailor it to suit your specific use cases and enhance its functionality. With Kafka’s robust messaging capabilities and the flexibility of Node.js, the possibilities are endless.</p><p>Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1fd4af0e7f73" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build a client app on Matrix network. Register user into the protocol.]]></title>
            <link>https://medium.com/@srijit29032001/build-a-client-app-on-matrix-network-register-user-into-the-protocol-27388d5263df?source=rss-6d5d41398c51------2</link>
            <guid isPermaLink="false">https://medium.com/p/27388d5263df</guid>
            <category><![CDATA[communication]]></category>
            <category><![CDATA[matrix]]></category>
            <category><![CDATA[instant-messaging]]></category>
            <category><![CDATA[app-development]]></category>
            <category><![CDATA[chat]]></category>
            <dc:creator><![CDATA[Srijit Das]]></dc:creator>
            <pubDate>Fri, 29 Dec 2023 19:20:42 GMT</pubDate>
            <atom:updated>2023-12-29T19:20:42.311Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HBD4IZFiZfj9jMZM" /><figcaption>Photo by <a href="https://unsplash.com/@robynnexy?utm_source=medium&amp;utm_medium=referral">Robynne Hu</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Let us start with a short introduction to [Matrix]:</h3><p><strong>Matrix,</strong> <em>aka</em> [<strong>matrix</strong>] is an open standard and communication protocol for real-time communication. It aims to make real-time communication work seamlessly between different service providers, in the way that standard Simple Mail Transfer Protocol email currently does for store-and-forward email service, by allowing users with accounts at one communications service provider to communicate with users of a different service provider via online chat, voice over IP, and videotelephony. It therefore serves a similar purpose to protocols like XMPP, but is not based on any existing communication protocol.</p><p>From a technical perspective, it is an application layer communication protocol for federated real-time communication. It provides HTTP APIs and open source reference implementations for securely distributing and persisting messages in JSON format over an open federation of servers.</p><h3>Let us talk about what we are building today</h3><p>In this tutorial, we will be analyzing the user registration flow. We will go through the APIs necessary for registering a user in a homeserver so that the user can be a part of the Matrix protocol.</p><p>We will gather the data required to make each API request and look at how to use the response to make subsequent API calls.</p><h3>User registration flow:</h3><p>The user registration flow is a multi-stage process where each step is necessary for the next. We have to follow a pattern to push the user on the Matrix protocol.</p><p><strong><em>Dependencies used in this demonstration:</em></strong></p><ol><li><strong>Synapse</strong> as the homeserver of the user.</li><li><strong>/_matrix/client/v3/register</strong> endpoint: <br>This API endpoint uses the <a href="https://spec.matrix.org/v1.9/client-server-api/#user-interactive-authentication-api">User-Interactive Authentication API</a>, except in the cases where a guest account is being registered.</li><li><strong><em>/_matrix/client/v3/account/3pid/email/requestToken </em></strong>endpoint for verifying the user’s email address.</li></ol><p>Registering a user on the Matrix homeserver is divided into 4 major steps.</p><h3>Step 1:</h3><p>Matrix provides a <strong>session id</strong> which is used throughout the flow. Matrix also expects the user to complete a set of authentications to really be a part of them.</p><ol><li>The first one is to stop robots from accessing the flow.</li><li>Verifying the user’s email address.</li><li>Accepting the privacy and policy of matrix homeserver.</li></ol><p>We have to make an API call to the <strong><em>/register</em></strong> endpoint to get the <strong>session id </strong>and the list of authentications we just discussed.</p><p>The API expects a query parameter, <strong>“kind” </strong>which accepts either user or guest. For the sake of simplicity, we will pass “<strong>user”</strong> in the kind query parameter.</p><pre>curl -X POST &quot;https://matrix.org/_matrix/client/v3/register?kind=user&quot; \<br> -H &quot;Accept: application/json&quot; \<br> -H &quot;Content-Type: application/json&quot;</pre><p>We also have to pass some data in the body of the request:</p><pre>{<br>  &quot;device_id&quot;: &quot;GHTYAJCE&quot;, // a unique id to identify the particular device of the user.<br>  &quot;inhibit_login&quot;: false, // toggles auto login<br>  &quot;initial_device_display_name&quot;: &quot;Jungle Phone&quot;, // A display name to assign to the newly-created device. <br>  &quot;password&quot;: &quot;ilovebananas&quot;, // password<br>  &quot;refresh_token&quot;: false, // If true, the client supports refresh tokens.<br>  &quot;username&quot;: &quot;cheeky_monkey&quot; // The basis for the localpart of the desired Matrix ID<br>}</pre><p>so the request stands as:</p><pre>curl -X POST &quot;https://matrix.org/_matrix/client/v3/register?kind=user&quot; \<br> -H &quot;Accept: application/json&quot; \<br> -H &quot;Content-Type: application/json&quot; \<br> -d &#39;{&quot;device_id&quot;:&quot;GHTYAJCE&quot;,&quot;inhibit_login&quot;:false,&quot;initial_device_display_name&quot;:&quot;Jungle Phone&quot;,&quot;password&quot;:&quot;ilovebananas&quot;,&quot;refresh_token&quot;:false,&quot;username&quot;:&quot;cheeky_monkey&quot;}&#39; \</pre><p>after calling this API, we will get:</p><pre>{<br>  &quot;session&quot;: &quot;A unique session id&quot;,<br>  &quot;flows&quot;: [<br>    {<br>      &quot;stages&quot;: [<br>        &quot;m.login.recaptcha&quot;, // 1st authentication<br>        &quot;m.login.terms&quot;, // Privacy policy acceptance step<br>        &quot;m.login.email.identity&quot; // 2st authentication<br>      ]<br>    }<br>  ],<br>  &quot;params&quot;: {<br>    &quot;m.login.recaptcha&quot;: {<br>      &quot;public_key&quot;: &quot;secret key to use as public key for google recaptcha&quot;<br>    },<br>    &quot;m.login.terms&quot;: {<br>      &quot;policies&quot;: {<br>        &quot;privacy_policy&quot;: {<br>          &quot;version&quot;: &quot;1.0&quot;,<br>          &quot;en&quot;: {<br>            &quot;name&quot;: &quot;Terms and Conditions&quot;,<br>            &quot;url&quot;: &quot;https://matrix-client.matrix.org/_matrix/consent?v=1.0&quot; // link to the terms and conditions of the matrix client<br>          }<br>        }<br>      }<br>    }<br>  }<br>}</pre><p>Now we will use this response data to make our next requests</p><h3>Step 2:</h3><p>Now we will use the <strong>session id</strong> and the recaptcha <strong>public_key</strong> to make our next API request to complete the 1st stage which is <strong>“m.login.recaptcha”.</strong></p><p>For this, we have to create a new <a href="https://www.google.com/recaptcha/admin/enterprise">google reCAPTCHA project</a> and set up keys there.</p><p>Use the reCAPTCHA <strong>public_key </strong>to get a captcha response.</p><p>Keep the captcha response handy, we will need that.</p><p>Again we will make an API call to the <strong><em>/register</em></strong> endpoint but this time with two additional keys <strong>“auth” </strong>and<strong> “session”.</strong></p><pre>{<br>  &quot;auth&quot;: {<br>     &quot;type&quot;: &quot;m.login.recaptcha&quot;, // type of the authetication we are trying to perform<br>     &quot;response&quot;: &quot;&lt;captcha response&gt;&quot;, // reCaptcha response<br>     &quot;session&quot;: &quot;session id&quot; // session id<br>  },<br>  &quot;session&quot;: &quot;session id&quot;, // session id<br><br>  // same as it was...<br>  &quot;device_id&quot;: &quot;GHTYAJCE&quot;,<br>  &quot;inhibit_login&quot;: false,<br>  &quot;initial_device_display_name&quot;: &quot;Jungle Phone&quot;,<br>  &quot;password&quot;: &quot;ilovebananas&quot;,<br>  &quot;refresh_token&quot;: false,<br>  &quot;username&quot;: &quot;cheeky_monkey&quot;<br>}</pre><p>so the request stands as:</p><pre>curl -X POST &quot;https://matrix.org/_matrix/client/v3/register?kind=user&quot; \<br> -H &quot;Accept: application/json&quot; \<br> -H &quot;Content-Type: application/json&quot; \<br> -d &#39;{&quot;auth&quot;:{&quot;type&quot;:&quot;m.login.recaptcha&quot;,&quot;response&quot;:&quot;&lt;captcha response&gt;&quot;,&quot;session&quot;:&quot;session id&quot;},&quot;session&quot;:&quot;session id&quot;,&quot;device_id&quot;:&quot;GHTYAJCE&quot;,&quot;inhibit_login&quot;:false,&quot;initial_device_display_name&quot;:&quot;Jungle Phone&quot;,&quot;password&quot;:&quot;ilovebananas&quot;,&quot;refresh_token&quot;:false,&quot;username&quot;:&quot;cheeky_monkey&quot;}&#39; \</pre><p>On successful attempt we will receive a response (do not panic if the status code is still <strong>401</strong>):</p><pre>{<br>  &quot;session&quot;: &quot;session id&quot;,<br>  &quot;flows&quot;: [<br>    {<br>      &quot;stages&quot;: [<br>        &quot;m.login.recaptcha&quot;,<br>        &quot;m.login.terms&quot;,<br>        &quot;m.login.email.identity&quot;<br>      ]<br>    }<br>  ],<br>  &quot;params&quot;: {<br>    &quot;m.login.recaptcha&quot;: {<br>      &quot;public_key&quot;: &quot;public key&quot;<br>    },<br>    &quot;m.login.terms&quot;: {<br>      &quot;policies&quot;: {<br>        &quot;privacy_policy&quot;: {<br>          &quot;version&quot;: &quot;1.0&quot;,<br>          &quot;en&quot;: {<br>            &quot;name&quot;: &quot;Terms and Conditions&quot;,<br>            &quot;url&quot;: &quot;https://matrix-client.matrix.org/_matrix/consent?v=1.0&quot;<br>          }<br>        }<br>      }<br>    }<br>  },<br>  &quot;completed&quot;: [<br>    &quot;m.login.recaptcha&quot; // now we can see recpatcha validation has been done<br>  ]<br>}</pre><p>As the <strong>“m.login.recaptcha”</strong> validation was successful we can now proceed to the next step.</p><h3>Step 3:</h3><p>In this step we have to use another API endpoint provided by Matrix to validate the email address of the user.</p><p>this API lets the homeserver check that the given email address is <strong>not</strong> already associated with an account on this homeserver. This API should be used to request validation tokens when adding an email address to an account. The homeserver will validate the email itself, either by sending a validation email itself or by using a service it has control over.</p><p>We will make a POST request with this as request body</p><pre>{<br>  &quot;client_secret&quot;: &quot;client secret&quot;, //we can generate this string and it must be a string consisting of the characters [0-9a-zA-Z.=_-]. Its length must not exceed 255 characters and it must not be empty.<br>  &quot;email&quot;: &quot;user&#39;s email id&quot;, // pass the email id of the user<br>  &quot;send_attempt&quot;: 1 // The server will only send an email if the send_attempt is a number greater than the most recent one which it has seen, scoped to that email + client_secret pair.<br>}</pre><p>so the request stands as:</p><pre>curl -X POST &quot;https://matrix.org/_matrix/client/v3/account/3pid/email/requestToken&quot; \<br> -H &quot;Accept: application/json&quot; \<br> -H &quot;Content-Type: application/json&quot; \<br> -d &#39;{&quot;client_secret&quot;:&quot;secret&quot;,&quot;email&quot;:&quot;user&#39;s email id&quot;,&quot;send_attempt&quot;:1}&#39; \</pre><p><strong>Note:</strong> We have to store the client secret as well for future reference.</p><p>after successful response we will receive:</p><pre>{<br>  &quot;sid&quot;: &quot;OvHekMIKKsEGjEqh&quot; // keep this sid handy as we will need this for the next API request<br>}</pre><p><strong>We have to prompt the user to check their inbox and verify their email address.</strong></p><p>Now we have to call the <strong>/register</strong> API again but with slightly different request body:</p><pre>{<br>  &quot;auth&quot;: {<br>    &quot;type&quot;: &quot;m.login.email.identity&quot;, // type of the authetication we are trying to perform<br>    &quot;threepid_creds&quot;: {<br>      &quot;sid&quot;: &quot;&lt;identity server session id&gt;&quot;, // sid we got in the previous response<br>      &quot;client_secret&quot;: &quot;&lt;identity server client secret&gt;&quot;, // the client secret we used in the previous request<br>    },<br>    &quot;session&quot;: &quot;session id&quot; // our old session id<br>  },<br>  &quot;session_id&quot;: &quot;session id&quot;,<br>  &quot;device_id&quot;: &quot;GHTYAJCE&quot;,<br>  &quot;inhibit_login&quot;: false,<br>  &quot;initial_device_display_name&quot;: &quot;Jungle Phone&quot;,<br>  &quot;password&quot;: &quot;ilovebananas&quot;,<br>  &quot;refresh_token&quot;: false,<br>  &quot;username&quot;: &quot;cheeky_monkey&quot;<br>}</pre><p>So the request stands as:</p><pre>curl -X POST &quot;https://matrix.org/_matrix/client/v3/register?kind=user&quot; \<br> -H &quot;Accept: application/json&quot; \<br> -H &quot;Content-Type: application/json&quot; \<br> -d &#39;{&quot;auth&quot;:{&quot;type&quot;:&quot;m.login.email.identity&quot;,&quot;threepid_creds&quot;:{&quot;sid&quot;:&quot;sid&quot;,&quot;client_secret&quot;:&quot;client secret&quot;},&quot;session&quot;:&quot;session id&quot;},&quot;session_id&quot;:&quot;session id&quot;,&quot;device_id&quot;:&quot;GHTYAJCE&quot;,&quot;inhibit_login&quot;:false,&quot;initial_device_display_name&quot;:&quot;Jungle Phone&quot;,&quot;password&quot;:&quot;ilovebananas&quot;,&quot;refresh_token&quot;:false,&quot;username&quot;:&quot;cheeky_monkey&quot;}&#39; \</pre><p>On successful attempt we will receive an response (do not panic if the status code is still <strong>401</strong>):</p><pre>{<br> &quot;session&quot;: &quot;session id&quot;,<br>  &quot;flows&quot;: [<br>    {<br>      &quot;stages&quot;: [<br>        &quot;m.login.recaptcha&quot;,<br>        &quot;m.login.terms&quot;,<br>        &quot;m.login.email.identity&quot;<br>      ]<br>    }<br>  ],<br>  &quot;params&quot;: {<br>    &quot;m.login.recaptcha&quot;: {<br>      &quot;public_key&quot;: &quot;public key&quot;<br>    },<br>    &quot;m.login.terms&quot;: {<br>      &quot;policies&quot;: {<br>        &quot;privacy_policy&quot;: {<br>          &quot;version&quot;: &quot;1.0&quot;,<br>          &quot;en&quot;: {<br>            &quot;name&quot;: &quot;Terms and Conditions&quot;,<br>            &quot;url&quot;: &quot;https://matrix-client.matrix.org/_matrix/consent?v=1.0&quot;<br>          }<br>        }<br>      }<br>    }<br>  },<br>  &quot;completed&quot;: [<br>    &quot;m.login.email.identity&quot;, // now we can see email validation has been done<br>    &quot;m.login.recaptcha&quot;<br>  ]<br>}</pre><p>As the <strong>“m.login.email.identity”</strong> validation was successful we can now proceed to the next step.</p><h3>Step 4:</h3><p>The last step in the flow, we have to use the same <strong>/register</strong> API endpoint to validate the <strong>“m.login.terms” </strong>stage. The <strong>auth</strong> key in the request body request body will be very simple this time.</p><pre>{<br>  &quot;auth&quot;: {<br>    &quot;type&quot;: &quot;m.login.terms&quot;, // type of the authetication we are trying to perform<br>    &quot;session&quot;: &quot;session id&quot; // our old session id<br>  },<br>  &quot;session_id&quot;: &quot;session id&quot;,<br>  &quot;device_id&quot;: &quot;GHTYAJCE&quot;,<br>  &quot;inhibit_login&quot;: false,<br>  &quot;initial_device_display_name&quot;: &quot;Jungle Phone&quot;,<br>  &quot;password&quot;: &quot;ilovebananas&quot;,<br>  &quot;refresh_token&quot;: false,<br>  &quot;username&quot;: &quot;cheeky_monkey&quot;<br>}</pre><p>So the request stands as:</p><pre>curl -X POST &quot;https://matrix.org/_matrix/client/v3/register?kind=user&quot; \<br> -H &quot;Accept: application/json&quot; \<br> -H &quot;Content-Type: application/json&quot; \<br> -d &#39;{&quot;auth&quot;:{&quot;type&quot;:&quot;m.login.terms&quot;,&quot;session&quot;:&quot;session id&quot;},&quot;session_id&quot;:&quot;session id&quot;,&quot;device_id&quot;:&quot;GHTYAJCE&quot;,&quot;inhibit_login&quot;:false,&quot;initial_device_display_name&quot;:&quot;Jungle Phone&quot;,&quot;password&quot;:&quot;ilovebananas&quot;,&quot;refresh_token&quot;:false,&quot;username&quot;:&quot;cheeky_monkey&quot;}&#39; \</pre><p>On successful attempt we will receive an response and finally with success status code <strong>200</strong>:</p><pre>{<br>  &quot;user_id&quot;: &quot;@,username&gt;:matrix.org&quot;, // user id of the user<br>  &quot;home_server&quot;: &quot;matrix.org&quot;, // home server of the user<br>  &quot;access_token&quot;: &quot;access token&quot;, // access token of the user<br>  &quot;device_id&quot;: &quot;GHTYAJCE&quot; // device id we provided<br>}</pre><p>The user account has been successfully registered, the user can use the provided credentials to login on any client including ours.</p><h3>Conclusion</h3><p>User registration is a complicated process and we have to follow each step to make it functional. This implmentation enabled us to understand few of the core concepts of Matrix protocol like <strong><em>session_id</em></strong>, <strong><em>access_token</em></strong>, <strong><em>client_secret</em></strong> and many more. We can use this understanding to build a fully functional client application.</p><h3>References</h3><ol><li><a href="https://spec.matrix.org/v1.9/client-server-api/#user-interactive-authentication-api">https://spec.matrix.org/v1.9/client-server-api/#user-interactive-authentication-api</a></li><li><a href="https://playground.matrix.org/#post-/_matrix/client/v3/account/3pid/email/requestToken">https://playground.matrix.org/#post-/_matrix/client/v3/account/3pid/email/requestToken</a></li><li><a href="https://playground.matrix.org/#post-/_matrix/client/v3/register">https://playground.matrix.org/#post-/_matrix/client/v3/register</a></li><li><a href="https://codesandbox.io/p/sandbox/keen-http-7h2fp5?file=%2Fsrc%2Findex.js%3A6%2C32">https://codesandbox.io/p/sandbox/keen-http-7h2fp5?file=%2Fsrc%2Findex.js%3A6%2C32</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=27388d5263df" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>