<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Márcio Duarte on Medium]]></title>
        <description><![CDATA[Stories by Márcio Duarte on Medium]]></description>
        <link>https://medium.com/@marcio.duarte89?source=rss-ccdfbda98ebf------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*5jQwm1w231pw_l6cnZTlHg.jpeg</url>
            <title>Stories by Márcio Duarte on Medium</title>
            <link>https://medium.com/@marcio.duarte89?source=rss-ccdfbda98ebf------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 12:45:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@marcio.duarte89/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Architect’s Workshop: MindLog — Conversations with MindLog]]></title>
            <link>https://medium.com/@marcio.duarte89/the-architects-workshop-mindlog-conversations-with-mindlog-43d2a23ac262?source=rss-ccdfbda98ebf------2</link>
            <guid isPermaLink="false">https://medium.com/p/43d2a23ac262</guid>
            <category><![CDATA[dotnet-core]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Márcio Duarte]]></dc:creator>
            <pubDate>Mon, 12 Jan 2026 21:02:09 GMT</pubDate>
            <atom:updated>2026-01-14T17:31:40.094Z</atom:updated>
            <content:encoded><![CDATA[<h3><a href="https://medium.com/p/1cae5e0b834d/edit">The Architect’s Worksho</a>p: MindLog — <em>Conversations with MindLog</em></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*IgiO-wx7xLVhF4pTOoymvw.jpeg" /></figure><p>It’s time to put an AI interface on my personal knowledge base!</p><p>If you aren’t a medium subscriber you can read this post for free <a href="https://marcioduarte89.github.io/article/mindlog-conversation-with-ai">here</a>.</p><h3>Closing the Loop</h3><p>In the previous two articles, I focused on laying the technical foundations for Mindlog.</p><p>In <strong>“</strong><a href="https://medium.com/@marcio.duarte89/the-architects-workshop-mindlog-vectors-in-the-vault-from-markdown-to-meaning-329dc7f9df80"><strong>Vectors in the Vault: From Markdown to Meaning</strong></a><strong>”</strong>, I walked through how raw Markdown notes from Obsidian are parsed, segmented, embedded, and stored in Weaviate. That step turned static text into something searchable at a semantic level.</p><p>In <strong>“</strong><a href="https://medium.com/@marcio.duarte89/the-architects-workshop-mindlog-searching-smarter-building-an-ai-ready-knowledge-base-8983a014ed6e"><strong>Searching Smarter: Building an AI-Ready Knowledge Base</strong></a><strong>”</strong>, I introduced the Model Context Protocol (MCP) and showed how a .NET-based MCP server can expose that knowledge base through a clean, standardized interface.</p><p>At that point, the system was technically complete — but still not particularly usable day to day. This final article is about the last step: putting a real interface on top of it, and and actually <em>use</em> it as a semantic knowledge base.</p><h3>Claude Desktop as the Interface</h3><p>For the final layer, I chose <strong>Claude Desktop</strong>. The main reason is because it supports MCP natively and works well as a local, focused environment.</p><p>At a high level, the setup looks like this:</p><ul><li>Claude Desktop runs locally.</li><li>A local MCP server exposes my Mindlog tools (search, filtering, etc.).</li><li>Claude connects to that server and can invoke those tools when needed.</li></ul><p>For this I didn’t need any cloud orchestration, no extra backend services — just a local client talking to a local server.</p><h3>Installing and Configuring Claude Desktop</h3><p>The installation itself is straightforward and <a href="https://support.claude.com/en/articles/10065433-installing-claude-desktop">well-documented</a>, so I won’t dwell on it here. The interesting part is the <strong>MCP configuration</strong>.</p><p>Claude Desktop allows you to register local MCP servers by pointing it to:</p><ul><li>The executable (or command) that starts the server.</li><li>The transport type (stdio in my case).</li><li>The tools exposed by the server.</li></ul><p>You can read about how to setup an MCP <a href="https://modelcontextprotocol.io/docs/develop/connect-local-servers">here</a>.</p><p>Once configured, Claude automatically discovers the available tools — including the KnowledgeBaseEngine tool from Mindlog — without any additional wiring.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LEmeBSCPmM5SS9V_wGsqPA.png" /><figcaption>Local MCP configuration</figcaption></figure><p>This is one of the strengths of MCP: once the contract is defined, the client experience is largely automatic.</p><h3>Seeing It in Action</h3><p>With everything wired up, the real test is simple: ask a question.</p><p>For example:</p><blockquote>“How can I use async/await mechanism in .NET? What are the best practises”</blockquote><p>Behind the scenes, Claude:</p><ol><li>Interprets the question.</li><li>Decides that it needs to call the KnowledgeBaseEnginetool.</li><li>Generates a structured query with:</li></ol><ul><li>A cleaned semantic query string.</li><li>Inferred keywords (CSharp, Dotnet, LanguageFeatures).</li><li>Optional date filters, if relevant.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Ll8kqX4lD3bNJZY0_MqUQ.png" /><figcaption>Claude adding specific keywords and query parameters</figcaption></figure><p>As you can see above, my question has been interpreted and parsed by Claude and sent to my MCP server in the format it expects. Because I didn’t provide any timeframes it didn’t include any (this will search across all my notes regardless of the when they were created), but it did include all the specific keywords and the broken down query.</p><p>That query is then passed to the MCP server, which:</p><ul><li>Generates embeddings.</li><li>Applies certainty thresholds.</li><li>Combines semantic search with structured filters.</li><li>Queries Weaviate.</li><li>Returns a ranked set of notes.</li></ul><p>Claude receives the results and presents them back in a readable, contextual way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*adPLNOGtM-blRxgjfeWIvw.png" /><figcaption>Partial example of the reply and interpretation by Claude</figcaption></figure><p>What’s important here is that <strong>Claude isn’t replacing search logic, </strong>but rather orchestrating it. The semantic search still lives in the architecture — Claude just provides a natural interface on top.</p><h3>What I Learned</h3><p>A few takeaways from building and using Mindlog:</p><ul><li><strong>Semantic search needs constraints.</strong> Embeddings alone aren’t enough; certainty thresholds and structured filters matter.</li><li><strong>MCP is a powerful abstraction.</strong> It keeps AI integration clean and decoupled from your core logic.</li><li><strong>.NET fits naturally here.</strong> With attributes, async APIs, and strong typing, building MCP tools feels like normal application development.</li><li><strong>The UX matters.</strong> The difference between “interesting project” and “daily tool” is the interface. I’ve started using Mindlog regularly to:<br>- Find older notes I’d forgotten about.<br>- Explore related ideas across topics.<br>- Support learning by resurfacing my own prior thinking.</li></ul><p>It’s become a quiet but reliable part of how I work nowdays.</p><h3>Wrapping Up</h3><p>This series started as a project for me to learn about building AI systems, applying architectural thinking to a real problems, making trade-offs, and learning by doing — which is the spirit I want for <em>The Architect’s Workshop</em>.</p><p>If you’ve followed along, I hope you’ve picked up ideas you can adapt to your own projects, whether that’s personal knowledge management, AI integration, or just building and hacking things away!</p><p>Stay tuned — keep building, keep learning.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=43d2a23ac262" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Architect’s Workshop: MindLog — Searching Smarter: Building an AI-Ready Knowledge Base]]></title>
            <link>https://medium.com/@marcio.duarte89/the-architects-workshop-mindlog-searching-smarter-building-an-ai-ready-knowledge-base-8983a014ed6e?source=rss-ccdfbda98ebf------2</link>
            <guid isPermaLink="false">https://medium.com/p/8983a014ed6e</guid>
            <category><![CDATA[dotnet-core]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[weaviate]]></category>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Márcio Duarte]]></dc:creator>
            <pubDate>Mon, 06 Oct 2025 19:27:53 GMT</pubDate>
            <atom:updated>2025-10-06T19:36:34.895Z</atom:updated>
            <content:encoded><![CDATA[<h3><a href="https://medium.com/p/1cae5e0b834d/edit">The Architect’s Worksho</a>p: MindLog — <em>Searching Smarter: Building an AI-Ready Knowledge Base</em></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*IgiO-wx7xLVhF4pTOoymvw.jpeg" /></figure><p>In the last <a href="https://medium.com/@marcio.duarte89/the-architects-workshop-mindlog-vectors-in-the-vault-from-markdown-to-meaning-329dc7f9df80">article</a>, we parsed markdown notes, created embeddings, and stored them in Weaviate. That gave us a powerful way to run semantic queries across our knowledge base.</p><p>But embeddings on their own aren’t the end goal — I want to <em>use</em> them. That means giving external tools (like Claude Desktop) a way to query my knowledge base data through a common interface. This is where the <strong>Model Context Protocol (MCP)</strong> comes in.</p><p>If you aren’t a medium subscriber you can read this post for free <a href="https://marcioduarte89.github.io/article/mindlog-searching-smarter">here</a>.</p><h3>What is MCP?</h3><p>The Model Context Protocol is a standard for connecting applications to external resources in a structured way. It lets AI models (or tools working with them) interact with backends as if they were first-class “extensions.”</p><p>MCP defines:</p><ul><li><strong>Servers</strong>: components that expose capabilities (like search, file access, or database queries).</li><li><strong>Clients</strong>: applications that consume those capabilities (e.g., Claude Desktop, custom tools).</li><li><strong>Transport</strong>: how client and server communicate (HTTP or STDIO).</li><li><strong>Schema/Contracts</strong>: a JSON-RPC–based contract describing what the server offers.</li></ul><p>In practice: An MCP server becomes a “plugin” that tools can connect to, query, and interact with — without having to embed application logic directly inside them.</p><h3>Why Use MCP in MindLog?</h3><ul><li><strong>Standardized access.</strong> Instead of building custom APIs for every integration, I expose my knowledge base once via MCP.</li><li><strong>Client-agnostic.</strong> Any MCP-compliant client can query my notes (Claude Desktop being the first).</li><li><strong>Composable architecture.</strong> MCP servers are small, focused services. It’s possible to run multiple servers for different backends and chain them together.</li></ul><p>For MindLog, the MCP server is the interface between my vector database (Weaviate) and the tools I actually use in my day-to-day.</p><h3>Transport Options: HTTP vs stdio</h3><p>MCP uses JSON-RPC to encode messages. JSON-RPC messages MUST be UTF-8 encoded.</p><p>Currently MCP supports two transport layers:</p><ul><li>HTTP</li><li>STDIO</li></ul><h4>HTTP</h4><p>In Streamable HTTP transport, the server operates as an independent process that can handle multiple client connections. This transport uses HTTP POST and GET requests. The server MUST provide a single HTTP endpoint path (the MCP endpoint) that supports both POST and GET methods. For example, this could be a URL like <a href="https://example.com/mcp.">https://example.com/mcp.</a></p><p>Servers can optionally make use of <a href="https://en.wikipedia.org/wiki/Server-sent_events">Server-Sent Events</a> (SSE) to stream multiple server messages. This permits basic MCP servers, as well as more feature-rich servers supporting streaming and server-to-client notifications and requests.</p><ul><li><strong>Pros:<br>- </strong>Familiar to most developers.<br>- Easy to integrate with web frameworks, reverse proxies, and load balancers.<br>- Well-suited for multi-user or distributed deployments.</li><li><strong>Cons:<br>- </strong>Slightly more overhead (latency, headers, networking).<br>- Requires hosting infrastructure (container, server, etc.).</li></ul><h4>STDIO</h4><p>This transport communicates over standard in and standard out. Some considerations with regards to STDIO:</p><ul><li>The server reads JSON-RPC messages from its standard input (stdin) and sends messages to its standard output (stdout).</li><li>Messages are delimited by newlines, and MUST NOT contain embedded newlines.</li><li>The server MUST NOT write anything to its stdin/stdout that is not a valid MCP message.</li><li><strong>Pros:<br>- </strong>Very simple transport: client and server talk via standard input/output.<br>- Easy to run locally, especially in dev/test environments.<br>- Zero network setup required.</li><li><strong>Cons:<br>- </strong>Best for single-user or local use cases.<br>- Harder to scale or expose remotely.</li></ul><p>For MindLog, and for now, I opted for <strong>STDIO </strong>because Claude Desktop supports it out of the box. But long-term, an HTTP-based server would allow me to run the service remotely and make it accessible across devices — It’s something I still need to think about.</p><h4>Authentication</h4><p>As I&#39;m running the server using STDIO transport and hosting it locally, I haven’t configured any authentication so far. I will include it if I move it to an HTTP base server and host it remotely.</p><p>If you want to learn more about the specification, you can read it <a href="https://modelcontextprotocol.io/specification/2025-03-26">here</a>.</p><h3>Building an MCP Server in .NET</h3><p>At its core, an MCP server is nothing more than a service that:</p><ol><li>Defines a schema of tools/resources.</li><li>Implements a transport (STDIO or HTTP).</li><li>Wires those definitions to actual logic.</li></ol><p>In .NET, we can use attributes to decorate tools that MCP should expose, and then let the server host them.</p><p>Here’s how I&#39;ve defined the <strong>MindLog MCP tool</strong> that powers my knowledge base search:</p><h4><strong>Program</strong>:</h4><pre>var builder = Host.CreateApplicationBuilder(args);<br>...<br>builder<br>    .Services<br>    .AddMcpServer()<br>    .WithStdioServerTransport()<br>    .WithToolsFromAssembly();<br>...<br>builder.Services.AddOptions();<br>builder.Services.AddScoped&lt;WeaviateClient&gt;();<br>builder.Services.AddSingleton&lt;SentenceTransformer&gt;();</pre><p>The snippet above adds the MCP server, configures the STDIO transport and registers the tools that it finds in the assembly. I also register my WeaviateClientand the SentenceTransformer so that I can use them in my tool.</p><h4>MCP Server Tool:</h4><pre>[McpServerToolType]<br>public class KnowledgeBaseSearch<br>{<br>    [McpServerTool, Description(&quot;Searches my personal article knowledge base for any notes and articles i&#39;ve written&quot;)]<br>    public static async Task&lt;IEnumerable&lt;Note&gt;&gt; Query(<br>        Helpers.WeaviateClient weaviateClient,<br>        SentenceTransformer sentenceTransformer,<br>        [Description(&quot;Query to be sent for article knowledge base search - this should be stripped of dates ranges or keywords&quot;)]<br>        string query,<br>        [Description(&quot;If a precise date time is provided, only this property should be populated, not the range - should be UTC format&quot;)]<br>        DateTime? specificDateTime,<br>        [Description(&quot;Start date time, if the original query includes date ranges - should be UTC format&quot;)]<br>        DateTime? startDateTime,<br>        [Description(&quot;End date time, if the original query includes date ranges - should be UTC format&quot;)]<br>        DateTime? endDateTime,<br>        [Description(&quot;Specific keywords used in the base query, ex: C#, dependency injection, and others. &quot; +<br>        &quot;These keywords should be translated to the available keywords Dotnet, Async, CSharp, LanguageFeatures, DependencyInjection, ASPNETCore, EFCore, Migrations&quot; +<br>        &quot;Inference should also be used, if the query relates to C#, CSharp and Dotnet should also be provided&quot;)]<br>        string[] keywords)<br>    {<br>...</pre><p>My Knowledge Base Search class is decorated withMcpServerToolType. This will be used by the server to discover all the existing tools within the assembly that are then decorate with McpServerTool.</p><p>The Description properties will be used to enhance any client connections to the server. This description helps the client determine which tool to call and will also provide guidance to our LLM on what and how to populate those values (which we will see in the next article).</p><p>It’s also worth noting that dependency injection also works by method injection convention, as here we are injecting WeaviateClient and SentenceTransformer in the tool method.</p><h4>Knowledge base search:</h4><pre>// We translate the simplified query and get the embeddings from the Sentence Transformer<br>var sentenceEmbeddings = await sentenceTransformer.GetEmbeddings(query);<br>var embedding = sentenceEmbeddings.Single().Embeddings.ToArray();<br><br>var weaviateClass = await weaviateClient.GetWeaviateClass();<br>var getQuery = weaviateClass.CreateGetQuery();<br><br>// Sets the graphql query to search for objects with similar embeddings<br>getQuery.Filter.NearVector(embedding);<br><br>var conditions = new List&lt;ConditionalAtom&lt;StorageModelWeaviate&gt;&gt;();<br><br>// Searches for any existing tags in the metadata<br>if (keywords is not null &amp;&amp; keywords.Any())<br>{<br>    conditions.Add(When&lt;StorageModelWeaviate, string[]&gt;.ContainsAny(&quot;tags&quot;, keywords));<br>}<br><br>// Provides date ranges if provided<br>if (startDateTime is not null &amp;&amp; endDateTime is not null)<br>{<br>    conditions.Add(When&lt;StorageModelWeaviate, DateTime&gt;.GreaterThanEqual(&quot;noteDate&quot;, startDateTime.Value));<br>    conditions.Add(When&lt;StorageModelWeaviate, DateTime&gt;.LessThanEqual(&quot;noteDate&quot;, endDateTime.Value));<br>}<br>else if (specificDateTime is not null)<br>{<br>    conditions.Add(When&lt;StorageModelWeaviate, DateTime&gt;.Equal(&quot;noteDate&quot;, specificDateTime.Value));<br>}<br><br>if (conditions.Any())<br>{<br>    getQuery.Filter.Where(Conditional&lt;StorageModelWeaviate&gt;.And(conditions.ToArray()));<br>}<br>else<br>{<br>    getQuery.Filter.Limit(10);<br>}<br><br>// Adds additional fields for the select<br>getQuery.Fields.Additional.Add(Additional.Id);<br>getQuery.Fields.Additional.Add(new Additional(&quot;certainty&quot;));<br><br>//the package doesn&#39;t yet support this, so would only work with direct graphql query<br>//{<br>//    When&lt;StorageModelWeaviate, float&gt;.GreaterThan([&quot;_additional&quot;, &quot;certainty&quot;], 0.7f)<br>//};<br><br>var graphQL = new GraphQLQuery<br>{<br>    Query = getQuery.ToString()<br>};<br><br>var queryResult = await weaviateClient.Db.Schema.RawQuery(graphQL);<br>var queryResultGet = queryResult.Data[&quot;Get&quot;];<br>var queryResultGetData = queryResultGet[&quot;StorageModel&quot;];<br>var weaviateModels = queryResultGetData.ToObject&lt;StorageModelWeaviate[]&gt;() ??<br>    throw new Exception(&quot;Weaviate model could not be deserialized&quot;);<br><br>return weaviateModel.Select(x =&gt; new Note()<br>{<br>    NoteDate = x.noteDate,<br>    NoteName = x.noteName,<br>    Section = x.section,<br>    Tags = x.tags,<br>    Text = x.text,<br>});</pre><p>With the code for the service above I&#39;m able to:</p><ol><li>Translate the simplified query Claude will send me to vector embeddings.</li><li>Construct the GraphQL query from the embeddings and the filtering parameters.</li><li>Call Weaviate database and get the notes that are relevant for the query and filtering parameters.</li><li>Return them to the client (Claude).</li></ol><h4>When “Near Enough” Isn’t Enough</h4><p>One thing I noticed during testing (and became quickly frustrating) is that vector search doesn’t always feel as precise as you might hope. Because my knowledge base is mostly technical — and weighted towards .NET — queries often surfaced results that were <em>semantically close</em> but not actually relevant.</p><p>For example, if I asked specifically about <strong>new features in C#</strong>, the system would sometimes return notes on dependency injection or ASP.NET Core. They’re related at a conceptual level (same language, same ecosystem), but not what I really wanted.</p><p>This is a natural side-effect of how embeddings work: vectors capture what is semantically close but not the “exact topic”. The model sees C#, ASP.NET, and dependency injection as part of the same neighbourhood in vector space. Without additional constraints, you can easily get neighbours that are near — but not on point.</p><h4>Introducing a Certainty Threshold</h4><p>The fix was to introduce a <strong>certainty (or confidence) threshold</strong> in the GraphQL query to Weaviate. Instead of accepting <em>any</em> close match, I required that results pass a minimum semantic similarity before they’re returned.</p><p>Here’s a an example query:</p><pre>{<br>    Get {<br>        StorageModel (<br>            nearVector: {<br>                vector: [........],<br>                certainty: 0.7<br>            }<br>            where: {<br>                operator: And,<br>                operands: [{<br>                    operator: ContainsAny,<br>                    path: [&quot;tags&quot;],<br>                    valueText: [&quot;CSharp&quot;,&quot;Dotnet&quot;]<br>                },<br>                {<br>                    operator: GreaterThanEqual,<br>                    path: [&quot;noteDate&quot;],<br>                    valueDate: &quot;2023-01-01T00:00:00Z&quot;<br>                },<br>                {<br>                    operator: LessThanEqual,<br>                    path: [&quot;noteDate&quot;],<br>                    valueDate: &quot;2023-12-31T23:59:59Z&quot;<br>                }]<br>            }<br>        ) {<br>            noteName<br>            noteDate<br>            text<br>            tags<br>            section<br>            _additional {<br>                id<br>                certainty<br>            }<br>        }<br>    }<br>}</pre><p>By setting certainty: 0.7, I forced the query and search to be stricter. Results that were only loosely related (say, ASP.NET Core when I really wanted C# features) would fall below that threshold and be excluded.</p><h4>Balancing Precision and Recall</h4><p>The right threshold isn’t one-size-fits-all. Too low, and I would drown “in noise”. Too high, and I would miss genuinely useful matches. I found that 0.7 worked well for my data — but the “best” number depends on:</p><ul><li>The <strong>diversity</strong> of the notes (more topics = lower threshold may help).</li><li>The <strong>embedding model</strong> used (some produce tighter clusters).</li><li>The <strong>use case</strong> (browsing vs. precise answering).</li></ul><p>In the context of MindLog, introducing certainty thresholds was a key step in moving towards more precise results.</p><h3>Testing the MCP Server</h3><p>MCP development is made much easier by the <strong>Inspector tool. </strong>You can run the inspector using:</p><pre>npx @modelcontextprotocol/inspector</pre><p>This launches a simple UI where we can:</p><ul><li>Browse the tools the server exposes.</li><li>Inspect parameter descriptions.</li><li>Run test queries and inspect responses.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xDvgTokY2rCLz4sgZvOphQ.png" /></figure><blockquote>You might have noticed I’m using HTTP Transport in the screenshot above. I usually find it much easier to test using this transport compared to STDIO.</blockquote><p>In the screenshot above I connected to my MCP server on port 5001.</p><p>The inspector also returns other relevant tools like tool/list , which is used to discover the existing tools in my server:</p><pre>{<br>  &quot;method&quot;: &quot;tools/list&quot;,<br>  &quot;params&quot;: {}<br>}</pre><p>which responds with:</p><pre>{<br>  &quot;tools&quot;: [<br>    {<br>      &quot;name&quot;: &quot;query&quot;,<br>      &quot;description&quot;: &quot;Searches my personal article knowledge base for any notes and articles i&#39;ve written&quot;,<br>      &quot;inputSchema&quot;: {<br>        &quot;type&quot;: &quot;object&quot;,<br>        &quot;properties&quot;: {<br>          &quot;query&quot;: {<br>            &quot;description&quot;: &quot;Query to be sent for article knowledge base search - this should be stripped of dates ranges or keywords&quot;,<br>            &quot;type&quot;: &quot;string&quot;<br>          },<br>          &quot;specificDateTime&quot;: {<br>            &quot;description&quot;: &quot;If a precise date time is provided, only this property should be populated, not the range - should be UTC format&quot;,<br>            &quot;type&quot;: [<br>              &quot;string&quot;,<br>              &quot;null&quot;<br>            ],<br>            &quot;format&quot;: &quot;date-time&quot;<br>          },<br>          &quot;startDateTime&quot;: {<br>            &quot;description&quot;: &quot;Start date time, if the original query includes date ranges - should be UTC format&quot;,<br>            &quot;type&quot;: [<br>              &quot;string&quot;,<br>              &quot;null&quot;<br>            ],<br>            &quot;format&quot;: &quot;date-time&quot;<br>          },<br>          &quot;endDateTime&quot;: {<br>            &quot;description&quot;: &quot;End date time, if the original query includes date ranges - should be UTC format&quot;,<br>            &quot;type&quot;: [<br>              &quot;string&quot;,<br>              &quot;null&quot;<br>            ],<br>            &quot;format&quot;: &quot;date-time&quot;<br>          },<br>          &quot;keywords&quot;: {<br>            &quot;description&quot;: &quot;Specific keywords used in the base query, ex: C#, dependency injection, and others. These keywords should be translated to the available keywords Dotnet, Async, CSharp, LanguageFeatures, DependencyInjection, ASPNETCore, EFCore, MigrationsInference should also be used, if the query relates to C#, CSharp and Dotnet should also be provided&quot;,<br>            &quot;type&quot;: &quot;array&quot;,<br>            &quot;items&quot;: {<br>              &quot;type&quot;: &quot;string&quot;<br>            }<br>          }<br>        },<br>        &quot;required&quot;: [<br>          &quot;query&quot;,<br>          &quot;specificDateTime&quot;,<br>          &quot;startDateTime&quot;,<br>          &quot;endDateTime&quot;,<br>          &quot;keywords&quot;<br>        ]<br>      }<br>    }<br>  ]<br>}</pre><p>queryis my existing tool that will perform the request.</p><p>I can then perform a test query to my MCP server:</p><pre>{<br>  &quot;method&quot;: &quot;tools/call&quot;,<br>  &quot;params&quot;: {<br>    &quot;name&quot;: &quot;query&quot;,<br>    &quot;arguments&quot;: {<br>      &quot;query&quot;: &quot;C# programming articles&quot;,<br>      &quot;specificDateTime&quot;: &quot;2023-01-01T00:00:00Z&quot;,<br>      &quot;startDateTime&quot;: &quot;2023-01-01T00:00:00Z&quot;,<br>      &quot;endDateTime&quot;: &quot;2023-12-31T23:59:59Z&quot;,<br>      &quot;keywords&quot;: [<br>        &quot;CSharp&quot;,<br>        &quot;Dotnet&quot;<br>      ]<br>    },<br>    &quot;_meta&quot;: {<br>      &quot;progressToken&quot;: 2<br>    }<br>  }<br>}</pre><p>and the response:</p><pre>{<br>  &quot;content&quot;: [<br>    {<br>      &quot;type&quot;: &quot;text&quot;,<br>      &quot;text&quot;: &quot;[{\&quot;noteName\&quot;:\&quot;cshap.md\&quot;,\&quot;noteDate\&quot;:\&quot;2023-05-25T00:00:00Z\&quot;,\&quot;text\&quot;:\&quot;C# 10 introduces several features aimed at reducing boilerplate and improving developer productivity.\&quot;,\&quot;tags\&quot;:[\&quot;Dotnet\&quot;,\&quot;CSharp\&quot;,\&quot;LanguageFeatures\&quot;],\&quot;section\&quot;:\&quot;What\\u2019s New in C# 10\&quot;},{\&quot;noteName\&quot;:\&quot;cshap.md\&quot;,\&quot;noteDate\&quot;:\&quot;2023-05-25T00:00:00Z\&quot;,\&quot;text\&quot;:\&quot;Simplifies namespace declarations by removing indentation: [csharp code] namespace MyApp; public class Program {   // ... }\&quot;,\&quot;tags\&quot;:[\&quot;Dotnet\&quot;,\&quot;CSharp\&quot;,\&quot;LanguageFeatures\&quot;],\&quot;section\&quot;:\&quot;File-scoped Namespaces\&quot;}, ....]&quot;<br>    }<br>  ]<br>}</pre><p>For MindLog, <strong>Inspector </strong>became an essential part of the workflow. It confirmed that:</p><ul><li>Tools were registered correctly.</li><li>Parameters like keywords and date ranges were interpreted properly.</li><li>The server returned valid responses before ever touching Claude Desktop.</li></ul><h3>Putting It All Together</h3><p>With the MCP server in place, MindLog now has a clean, reusable interface for accessing my knowledge base. Instead of thinking about embeddings, certainty thresholds, or GraphQL queries, I can just expose a KnowledgeBaseSearch tool and let MCP handle the rest.</p><p>At this point, the system is “<em>AI-ready”</em>. Semantic search is available through a standard protocol that any compliant client can consume. For now, I’ve focused on making sure the foundation is solid: notes are parsed, embeddings are generated, results are filtered, and the MCP server ties everything together.</p><p>The next step is where things get really interesting: bringing Claude into the picture. That’s the focus of the final article in the series, <strong>“Conversations with MindLog”</strong>, where we’ll explore what it feels like to actually chat with your own second brain.</p><p>I hope this article has given you both insight and practical guidance into how MCP can turn a vector database into an accessible, AI-ready service. More importantly, I hope you’ve enjoyed following along and learned something you can apply in your own projects.</p><p><strong>Stay tuned — keep building, keep learning.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8983a014ed6e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Architect’s Workshop: MindLog — Vectors in the Vault: From Markdown to Meaning]]></title>
            <link>https://medium.com/@marcio.duarte89/the-architects-workshop-mindlog-vectors-in-the-vault-from-markdown-to-meaning-329dc7f9df80?source=rss-ccdfbda98ebf------2</link>
            <guid isPermaLink="false">https://medium.com/p/329dc7f9df80</guid>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[embedded-systems]]></category>
            <category><![CDATA[dotnet-core]]></category>
            <category><![CDATA[weaviate]]></category>
            <dc:creator><![CDATA[Márcio Duarte]]></dc:creator>
            <pubDate>Tue, 02 Sep 2025 19:23:09 GMT</pubDate>
            <atom:updated>2026-01-12T20:17:41.688Z</atom:updated>
            <content:encoded><![CDATA[<h3><a href="https://medium.com/p/1cae5e0b834d/edit">The Architect’s Worksho</a>p: MindLog — Vectors in the Vault: From Markdown to Meaning</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3CgT5v6XpBifADcMiflpSw.jpeg" /></figure><p>In order for me to be able to “chat” with my notes I need to start with a a parser. In the MindLog pipeline, the parser ensures that notes from the knowledge base are consistently extracted, segmented, and enriched before they ever reach the embedding stage.</p><h3>A Scheduled Service</h3><p>The parser is designed as a lightweight <strong>.NET Core CRON service</strong>, scheduled to run at a certain interval. The reasoning here is that my knowledge base (using Obsidian) usually evolves on a daily basis, but in reality it doesn’t need real-time indexing. For my notes, a 12–24 hour indexing cadence ensures that new insights are captured in a timely manner without wasting compute cycles on excessive updates.</p><p>A <strong>log file</strong> stored in my knowledge base acts as a lastUpdated bookmark. Instead of re-parsing the entire knowledge base every run, generating and persisting the embeddings in the vector database, the service gets all the files in my knowledge base and only processes the ones that have been modified since the last recorded timestamp. This dramatically reduces processing overhead while keeping the vector store in sync with ongoing knowledge capture.</p><h3>Markdown and Frontmatter Parsing</h3><p>The structure of each note matters. Markdown itself provides the textual flow, but my knowledge notes begin with <strong>YAML frontmatter</strong> — metadata describing date, resource, tags etc.<br>My notes usually focus on a specific topic (are usually very targeted) and roughly follow the structure (below is just an illustration — Different items and order might be used):</p><pre>---<br>date:<br>resource:<br>tags:<br>---<br><br># Heading level 1<br>Paragraph / Lists / Images / Links / Code<br><br>## Heading level 2<br>Paragraph / Lists / Images / Links / Code<br><br># Heading level 1<br>Paragraph / Lists / Images / Links / Code</pre><p>A more specific example could be:</p><pre>---<br>date: 2023-05-25<br>resource: https://learn.microsoft.com/en-us/ef/core/managing-schemas/migrations/<br>tags:<br>  - Dotnet<br>  - EFCore<br>  - Migrations<br>  - CSharp<br>---<br><br># Managing Database Schema with EF Core Migrations<br><br>EF Core Migrations help manage changes to your database schema over time, keeping your code and database in sync.<br><br>## Common Commands<br><br>- Add a new migration:<br><br>```bash<br>dotnet ef migrations add AddCustomerTable<br>```<br><br>- Update the database:<br><br>```bash<br>dotnet ef database update<br>```<br><br>- Remove the last migration:<br><br>```bash<br>dotnet ef migrations remove<br>```<br><br>### Workflow<br>- Modify your entity classes.<br>- Add a migration to capture schema changes.<br>- Apply migration to the database.<br><br>Migrations generate code files describing schema changes, allowing version control and collaboration.<br><br>Check official docs: [EF Core Migrations](https://learn.microsoft.com/en-us/ef/core/managing-schemas/migrations/).</pre><p>I like to have my notes organised within different folders based on top level topics, but this isn’t really needed as Obsidian search is great if you know what you are looking for and have notes well categorized.</p><p>To handle parsing reliably in .NET, two well-supported libraries are used:</p><ul><li><a href="https://github.com/xoofx/markdig?utm_source=chatgpt.com"><strong>Markdig</strong></a> — a robust Markdown parser for .NET. It transforms raw Markdown into an abstract syntax tree, which makes extracting paragraphs, headings, and inline structures straightforward.</li><li><a href="https://github.com/aaubry/YamlDotNet?utm_source=chatgpt.com"><strong>YamlDotNet</strong></a> — a YAML parser for .NET, ideal for reading the frontmatter section at the top of each note. This metadata can later enrich embeddings with higher-level context, improving search relevance.</li></ul><h3>Splitting Notes into Chunks?</h3><p>One key design choice is how to represent a note for embeddings. The naive option would be to feed the entire note into a single embedding. But this comes with pitfalls:</p><ul><li><strong>Semantic dilution</strong> — Although I try to keep my notes targeted, semantically, a single note might contain multiple concepts, and lumping them together risks blurring distinctions. For example, a note titled <em>Entity Framework </em>could<em> </em>discuss both <em>fluent api </em>and <em>optimistic concurrency</em>. One embedding would mix these unrelated ideas.</li><li><strong>Granularity of retrieval</strong> — When querying the vector store, smaller chunks improve precision. A question about <em>optimistic concurrency </em>should surface only the relevant section, not the entire note.</li><li><strong>Scalability</strong> — Embedding smaller paragraphs allows us to reuse relevant sections across multiple queries without regenerating embeddings.</li></ul><p>The chosen approach is to split each note by the “items”, i.e.. paragraphs under <strong>headings or subheadings, if they exist</strong>. Paragraphs under the <strong>headings</strong>/<strong>subheadings</strong> become a discrete semantic unit with its own embedding. This strikes a balance between preserving context (headings are carried along as metadata) and ensuring embeddings remain tightly focused. Here’s an example split taking into consideration the above markdown:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*HtdTuqMUNwc9NWKOormBuw.png" /><figcaption>Embeddings split</figcaption></figure><p>The following type holds the representation of the markdown section:</p><pre>internal record MarkdownSection<br>{<br>    public Guid Id { get; init; }<br><br>    public string Text { get; init; } = default!;<br><br>    public string NoteName { get; init; } = default!;<br><br>    public DateTime NoteDate { get; init; }<br><br>    public IEnumerable&lt;string&gt; Tags { get; init; }<br><br>    public string? Section { get; init; }<br>}</pre><p>Here’s an example of the data held in this type:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H9jbhgt3C75ha9XlFBdSJA.png" /></figure><p>Each note has a Id which is auto-generated, this Id is the identifier for the Markdown section which will be sent to the Embeddings service (together with the Text to generate the embedding).</p><h3>Embeddings via Python</h3><p>Once the text chunks are identified, they need to be converted into embeddings. For this stage, we deliberately decouple from the .NET ecosystem and use a <strong>Python microservice</strong>. The reasoning is threefold:</p><ul><li><strong>Model ecosystem</strong> — Python is the de-facto ecosystem for machine learning, with mature libraries and tooling.</li><li><strong>SentenceTransformers</strong> — Specifically, we use all-MiniLM-L6-v2, a well-regarded open-source embedding model. It offers an excellent trade-off:<br>- Free to use and self-hosted (no dependency on proprietary APIs).<br>- Compact (384-dimensional embeddings), making it efficient in storage and retrieval.<br>- High semantic performance relative to its size.<br>- <strong>Avoiding wheel reinvention</strong> — While technically possible to wrap a transformer model in .NET, it would require significant overhead in integration, performance tuning, and model compatibility. Python allows us to adopt the model out-of-the-box and focus engineering energy on the surrounding pipeline instead.</li></ul><p>The Python service is built using <a href="https://flask.palletsprojects.com/en/stable/">Flask</a>. Flask is lightweight WSGI web application framework. It is designed to make getting started quick and easy, with the ability to scale up to complex applications.<br>This Python service receives a:</p><pre>[<br>    { <br>        &quot;id&quot;: &quot;384b03bf-85e1-4bfb-a922-76a61ca3581a&quot;,<br>        &quot;text&quot;: &quot;the text section to generate the embedding&quot;<br>    },<br>    ..<br>]</pre><p>it then generates the embeddings for the textand associates the embeddings to the id, this has the following advantages:</p><ul><li>Smaller payloads — We only send the Markdown section id and the text for the section, instead of the entire markdown metadata which the Embeddings service doesn’t need.</li><li>We send all Markdown sections (list of id and text) for a note — instead of calling the service per Markdown section, this reduces the overall number of API calls we make.</li></ul><p>The Python service is written as follows:</p><pre>from flask import Flask, request, jsonify<br>from sentence_transformers import SentenceTransformer<br>import numpy as np<br><br>app = Flask(__name__)<br><br># Load model once at startup<br>model = SentenceTransformer(&#39;all-MiniLM-L6-v2&#39;)<br><br>@app.route(&#39;/sentences-transform&#39;, methods=[&#39;POST&#39;])<br>def transform_sentences():<br>    try:<br>        data = request.get_json()<br>        <br>        if not isinstance(data, list):<br>            return jsonify({&quot;error&quot;: &quot;Input must be a list&quot;}), 400<br><br>        response = []<br>        for item in data:<br>            if not isinstance(item, dict) or &#39;id&#39; not in item or &#39;text&#39; not in item:<br>                return jsonify({&quot;error&quot;: &quot;Each item must be a list with &#39;id&#39; and &#39;text&#39;&quot;}), 400<br><br>            sentence_id = item[&#39;id&#39;]<br>            text = item[&#39;text&#39;]<br>            embedding = model.encode(text).tolist()  # Convert numpy array to list<br><br>            response.append({<br>                &quot;id&quot;: sentence_id,<br>                &quot;embeddings&quot;: embedding<br>            })<br><br>        return jsonify(response), 200<br><br>    except Exception as e:<br>        return jsonify({&quot;error&quot;: str(e)}), 500<br><br>if __name__ == &#39;__main__&#39;:<br>    app.run(debug=True)</pre><p>Once the embeddings are generated, its then time to persist them in our Vector Database.</p><h3>The Vault: Why Weaviate for Vectors</h3><p>Once embeddings are generated, they need a home — a system designed not just to store them, but to make them retrievable in a way that preserves their semantic power. This is where a vector database enters the pipeline.</p><h4>Why a Vector Database at All?</h4><p>Traditional databases excel at exact matches and well-structured queries. If you need to know <em>“what notes did I write on August 12th?”</em>, a relational database is perfect. But if you need to ask <em>“what notes talk about semantic search even if the word ‘semantic’ isn’t used?”</em>, traditional indexing falls short.</p><p>A vector database addresses this gap by supporting <strong>approximate nearest neighbor (ANN) search</strong> over high-dimensional embeddings. Instead of checking for keyword overlap, the database finds the embeddings closest in vector space to a query. This enables:</p><ul><li><strong>Semantic retrieval</strong> — results are ranked by meaning, not exact token match.</li><li><strong>Contextual discovery</strong> — related notes can be surfaced even if phrased differently.</li><li><strong>Scalability</strong> — ANN indexes are optimized for large collections of embeddings, something relational databases struggle with at scale.</li></ul><p>Could one retrofit a SQL or NoSQL database for embeddings? Yes — you could store embeddings as JSON arrays or blobs and run cosine similarity computations manually. But this would quickly become unmanageable for non-trivial datasets. Without native ANN indexing, query latency and cost would skyrocket.</p><h4>Why Weaviate?</h4><p>The vector database space has grown crowded: Pinecone, Milvus, Vespa, Qdrant — all offer ANN capabilities. Choosing Weaviate came down to three considerations:</p><ol><li><strong>Integrated Hybrid Search</strong><br> Weaviate combines vector search with traditional filters (structured metadata, boolean queries, keyword matching). This is vital for knowledge bases, where I often want queries like <em>“What do I know about partition keys when using DynamoDb”</em>. (This query would automatically add #DynamoDb tag and filter notes further).</li><li><strong>Modular Deployment</strong><br> Unlike some commercial SaaS-first solutions, Weaviate is open-source and deployable on-prem or in the cloud. This flexibility avoids lock-in and makes it easy to start small, then scale.</li><li><strong>Ecosystem and Maturity</strong><br> Weaviate has strong documentation, an active community, and production-grade stability. It has features like schema enforcement, modularity, and hybrid search integration.</li></ol><p>Ultimately, Weaviate balances <strong>open-source control</strong> with <strong>production readiness</strong>, making it a natural fit for my personal knowledge vault.</p><h4>Abstracting Access with WeaviateNET</h4><p>I could have written a client to interact with Weaviate via REST endpoints, but it wouldn’t have been the best usage of my (limited) time for my “pet projects”. Instead, after reading a good portion of the source code and doing some feature testing I was happy to use <a href="https://github.com/Unipisa/WeaviateNET">WeaviateNET</a> as the client for my project. It provides:</p><ul><li><strong>Schema management</strong> — defining object classes, properties, and vector fields directly from C#.</li><li><strong>Data ingestion</strong> — pushing embeddings (paragraphs + metadata) into Weaviate without writing raw HTTP calls.</li><li><strong>Querying</strong> — running both semantic vector searches and hybrid keyword queries with LINQ-like syntax.</li></ul><h4>Usage and Testing</h4><p>I ran my Weaviate instance in a Docker <a href="https://docs.weaviate.io/academy/py/starter_multimodal_data/setup_weaviate/create_docker">container</a>.</p><p>I can always query data through the Weaviate API.. but having a GUI client is always better. For this I used <a href="https://github.com/Shah91n/WeaviateDB-Cluster-WebApp">WeaviateCluster</a> — It’s not an official GUI client, but good enough for me.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CdeXhWuYFoxrXuh03eJOIg.png" /><figcaption>Storage model collection schema</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ocRcS7fOYKbmgrEXQGH2XA.png" /><figcaption>Testing collection data</figcaption></figure><h3>What’s Next?</h3><p>We’ve walked through the first steps of turning Markdown notes into “meaning”: a <strong>.NET Core parser</strong> to process and split notes, a <strong>Python embedding service</strong> powered by SentenceTransformers, and a <strong>vector database in Weaviate</strong> to store and retrieve knowledge at a semantic level. These pieces lay the foundation for a system that doesn’t just capture text, but understanding.</p><p>I hope this helps you think about your own knowledge pipelines, and maybe gives you ideas for how to bring structure and semantics to your notes. Next up in: <strong>Searching Smarter: Building an AI-Ready Knowledge Base</strong>, where we’ll dive into how to query, retrieve, and unlock the potential of vectors.</p><p>Stay tuned — keep building, keep learning.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=329dc7f9df80" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Architect’s Workshop: MindLog — Architecting an AI-Powered Knowledge Base]]></title>
            <link>https://medium.com/@marcio.duarte89/the-architects-workshop-mindlog-architecting-an-ai-powered-knowledge-base-d57f6fbc3b35?source=rss-ccdfbda98ebf------2</link>
            <guid isPermaLink="false">https://medium.com/p/d57f6fbc3b35</guid>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[dotnet-core]]></category>
            <dc:creator><![CDATA[Márcio Duarte]]></dc:creator>
            <pubDate>Thu, 21 Aug 2025 12:31:15 GMT</pubDate>
            <atom:updated>2025-08-21T12:31:15.249Z</atom:updated>
            <content:encoded><![CDATA[<h3><a href="https://medium.com/p/1cae5e0b834d/edit">The Architect’s Worksho</a>p: MindLog — Architecting an AI-Powered Knowledge Base</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3CgT5v6XpBifADcMiflpSw.jpeg" /></figure><p>Welcome back to <em>The Architect’s Workshop</em>, where we learn about software development and architecture by building. Instead of working only with theory or abstract diagrams, each post series attempts to take on practical problems and explore how to design, implement, and evolve a solution.</p><p>The focus isn’t on creating the “perfect” architecture, but on making decisions, understanding trade-offs, and seeing how different components fit together in practice.</p><p>If you aren’t a medium subscriber you can read this post for free <a href="https://marcioduarte89.github.io/article/mindlog-architecting-ai-powered-knowledge-base">here</a>.</p><h3>Mindlog: Architecting an AI-Powered Knowledge Base</h3><p>Like many people in tech, I take a lot of (technical) notes and do my own journaling using Obsidian. It’s literally my second brain.</p><figure><img alt="Obsidian Knowledge graph" src="https://cdn-images-1.medium.com/max/1021/1*7RJowAaYZik7_Go61AmyTA.png" /></figure><p>It’s great for capturing notes, linking ideas, and keeping knowledge portable. But… it’s static. Search and indexing works, but it’s “literal”. If I want to ask deeper questions, spot patterns, or get AI-assisted summaries, I quickly hit a wall.</p><p>In short:</p><ul><li><strong>I don’t just want to <em>store</em> knowledge -</strong> I want to be able to <em>query</em> it in smarter ways.</li><li>I want AI to help me surface connections across hundreds of markdown files.</li><li>And I’d love to use tools like Claude to converse with my knowledge base, not just browse it.</li><li>I usually revisit notes I&#39;ve written recently as I find its easier for me to retain information.</li></ul><h3>Enter Mindlog</h3><p><em>Mindlog</em> is my attempt to build an <strong>AI-augmented personal knowledge system</strong>. At its heart, it’s about bridging the gap between traditional note-taking and modern GenAI workflows.</p><p>Instead of leaving my notes locked in folders, I want to:</p><ul><li><strong>Parse and embed markdown notes</strong> into vector space.</li><li><strong>Store them in a vector database</strong> (Weaviate in my case) for semantic search.</li><li><strong>Expose them via an MCP server</strong> so other tools (like Claude desktop) can query them.</li><li><strong>Add an AI layer</strong> on top, so I can actually “talk to” my knowledge base.</li></ul><h3>High-Level Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*895wmE9BDscgqQsdgZfGNg.png" /><figcaption>MindLog Architecture</figcaption></figure><p>Let’s look at the moving parts (without diving too deep yet — that’s what later posts are for).</p><ul><li><strong>Obsidian (Notes Source):</strong> Where all my markdown notes live. Nothing changes here; it stays my central writing environment.</li><li><strong>Embedding Service:</strong> A lightweight Python microservice that parses notes, generates embeddings with sentence-transformers, using all-MiniLM-L6-v2.</li><li><strong>Markdown Parser: </strong>Parses all the markdown files, uses the <strong>Embeddings Service </strong>to generate the embeddings, then ships them to Weaviate.</li><li><strong>Weaviate (Vector Database):</strong> The brain of the system, where semantic search happens.</li><li><strong>MCP Server:</strong> A bridge that lets external apps (like Claude) query my notes.</li><li><strong>Claude Desktop:</strong> The “front-end intelligence” — letting me chat with my knowledge base, ask for summaries, or explore connections across notes, etc.</li></ul><p>This layered approach keeps each component focused: parsing and embedding is one job, search and retrieval is another, and the AI interface is yet another. That separation makes the system easier to evolve — if I swap out Weaviate later, or switch embedding models, the rest can stay intact.</p><h4>Why This Architecture?</h4><p>A few guiding principles shaped the design:</p><ul><li><strong>Keep notes portable.</strong> My Obsidian vault remains the single source of truth. No lock-in.</li><li><strong>Small, focused services.</strong> The embedding logic lives in a tiny Python service (for easier access to the SentenceTransformer model), not buried in the database or a massive app.</li><li><strong>Composable layers.</strong> Each part can be swapped: Claude could be replaced by another LLM, Weaviate by another vector DB, etc.</li><li><strong>Learn by doing.</strong> Each stage is an opportunity to deepen understanding: embeddings, vector search, AI interfaces — but grounded in a real project.</li></ul><h3>What’s Next?</h3><p>This post lays the foundation. In the next ones, I’ll dive into each part of the stack:</p><ul><li><em>Vectors in the Vault: From Markdown to Meaning</em> — turning notes into embeddings and introducing Weaviate.</li><li><em>Searching Smarter: Building an AI-Ready Knowledge Base</em> — wiring up the MCP server for semantic queries.</li><li><em>Conversations with MindLog</em> — plugging it all into Claude and exploring what it feels like to chat with your own second brain.</li></ul><p>Stay tuned — keep building, keep learning.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d57f6fbc3b35" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Architect’s Workshop: Navora — Reflections on Architecture, Trade-offs, and What I’d Do…]]></title>
            <link>https://medium.com/@marcio.duarte89/the-architects-workshop-navora-reflections-on-architecture-trade-offs-and-what-i-d-do-3ea7db7f4fd8?source=rss-ccdfbda98ebf------2</link>
            <guid isPermaLink="false">https://medium.com/p/3ea7db7f4fd8</guid>
            <category><![CDATA[dotnet]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[serverless-architecture]]></category>
            <category><![CDATA[c-sharp-programming]]></category>
            <category><![CDATA[software-architecture]]></category>
            <dc:creator><![CDATA[Márcio Duarte]]></dc:creator>
            <pubDate>Mon, 28 Jul 2025 19:44:52 GMT</pubDate>
            <atom:updated>2026-01-12T20:14:41.372Z</atom:updated>
            <content:encoded><![CDATA[<h3><a href="https://medium.com/p/1cae5e0b834d/edit">The Architect’s Worksho</a>p: Navora — Reflections on Architecture, Trade-offs, and What I’d Do Differently</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hja5V5Ls7002jnRfYdPQsQ.jpeg" /></figure><p>As I wrap up this blog series on Navora, I want to take a step back and reflect on the journey — what I set out to do, the architectural decisions I made, the trade-offs I accepted, and what I’d approach differently if I were to do it all over again.</p><h3>First Things First: This Was a Side Project</h3><p>Let’s be clear — Navora isn’t a real product. It’s a side project I’ve been building during the evenings, usually with just a couple of hours (being generous here..) to spare after work. That constraint shaped a lot of the decisions I made. Some features were designed at a high level but never implemented. Others were simplified or skipped entirely to keep the momentum going. The code was never intended to be production ready code — as the intent was to make the system work.</p><p>This series was never about building a production-ready system. It was about exploring architecture, experimenting with AWS services, and sharing what I learned along the way.</p><h3>Architectural Intent vs. Reality</h3><p>The architecture I designed was intentionally ambitious. I wanted to simulate a real-world telemetry platform with:</p><ul><li>IoT ingestion</li><li>Stream processing</li><li>Alerting</li><li>Admin APIs</li><li>Role-based access control</li></ul><p>But not everything made it into the implementation. Here are some of the things I considered but didn’t build:</p><ul><li>Dead Letter Queues (DLQs): I planned to add DLQs for SQS and Lambda to handle failed telemetry and alert messages. It’s a best practice, but it would’ve added complexity and time I didn’t have.</li><li>Tests: I scaffolded test projects for all services to support CI, but I didn’t write actual tests. In a real-world scenario, this would be a non-starter!</li><li>Alert De-duplication: I mentioned de-duplicating alerts and notifications, but this logic wasn’t implemented due to time constraints.</li><li>IoT Rule Simplification: Instead of using AWS IoT rules to route messages directly, I chose to forward everything to SQS and process it in Lambda. This gave me more architectural flexibility, but in a real system (at least for the beginning), I’d likely opt for a simpler, more cost-effective path where messages that contain alert constraints would be forwarded directly into a specific queue, and we could potentially even bypass SNS flow.</li><li>Authorization: Cognito and API Gateway were set up with basic auth flows. In production, I’d explore fine-grained access control, token scopes, and integration with enterprise identity providers.</li><li>Strategies to reduce infrastructure costs like storage, reads/writes (ex: introduction of cache; TTL to expire old data; Archive older data to colder storages for cheaper long term storage).</li></ul><h3>The Constraints of Testing in AWS</h3><p>Testing AWS services locally was not always trivial on this project. LocalStack helped, but not all services are supported (or free) or behave identically. SAM CLI was inconsistent, and I often had to deploy to AWS just to test things properly (which would incur in expenses).</p><p>Even when running Lambdas locally against real AWS resources (like DynamoDB or SNS), I’d hit subtle issues that only appeared after deployment, ex:</p><blockquote>When processing Telemetry history and getting the SNS topic to publish Telemetry alerts to SNS, the entire execution worked locally but it failed when running on AWS. It turned out to be the Lambda function didn’t have the necessary sns:Publish or sns:ListTopics permissions. AWS Cloudwatch was also not logging what was actually happening.</blockquote><p>That lack of confidence in local testing slowed me down and made iterations harder than it should’ve been.</p><p>I was not always confident if the entire stack would continue to operate correctly when deploying individual services. So end to end testing strategies would need to be put in place in a real production environment to mitigate these concerns.</p><h3>Vendor Lock-In: A Real Concern?</h3><p>Going all-in on AWS made development easier in some ways, but it also introduced tight coupling to the platform. Even with patterns like the repository pattern and service abstraction, some parts of the system — like DynamoDB queries — are inherently AWS-specific.</p><p>There are tools like AWS Annotations and frameworks that aim to abstract cloud providers code, but the reality is: <strong>if you build for AWS, you’re building <em>on</em> AWS</strong>. That’s not necessarily bad, <strong>but it’s a trade-off worth acknowledging</strong>.</p><h3>Let’s Talk About Cost</h3><p>One of the most eye-opening exercises was estimating how much this architecture would cost at scale for Navora. Here’s a rough breakdown based on:</p><ul><li>10,000 IoT devices publishing telemetry every 10 seconds (assuming we would always have drivers operating the vehicles 24/7).</li><li>AWS IoT Core → SQS → Lambda → DynamoDB.</li><li>5% of messages triggering alerts via SNS and another Lambda.</li><li>3–5 back-office users querying telemetry and alerts via API Gateway.</li></ul><p>The pricing values used in the previous breakdown are based on public AWS pricing documentation (approximate and simplified for estimation):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/341f70e22f560abe052c9480d5b09b12/href">https://medium.com/media/341f70e22f560abe052c9480d5b09b12/href</a></iframe><p>These are simplified averages and may vary by region or usage tier. For production-grade estimates, you’d want to use the <a href="https://calculator.aws/">AWS Pricing Calculator</a>.</p><h4>Monthly message volume:</h4><p><strong>Telemetry messages per month:</strong></p><ul><li>10,000 devices × 6 (each 10 second per min) × 60 × 24 × 30 = 259,200,000 messages.</li><li>Alert messages (5%):<br>259,200,000 × 0.05 = 12,960,000 alert messages.</li></ul><p>With the above message volume in mind and the cost per compute highlighted earlier— processing costs would equate to:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3ace12965a3ca8ed7b89b710fe4c8700/href">https://medium.com/media/3ace12965a3ca8ed7b89b710fe4c8700/href</a></iframe><p>That’s nearly $9,000/month — and that’s without factoring in storage, monitoring, or data transfer costs.</p><p><strong>Storage Cost estimation:</strong></p><p>Payload size — The payload we used averages around 300bytes per message. For DynamoDB, AWS charges based on item size rounded up to the nearest 1 KB. So it will be billed as 1 KB per item.</p><p>1 GB = 1,024 MB = 1,048,576 KB</p><ul><li>259,200,000 KB of Telemetry History ÷ 1,048,576 ≈ 247.2 GB</li><li>5 % of alerts = 259,200,000 × 0.05 = 12,960,000 KB ÷ 1,048,576 ≈ ~12.4 GB</li></ul><p>This equates to:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dd78622e2ba70d6c559a769377707965/href">https://medium.com/media/dd78622e2ba70d6c559a769377707965/href</a></iframe><p><strong>Updated Total Monthly Costs:</strong></p><p><strong>Compute costs: </strong>$8,855.05 + <strong>Storage costs: </strong>$64.90 = $8,919.95/month.</p><p>This would be one of the most important factors if we were pitching this architecture to a potential real client.</p><h4>Exploring Hybrid and Cost-Conscious Alternatives</h4><p>If this was a real-world project with a client concerned about infrastructure costs, nearly $9,000/month could be a tough sell. While the architecture we explored is scalable and cloud-native, it’s also heavily reliant on on-demand services like Lambda, DynamoDB, and IoT Core — which are convenient but not always the most cost-effective at scale. We should discuss all different types of trade-offs with the clients.</p><p><strong>Pragmatic alternatives:</strong></p><p>A hybrid approach might look like:</p><ul><li>Use IoT Core for ingestion (still cost-effective for device connectivity).</li><li>Replace Lambda with a Fargate service for telemetry processing and Admin Portal:<br>- More predictable pricing.<br>- Easier to manage shared state or in-memory caching.<br>- Lose automatic scaling granularity of Lambda.<br>- Need to manage container lifecycle, health checks, and deployments — Potentially higher operational overhead.</li><li>Use RDS or MongoDB for telemetry storage:<br>- Lower storage costs.<br>- More flexible querying in some cases.<br>- You manage scaling, backups, and availability.<br>- May require more infrastructure and monitoring.</li><li>Keep SNS for alert fan-out (cheap and reliable).</li><li>If the Admin Portal was low traffic and internal we could:<br>- Skip Cognito and use Microsoft Identity for authentication and replace API Gateway with a self-hosted Ocelot API Gateway, for example.<br>- Alternatively, we could use other cloud native Identity platforms which have free tiers available like Azure AD B2C or Auth0.</li></ul><h3>What I’d Do Differently</h3><p>If I were to start over or take this project further, here’s what I’d change:</p><ul><li>Simplify the architecture: Use IoT rules to route messages directly to DynamoDB or SNS where possible.</li><li>Add observability early: Logging, tracing, and metrics would’ve saved me hours of debugging.</li><li>Invest in end to end testing: This is a big one — Even basic integration tests would’ve improved confidence and reduced deployment surprises.</li><li>Explore cost-saving options: Use batching, compression, caching, and reserved capacity where appropriate. Usage of non-serverless approaches.</li><li>Design for portability: Abstract cloud-specific logic more aggressively to reduce lock-in.</li></ul><h3>Final Thoughts</h3><p>Navora was never about perfection. It was about <strong>learning</strong>, <strong>experimenting</strong>, and <strong>sharing</strong>. I hope this series gave you a glimpse into the thought process behind building cloud-native systems — and the trade-offs that come with it. The repository can be found at <a href="https://github.com/marcioduarte89/Navora">https://github.com/marcioduarte89/Navora</a>.</p><p>If you’ve followed along, thank you! I’d love to hear your thoughts, questions, or even your own war stories from the cloud trenches.</p><p>Until next time — keep building, keep learning.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3ea7db7f4fd8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Architect’s Workshop: Navora — Control Center - Designing the Admin Service for Telemetry…]]></title>
            <link>https://medium.com/@marcio.duarte89/the-architects-workshop-navora-control-center-designing-the-admin-service-for-telemetry-df2e57a54482?source=rss-ccdfbda98ebf------2</link>
            <guid isPermaLink="false">https://medium.com/p/df2e57a54482</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[c-sharp-programming]]></category>
            <category><![CDATA[serverless-architecture]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[dotnet]]></category>
            <dc:creator><![CDATA[Márcio Duarte]]></dc:creator>
            <pubDate>Thu, 17 Jul 2025 18:56:27 GMT</pubDate>
            <atom:updated>2025-07-17T18:56:27.850Z</atom:updated>
            <content:encoded><![CDATA[<h3><a href="https://medium.com/p/1cae5e0b834d/edit">The Architect’s Worksho</a>p: Navora — Control Center - Designing the Admin Service for Telemetry Management</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hja5V5Ls7002jnRfYdPQsQ.jpeg" /></figure><p>In Navora’s architecture, the <strong>Admin Service</strong> acts as the control center — a secure interface where administrators can monitor vehicle activity, review historical telemetry, and respond to alerts as they happen. Navoras Control Center brings observability and operational control to the heart of the system. In this post, we’ll explore how the Admin Service is designed and built using AWS serverless components. We’ll focus on how we provide secure access, expose telemetry data efficiently, and maintain a scalable, low-maintenance architecture that can evolve as the system grows.</p><h3>A Single Entry Point for Admin Operations</h3><p>Unlike user-facing APIs, which often split functionality across multiple endpoints and microservices, the Admin Service consolidates its logic into <strong>a single Lambda function</strong> behind an <strong>Amazon API Gateway (HTTP API)</strong>. This decision was intentional.</p><p>Because this service is relatively low-traffic and used primarily by internal staff, we optimized for simplicity over micro-level modularity. The Lambda function serves as the backend for multiple admin operations — including retrieving recent alerts, accessing historical telemetry, and filtering vehicle data. Each API route is routed internally to the appropriate handler within the same Lambda.</p><p>This design provides a clean, cohesive boundary around the admin domain and makes it easy to share infrastructure like database clients, logging utilities, and authentication checks — all without sacrificing performance for our expected load.</p><p>Of course, we’re aware that single-function architectures can grow unwieldy. To counteract this, the function should be modularized internally with clear separation between routes, domain logic, and integrations. This should make the service easy to navigate and extend while keeping the deployment footprint minimal.</p><h3>Securing Access with Cognito and API Gateway</h3><p>Given the sensitive nature of the data involved — including real-time location and alert information — strong access control is non-negotiable.</p><p>To achieve this, we integrated <strong>Amazon Cognito</strong> for user authentication. Admins sign in using Cognito’s hosted UI or through federated identity providers (such as an enterprise IdP), and their identity is validated via <strong>API Gateway’s Cognito authorizer</strong>.</p><p>This approach offers several advantages. First, it keeps the Lambda code clean — authentication is handled at the API Gateway layer, so only verified users can even reach the function. Second, it aligns with a broader serverless philosophy: offload what you can to managed services. Finally, Cognito scales easily with the organization, supporting user groups, role-based access, and multi-factor authentication out of the box.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/1*JsJXGINqyVQz0w6O0hg_Iw.png" /><figcaption>Navora Admin Service architecture</figcaption></figure><h4>Best Practices for Cognito Authentication in Admin Services</h4><p>When building admin interfaces — especially for internal control panels like Navora’s — it’s essential to follow <strong>secure, modern OAuth2 standards</strong>. The recommended pattern is:</p><p><strong>Authorization Code Flow with PKCE (Proof Key for Code Exchange).</strong></p><p>This flow is secure even in <strong>single-page apps (SPA)</strong> or <strong>mobile clients</strong>, where storing a client secret is not safe.</p><h4>Overview: Cognito Code Flow with PKCE</h4><p>Here’s how the full flow works step-by-step:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/505/1*gn5Ul6f0Nu36nhbtZO-t8w.png" /><figcaption>Code flow with PKCE</figcaption></figure><p><strong>1. Admin User Tries to Log In</strong></p><ul><li>Admin is redirected to Cognito’s <strong>Hosted UI</strong>.</li><li>The frontend app initiates an <strong>OAuth 2.0 Authorization Code Flow with PKCE</strong>, generating:</li><li>A <strong>code_verifier</strong> (random secret).</li><li>A <strong>code_challenge</strong> (hashed from code_verifier, usually using SHA-256).</li><li>The frontend app sends the user to:</li></ul><pre>https://&lt;your-domain&gt;.auth.&lt;region&gt;.amazoncognito.com/oauth2/authorize</pre><p>With query parameters like:</p><pre>response_type=code<br>client_id=...<br>redirect_uri=...<br>code_challenge=... (from code_verifier)<br>code_challenge_method=S256</pre><p><strong>2</strong>. <strong>User Logs In via Hosted UI</strong></p><ul><li>The user authenticates (e.g., with username/password or enterprise IdP via SAML/OIDC).</li><li>Cognito then <strong>redirects back to your app</strong> with a ?code=... query param.</li></ul><p><strong>3. Token Exchange (Backend or Frontend)</strong></p><ul><li>The app sends a POST request to:</li></ul><pre>https://&lt;your-domain&gt;.auth.&lt;region&gt;.amazoncognito.com/oauth2/token</pre><p>with:</p><pre>grant_type=authorization_code<br>code=&lt;authorization_code&gt;<br>client_id=&lt;your app client id&gt;<br>redirect_uri=&lt;same as before&gt;<br>code_verifier=&lt;original verifier&gt;</pre><p>If the code_verifier matches the previously submitted code_challenge, Cognito returns:</p><ul><li><strong>ID Token</strong> (JWT describing the user, used for identity)</li><li><strong>Access Token</strong> (JWT used to authorize API requests)</li><li><strong>Refresh Token</strong> (used to silently refresh tokens later)</li></ul><p><strong>4.</strong> <strong>Calling the Admin API</strong></p><p>The frontend/client app <strong>attaches the Access Token</strong> as a bearer token in an Authorization header:</p><pre>Authorization: Bearer eyJ... (access_token)</pre><ul><li><strong>API Gateway</strong> uses the <strong>Cognito Authorizer</strong> to validate the token.</li><li>If valid, API Gateway passes the request to the Lambda function with user claims.</li></ul><h4>Why Use Code Flow with PKCE?</h4><ul><li>No client secret needed — Ideal for public clients (SPAs, mobile).</li><li>Mitigates code interception — The code can’t be used without the verifier.</li><li>Leverages Cognito Hosted UI — You avoid building and securing your own login form.</li><li>Multi-Provider Support — Cognito can federate to enterprise IdPs, Google, Facebook, etc.</li><li>Standards-compliant — Aligns with OAuth 2.1 and OpenID Connect.</li></ul><h3>Reading from DynamoDB: Alerts and History</h3><p>The Admin Service primarily reads from two DynamoDB tables: The <strong>Telemetry</strong> A<strong>lerts</strong> and <strong>Telemetry History</strong>.</p><p>The <strong>Telemetry</strong> <strong>Alerts </strong>table captures all triggered events — from high temperature, low fuel — tagged with metadata such as severity, vehicle ID, timestamps and vehicle position. These are queried frequently to populate the dashboard with recent incidents.</p><p>The <strong>Telemetry</strong> <strong>History </strong>table stores rolling telemetry snapshots per vehicle — position, speed and other contextual signals over time. When administrators want to investigate the lead-up to a particular alert, they can retrieve this history to build a full picture of what was happening in the vehicle before and after the event.</p><p>We chose DynamoDB for its performance characteristics and fit for our access patterns. Admin operations tend to involve key-based or time-range queries — tasks DynamoDB excels at. It’s fully managed, scales elastically, and with on-demand capacity mode, we only pay for what we use.</p><p>While DynamoDB is optimized for these access patterns, we’re mindful of limitations around ad-hoc querying and indexing. That said, for this service’s use case — targeted lookups driven by admin workflows — it’s the right tool for the job.</p><p><strong>Example of query history by vehicle id:</strong></p><pre>private async Task&lt;List&lt;T&gt;&gt; Query&lt;T&gt;(long vehicleId)<br>{<br>    var telemetryData = new List&lt;T&gt;();<br>    var queryResults = _dynamoDbContext.QueryAsync&lt;T&gt;(vehicleId);<br>    while (!queryResults.IsDone)<br>    {<br>        telemetryData.AddRange(await queryResults.GetNextSetAsync());<br>    }<br><br>    return telemetryData;<br>}</pre><p>This is a classic usage of <strong>AWS SDK for .NET’s DynamoDBContext</strong>, which implements a <strong>lazy-loading paginated query</strong> under the hood. <br>How it works:</p><ul><li>QueryAsync&lt;T&gt;() returns a <strong>AsyncSearch&lt;T&gt; object</strong> — a wrapper that handles paging through DynamoDB results.</li><li>DynamoDB queries are <strong>paginated by default</strong>, with a <strong>maximum page size of 1MB</strong> or a user-specified Limit.</li><li>The SDK’s pattern is:<br>- Check IsDone<br>- If false, call GetNextSetAsync() to fetch the next page.<br>- Aggregate the results until done.</li></ul><p>This lazy iteration model (looping with IsDone) offers:</p><ul><li>Streamlined memory use — We don’t need to buffer everything at once — but you can if you want.</li><li>AWS-optimized SDK — Respects read capacity units (RCUs), retries, exponential backoff, etc.</li><li>Transparent pagination — We avoid manually stitching LastEvaluatedKey tokens.</li></ul><h4>DynamoDBContext vs DynamoDBClient</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2396bf8b134b3648b7cef4e1f34b1804/href">https://medium.com/media/2396bf8b134b3648b7cef4e1f34b1804/href</a></iframe><p><strong>Use </strong><strong>DynamoDBContext when:</strong></p><ul><li>You want quick mapping of objects (e.g., telemetry DTOs).</li><li>You’re doing basic Get/Query/Scan operations.</li></ul><p><strong>Use </strong><strong>DynamoDBClient when:</strong></p><ul><li>You need <strong>precise control</strong> over capacity units, filters, pagination, or consistency.</li><li>You want to use <strong>expression-based queries</strong> or PartiQL.</li><li>You’re building <strong>general-purpose data access layers</strong> or admin tooling.</li></ul><h3>Performance, Scalability, and Cost Considerations</h3><p>Despite its simplicity, this architecture scales remarkably well.</p><p>The single Lambda function handles all admin requests and scales horizontally with load. Cold starts are minimal thanks to AWS’s recent improvements and our use of lightweight dependencies. If traffic increases — say, during major fleet incidents — the service can handle it without manual intervention.</p><p>API Gateway’s HTTP API tier was chosen for its cost-effectiveness and lower latency compared to the REST API tier. And because everything is event-driven and pay-per-use, the system incurs no idle costs. This aligns with our goal of keeping the platform lean while still delivering enterprise-grade responsiveness.</p><h3>Putting It All Together</h3><p>Once we simulate a code change in the TelemetryAdmin service, open a PR, and merge it (we need to have a milestone in the PR):</p><p>The CI/CD pipeline will be triggered.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*vSSAZVnX2Mm2F8vWcEbUoQ.png" /><figcaption>CI/CD pipeline</figcaption></figure><p>Let’s see what was executed as part of the publish step:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/391/1*Z_RQBowbcf_NhFj5l0O1gg.png" /><figcaption>CD Publishing steps</figcaption></figure><p>A versioned release is created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*F4GbWYNx7TWVVdYp5GDMRw.png" /><figcaption>New Telemetry Admin service release</figcaption></figure><p>The CloudFormation stack is created/updated with all the resources to make this service work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*6Ju3XYENSd3DiGY8LSm4FQ.png" /><figcaption>Telemetry Admin Stack creation</figcaption></figure><p>Cognito has now been created</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*GIiXt13KPuVmrIC9CcLIUA.png" /><figcaption>User pool</figcaption></figure><p>and the AppClients been configured:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/1*cDPakywOOgfcN3WOGl88Ng.png" /><figcaption>Cognito and AppClient</figcaption></figure><p>We can now use the AppClient to verify the Identity Provider, OAuth grant types and OpenId scopes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bsW-6ii1OuoAhM_nsrI_LQ.png" /><figcaption>Cognito Login pages</figcaption></figure><blockquote>As you might have noticed, I&#39;ve left ImplicitGrant as an available grant type (instead of just having Authorization code with PKCE). As we haven’t created a real client (frontend) application, I&#39;ll be using this grant type for a little hack do get a valid access token to demo the service.</blockquote><p>We can use the Cognito’s Login page to create a user for testing:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/348/1*E-P17-Z8WnFP-9MzjNLjuA.png" /><figcaption>Out of the box Login pages</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/222/1*We3YLR102AJHsNQ3rqFtFQ.jpeg" /><figcaption>User creation</figcaption></figure><p>Once the user is created and verified, we can choose to login with the user. We will be redirected to the following page:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gi5DR5wVPOGCxO45xPu24w.png" /><figcaption>Authentication</figcaption></figure><p>For the little hack I mentioned above, we can update the response_type=codeabove with response_type=token.This will return to the callback URL with the access tokens in the URL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iIWQwj_lqo4-iPZYf-jg1w.png" /><figcaption>Callback to get the access tokens</figcaption></figure><p>We can now grab the token and check the claims provided.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*ogHopAq1N-BhAJCEIoIEfA.png" /></figure><p>Let’s now check our API Gateway. We can see the different routes, its authorizers and lambda integrations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/1*pH4a4RitgymaIYb1uZRjJA.png" /><figcaption>API Gateway Routes and authorizer</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/1*LjFxr4m4x1K3BFH6hUjScg.png" /><figcaption>API Gateway Lambda Integrations</figcaption></figure><p>As part of the CloudFormation deployment, the API Gateway was automatically deployed to a dev stage — so its ready to use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zJ_XsCGvzZVYLX3BRzuDmw.png" /><figcaption>API Gateway stage</figcaption></figure><p>Let’s now try our API’s — The first test is calling any of the routes without a valid access token:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*XtzZgFidljGuIUKjsP_WeA.png" /><figcaption>Unauthorized API call</figcaption></figure><p>Once we provide the Access token we got before and call the live Telemetry API we get the current live positions and details for our vehicles:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fMeXo5LlncA8XVmQ5jUYjA.png" /><figcaption>Live telemetry</figcaption></figure><p>We can get a specific vehicle Telemetry History:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/1*R2YgK-Wf138Bi5CaW_MlPw.png" /><figcaption>Vehicle Telemetry history</figcaption></figure><p>We can also get all Vehicle information that emitted an alert:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*BOGqOcS2a-oz8G6WnfhUzw.png" /><figcaption>Vehicle Alerts Telemetry</figcaption></figure><p>And finally we can see specific vehicle Alerts:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/824/1*sRdU1aVasB5zZr-yERp7Tw.png" /></figure><h3>Looking Ahead</h3><p>The Admin Service could continue to evolve. As alert volumes grow and admin workflows become more sophisticated, we could explore the addition of server-side filtering, paginated queries, and potentially a caching layer (such as Lambda-level in-memory caching or ElastiCache) for frequently accessed alert views.</p><p>We could also consider integration with AWS EventBridgeor WebSocketsto support real-time push notifications for critical alerts — giving admins faster situational awareness.</p><h3>Wrapping Up</h3><p>As we’ve seen, the Admin Service leverages a minimal, serverless design — centered around a single, secure Lambda function and DynamoDB-backed data access — we’ve built a reliable and extensible control center for fleet monitoring.</p><p>It’s a great example of how a thoughtful approach to simplicity can yield powerful results, especially when backed by the flexibility and scalability of AWS’s serverless ecosystem.</p><p>In the next post — which will be a very important ones — we’ll dive on what we have learned, reflections on Architecture, trade-offs, and what we would do differently, I hope to see you there.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df2e57a54482" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Architect’s Workshop: Navora — Detect and Respond — Architecting the Alerts Processing Service]]></title>
            <link>https://medium.com/@marcio.duarte89/the-architects-workshop-navora-detect-and-respond-architecting-the-alerts-processing-service-a2b4e7cc3d7f?source=rss-ccdfbda98ebf------2</link>
            <guid isPermaLink="false">https://medium.com/p/a2b4e7cc3d7f</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[csharp-programming]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[dotnet]]></category>
            <category><![CDATA[serverless-architecture]]></category>
            <dc:creator><![CDATA[Márcio Duarte]]></dc:creator>
            <pubDate>Tue, 08 Jul 2025 20:16:53 GMT</pubDate>
            <atom:updated>2025-07-08T20:16:53.057Z</atom:updated>
            <content:encoded><![CDATA[<h3><a href="https://medium.com/p/1cae5e0b834d/edit">The Architect’s Worksho</a>p: Navora — Detect and Respond — Architecting the Alerts Processing Service</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hja5V5Ls7002jnRfYdPQsQ.jpeg" /></figure><p>In any telemetry-driven system, detecting anomalies is only half the work — responding to them quickly and reliably is just as critical. In Navora, the alerting pipeline is designed to be scalable, loosely coupled, and extensible, using Amazon SNS, Lambda, and a purpose-built DynamoDB Alerts table.</p><h3>Why SNS for Alerts Distribution?</h3><p>We use Amazon SNS (Simple Notification Service) as the backbone of our alert distribution system. It allows us to fan out alert messages to multiple subscribers in real time, without coupling producers to consumers.</p><p>Amazon SNS delivers alert messages directly to a Lambda function (Telemetry Alerts). This integration is powerful because it’s:</p><ul><li><strong>Fully managed</strong>: No need to poll or manage consumers.</li><li><strong>Highly scalable</strong>: SNS can handle millions of messages per second, ensuring that alert throughput scales with telemetry volume. SNS fans out messages to all subscribers in parallel.</li><li><strong>Low latency</strong>: Messages are typically delivered to Lambda within milliseconds.</li><li><strong>Flexibility: </strong>Multiple subscribers can independently consume alerts.</li></ul><h3>What Happens on Failure?</h3><p>When SNS invokes a Lambda function, it expects a successful response. If the Lambda fails (e.g., due to an exception or timeout), SNS will:</p><ol><li><strong>Retry the message</strong>:<br>- SNS retries up to 3 times with exponential backoff.<br>- If the Lambda continues to fail, the message is considered undeliverable.</li><li><strong>Dead-Letter Queue (DLQ) (optional but recommended)</strong>:<br>- You can configure an SQS DLQ for the Lambda subscription.<br>- Failed messages are sent to the DLQ after all retries are exhausted.<br>- This allows for manual inspection, reprocessing, or alerting on failures.</li><li><strong>Monitoring</strong>:<br>- CloudWatch metrics and alarms can be used to track invocation failures and retry attempts.<br>- This is critical for maintaining observability and operational confidence.</li></ol><h3>DynamoDB Alerts Table: Designing for Query Efficiency</h3><p>Unlike the main telemetry table (which uses VehicleId and Timestamp as Partion and sort keys), the Alerts table is optimized for a different access pattern: querying alerts across all vehicles for a given day.</p><p>To support this, we use:</p><ul><li>Partition Key: AlertDate (e.g., 2025-06-13).</li><li>Sort Key: VehicleId#Timestamp.</li></ul><p>This schema allows us to:</p><ul><li>Efficiently retrieve all alerts for a specific day.</li><li>Avoid full table scans.</li><li>Maintain a logical grouping of alerts by time and source.</li></ul><p>This is a great example of designing DynamoDB tables based on access patterns, not just data structure.</p><h3>Idempotency and Deduplication in Alert Processing</h3><p>In distributed systems, especially those involving retries and asynchronous messaging (like SNS → Lambda), idempotency is essential to ensure that the same alert isn’t processed or stored multiple times due to retries or duplicate messages.</p><h4>How Navora Handles Idempotency</h4><p>To prevent duplicate alert records in DynamoDB, the Telemetry Alert Processor Lambda uses a conditional write:</p><pre>var request = new PutItemRequest<br>{<br>    TableName = &quot;Alerts&quot;,<br>    Item = item,<br>    ConditionExpression = &quot;attribute_not_exists(AlertId)&quot;<br>};</pre><p>This ensures that if an alert with the same AlertId already exists, the write is skipped. This is particularly useful when:</p><ul><li>SNS retries a failed Lambda invocation.</li><li>The same alert is accidentally published more than once.</li></ul><h4>Deduplication Strategy</h4><ul><li>AlertId could be generated deterministically (e.g., a hash of VehicleId + Timestamp + AlertType). This uniquely identifies an alert.</li><li>This allows the system to safely retry without risk of duplication.</li><li>It also enables idempotent downstream processing, such as logging, analytics, or notifications.</li></ul><h3>Strategies to Control Notification Volume:</h3><p>Some potential strategies to control notifications:</p><blockquote>These strategies weren’t implemented as part of the project.</blockquote><ol><li><strong>Track Notification State:</strong><br>- Store a Notified flag or timestamp in the Telemetry Data table (and ignore notification in case of retrying).<br>- Only send a notification if it hasn’t been sent yet.</li><li><strong>Throttle or Debounce Alerts:</strong><br>- Implement logic to suppress repeated alerts for the same condition within a time window (e.g., 5 minutes).<br>- Useful for noisy sensors or rapidly fluctuating metrics.</li><li><strong>Batch Notifications:<br>- </strong>Aggregate alerts and send summaries instead of individual messages.<br>- Especially useful for email or dashboard-based alerting.</li><li><strong>Use SNS Message Attributes:<br>- </strong>Include metadata like AlertType, Severity, or Source to allow subscribers to filter or route messages intelligently.</li></ol><p>If combining and implementing the above strategies, we can ensure alerts are:</p><ul><li>Processed exactly once.</li><li>Stored without duplication.</li><li>Notified responsibly.</li></ul><p>This design not only improves system reliability but also enhances the user experience by reducing noise, costs and ensuring meaningful, actionable alerts.</p><blockquote>Given this feature architecture and details is very similar to the previous post, I won’t add any screenshots or code snippets.</blockquote><p>With the alerting pipeline in place, Navora is now equipped to detect and respond to critical telemetry events in real time. But alerts are only part of the story — administrators need visibility and control over the system to manage configurations, review historical data, and interact with telemetry in meaningful ways. In the next post, we’ll explore the Telemetry Admin API, how it’s secured using Cognito, and how it exposes key functionality through a scalable, serverless interface. Stay tuned as we continue building out the operational backbone of Navora.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a2b4e7cc3d7f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Architect’s Workshop: Navora — Streaming the Signal: Designing the IoT Telemetry Ingestion…]]></title>
            <link>https://medium.com/@marcio.duarte89/the-architects-workshop-navora-streaming-the-signal-designing-the-iot-telemetry-ingestion-bd1492de9298?source=rss-ccdfbda98ebf------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd1492de9298</guid>
            <category><![CDATA[serverless-architecture]]></category>
            <category><![CDATA[csharp-programming]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[dotnet]]></category>
            <dc:creator><![CDATA[Márcio Duarte]]></dc:creator>
            <pubDate>Mon, 30 Jun 2025 20:21:50 GMT</pubDate>
            <atom:updated>2025-06-30T20:21:50.875Z</atom:updated>
            <content:encoded><![CDATA[<h3><a href="https://medium.com/p/1cae5e0b834d/edit">The Architect’s Worksho</a>p: Navora — Streaming the Signal: Designing the IoT Telemetry Ingestion Pipeline</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hja5V5Ls7002jnRfYdPQsQ.jpeg" /></figure><p>In the previous <a href="https://medium.com/@marcio.duarte89/the-architects-workshop-navora-laying-the-foundation-8f3242cb1796">post</a>, we presented the functional and non functional requirements for Navora, as well as the proposed architecture. In this post we will breakdown how we’ve designed and implemented the IoT Telemetry ingestion pipeline.</p><p>In IoT systems, where devices continuously transmit data, the ingestion pipeline plays a critical role. It’s the entry point for telemetry — responsible for reliably capturing, transporting, and preparing data for downstream processing. In Navora, we needed a solution that could:</p><ul><li>Handle high-frequency input from thousands of vehicles.</li><li>Maintain low latency.</li><li>Scale without manual intervention.</li></ul><p>To meet these demands, we designed a serverless, event-driven pipeline using <strong>MQTT, AWS IoT Core, SQS, Lambda</strong>, and <strong>DynamoDB</strong>. This architecture allows us to decouple ingestion from processing, ensure message durability, and handle telemetry efficiently — while remaining flexible enough to support alerting and long-term storage.</p><h3>Why MQTT?</h3><p>We chose MQTT for telemetry ingestion because it is a lightweight, publish-subscribe protocol ideal for constrained devices. Compared to HTTP or WebSocket&#39;s, MQTT offers:</p><ul><li><strong>Low overhead:</strong> Ideal for frequent, small payloads like telemetry data.</li><li><strong>Persistent connections:</strong> Reduces connection churn and latency.</li><li><strong>Built-in QoS levels:</strong> Supports delivery guarantees (at most once, at least once, exactly once).</li><li><strong>Efficient bandwidth usage:</strong> Especially important when scaling to thousands of devices.</li></ul><h4>MQTT Architecture</h4><p>MQTT uses a broker-based architecture:</p><ol><li><strong>Publisher</strong>: Sends messages to a topic.</li><li><strong>Broker</strong>: Receives, filters and distributes messages.</li><li><strong>Subscriber</strong>: Receives messages from subscribed topics.</li></ol><h4>MQTT Consistency Guarantees</h4><p>MQTT has different Quality of Services (QoS) to define delivery guarantees:</p><p>QoS 0 — At most once:</p><ul><li>No guarantee of delivery.</li><li>Message is sent once, no acknoledgement.</li><li>No retry if message is lost.</li><li>Fastest, lowest overhead.</li></ul><p>QoS 1 — At least once:</p><ul><li>Message is <strong>guaranteed to arrive</strong>, but may be <strong>duplicated.</strong></li><li>Sender retries until it gets an acknowledgment (PUBACK).</li><li>Duplicates possible — subscriber must handle idempotency.</li></ul><p>QoS 2 — Exactly once:</p><ul><li>Message is guaranteed to arrive <strong>exactly once</strong>.</li><li>Uses a four-step handshake.</li><li>Most reliable, but highest overhead.</li></ul><h4>MQTT Ordering Guarantees</h4><p>MQTT guarantees <strong>per-topic, per-client session</strong> ordering for QoS 1 and 2. To maintain order, the broker waits for acknowledgments before sending the next message. If persistent sessions are enabled, unacknowledged messages can be redelivered when the client reconnects.</p><p>Here’s a little summary for MQTT consistency and order guarantees:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/af23275c6bb5ee2856fccf1bdb42bac5/href">https://medium.com/media/af23275c6bb5ee2856fccf1bdb42bac5/href</a></iframe><h3>How AWS IoT Core Fits In</h3><p>AWS IoT Core acts as a <strong>fully managed MQTT broker</strong>, handling:</p><ul><li>Device authentication and authorization.</li><li>Topic-based message routing.</li><li>Scalable ingestion.</li><li>Seamless integration with AWS services (SQS, Lambda, etc.)..</li></ul><p>This lets us focus on business logic instead of managing broker infrastructure.</p><h4>Publishing Telemetry with MQTTnet</h4><p>Now, let’s look at how we publish telemetry from a device using <a href="https://github.com/dotnet/MQTTnet">MQTTnet</a>.</p><blockquote>Prerequisite: The device (Thing) must be registered in AWS, with its certificate, policy, and endpoint properly configured.</blockquote><pre>var config = new ConfigurationBuilder()<br>     .AddJsonFile($&quot;appsettings.json&quot;)<br>     .Build();<br><br>var iotSettings = config.GetSection(&quot;iotSettings&quot;).Get&lt;IoTSettings&gt;();<br><br>var caCert = X509Certificate.CreateFromCertFile(&quot;Path to *.pem file&quot;);<br>var clientCert = new X509Certificate2(&quot;Path to *.cert.pfx&quot;, &quot;AWS-Cert-Password&quot;);<br><br>var caChain = new X509Certificate2Collection();<br>caChain.ImportFromPem(caCert.GetCertHashString());<br><br>var mqttClientOptions = new MqttClientOptionsBuilder()<br>    .WithClientId(Guid.NewGuid().ToString())<br>    .WithTcpServer(iotSettings.Endpoint, iotSettings.Port)<br>    .WithTlsOptions(<br>        new MqttClientTlsOptionsBuilder()<br>        .WithTrustChain(caChain)<br>        .WithClientCertificates(new List&lt;X509Certificate2&gt; { clientCert })<br>        .WithSslProtocols(System.Security.Authentication.SslProtocols.Tls12)<br>        .Build())<br>    .WithCleanStart(false) // persistent session<br>    .WithProtocolVersion(MqttProtocolVersion.V500)<br>    .Build();<br><br>var mqttFactory = new MqttClientFactory();<br>var mqttClient = mqttFactory.CreateMqttClient();<br><br>var connection = await mqttClient.ConnectAsync(mqttClientOptions, CancellationToken.None);<br><br>var vehicleTelemetryData = new VehicleTelemetryData()<br>{<br>    VehicleId = 1,<br>    TimeStamp = DateTime.UtcNow,<br>    Speed = 60,<br>    CargoTemperature = 50,<br>    EngineTemperature = 50,<br>    FuelLevel = 90,<br>    IgnitionStatus = true,<br>    Location = new Nest.GeoCoordinate(51.5072, 0.1276)<br>};<br><br> var serializedVehicle = JsonSerializer.Serialize(vehicleTelemetryData);<br><br> var message = new MqttApplicationMessageBuilder()<br>     .WithTopic($&quot;{iotSettings.Topic}/{vehicleTelemetryData.VehicleId}&quot;)<br>     .WithPayload(serializedVehicle)<br>     .WithQualityOfServiceLevel(MqttQualityOfServiceLevel.AtLeastOnce)<br>     .Build();<br><br> await mqttClient.PublishAsync(message, CancellationToken.None);</pre><p>This client uses:</p><ul><li>TLS for secure communication.</li><li><strong>Persistent sessions</strong> for ordered delivery.</li><li><strong>QoS 1 </strong>for at-least-once delivery.</li></ul><p>To test if messages are being received:</p><ul><li>Create a subscriber client with the topic format: {iotSettings.Topic}/{vehicleId}.</li><li>Or use the AWS IoT MQTT test client to monitor published messages (also needs to have a topic specified).</li></ul><h3>Decoupling with SQS</h3><p>Once messages are received by IoT Core, they are forwarded to <strong>Amazon SQS</strong>. This decoupling is critical for:</p><ul><li><strong>Scalability:</strong> SQS can buffer spikes in telemetry traffic, preventing downstream overload.</li><li><strong>Reliability:</strong> Messages are durably stored until processed.</li><li><strong>Flexibility:</strong> Enables asynchronous processing and retries without blocking ingestion.</li></ul><h3>Processing with Lambda and DynamoDB</h3><p>Our <strong>Telemetry Processor Lambda</strong> is triggered by SQS and processes messages in <strong>batches</strong> (typically 10 per batch).</p><p>Why batching helps:</p><ul><li>Reduces Lambda invocation overhead.</li><li>Increases throughput.</li></ul><p>But batching has caveats:</p><ul><li>If one message fails, the entire batch is retried after the visibility timeout.</li><li>Without <strong>idempotency</strong>, duplicates can corrupt data.</li><li>This can lead to <a href="https://docs.aws.amazon.com/prescriptive-guidance/latest/lambda-event-filtering-partial-batch-responses-for-sqs/best-practices-partial-batch-responses.html#snowball-anti-patterns"><strong>snowball effects</strong></a> where failed messages cause growing retries.</li></ul><p>Mitigating the risks:</p><ul><li><strong>Selective acknowledgment:</strong> We only delete messages from the queue that were successfully processed, allowing partial batch success.</li><li><strong>Idempotency:</strong> We use conditional writes in DynamoDB.</li></ul><p>In order to configure Partial batch response we must also enable ReportBatchItemFailures in the Lambda’s event source mapping to make only the failed messages visible again.</p><blockquote>The above settings will be set in the CloudFormation Stack.</blockquote><p>The code snippet below shows how we are handling the processing of messages from the SQS Queue in the Telemetry Processor Lambda Function.</p><pre>[LambdaFunction]<br>public async Task&lt;SQSBatchResponse&gt; FunctionHandler([FromServices] IAmazonDynamoDB dynamoDbClient, SQSEvent evnt, ILambdaContext context)<br>{<br>    List&lt;BatchItemFailure&gt; batchItemFailures = [];<br><br>    // Iterate over the messages in the batch<br>    foreach (var message in evnt.Records)<br>    {<br>        try<br>        {<br>            await ProcessMessageAsync(dynamoDbClient, message, context);<br>        }<br>        catch (Exception ex)<br>        {<br>            // TODO: handle the exception appropriately!<br><br>            //Add failed message identifier to the batchItemFailures list<br>            batchItemFailures.Add(new BatchItemFailure { ItemIdentifier = message.MessageId });<br>        }<br>    }<br><br>    // return only the message that failed<br>    return new SQSBatchResponse(batchItemFailures);<br>}<br><br>private async Task ProcessMessageAsync(IAmazonDynamoDB dynamoDbClient, SQSEvent.SQSMessage message, ILambdaContext context)<br>{<br>    var payload = JsonSerializer.Deserialize&lt;VehicleTelemetryData&gt;(message.Body, OPTIONS);<br><br>    // set the payload to be sent to DynamoDB<br>    var item = new Dictionary&lt;string, AttributeValue&gt;<br>    {<br>        { &quot;vehicleId&quot;, new AttributeValue { N = payload.VehicleId.ToString() } },<br>        { &quot;timestamp&quot;, new AttributeValue { S = payload.Timestamp.ToString(&quot;o&quot;) } },<br>        { &quot;location&quot;, new AttributeValue { S = payload.Location.ToString() } },<br>        { &quot;speed&quot;, new AttributeValue { S = payload.Speed.ToString() } },<br>        { &quot;fuelLevel&quot;, new AttributeValue { S = payload.FuelLevel.ToString() } },<br>        { &quot;engineTemperature&quot;, new AttributeValue { S = payload.EngineTemperature.ToString() } },<br>        { &quot;cargoTemperature&quot;, new AttributeValue { S = payload.CargoTemperature.ToString() } },<br>        { &quot;ignitionStatus&quot;, new AttributeValue { S = payload.IgnitionStatus.ToString() } },<br>        { &quot;messageId&quot;, new AttributeValue { S = message.MessageId.ToString() } },<br>    };<br><br>    // by checking the Message Id in the condition we make sure we are not<br>    // inserting duplicated data.<br>    var request = new PutItemRequest<br>    {<br>        TableName = &quot;TelemetryData&quot;,<br>        Item = item,<br>        ConditionExpression = &quot;attribute_not_exists(MessageId)&quot;<br>    };<br><br>    var response = await dynamoDbClient.PutItemAsync(request);<br><br>    // If the item was inserted in DynamoDB and has an Alert, then publish the message to the alert topic<br>    if (response.HttpStatusCode == HttpStatusCode.OK &amp;&amp; payload.HasAlert())<br>    {<br>        // Gets the AlertsTopic Arn<br>        var topicArn = await _topicsService.GetTopicArnByName(&quot;AlertsTopic&quot;);<br><br>        if (topicArn is null)<br>        {<br>            // TODO: handle unexpected flow here (move message to error queue for example)<br>            // TODO: Log the error<br>            Console.WriteLine(&quot;No topic has been found&quot;);<br>            return;<br>        }<br><br>        var publishRequest = new PublishRequest<br>        {<br>            TopicArn = topicArn,<br>            Message = message.Body,<br>        };<br><br>        // TODO: Handle the result of the publish<br>        await _snsService.PublishAsync(publishRequest);<br>    }<br>}</pre><h4>A Note on AWS Lambda Annotations</h4><p>We use <a href="https://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/aws-lambda-annotations.html"><strong>AWS Lambda Annotations</strong></a> to simplify function setup. This approach eliminates boilerplate code for:</p><ul><li>Dependency injection.</li><li>Input/output mapping.</li><li>Validation.</li></ul><p>The annotations generate the necessary handlers during build time using Source Code generators. We need to configure the Lambda to point to the generated function entry point, otherwise it will attempt to use the main one in the project.</p><h3>Putting It All Together</h3><p>Once we simulate a code change in the TelemetryProcessor service, open a PR, and merge it (we need to have a milestone in the PR):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/336/1*6IheuC6-Pf3NpHmD7v1EuA.png" /><figcaption>PR open with Service milestone</figcaption></figure><ol><li>The CI/CD pipeline will be triggered.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*-ohi7B-7NOPcIcnnLg_Pvg.png" /><figcaption>CI/CD pipeline</figcaption></figure><p>Let’s see what was executed as part of the publish step:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/259/1*r8QAMTv8P2iASAW6fv7EXg.png" /><figcaption>CI pipeline</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/1*dtUqCsq8mxjapy_9OViO1g.png" /><figcaption>CD pipeline</figcaption></figure><p>2. A versioned release is created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*h9yew_wwEd2juzwQnO_m7A.png" /><figcaption>Service release</figcaption></figure><p>3. A new Docker image is published to ECR:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/1*GrZMyE1lea6ANtRLE-hbaQ.png" /><figcaption>Telemetry Processor in ECR</figcaption></figure><p>The CloudFormation stack is created/updated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PONuI4JXsyEKvx2i8Gjrlw.png" /><figcaption>Created stacks</figcaption></figure><p>4. All required resources — Thing, SQS, Lambda, DynamoDB, Roles— are deployed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*Gw7EwQ2_N6bQDGa2VMGZSw.png" /><figcaption>Aws Thing</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o8zqO_gDJdmUyDtyHnmVew.jpeg" /><figcaption>Forwarding rule</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YBTbhVMzMHuPHi-CxZQgww.png" /><figcaption>Lambda Function</figcaption></figure><p>We can now use the <strong>IoTDeviceApp</strong> (or similar) to generate real telemetry traffic. While this data won’t trigger alerts (yet), it proves the ingestion flow is working end-to-end.</p><p>We can validate this by:</p><ul><li>Inspecting <strong>DynamoDB</strong> to see new entries appear every 10 seconds.</li><li>Monitoring <strong>CloudWatch</strong> logs and metrics for throughput and billing.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vGJYmdKZC_U-vvi-JHp2-A.png" /><figcaption>TelemetryData DynamoDB table</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m-qRIRYXt1vYS753c5N_Yg.png" /><figcaption>Telemetry Processor Lambda Function monitoring</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tdf3Ku59vDLh0by79V6iHg.jpeg" /><figcaption>Billing</figcaption></figure><h3>Wrapping Up</h3><p>This has been a deep dive into telemetry ingestion — from MQTT design decisions to AWS implementation details and resilience patterns. We’ve explored the critical role of decoupling and idempotency in building robust serverless data pipelines.</p><p>In the next post, we’ll explore the <strong>Alerts Processing Service</strong>, which kicks in when telemetry indicates abnormal vehicle behavior.</p><p><strong>Thanks for reading — see you in the next post!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd1492de9298" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Architect’s Workshop: Navora — Laying the Foundation]]></title>
            <link>https://medium.com/@marcio.duarte89/the-architects-workshop-navora-laying-the-foundation-8f3242cb1796?source=rss-ccdfbda98ebf------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f3242cb1796</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[dotnet]]></category>
            <category><![CDATA[csharp-programming]]></category>
            <category><![CDATA[serverless-architecture]]></category>
            <dc:creator><![CDATA[Márcio Duarte]]></dc:creator>
            <pubDate>Mon, 23 Jun 2025 21:30:16 GMT</pubDate>
            <atom:updated>2025-06-30T14:24:24.756Z</atom:updated>
            <content:encoded><![CDATA[<h3><a href="https://medium.com/p/1cae5e0b834d/edit?source=your_stories_page--------------------------------------------">The Architect’s Worksho</a>p: Navora — Laying the Foundation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hja5V5Ls7002jnRfYdPQsQ.jpeg" /></figure><p>In the previous <a href="https://medium.com/@marcio.duarte89/welcome-to-the-architects-workshop-1cae5e0b834d">post, </a>we introduced the concept of The Architect’s Workshop and its first Series — Navora. In this post we will define the functional and non-functional requirements for our fictional company.</p><h3>Project Overview</h3><blockquote>We operate a logistics company with a fleet of delivery vehicles across the UK. We need a cloud-native system to monitor our fleet in real time. The system should ingest telemetry data from vehicle-mounted IoT devices, process and store this data, and provide real-time alerts and analytics to fleet managers.</blockquote><h3>Functional Requirements</h3><h4>Actors:</h4><p><strong>Vehicle device (IoT Sensor):</strong></p><ul><li>Sends telemetry data every 10 seconds.</li><li>Data includes:<br>- Vehicle Id (number)<br>- Timestamp (ISO 8601)<br>- Location (lat, long)<br>- Speed (km/h)<br>- Fuel Level (%)<br>- Engine Temp (°C)<br>- Cargo Temp (°C)<br>- Ignition Status (on/off)</li></ul><p><strong>Fleet Manager:</strong></p><ul><li>Views real-time vehicle locations</li><li>Receives alerts for:<br>- Overheating engine (&gt; 100ºC)<br>- Low fuel (&lt;15%)<br>- Cargo temperature out of range (2ºC — 8ºC)</li><li>Views historical data.</li></ul><h3>Non-Functional Requirements:</h3><ul><li>Scalability: Must support up to 10,000 vehicles sending data every 10 seconds.</li><li>Latency: Real-time data should be available within 3 seconds of ingestion.</li><li>Durability: All telemetry must be stored for at least 1 year.</li><li>Security: Data must be encrypted in transit and at rest.</li><li>Availability: Ensure 99.9% uptime for APIs and alerting.</li></ul><h3>Technical Requirements:</h3><p>To meet the functional and non-functional requirements, a cloud-based serverless architecture is proposed.</p><p>Project and architecture setup:</p><ul><li>A monorepo in GitHub with CI/CD via GitHub Actions.</li><li>CloudFormation for infrastructure as code.</li><li>AWS IoT Core → SQS → Lambda for telemetry ingestion and processing.</li><li>SNS + Lambda(s) for alerting.</li><li>API Gateway + Cognito + Lambda for admin APIs.</li><li>DynamoDB to persist telemetry time-series data.</li><li>Cloudwatch for Logging, Monitoring and tracing.</li><li>S3 for persistent storage.</li></ul><h4>Benefits of the Proposed Architecture</h4><p><strong>Scalability</strong></p><blockquote>Requirement: Support up to 10,000 vehicles sending data every 10 seconds.</blockquote><ul><li>AWS Lambda and SQS scale automatically with demand. We don’t need to provision or manage servers to handle spikes in telemetry data.</li><li>IoT Core can handle millions of messages per second, and SQS decouples ingestion from processing, ensuring that bursts of data don’t overwhelm downstream services.</li><li>DynamoDB supports millisecond latency at scale, which is essential for querying recent telemetry data in near real-time.</li><li>With on-demand capacity mode, it automatically adjusts to traffic patterns — ideal for unpredictable IoT workloads.</li></ul><p><strong>Low Latency</strong></p><blockquote>Requirement: Real-time data should be available within 3 seconds of ingestion.</blockquote><ul><li>AWS Lambda functions are event-driven and near instantaneous in execution. Combined with SQS and SNS, this architecture ensures that telemetry is processed and alerts are triggered within milliseconds to seconds.</li><li>No cold start issues for frequently invoked functions due to AWS’s provisioned concurrency options if needed.</li></ul><p><strong>Durability</strong></p><blockquote>Requirement: Store telemetry for at least 1 year.</blockquote><ul><li>We can integrate AWS Lambda with DynamoDB or S3 for long-term storage. These services are highly durable (11’s for S3) and cost-effective for large volumes of time-series data.</li><li>DynamoDB offers high durability with automatic multi-AZ replication.</li><li>Data is stored redundantly across multiple facilities, ensuring it remains safe for the required 1-year retention period.</li></ul><p><strong>Security</strong></p><blockquote>Data must be encrypted in transit and at rest.</blockquote><ul><li>AWS services like IoT Core, SQS, SNS, and Lambda support TLS encryption in transit and KMS encryption at rest by default.</li><li>Cognito provides secure authentication and authorization for the Admin API, integrating seamlessly with API Gateway.</li></ul><p><strong>High Availability</strong></p><blockquote>Requirement: 99.9% uptime for APIs and alerting.</blockquote><ul><li>AWS manages the availability and fault tolerance of services like Lambda, API Gateway, DynamoDb and SNS. These services are deployed across multiple Availability Zones by default.</li><li>No need to manage failover, load balancing, or patching — AWS handles it.</li></ul><p><strong>Cost Consideration</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e9ce2bc7da29ed425d89746dd727793/href">https://medium.com/media/6e9ce2bc7da29ed425d89746dd727793/href</a></iframe><p><strong>Architecture Considerations</strong></p><ul><li>No idle infrastructure: Costs are incurred only when the system is active.</li><li>Granular scaling: Each component scales independently, avoiding over-provisioning.</li><li>Predictable billing: Especially with DynamoDB on-demand and Lambda’s transparent pricing.</li><li>Meets all non-functional requirements: scalability, latency, durability, security, and availability.</li><li>Minimizes operational overhead: no servers to patch, scale, or monitor.</li><li>Future-proof: easily extendable with services like DynamoDB Streams, Step Functions, or EventBridge.</li></ul><h4>Architecture Diagram &amp; Workflows</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wB_AbMXEiXUC_Tf3AMR_FQ.png" /><figcaption>Navora’s Architecture</figcaption></figure><p><strong>CI/CD Pipeline Description:</strong></p><ol><li>A pull request (PR) is created in GitHub, triggering the CI pipeline:<br>1.1 The milestone and version are verified.<br>1.2 The project is built.<br>1.3 Tests are executed.</li><li>Once the PR is merged to main, the CD pipeline is triggered:<br>2.1 The deployment agent logs in to AWS ECR.<br>2.2 The version is retrieved from the milestone.<br>2.3 A new tag is created.<br>2.4 A new Release is created.<br>2.5 The Docker image is built with the new version and published to ECR.<br>2.6 CloudFormation stack is deployed for the service.</li></ol><p><strong>Workflow description:</strong></p><ol><li>IoT device sends messages to the MQTT broker using the Telemetry topic.</li><li>The broker forwards the message to SQS.</li><li>The Telemetry Processor Lambda Function:<br>3.1 Pulls telemetry entries from the SQS queue.<br>3.2 Enriches the data set.<br>3.3 Persists the telemetry data in the TelemetryData DynamoDB table.<br>3.4 Verifies if any limits have been reached:<br> 3.4.1 If limits have been reached, a new event is published to the <em>Alerts Topic</em>.</li><li>The <em>Alerts Topic </em>listens to any events published and forwards the events to a Telemetry Alerts Processor Lambda Function and to the Email service.<br>4.1 The Telemetry Alerts Processor, processes the alert and persists the data in the TelemetryAlerts DynamoDB table.<br>4.2 The Email service sends an email to the Administration and to the driver.</li><li>The Administration platform:<br>5.1) Authenticates using user credentials from a user pool in Cognito.<br>5.2) Accesses historical and alerts data using API Gateway which is backed up by a Telemetry Administration Lambda Function.</li></ol><h4>Monorepo</h4><p>We’ve chosen a Monorepo setup because Monorepos excel at code sharing and consistency, simplifying dependency management and collaboration across projects. Another aspect we wanted to explore with the monorepo setup was centralized build and release management across different projects (ie. we don’t want to build and deploy services when they haven’t been changed).</p><p><strong>Monorepo and service folder structure</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*yWvEWRQnBVC1UM5TGxOMmw.png" /></figure><p>Services are organized within their specific folders. Each service contains the corresponding <em>Dockerfile</em> and <em>cloudformation-template</em> for deployment.</p><p><strong>GitHub actions</strong></p><p>GitHub actions and workflows are located under .github &gt; workflowsand<em> </em>.github &gt; actions. Each workflow corresponds to a separate service which will build and deploy the corresponding service, ex:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*sHi2xapWqxnFy3Lad1YsEg.png" /></figure><h4>Release management</h4><p>Release management happens through GitHub milestones with the following template: {service}-{SemVer}, ex: TelemetryProcessor-V1.0.0.<br>The milestone will be used to generate the appropriate artifacts, create docker images, deployments and create a new tag and a new release.</p><blockquote>In a later chapter we’ll discuss potential alternatives/improvements to this approach, especially around the tagging of the milestone with a specific SemVer version as this wouldn’t be scalable in bigger teams developing multiple features in the same service.</blockquote><h3>GitHub and AWS Configuration Setup</h3><p>In order to be able to build, test, publish images to ECR and deploy CloudFormation stacks, we need to add some configuration to both AWS and GitHub:</p><h4>AWS</h4><p>In order to publish images and deploy CloudFormation Stacks we have to create a new user with the appropriate permissions to be able to perform all of our CD tasks. We called ours <em>github-actions. </em>We also need to create an Access key for this user so that we can use it in GitHub.<br>Once the user is created we also created two user groups: <em>cloudformation-stack-operations</em> and <em>push-pull-images.</em></p><p><strong>push-pull-images — </strong>Has all the necessary permissions to manage ECR images on the Navora ECR repository.</p><p><strong>cloudformation-stack-operations — </strong>Has all the necessary permissions to deal with CloudFormation stacks as well as to manage the creation, update and deletion of the involved services.</p><h4>GitHub</h4><p>For GitHub we need to create a few secrets that will be used to authenticate and communicate with AWS:</p><ul><li>AWS_ACCOUNT_ID</li><li>AWS_ACCESS_KEY_ID</li><li>AWS_SECRET_ACCESS_KEY</li><li>AWS_SECRET_REGION</li></ul><p>So far we’ve covered quite a lot in the first post. Thanks for reading and stay tuned for the next post, where we’ll dive deeper into the IoT device integration and the telemetry ingestion pipeline.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f3242cb1796" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Welcome to The Architect’s Workshop]]></title>
            <link>https://medium.com/@marcio.duarte89/welcome-to-the-architects-workshop-1cae5e0b834d?source=rss-ccdfbda98ebf------2</link>
            <guid isPermaLink="false">https://medium.com/p/1cae5e0b834d</guid>
            <category><![CDATA[dotnet]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[serverless-architecture]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[c-sharp-programming]]></category>
            <dc:creator><![CDATA[Márcio Duarte]]></dc:creator>
            <pubDate>Thu, 19 Jun 2025 19:40:46 GMT</pubDate>
            <atom:updated>2026-01-23T15:04:58.574Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hja5V5Ls7002jnRfYdPQsQ.jpeg" /></figure><p>In the evolving world of software and cloud architecture, theory is everywhere and it’s easy to get caught up in it— but practical, real-world application is where real learning happens. The Architect’s Workshop is a space dedicated to that.</p><p>I don’t intend this to be just another blog, but a space where we explore the design and implementation of systems that are scalable, resilient, and thoughtfully engineered. Each series within <em>The Architect’s Workshop</em> dives deep into specific architectural challenges or domains, simulating real-world scenarios and walking through the decisions, trade-offs, and lessons learned that, hopefully shape good systems.</p><p>Whether you’re an aspiring architect, a senior engineer, or simply someone passionate about building better software, I hope you get something out of these workshops.</p><h3>Series 1: Navora — Simulating a Real-World Architecture Journey</h3><p>On our first series, we will introduce Navora, a logistics company with a fleet of delivery vehicles that has real time monitoring. This is a fictional but realistic project designed to simulate the full lifecycle of a modern cloud-native system — from development to deployment, infrastructure to observability.</p><p>In this series, we’ll explore:</p><ul><li>How to define and fulfill both functional and non-functional requirements.</li><li>Designing modular, scalable architecture.</li><li>Implementing CI/CD pipelines and infrastructure as code.</li><li>Building and integrating services like telemetry ingestion, alerting, and admin tools.</li><li>And ultimately, reflecting on what worked, what didn’t, and most importantly what we learned.</li></ul><h4>What to expect in this first series</h4><p>This blog series will walk through the journey of building Navora, one step at a time. Each post will focus on a specific aspect of the project — from infrastructure design to service communication, from CI/CD to monitoring.</p><p>Here’s a sneak peek at what’s coming in the next posts:</p><ul><li>Laying the Foundation — Functional and Non-Functional Requirements</li><li>Streaming the Signal: Designing the IoT Telemetry Ingestion Pipeline</li><li>Detect and Respond: Architecting the Alerts Processing Service</li><li>Control Center: Designing the Admin Service for Telemetry Management</li><li>Reflections on Architecture, Trade-offs, and What I’d Do Differently</li></ul><p>I hope to see you in our next post when we breakdown functional and non-functional requirements.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1cae5e0b834d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>