<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Amir Mehrabi Jorshari on Medium]]></title>
        <description><![CDATA[Stories by Amir Mehrabi Jorshari on Medium]]></description>
        <link>https://medium.com/@amir.mehrabi.j?source=rss-faea5a04b1cf------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*lao4dD3kIJbvjaJX_TP-mg.jpeg</url>
            <title>Stories by Amir Mehrabi Jorshari on Medium</title>
            <link>https://medium.com/@amir.mehrabi.j?source=rss-faea5a04b1cf------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 19:17:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@amir.mehrabi.j/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Note-Taking Apps Didn’t Fail. Their Foundational Idea Did.]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-snippet">For thirty years, productivity tools have helped us store more information. But storing thoughts is not the same as thinking.</p><p class="medium-feed-link"><a href="https://medium.com/@amir.mehrabi.j/note-taking-apps-didnt-fail-their-foundational-idea-did-343897d05293?source=rss-faea5a04b1cf------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@amir.mehrabi.j/note-taking-apps-didnt-fail-their-foundational-idea-did-343897d05293?source=rss-faea5a04b1cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/343897d05293</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[note-taker]]></category>
            <category><![CDATA[personal-intelligence]]></category>
            <dc:creator><![CDATA[Amir Mehrabi Jorshari]]></dc:creator>
            <pubDate>Mon, 03 Aug 2026 11:06:13 GMT</pubDate>
            <atom:updated>2026-08-03T11:06:13.079Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[The connection between your January reading and your July argument already exists.]]></title>
            <link>https://medium.com/@amir.mehrabi.j/the-connection-between-your-january-reading-and-your-july-argument-already-exists-dfc2dc1c10b9?source=rss-faea5a04b1cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/dfc2dc1c10b9</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[personal-intelligence]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Amir Mehrabi Jorshari]]></dc:creator>
            <pubDate>Sat, 18 Jul 2026 16:36:08 GMT</pubDate>
            <atom:updated>2026-07-18T16:36:08.489Z</atom:updated>
            <content:encoded><![CDATA[<p>Every researcher, writer, and academic I know is sitting on an answer they cannot see.</p><p>You read for a living, or close to it. You take notes, you highlight, you save the paper, you capture the quote that made you stop. Over a year, over five years, you accumulate an enormous private library of what you have read and what you thought while reading it. And then you sit down in July to make an argument, and the one insight that would sharpen it, the insight already sitting inside something you read in January, does not arrive. It is there. You captured it faithfully. It simply stays silent at the exact moment it would have mattered most.</p><p>I want to spend this first issue making the case that this is not a personal failing, not a discipline problem, not a sign you chose the wrong app. It is a structural failure in the way we have been taught to work, and once you see it clearly, most of the advice the productivity world has given knowledge workers for the last thirty years falls apart in your hands.</p><p><strong>The misdiagnosis</strong></p><p>When the synthesis does not come, we reach for the wrong explanation. We assume we have not read enough. Or we assume we read enough but filed it carelessly, and that a cleaner structure would have saved us. So we do the two things that feel like progress. We read more, and we file more carefully.</p><p>Watch what happens. The library grows. The tags multiply. The folders nest more elegantly. And the synthesis still does not happen, because more input was never the missing ingredient. You did not fail to make the connection between the spring paper and the summer argument because you were short one more source. You failed to make it because nothing you own was doing the connective work, and reading a further paper only adds another silent entry to a collection that was already too large to hold in a single mind at once.</p><p>This is the loop the whole field is trapped in. We diagnose a synthesis problem and prescribe an accumulation cure. We treat a failure of connection as if it were a failure of collection. And because collecting feels productive, and because each new tool is genuinely better at collecting than the last, we never notice that we have been answering the wrong question with increasing sophistication.</p><p><strong>Accumulation is not synthesis</strong></p><p>Here is the distinction the entire field blurs, and it is worth slowing down on, because everything else follows from it.</p><p>Accumulating and synthesizing are different acts. They use different faculties, they happen at different times, and being good at one does you almost no good at the other. Accumulation is collection: taking in, capturing, storing. Synthesis is seeing what a collection means when its pieces are held together across time. Your research compounds in volume every single year. It compounds in understanding almost not at all, and the gap between those two curves is the quiet tragedy of a serious reading life.</p><p>The reason the gap exists is that synthesis is genuinely hard, and it is hard in a specific way. It requires holding many moments of your own thinking in view at once: the thing you believed in spring, the source that unsettled it in June, the half-formed objection you noted in a margin and left. Synthesis is the act of noticing the line between two ideas that were captured months apart and never sat on the same page. It is inherently a cross-time operation, and human attention is stubbornly single-time. We live in the present tense. The version of you reading this sentence has poor access to the hundreds of earlier versions who did the reading you now need.</p><p>That is not a character flaw. It is the architecture of a mind that evolved to handle what is in front of it. But it means the raw material of your best future arguments is scattered across a self you can no longer fully reach, and the tools you have adopted to help do nothing to reach it for you.</p><p><strong>The posture of your tools</strong></p><p>Look closely at every instrument you use to think, and you will notice they share one fatal posture. They wait.</p><p>Your notes wait for you to search. This sounds like a feature and is actually the whole limitation, because search can only ever return what you already know to look for. To find the January paper, you must first suspect that a January paper exists and bears on today’s problem. But the entire difficulty is that you do not suspect it. The source whose relevance you cannot yet see is precisely the one you will never query, because querying requires you to already hold the connection the query was supposed to surface. The intelligence has to be in your head before the tool will help, and if it were already in your head, you would not need the tool.</p><p>The newer instruments wait too, in a more seductive way. A model that answers questions is extraordinary, and it is entirely reactive. It sits perfectly still until you arrive with a prompt, and then it performs. The initiative is always yours. You still have to know what to ask, which means these tools inherit the exact limitation of the notes app, dressed in far more impressive clothing. They are magnificent at responding and incapable of initiating, and initiation is the part you cannot supply about the ideas you have lost track of.</p><p>So consider what a working scholar is actually left with. A vast collection they cannot fully hold in mind. A search box that answers only questions they already know to pose. And an answer-engine that waits for a prompt they cannot write, because writing it would require the very synthesis they came for. Every tool in the stack is downstream of an insight the researcher does not yet have. That is the trap, stated precisely. Not lost information. Present, captured, complete information that stays silent because everything around it is reactive, and the one thing a reactive tool cannot do is hand you a connection you have not yet made yourself.</p><p><strong>The particular texture of research loss</strong></p><p>It is worth naming the specific ways this failure shows up in a reading life, because in the abstract it is easy to nod at and hard to feel, and I want you to feel it.</p><p>There is the forgotten source. The study you read with real attention eighteen months ago, whose central finding would settle a debate you are having with yourself this week, and which never comes to mind, because nothing brings it. It does not resurface. It waits in a folder for a search you will never think to run.</p><p>There is the contradiction between sources. You trusted two authors, at two different times, and they cannot both be right, and you have never once put their claims side by side, because they entered your collection months apart and no force in your system ever draws them together. You carry an unexamined contradiction in your own foundations and do not know it.</p><p>There is the pattern across months. A theme has been forming across a dozen things you have read, a slow convergence that would be obvious if you could see all twelve at once, and invisible because you only ever encounter them one at a time, spaced across a year. The pattern is real and you will likely never notice it, because noticing would require a vantage point above your own timeline that you do not have.</p><p>And there is the thread across fields. The idea from a discipline you read for pleasure that would transform the argument in the discipline you work in, a connection that only exists because both readings happen to live in the same mind. That is the most valuable synthesis of all, and the one least likely to occur, because the two halves are filed a world apart and nothing reaches across to join them.</p><p>None of these is a storage failure. In every case the information was captured perfectly and remained perfectly useless, because capture was never the bottleneck. Connection was.</p><p><strong>We have known this for centuries, and never solved it</strong></p><p>None of this is new. Scholars have felt it for as long as scholarship has existed, and the history of note-taking is really a long history of people trying, and failing, to make their past reading work on their present problem.</p><p>The Renaissance had the commonplace book, in which readers copied passages under headings so that wisdom could be retrieved by theme. It was a genuine advance, and it was still fundamentally a retrieval system: it answered the questions you knew to ask of it. The card index that scholars refined over the following centuries, and that later thinkers turned into elaborate cross-referenced systems, promised something more, a web of links dense enough that ideas would collide and generate new ones. In practice, for almost everyone who has ever kept one, it became another archive to maintain, another collection whose upkeep consumed the very hours it was meant to free. The tools got more sophisticated across five hundred years, and the underlying bargain never changed. They all asked you to do the connecting. They all waited.</p><p>That is the through-line I want you to see. Every generation of readers has built a slightly better container and called it a solution, and every generation has discovered that a better container does not think. The problem has survived every technology we have thrown at it, because we kept throwing storage at a synthesis problem, and storage, however elegant, is still just a place for things to sit.</p><p><strong>A different posture</strong></p><p>So the way out is not a better place to store research, and not a cleverer scheme for arranging it. Both of those are more storage. The way out is a change in posture, from waiting to moving first.</p><p>Imagine a system that reads across the whole of what you have gathered, continuously, and surfaces the connections proactively rather than waiting to be asked. It brings you the pattern forming across months of reading before you have consciously noticed it. It sets the two contradicting sources side by side and shows you that you have been holding both. It carries the thread from January forward to the argument you are drafting in July, unprompted, because it does not depend on you already knowing the thread is there. It does not organize your research for you. It compiles it, in the older sense of the word: it draws together what is scattered into something that can finally be read as a whole.</p><p>That is the shift this newsletter exists to argue for. From accumulation to synthesis. From reactive to proactive. From a collection you maintain to an intelligence that compiles what you have already taken in and keeps it working on the questions in front of you. It is a category shift, not a feature, and I think it is the most important change coming to knowledge work in our lifetimes.</p><p><strong>What this newsletter is</strong></p><p>Each week I will send you one honest idea about how we read, think, and lose what we know, and what changes when a system does the connective work instead of leaving it to a present-tense mind that cannot reach its own past. Some weeks that will be an argument. Some weeks it will be something I have learned building toward this. It will always be short, except this once, and it will always earn its claims rather than assert them.</p><p>I am writing it in the open because I would rather make this case in public, over months, than pretend I arrived at it from a neutral distance. I did not. I arrived at it by living the failure I have just described, watching my own best reading go silent exactly when I needed it, and then setting out to build the way through.</p><p>So let me end where the whole thing began for me, with a question. What is something you read this year that you know is important, and have not been able to bring to bear on a single thing since? Reply and tell me. That gap, between what you have taken in and what you can actually use, is the entire subject of this publication, and I read every reply.</p><p><em>I am building toward exactly this: research that compiles itself, and surfaces the connection before you know to ask. It is called </em><a href="https://5guys-co.slack.com/archives/C0B2X4XBN4E/p1783247189272989"><em>Ovandor</em></a><em>. If you want to see it before we launch: ovandor.com.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dfc2dc1c10b9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Synthesis, Not Retrieval, Is the Hard Problem]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@amir.mehrabi.j/why-synthesis-not-retrieval-is-the-hard-problem-d763e206c8bb?source=rss-faea5a04b1cf------2"><img src="https://cdn-images-1.medium.com/max/1672/1*Fw73CMSsHIfR22vi4DhT3w.png" width="1672"></a></p><p class="medium-feed-snippet">What building Ovandor taught us about turning captured information into evidence-backed, revisable knowledge.</p><p class="medium-feed-link"><a href="https://medium.com/@amir.mehrabi.j/why-synthesis-not-retrieval-is-the-hard-problem-d763e206c8bb?source=rss-faea5a04b1cf------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@amir.mehrabi.j/why-synthesis-not-retrieval-is-the-hard-problem-d763e206c8bb?source=rss-faea5a04b1cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/d763e206c8bb</guid>
            <category><![CDATA[personal-intelligence]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[notetaking]]></category>
            <dc:creator><![CDATA[Amir Mehrabi Jorshari]]></dc:creator>
            <pubDate>Wed, 15 Jul 2026 21:21:43 GMT</pubDate>
            <atom:updated>2026-07-15T21:21:43.891Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[The Architecture of Personal Intelligence: Why Notes Apps Are Not Enough]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@amir.mehrabi.j/the-architecture-of-personal-intelligence-why-notes-apps-are-not-enough-c855c84b178d?source=rss-faea5a04b1cf------2"><img src="https://cdn-images-1.medium.com/max/1672/1*SbLvG5OPyB870EL-bd98nw.png" width="1672"></a></p><p class="medium-feed-snippet">For years, personal knowledge management has been framed as a storage problem.</p><p class="medium-feed-link"><a href="https://medium.com/@amir.mehrabi.j/the-architecture-of-personal-intelligence-why-notes-apps-are-not-enough-c855c84b178d?source=rss-faea5a04b1cf------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@amir.mehrabi.j/the-architecture-of-personal-intelligence-why-notes-apps-are-not-enough-c855c84b178d?source=rss-faea5a04b1cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/c855c84b178d</guid>
            <category><![CDATA[note-taking-app]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[personal-intelligence]]></category>
            <dc:creator><![CDATA[Amir Mehrabi Jorshari]]></dc:creator>
            <pubDate>Mon, 06 Jul 2026 15:01:07 GMT</pubDate>
            <atom:updated>2026-07-06T15:01:07.168Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Rethinking Team Performance Reviews: Growth, Fairness, and Self-Improvement]]></title>
            <link>https://medium.com/@amir.mehrabi.j/rethinking-team-performance-reviews-growth-fairness-and-self-improvement-621898e3856c?source=rss-faea5a04b1cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/621898e3856c</guid>
            <category><![CDATA[team-leader]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[performance-reviews]]></category>
            <category><![CDATA[management]]></category>
            <dc:creator><![CDATA[Amir Mehrabi Jorshari]]></dc:creator>
            <pubDate>Tue, 17 Feb 2026 19:03:29 GMT</pubDate>
            <atom:updated>2026-02-17T19:03:29.293Z</atom:updated>
            <content:encoded><![CDATA[<p>As a team lead, I’ve always seen performance reviews as a pivotal moment — not for punishment, but for growth. Last year, we shared our experiences, successes, and challenges. While the past cannot be changed, it serves as a foundation for planning the future. Here’s how I approach performance reviews to ensure they are fair, actionable, and inspiring.</p><h3>1. Performance Reviews as a Growth Opportunity</h3><p>Performance reviews shouldn’t feel like a judgment day. Instead, they are a chance to reflect on achievements, identify areas for improvement, and set clear, attainable goals for the year ahead. By framing reviews this way, engineers feel empowered rather than scrutinized.</p><h3>2. A Structured, Transparent Approach</h3><p>To make the process fair and clear, I designed a checklist covering eight key topics:</p><ul><li><strong>Technical Design &amp; Decision Making</strong></li><li><strong>Analytical Thinking &amp; Complex Problem Solving</strong></li><li><strong>Code Quality &amp; Technical Leadership</strong></li><li><strong>Delivery Ownership (End-to-End)</strong></li><li><strong>Stakeholder Management</strong></li><li><strong>Communication &amp; Documentation</strong></li><li><strong>Collaboration &amp; Leadership</strong></li></ul><p>The process worked in three steps:</p><ol><li><strong>Self-Assessment:</strong> Each engineer first scored themselves against these topics.</li><li><strong>Lead Assessment:</strong> I then scored them independently, giving an external perspective.</li><li><strong>Discussion Meeting:</strong> Finally, we met to discuss strengths, weaknesses, and improvement areas.</li></ol><p>This checklist not only made expectations clear from my point of view but also gave engineers a structured way to reflect on their own growth and identify opportunities for development.</p><h3>3. Encouraging Self-Reliance and Collaboration</h3><p>A healthy team thrives when individuals take ownership of their growth — and support their peers along the way. The review process encouraged engineers to:</p><ul><li>Reflect on their contributions and challenges.</li><li>Share knowledge and help teammates overcome blockers.</li><li>Celebrate successes collectively, not just individually.</li></ul><h3>4. Fairness Across the Team</h3><p>Consistency and transparency are key. Everyone is evaluated against the same criteria, with clear examples of performance expectations. By keeping the process structured and unbiased, we create an environment where each engineer feels respected and fairly assessed.</p><h3>5. Learning From the Past, Planning for the Future</h3><p>While we cannot change what happened in the previous year, we can extract valuable lessons. The review becomes a bridge: connecting past experiences with future objectives. Engineers can identify patterns, acknowledge accomplishments, and define the steps they want to take next. Through this process, the ultimate goal is to prepare them for their next role — team lead — ensuring they are ready for higher responsibilities.</p><h3>6. Self-Monitoring for Continuous Improvement</h3><p>One of the most powerful outcomes of a performance review is self-awareness. Engineers can track their growth throughout the year by:</p><ul><li>Setting measurable goals and key milestones.</li><li>Regularly reviewing their progress.</li><li>Seeking feedback from peers and leads.</li><li>Adjusting strategies as needed to stay aligned with personal and team objectives.</li></ul><h3>Closing Thoughts</h3><p>Performance reviews are more than an annual evaluation — they are a roadmap for growth, learning, and collaboration. When approached with fairness, transparency, and forward-thinking, they empower engineers to take charge of their development, contribute meaningfully to the team’s success, and confidently step into future leadership roles.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=621898e3856c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Offset Pagination Breaks Your Inbox (And How Cursor Pagination Fixes It)]]></title>
            <link>https://medium.com/@amir.mehrabi.j/why-offset-pagination-breaks-your-inbox-and-how-cursor-pagination-fixes-it-fdf670d25e1e?source=rss-faea5a04b1cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/fdf670d25e1e</guid>
            <category><![CDATA[database-design]]></category>
            <category><![CDATA[design]]></category>
            <category><![CDATA[notification-system]]></category>
            <category><![CDATA[architecture]]></category>
            <dc:creator><![CDATA[Amir Mehrabi Jorshari]]></dc:creator>
            <pubDate>Thu, 12 Feb 2026 09:34:09 GMT</pubDate>
            <atom:updated>2026-02-12T09:34:09.299Z</atom:updated>
            <content:encoded><![CDATA[<h3>📌 Abstract</h3><p>When we first designed our notification service, everything lived in a single messages table. Every notification — personal alerts, transactional updates, and marketing messages — was stored per user.</p><p>At small scale, this worked perfectly fine.</p><p>But once broadcast campaigns entered the system, the data model began collapsing under its own weight.</p><p>A single marketing campaign sent to 1 million users meant:</p><ul><li>1 million inserts</li><li>1 million stored rows</li><li>1 million read-state updates</li><li>Explosive index growth</li></ul><p>The database was doing enormous unnecessary work duplicating identical content for every single user.</p><p>To fix this, we separated the model into two tables:</p><ul><li>messages → user-specific notifications</li><li>broadcast_push_notification_message → shared broadcast campaigns</li></ul><p>Instead of replicating broadcast rows per user, we stored them once and tracked user read-state separately. This dramatically reduced:</p><ul><li>Storage consumption</li><li>Write amplification</li><li>Index pressure</li><li>Replication overhead</li></ul><p>But this architectural improvement introduced a new challenge:</p><blockquote><em>How do you paginate an inbox that merges two different data sources without breaking order or performance?</em></blockquote><p>This is where cursor-based pagination becomes not just an optimization — but a necessity.</p><p>In this article, we’ll walk through:</p><ul><li>Why we separated broadcast and personal messages</li><li>The scalability benefits of this design</li><li>Why offset pagination fails in real-time inbox systems</li><li>How cursor-based pagination solves it cleanly and correctly</li><li>How to implement it for multi-source feeds</li></ul><h3>🚀 Designing a Scalable Inbox</h3><p>At first glance, pagination looks trivial:</p><pre>GET /inbox?skip=0&amp;take=50</pre><p>Simple. Familiar. Works in CRUD dashboards.</p><p>But once you move into real systems — especially ones that merge multiple data sources and receive real-time inserts — offset pagination begins to fail in subtle and dangerous ways.</p><p>Let’s break it down.</p><h3>📦 The Use Case: A Real Notification System</h3><p>Our system supports two types of notifications:</p><ol><li><strong>Personal messages</strong> (user-specific)</li><li><strong>Broadcast messages</strong> (shared across users)</li></ol><p>To avoid duplicating broadcast rows per user, we separated them into two tables.</p><h3>Personal Notifications</h3><pre>@Entity(&#39;messages&#39;)<br>export class MessageEntity {<br>  @Column()<br>  userId: number;<br>  @Column()<br>  message: string;<br>  @Column()<br>  timestamp: Date;<br>  @Column()<br>  read: boolean;<br>}</pre><h3>Broadcast Campaigns</h3><pre>@Entity(&#39;broadcast_push_notification_mtessage&#39;)<br>export class BroadcastMessageEntity {<br>  @Column()<br>  subject: string;<br>  @Column()<br>  body: string;<br>  @Column()<br>  sentAt: Date;<br>  @Column()<br>  status: &#39;pending&#39; | &#39;sent&#39; | &#39;failed&#39;;<br>}</pre><p>The inbox endpoint must merge both tables into a single, chronologically ordered feed.</p><p>That’s where pagination gets tricky.</p><h3>❌ Why Offset Pagination Breaks</h3><p>Offset pagination looks like this:</p><pre>SELECT *<br>FROM messages<br>ORDER BY timestamp DESC<br>LIMIT 50 OFFSET 100;</pre><p>This works fine in static datasets.</p><p>It breaks in dynamic systems.</p><h3>Problem 1: Real-Time Inserts</h3><p>Imagine this:</p><ol><li>User loads page 1 (OFFSET 0)</li><li>A new notification arrives</li><li>User loads page 2 (OFFSET 50)</li></ol><p>Because new rows are inserted at the top, the dataset shifts.</p><p>Now one record is either:</p><ul><li>Skipped</li><li>Or duplicated</li></ul><p>Your inbox becomes inconsistent.</p><p>Users may miss important notifications.</p><p>In financial or transactional systems, this is unacceptable.</p><h3>Problem 2: Merging Multiple Tables</h3><p>Offset pagination assumes a single ordered dataset.</p><p>But our inbox merges:</p><ul><li>messages</li><li>broadcast_push_notification_message</li></ul><p>Each table:</p><ul><li>Has its own auto-increment ID</li><li>Has its own insert pattern</li><li>Has independent write frequency</li></ul><p>There is no globally consistent offset across two tables.</p><p>Trying to paginate each independently and merge later causes:</p><ul><li>Ordering instability</li><li>Duplicate entries</li><li>Missing rows</li><li>Complex reconciliation logic</li></ul><h3>Problem 3: Performance Degradation</h3><p>Even worse — MySQL still scans skipped rows internally.</p><p>OFFSET 100000 does not magically jump to row 100000.</p><p>It scans and discards 100000 rows first.</p><p>As offsets grow:</p><ul><li>Query time increases</li><li>CPU usage rises</li><li>Buffer pool pressure increases</li></ul><p>At scale, offset pagination becomes expensive.</p><h3>✅ The Solution: Cursor-Based Pagination (Keyset Pagination)</h3><p>Instead of saying:</p><blockquote><em>“Give me page 3.”</em></blockquote><p>You say:</p><blockquote><em>“Give me items older than this specific item.”</em></blockquote><p>That shift in mindset changes everything.</p><h3>🧠 Stable Ordering</h3><p>To guarantee deterministic results, we sort by:</p><pre>ORDER BY timestamp DESC, id DESC</pre><p>The cursor contains:</p><ul><li>timestamp</li><li>id</li></ul><p>Example cursor:</p><pre>2026-02-11T09:22:10.000Z_345</pre><h3>🔎 Personal Messages Query</h3><h3>First Request</h3><pre>SELECT *<br>FROM messages<br>WHERE userId = :userId<br>ORDER BY timestamp DESC, id DESC<br>LIMIT 50;  </pre><h3>Next Request</h3><pre>SELECT *<br>FROM messages<br>WHERE userId = :userId<br>  AND (<br>        timestamp &lt; :cursorTimestamp<br>        OR (timestamp = :cursorTimestamp AND id &lt; :cursorId)<br>      )<br>ORDER BY timestamp DESC, id DESC<br>LIMIT 50;</pre><p>No shifting.<br> No duplication.<br> No performance degradation.</p><p>The database performs an index range scan — not a full scan + discard.</p><h3>🤯 The Real Challenge: Merging Two Tables</h3><p>The inbox must merge:</p><ul><li>Personal messages</li><li>Broadcast messages</li></ul><p>You cannot rely on a single ID because:</p><ul><li>IDs are auto-incremented per table</li><li>They are not globally unique</li><li>Comparing IDs across tables is meaningless</li></ul><p>So what should the cursor contain?</p><h3>✅ The Correct Cursor for Multi-Source Feeds</h3><p>Your cursor must carry enough information to allow each source to continue independently.</p><p>For example:</p><pre>{<br>  &quot;lastTimestamp&quot;: &quot;2026-02-11T09:22:10.000Z&quot;,<br>  &quot;lastMessageId&quot;: 345,<br>  &quot;lastBroadcastId&quot;: 89<br>}</pre><p>Encoded (e.g., base64) before sending to the frontend.</p><p>Each table then applies its own keyset condition:</p><h3>Personal Messages</h3><pre>WHERE (<br>  timestamp &lt; :lastTimestamp<br>  OR (timestamp = :lastTimestamp AND id &lt; :lastMessageId)<br>)</pre><h3>Broadcast Messages</h3><pre>WHERE (<br>  sentAt &lt; :lastTimestamp<br>  OR (sentAt = :lastTimestamp AND id &lt; :lastBroadcastId)<br>)</pre><p>You fetch from both tables, merge in memory, sort by timestamp, and return the top N.</p><p>This guarantees:</p><ul><li>Stable pagination</li><li>No skipped rows</li><li>No duplicates</li><li>No cross-table ID corruption</li></ul><h3>📈 Performance Benefits</h3><p>Cursor pagination:</p><ul><li>Uses index range scans</li><li>Avoids large OFFSET scans</li><li>Handles real-time inserts safely</li><li>Works perfectly with infinite scroll</li></ul><p>With proper indexing:</p><pre>CREATE INDEX idx_messages_user_timestamp_id<br>ON messages (userId, timestamp DESC, id DESC);</pre><p>The query becomes extremely efficient — even at millions of rows.</p><h3>🎯 When Should You Use Cursor Pagination?</h3><p>Use cursor pagination when:</p><ul><li>Data is frequently inserted at the top</li><li>You build feeds, timelines, or inboxes</li><li>You merge multiple data sources</li><li>You care about correctness</li><li>You operate at scale</li></ul><p>Avoid offset pagination in:</p><ul><li>Real-time feeds</li><li>Messaging systems</li><li>Financial systems</li><li>High-traffic applications</li></ul><p>Offset pagination is acceptable for:</p><ul><li>Admin dashboards</li><li>Static reports</li><li>Small datasets</li></ul><h3>🏁 Final Thoughts</h3><p>Cursor-based pagination is not just a performance optimization.</p><p>It’s a <strong>correctness guarantee</strong>.</p><p>As your system grows, offset pagination becomes a liability:</p><ul><li>It breaks ordering</li><li>It introduces duplication</li><li>It skips data</li><li>It slows down at scale</li></ul><p>Cursor pagination, on the other hand:</p><ul><li>Preserves consistency</li><li>Scales efficiently</li><li>Handles real-time inserts</li><li>Works naturally with infinite scroll</li></ul><p>If you’re building an inbox, timeline, or notification system — start with cursors from day one.</p><p>Future you (and your database) will be grateful.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fdf670d25e1e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS ECS Autoscaling with Custom CloudWatch Metrics via Terraform]]></title>
            <link>https://awstip.com/aws-ecs-autoscaling-with-custom-cloudwatch-metrics-via-terraform-e44f8374b1dd?source=rss-faea5a04b1cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/e44f8374b1dd</guid>
            <category><![CDATA[autoscaling]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[cloudwatch]]></category>
            <category><![CDATA[aws-ecs]]></category>
            <dc:creator><![CDATA[Amir Mehrabi Jorshari]]></dc:creator>
            <pubDate>Wed, 30 Jul 2025 04:31:32 GMT</pubDate>
            <atom:updated>2025-07-30T18:28:39.062Z</atom:updated>
            <content:encoded><![CDATA[<p>Autoscaling is an essential feature for enterprise software. It allows us to dynamically adjust processing power and other resources based on specific factors. Depending on the launch type, several approaches are available to achieve this. In this article, I’ll describe one such approach: autoscaling for ECS services based on a custom metric that isn’t among AWS’s built-in options.</p><h3>Key Concepts:</h3><p><strong>Scalable Target:</strong></p><ul><li>The AWS resource that Application Auto Scaling will manage. This is typically defined using the aws_appautoscaling_target resource.</li></ul><p><strong>Policy Types:</strong></p><ul><li><strong>Step Scaling:</strong> Adjusts the capacity in increments based on metric breaches. You define step adjustments for different alarm thresholds.</li><li><strong>Target Tracking Scaling:</strong> Scales the target to maintain a specified metric at a desired value (e.g., keep CPU utilization at 70%).</li></ul><p><strong>Metrics:</strong></p><ul><li>The CloudWatch metrics that trigger the scaling actions (e.g., CPU utilization, network I/O, custom metrics).</li></ul><p><strong>Alarms:</strong></p><ul><li>CloudWatch alarms are often used in conjunction with step scaling policies to trigger the scaling actions when a metric threshold is crossed.</li></ul><p>While CloudWatch automatically collects numerous metrics that can be used for autoscaling, this article will focus on demonstrating how to utilize a custom metric that might be more suitable for specific autoscaling needs. For example, if we needed to increase processing power based on the number of remaining messages in a queue? In such a scenario, defining a custom metric becomes necessary.</p><p>To utilize a custom metric, the application first needs to calculate the metric and then <strong>publish it to CloudWatch</strong>. Once the custom metric is in CloudWatch, we can define an alarm. An autoscaling policy continuously monitors this alarm, and if it detects that the predefined criteria have been met, it will trigger the <strong>scaling in or out</strong> of the ECS service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5-eiBhdzmPiV_fgnLe3YXw.png" /><figcaption>solution overview</figcaption></figure><h3>1. Publishing Custom Metric</h3><p>Using AWS SDK, u can publish the metric as follows:</p><pre>import { ConfigurationService } from &#39;@configs/configuration.service&#39;;<br>import { GeneralService } from &#39;@core/shared/shared-service&#39;;<br>import { Injectable, OnModuleInit } from &#39;@nestjs/common&#39;;<br>import {<br>  CloudWatchClient,<br>  PutMetricDataCommand,<br>} from &#39;@aws-sdk/client-cloudwatch&#39;;<br><br>@Injectable()<br>export class AwsCloudWatchService<br>  extends GeneralService<br>  implements OnModuleInit<br>{<br>  client: CloudWatchClient;<br>  constructor(private config: ConfigurationService) {<br>    super();<br>  }<br>  onModuleInit() {<br>    this.client = new CloudWatchClient({ region: this.config.app.awsRegion });<br>  }<br><br>  publishCustomMetric = async (<br>    metricName: string,<br>    namespace: string,<br>    value: number,<br>    unit: &#39;Seconds&#39; | &#39;Microseconds&#39; | &#39;Milliseconds&#39;,<br>    dimensions: { Value: string; Name: string }[],<br>  ) =&gt; {<br>    const params = {<br>      MetricData: [<br>        {<br>          MetricName: metricName,<br>          Dimensions: dimensions,<br>          Unit: unit,<br>          Value: value,<br>          Timestamp: new Date(), // Use current time or a specific timestamp<br>        },<br>      ],<br>      Namespace: namespace,<br>    };<br><br>    const command = new PutMetricDataCommand(params);<br><br>    try {<br>      const response = await this.client.send(command);<br>      this.logger.log(<br>        `Successfully published metric: ${metricName} with value ${value}`,<br>        { metricName, namespace, value, unit, dimensions },<br>      );<br>      return response;<br>    } catch (err) {<br>      this.logger.error(`Error publishing metric ${metricName}`, err.stack, {<br>        error: err.message,<br>        metricName,<br>        namespace,<br>        value,<br>        unit,<br>        dimensions,<br>      });<br>    }<br>  };<br>}</pre><p>For publishing a custom metric, we need to define a metric name, namespace, unit, and value</p><h3>2. CloudWatch Alarm and policies in Terraform</h3><p>The assumption for the following Terraform is that we have defined ECS task definition and service before. Therefore, we first need to define the app autoscaling target using <strong>aws_appautoscaling_target </strong>and set the scalable dimension that is the desired count for our case.</p><pre>resource &quot;aws_appautoscaling_target&quot; &quot;api_notification_v2_target&quot; {<br>  service_namespace  = &quot;ecs&quot;<br>  # Format: service/cluster-name/service-name<br>  resource_id        = &quot;service/${local.ecs_cluster_name}/${local.name}&quot;<br>  scalable_dimension = &quot;ecs:service:DesiredCount&quot;<br>  min_capacity       = var.task_count # Matches your initial desired_count<br>  max_capacity       = var.autoscaling_properties.MAX_TASKS_COUNT # Define a max_task_count variable<br>}</pre><p>Then we need to define two policies, scale in and scale out.</p><h3>Scale-out policy</h3><pre><br>resource &quot;aws_appautoscaling_policy&quot; &quot;api_notification_v2_scale_out_policy&quot; {<br>  name                   = &quot;${local.name}-scale-out-policy&quot;<br>  service_namespace      = aws_appautoscaling_target.api_notification_v2_target.service_namespace<br>  resource_id            = aws_appautoscaling_target.api_notification_v2_target.resource_id<br>  scalable_dimension     = aws_appautoscaling_target.api_notification_v2_target.scalable_dimension<br>  policy_type            = &quot;StepScaling&quot;<br><br>  step_scaling_policy_configuration {<br>    adjustment_type         = &quot;ChangeInCapacity&quot;<br>    cooldown                = 300 # Wait 5 minutes before evaluating again<br>    metric_aggregation_type = &quot;Average&quot;<br><br>    step_adjustment {<br>      metric_interval_lower_bound = 0<br>      scaling_adjustment          = 1 # Add 1 task<br>    }<br>  }<br>}</pre><h3>Property Descriptions:</h3><p>1. <strong>name</strong>:</p><ul><li>Description: A unique name for the scaling policy within the AWS account and region.</li><li>Value in Example: ${local.name}-scale-out-policy. This dynamically generates a name based on a local variable, ensuring it’s descriptive and unique (e.g., my-app-api-notification-v2-scale-out-policy).</li></ul><p>2. <strong>service_namespace</strong>:</p><ul><li>Description: The namespace of the AWS service that the scalable target belongs to. This identifies the type of resource being scaled.</li><li>Value in Example: aws_appautoscaling_target.api_notification_v2_target.service_namespace. For ECS services, this will always resolve to “ecs”. Other namespaces include “ec2” (for EC2 Auto Scaling Groups, though less common to directly define with App Auto Scaling), “rds” (for RDS Aurora replicas), “custom-resource” (for custom resources), etc.</li></ul><p>3. <strong>resource_id</strong>:</p><ul><li>Description: The identifier of the scalable target. This points to the specific resource that will be scaled.</li><li>Value in Example: aws_appautoscaling_target.api_notification_v2_target.resource_id. For an ECS service, this will be in the format service/cluster-name/service-name (e.g., service/my-fargate-cluster/api-notification-v2).</li></ul><p>4. <strong>scalable_dimension</strong>:</p><ul><li>Description: The scalable dimension associated with the scalable target. This indicates what specific attribute of the resource is being scaled.</li><li>Value in Example: aws_appautoscaling_target.api_notification_v2_target.scalable_dimension. For ECS services, this will always resolve to “ecs:service:DesiredCount”, meaning the policy will adjust the desiredCount property of the specified ECS service.</li></ul><p>5. <strong>policy_type</strong>:</p><ul><li>Description: The type of scaling policy.</li><li>Value in Example: “<strong>StepScaling</strong>”.</li><li>Possible Values:</li><li>“<strong>StepScaling</strong>”: Adjusts the capacity of the scalable target based on the current metric value relative to predefined thresholds (steps). This is what’s used here.</li><li>“<strong>TargetTrackingScaling</strong>”: Automatically adjusts the capacity to maintain a specified target value for a chosen metric (e.g., maintain 70% CPU utilization). This is generally simpler to configure for many common metrics but doesn’t allow for multiple “steps” of adjustment.</li></ul><p>6. <strong>step_scaling_policy_configuration</strong> (Block):</p><ul><li>Description: This block is required when policy_type is set to “StepScaling”. It defines the detailed behavior of a step scaling policy.</li></ul><p>6a. adjustment_type:</p><ul><li>Description: The method by which the capacity is adjusted.</li><li>Value in Example: “<strong>ChangeInCapacity</strong>”.</li><li>Possible Values:</li><li>“<strong>ChangeInCapacity</strong>”: Adds or removes the specified number of units (e.g., instances, tasks). This is an absolute adjustment.</li><li>“<strong>PercentChangeInCapacity</strong>”: Adds or removes a percentage of the current capacity.</li><li>“<strong>ExactCapacity</strong>”: Sets the scalable target’s capacity to an exact number.</li></ul><p>6b. <strong>cooldown</strong>:</p><ul><li>Description: The amount of time, in seconds, after a scaling activity completes that Auto Scaling waits before initiating another scaling activity based on this policy. This prevents rapid, undesired fluctuations (flapping).</li><li>Value in Example: 300 (5 minutes). For scale-out, this is typically shorter than scale-in to react quickly to increased load.</li></ul><p>6c. <strong>metric_aggregation_type</strong>:</p><ul><li>Description: The method to use to aggregate metric data points for the alarm. This property applies only if the metric_alarm_name (the alarm triggering this policy) specifies a metric with multiple data points within a single period.</li><li>Value in Example: “Average”.</li><li>Possible Values: “Average”, “Minimum”, “Maximum”, “Sum”.</li></ul><p>6d. <strong>step_adjustment</strong> (Block):</p><ul><li>Description: This block defines one or more individual “steps” that determine how much to scale based on the metric’s value relative to the alarm’s threshold. You can define multiple step_adjustment blocks within <strong>step_scaling_policy_configuration</strong> for different ranges.</li></ul><p>6d.i. <strong>metric_interval_lower_bound</strong>:</p><ul><li>Description: The lower bound for the difference between the CloudWatch metric value and the alarm threshold. If the difference is greater than or equal to this value, the scaling_adjustment is applied.</li><li>Value in Example: 0. When used in a scale-out policy with GreaterThanThreshold alarm, 0 means “if the metric is greater than or equal to the threshold (i.e., difference is 0 or positive)”.</li></ul><p>6d.ii. <strong>scaling_adjustment</strong>:</p><ul><li>Description: The number of units (tasks in this case) to add or remove when this step adjustment is applied.</li><li>Value in Example: 1. In this scale-out policy, it means “add 1 task”. For a scale-in policy, it would typically be a negative number (e.g., -1).</li></ul><p>In summary, this <strong>aws_appautoscaling_policy</strong> resource defines a Step Scaling policy named ${local.name}-scale-out-policy. When this policy is triggered by a CloudWatch alarm (which would be api_notification_v2_alarm_scale_out in our full example), it will increase the DesiredCount of the ECS service api_notification_v2 by 1 task. It will then wait for a cooldown period of 5 minutes before allowing further scaling actions from this specific policy. The step_adjustment of metric_interval_lower_bound = 0 means this action applies whenever the metric that triggered the alarm is at or above the alarm’s threshold.</p><h3>Scale in policy</h3><pre><br>resource &quot;aws_appautoscaling_policy&quot; &quot;api_notification_v2_scale_in_policy&quot; {<br>  name                   = &quot;${local.name}-scale-in-policy&quot;<br>  service_namespace      = aws_appautoscaling_target.api_notification_v2_target.service_namespace<br>  resource_id            = aws_appautoscaling_target.api_notification_v2_target.resource_id<br>  scalable_dimension     = aws_appautoscaling_target.api_notification_v2_target.scalable_dimension<br>  policy_type            = &quot;StepScaling&quot;<br><br>  step_scaling_policy_configuration {<br>    adjustment_type         = &quot;ChangeInCapacity&quot;<br>    cooldown                = 600 # Longer cooldown for scale-in<br>    metric_aggregation_type = &quot;Average&quot;<br><br>    step_adjustment {<br>      metric_interval_upper_bound = 0<br>      scaling_adjustment          = -1 # Remove 1 task<br>    }<br>  }<br>}</pre><h3>Scale out alarm</h3><pre><br>resource &quot;aws_cloudwatch_metric_alarm&quot; &quot;api_notification_v2_alarm_scale_out&quot; {<br>  alarm_name          = &quot;${local.name}-SQS-Queue-High-Alarm&quot;<br>  comparison_operator = &quot;GreaterThanThreshold&quot;<br>  evaluation_periods  = 2 # If threshold is breached for 2 consecutive periods (2 minutes)<br>  metric_name         = &quot;${var.autoscaling_properties.METRIC_NAME}&quot; # SQS metric<br>  namespace           = &quot;${var.autoscaling_properties.NAMESPACE}&quot; # Standard SQS namespace<br>  period              = 60 # Check every 60 seconds<br>  statistic           = &quot;Average&quot;<br>  threshold           = 50 # Trigger if average messages visible &gt; 50<br>  unit                = &quot;Count&quot;<br><br>  dimensions = {<br>    QueueName = &quot;TEST&quot;<br>  }<br><br>  # THIS IS THE CRITICAL LINK: Alarm actions reference the scaling policy&#39;s ARN<br>  alarm_actions = [aws_appautoscaling_policy.api_notification_v2_scale_out_policy.arn]<br>  ok_actions    = [] # Typically leave empty, the scale-in policy handles OK state<br><br>  tags = {<br>    Name = &quot;${local.name} SQS Scale Out Alarm&quot;<br>  }<br>}</pre><h3>Scale in alarm</h3><pre><br>resource &quot;aws_cloudwatch_metric_alarm&quot; &quot;api_notification_v2_alarm_scale_in&quot; {<br>  alarm_name          = &quot;${local.name}-SQS-Queue-Low-Alarm&quot;<br>  comparison_operator = &quot;LessThanThreshold&quot;<br>  evaluation_periods  = 5 # Check over 5 periods (5 minutes) before scaling in<br>  metric_name         = &quot;${var.autoscaling_properties.METRIC_NAME}&quot; # SQS metric<br>  namespace           = &quot;${var.autoscaling_properties.NAMESPACE}&quot; # Standard SQS namespace<br>  period              = 60<br>  statistic           = &quot;Average&quot;<br>  threshold           = 10 # Trigger if average messages visible &lt; 10<br>  unit                = &quot;Count&quot;<br><br>  dimensions = {<br>    QueueName = &quot;TEST&quot;  <br>  }<br><br>  # Link to the scale-in policy&#39;s ARN<br>  alarm_actions = [aws_appautoscaling_policy.api_notification_v2_scale_in_policy.arn]<br>  ok_actions    = []<br><br>  tags = {<br>    Name = &quot;${local.name} SQS Scale In Alarm&quot;<br>  }<br>}</pre><h3>3. Required permissions for ur account:</h3><pre>&quot;application-autoscaling:RegisterScalableTarget&quot;,<br>&quot;application-autoscaling:DescribeScalableTargets&quot;,<br>&quot;application-autoscaling:ListTagsForResource&quot;,<br>&quot;application-autoscaling:DeregisterScalableTarget&quot;,<br>&quot;application-autoscaling:PutScalingPolicy&quot;,<br>&quot;application-autoscaling:DescribeScalingPolicies&quot;,<br>&quot;application-autoscaling:DeleteScalingPolicy&quot;,<br>&quot;iam:CreateServiceLinkedRole&quot;,<br>&quot;cloudwatch:DescribeAlarms&quot;,<br>&quot;cloudWatch:DeleteAlarms&quot;,<br>&quot;cloudwatch:ListTagsForResource&quot;<br><br>ECSTaskRole permission<br>~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>cloudwatch:PutMetricData ( for sending custom metric to cloud watch )<br>ecs:DescribeServices  ( for getting running tasks count )</pre><h3>4. Notes:</h3><ul><li>Please note that when you want to publish a custom metric, the unit and dimensions should be the same as the alert definition in your Terraform configuration</li></ul><p>#AWS #ECS #CUSTOM-METRIC #Terraform</p><h3>References:</h3><ul><li><a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/appautoscaling_policy#attribute-reference">Terraform Registry</a></li><li><a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring_get_statistics.html">Statistics for CloudWatch metrics for your instances</a></li><li><a href="https://github.com/aws-samples/aws-ecs-auto-scaling-with-custom-metrics/blob/main/README.md">aws-ecs-auto-scaling-with-custom-metrics/README.md at main · aws-samples/aws-ecs-auto-scaling-with-custom-metrics</a></li><li><a href="https://aws.amazon.com/blogs/containers/autoscaling-amazon-ecs-services-based-on-custom-cloudwatch-and-prometheus-metrics/">Autoscaling Amazon ECS services based on custom CloudWatch and Prometheus metrics | Amazon Web Services</a></li><li><a href="https://judoscale.com/blog/ecs-autoscaling">Understanding How ECS Autoscaling Works</a></li><li><a href="https://docs.aws.amazon.com/autoscaling/application/APIReference/API_PredefinedMetricSpecification.html">PredefinedMetricSpecification</a></li><li><a href="https://aws.amazon.com/blogs/containers/autoscaling-amazon-ecs-services-based-on-custom-metrics-with-application-auto-scaling/">Autoscaling Amazon ECS services based on custom metrics with Application Auto Scaling | Amazon Web Services</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e44f8374b1dd" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/aws-ecs-autoscaling-with-custom-cloudwatch-metrics-via-terraform-e44f8374b1dd">AWS ECS Autoscaling with Custom CloudWatch Metrics via Terraform</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Developing a reusable Terraform module to streamline the Terraform configuration process]]></title>
            <link>https://medium.com/@amir.mehrabi.j/developing-a-reusable-terraform-module-to-streamline-the-terraform-configuration-process-7ea62300f9b8?source=rss-faea5a04b1cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ea62300f9b8</guid>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Amir Mehrabi Jorshari]]></dc:creator>
            <pubDate>Sun, 27 Oct 2024 15:41:06 GMT</pubDate>
            <atom:updated>2024-10-27T15:41:06.543Z</atom:updated>
            <content:encoded><![CDATA[<p>As a developer, I frequently use Terraform to provision infrastructure on AWS. However, setting up Terraform configurations can be a time-consuming and tedious task. To streamline my workflow, I created reusable modules for myself, which significantly accelerated my development speed. I found that using modules not only saved time but also helped me avoid errors by ensuring that I didn’t forget critical specifications and configurations. In this section, I will demonstrate how to write and utilize Terraform modules, highlighting the benefits of this approach for both beginners and experienced users.</p><h3>Terraform folder structure:</h3><p>A well-structured Terraform module is essential for creating reusable and modular configurations that are easy to maintain and update. A typical Terraform module consists of three key components: input/variable, main, and output.</p><p>The input/variable file defines the variables required to set up the module, such as the VPC, permissions, memory size, layers, and other necessary settings. For example, in a module for deploying a Lambda function on AWS, the variables.tf file might define variables for the VPC, IAM role, and memory size.</p><p>The main file contains the necessary settings and configurations for the module. In the case of the Lambda function module, the main.tf file would contain the Terraform configuration that sets up the Lambda function name, timeout, VPC configuration, IAM role, and other necessary resources.</p><p>The output file specifies the references and variables that need to be exported, such as the function name and ARN. These exported values can be used to access and manage the deployed resources.</p><p>To better illustrate this concept, let’s consider an example. Suppose we’re creating a Terraform module for deploying a Lambda function on AWS. Our module might have the following structure:</p><ul><li>variables.tf: This file defines the variables required to set up the Lambda function, such as the VPC, permissions, memory size, layers, and other necessary settings.</li><li>main.tf: This file contains the Terraform configuration that sets up the Lambda function, VPC configuration, IAM role, and other necessary resources. For example, it might create an EC2 instance, security group, and IAM role.</li><li>output.tf: This file specifies the references and variables that need to be exported, such as the function name and ARN. These exported values can be used to access and manage the deployed resources.</li></ul><p>By following this structure, we can create reusable and modular Terraform configurations that are easy to maintain and update</p><h3>EXAMPLE: Lambda Module</h3><p>This Lambda module is designed to deploy a lambda function on AWS. Below, you can view the directory structure that comprises this module.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*fpLvSdPht1B4L22DUoOUlw.png" /></figure><h3>variable.tf</h3><pre>variable &quot;roleName&quot; {<br>  type        = string<br>  description = &quot;Lambda function IAM role name&quot;<br>}<br><br>variable &quot;rolePolicies&quot; {<br>  type        = list(string)<br>  description = &quot;Lambda function IAM role policies&quot;<br>}<br><br>variable &quot;functionName&quot; {<br>  type        = string<br>  description = &quot;Lambda function name&quot;<br>}<br><br>variable &quot;handler&quot; {<br>  type        = string<br>  description = &quot;Lambda function handler&quot;<br>}<br><br>variable &quot;runtime&quot; {<br>  type        = string<br>  description = &quot;Lambda function runtime&quot;<br>}<br><br>variable &quot;layers&quot; {<br>  type        = list(string)<br>  description = &quot;Lambda function layers&quot;<br>}<br><br>variable &quot;sourceCodeHash&quot; {<br>  type        = string<br>  description = &quot;Lambda DynamoDB table name&quot;<br>}<br><br>variable &quot;bucketArn&quot; {<br>  type        = string<br>  description = &quot;bucket arn for lambda function&quot;<br>}<br><br>variable &quot;bucketKey&quot; {<br>  type        = string<br>  description = &quot;bucket key for lambda function&quot;<br>}<br><br>variable &quot;environment&quot; {<br>  type        = any<br>  description = &quot;environment variable for lambda function&quot;<br>  default     = {}<br>}<br><br>variable &quot;permissions&quot; {<br>  type        = list(object({ Action : list(string), Effect = string, Resource = any }))<br>  description = &quot;permission that lambda may need to be run&quot;<br>  default     = []<br>}<br><br>variable &quot;memory&quot; {<br>  type        = number<br>  description = &quot;memory for lambda function&quot;<br>  default     = 128<br>}<br><br>variable &quot;timeout&quot; {<br>  type        = number<br>  description = &quot;timeout of execution&quot;<br>  default     = 15 * 60<br>}<br><br>variable &quot;tags&quot; {<br>  type        = map(string)<br>  description = &quot;lambda function tags&quot;<br>  default     = {}<br>}<br><br><br>variable &quot;vpc&quot; {<br>  type        = object({ subnet_ids : list(string), security_group_ids : list(string) })<br>  description = &quot;lambda security group ids&quot;<br>  default     = {<br>    subnet_ids         = []<br>    security_group_ids = []<br>  }<br>}</pre><h3>main.tf</h3><pre># Create a lambda function IAM role<br>resource &quot;aws_iam_role&quot; &quot;lambda_function_role&quot; {<br>  name = var.roleName<br>  assume_role_policy = jsonencode({<br>    Version = &quot;2012-10-17&quot;<br>    Statement = [<br>      {<br>        Action : &quot;sts:AssumeRole&quot;<br>        Effect : &quot;Allow&quot;<br>        Sid : &quot;&quot;<br>        Principal = {<br>          Service = &quot;lambda.amazonaws.com&quot;<br>        }<br>      }<br>    ]<br>  })<br>  managed_policy_arns = var.rolePolicies<br>  inline_policy {<br>    name = &quot;lambda_function_execution_permissions&quot;<br>    policy = jsonencode({<br>      Version   = &quot;2012-10-17&quot;<br>      Statement = var.permissions<br>    })<br>  }<br>}<br><br># Create a lambda function<br>resource &quot;aws_lambda_function&quot; &quot;lambda_function&quot; {<br>  function_name    = var.functionName<br>  role             = aws_iam_role.lambda_function_role.arn<br>  handler          = var.handler<br>  runtime          = var.runtime<br>  source_code_hash = var.sourceCodeHash<br>  layers           = var.layers<br>  s3_bucket        = var.bucketArn<br>  s3_key           = var.bucketKey<br>  environment {<br>    variables = var.environment<br>  }<br>  memory_size = var.memory<br>  timeout     = var.timeout<br>  tags        = var.tags<br>  vpc_config {<br>    security_group_ids = var.vpc.security_group_ids<br>    subnet_ids         = var.vpc.subnet_ids<br>  }<br>}</pre><h3>output.tf</h3><pre>output &quot;function_arn&quot; {<br>    value       = aws_lambda_function.lambda_function.arn<br>    description = &quot;Lambda function ARN&quot;<br>}<br><br>output &quot;function_name&quot; {<br>    value       = aws_lambda_function.lambda_function.function_name<br>    description = &quot;Lambda function name&quot;<br>}</pre><h3>How to use the lambda module</h3><p>To utilize this Lambda module in your Terraform configuration, simply import it and configure its variables according to your specific requirements. This can be achieved by specifying the module’s source path and setting the necessary input variables, as demonstrated below:</p><pre><br><br>module &quot;sample_lambda&quot; {<br>  source       = &quot;./modules/lambda&quot;   # OUR LAMBDA MODULE EXISTS IN MODULES FOLDER<br>  functionName = &quot;test-connection-handler&quot;<br>  roleName     = &quot;test-lambda-role&quot;<br>  rolePolicies = [<br>    &quot;arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole&quot;,<br>    &quot;arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole&quot;<br>  ]<br>  handler        = &quot;datadog_lambda.handler.handler&quot;<br>  runtime        = &quot;python3.9&quot;<br>  sourceCodeHash = module.lambda_py_deployment_into_s3.base64sha256<br>  bucketArn      = module.lambda_py_deployment_into_s3.bucket_id<br>  bucketKey      = module.lambda_py_deployment_into_s3.bucket_key<br>  layers = [&quot;arn:aws:lambda:ap-southeast-1:336392948345:layer:AWSDataWrangler-Python39:3&quot;]<br>  memory  = 2048<br>  timeout = 900<br>  permissions = [<br>    {<br>      Action   = [&quot;secretsmanager:GetSecretValue&quot;]<br>      Resource = [var.datadog.DD_API_KEY]<br>      Effect   = &quot;Allow&quot;<br>    },<br><br>  ]<br>  vpc = {<br>    subnet_ids = [<br>      module.imports.vpc.outputs.PrivateSubnet1,<br>      module.imports.vpc.outputs.PrivateSubnet2<br>    ]<br>    security_group_ids = [<br>      module.imports.vpc.outputs.AppSecurityGroup,<br>    ]<br>  }<br>  environment = merge(var.datadog, {<br>    ENV               = &quot;staging&quot;<br>    DD_LAMBDA_HANDLER = &quot;connection-handler.lambda_handler&quot;<br>  })<br>}</pre><p>#terraform #aws #lambda</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ea62300f9b8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Streamline Your Data Processing: How to Build a Custom Lambda Layer for Kafka Connectivity in…]]></title>
            <link>https://awstip.com/streamline-your-data-processing-how-to-build-a-custom-lambda-layer-for-kafka-connectivity-in-d6f13b5c5123?source=rss-faea5a04b1cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6f13b5c5123</guid>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[bitbucket]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Amir Mehrabi Jorshari]]></dc:creator>
            <pubDate>Sun, 20 Oct 2024 18:49:45 GMT</pubDate>
            <atom:updated>2024-10-26T16:36:38.203Z</atom:updated>
            <cc:license>https://creativecommons.org/publicdomain/mark/1.0/</cc:license>
            <content:encoded><![CDATA[<h3>Streamline Your Data Processing: How to Build a Custom Lambda Layer for Kafka Connectivity in Python</h3><p>I recently embarked on a project that required a Lambda function to produce messages to a Kafka topic on AWS. While this task might seem straightforward in a traditional ECS/EC2 environment, where you can simply install and use a library, it proved to be a more complex challenge in a Lambda function. The constraints of a serverless environment, combined with the need for a reliable and efficient connection to Kafka, made this task a unique puzzle to solve. In this post, I’ll share my experience and knowledge on how I created a Lambda layer for my lambda function using Bitbucket Pipelines and Terraform, and the lessons I learned along the way</p><h3>Libraries:</h3><p>After researching the available libraries, I decided to focus on two of the most popular ones. I created a Lambda layer that bundles both libraries, making it easy for future developers to use without having to recreate the layer from scratch. This way, our company can benefit from a pre-built solution, reducing the time and effort required to set up a Lambda function for producing messages to Kafka.</p><ul><li><a href="https://pypi.org/project/confluent-kafka/"><em>confluent-kafka</em></a></li><li><a href="https://pypi.org/project/kafka-python/"><em>kafka-python</em></a></li></ul><h3>Packaging the libraries</h3><p>I leveraged Bitbucket Pipelines to automate the packaging and uploading of libraries to S3 for my Lambda layer. The pipeline consists of a few steps, including installing the libraries and creating a ZIP archive file. Since this process is well-documented online, I won’t dive into the details here. The output of this step is a ZIP file named layer.zip, which will be used in the next step to create the Lambda layer using Terraform</p><p>One crucial consideration when packaging a Lambda layer is that the folder name must be within the <strong><em>python</em></strong> directory. As shown in the script below, we create a specific folder for installing and packaging the libraries</p><pre>  - step:<br>          name: Packaging lambda Layer<br>          image: python:slim<br>          script:<br>            - apt-get update -y &amp;&amp; apt-get upgrade -y &amp;&amp; apt-get install -y zip<br>            - mkdir -p lambda-layer/python<br>            - cp captures/requirements.txt lambda-layer/python/requirements.txt<br>            - cd lambda-layer/python<br>            - pip3 install --platform manylinux2014_x86_64 -r requirements.txt --target .  --python-version 3.11 --only-binary=&quot;:all:&quot;<br>            - cd ..<br>            - zip -r layer.zip python<br>          artifacts:<br>            - lambda-layer/layer.zip</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AHDnz5vxeOapHDwvyMQsBA.png" /><figcaption>visual representation of bitbucket pipleline</figcaption></figure><h3>Terraform Configurations:</h3><p>For creating a Lambda layer, I used Terraform to execute three steps:</p><ul><li>creating an S3 bucket,</li><li>uploading the layer.zip file to the bucket,</li><li>and creating the Lambda layer from the S3 URL.</li></ul><p>The Terraform configuration is straightforward and can be found below. If you&#39;re new to Terraform, you can refer to the <a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_layer_version">documentation</a> for more information</p><pre><br># define existing bucket for storing lambda layers<br>resource &quot;aws_s3_bucket&quot; &quot;lambda_layer&quot; {<br>  bucket_prefix = &quot;lambda-layer-kafka-${var.environment}&quot;<br>}<br><br># upload zip file to s3<br>resource &quot;aws_s3_object&quot; &quot;lambda_layer_zip&quot; {<br>  bucket = aws_s3_bucket.lambda_layer.id<br>  key    = &quot;lambda_layers/${local.layer_name}/${local.layer_zip_name}&quot;<br>  source = &quot;../lambda-layer/layer.zip&quot;<br>  etag   = filesha1(&quot;../lambda-layer/layer.zip&quot;)<br>}<br><br>resource &quot;aws_lambda_layer_version&quot; &quot;lambda_layer&quot; {<br>  s3_bucket           = aws_s3_bucket.lambda_layer.id<br>  s3_key              = aws_s3_object.lambda_layer_zip.key<br>  layer_name          = local.layer_name<br>  description         = &quot;python libraries for kafka&quot;<br>  license_info        = &quot;MIT&quot;<br>  compatible_runtimes = [&quot;python3.11&quot;, &quot;python3.9&quot;, &quot;python3.10&quot;]<br>  depends_on          = [aws_s3_object.lambda_layer_zip] # triggered only if the zip file is uploaded to the bucket<br>}</pre><p>Check out the screenshot below to see the Lambda layer in action!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C5wo72NvM587Mv1Cl1GtFA.png" /><figcaption>one the lambda layer is crated we should be able to see it in layer section</figcaption></figure><p>With the Lambda layer created, we can now easily import and use these libraries in our Lambda function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L4g2n4vsM3OH4AK9U3NNVg.png" /></figure><p>#python #bitbucket #terraform #lambda #aws #lambda-layer</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6f13b5c5123" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/streamline-your-data-processing-how-to-build-a-custom-lambda-layer-for-kafka-connectivity-in-d6f13b5c5123">Streamline Your Data Processing: How to Build a Custom Lambda Layer for Kafka Connectivity in…</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Retrieval-Augmented Generation aka RAG]]></title>
            <link>https://medium.com/@amir.mehrabi.j/retrieval-augmented-generation-aka-rag-729d58118f9b?source=rss-faea5a04b1cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/729d58118f9b</guid>
            <category><![CDATA[chromadb]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[rag-architecture]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[ollama]]></category>
            <dc:creator><![CDATA[Amir Mehrabi Jorshari]]></dc:creator>
            <pubDate>Thu, 01 Aug 2024 15:51:53 GMT</pubDate>
            <atom:updated>2024-08-01T16:06:32.678Z</atom:updated>
            <cc:license>http://creativecommons.org/publicdomain/zero/1.0/</cc:license>
            <content:encoded><![CDATA[<p>I became fascinated with AI and argument generation for a few months, and I decided to take it a step further by building a minimal prototype using Node.js to gain a deeper understanding of the concept. As a hands-on learner, I find that writing code helps me absorb information better than just reading about it. This article is intended for developers who have a background in coding and are interested in exploring AI. I’ll start by providing a brief introduction to RAG and its applications, followed by an explanation of its architecture. Finally, I’ll guide readers through the process of building a small-scale version of RAG using Node.js, ChromaDB, Docker, and Ollama</p><h3>What is RAG?</h3><p>Retrieval-Augmented Generation (RAG) is a cutting-edge language model that leverages the best of both worlds by combining retrieval-based and generation-based approaches. Unlike traditional language models, which generate text from scratch, and retrieval-based models, which rely solely on pre-existing text, RAG models employ a hybrid strategy that integrates both methods.</p><p>Here’s how it works:</p><ol><li><strong>Retrieval</strong>: The model searches a database or knowledge graph for relevant text related to the input prompt or query, drawing upon existing knowledge.</li><li><strong>Generation</strong>: The model then uses a language generation algorithm, such as transformer-based models, to create new text based on the retrieved information. This generated text is informed by the retrieved text, allowing for a more accurate and informative output.</li></ol><h3>SCOPE</h3><p>Understanding the scope of Retrieval-Augmented Generation (RAG) is crucial, as it enables us to determine when and where to apply this approach. The retrieved text serves as a foundation or “prompt” for the generation process, offering several benefits:</p><ul><li><strong>Improved accuracy</strong>: By leveraging the knowledge and accuracy of the retrieved text, RAG models can produce more accurate and informative generated text.</li><li><strong>Increased diversity</strong>: Combining retrieved text with generated text enables RAG models to generate more diverse and varied text, unbound by a single perspective or style.</li><li><strong>Efficiency</strong>: RAG models can be more efficient than traditional generation-based models, as they don’t require generating text from scratch.</li></ul><p>However, RAG models may not be suitable for all situations, particularly when:</p><ul><li><strong>Limited or irrelevant data</strong>: If the retrieved data is limited or irrelevant, the generated text may not be accurate or informative.</li><li><strong>Complex tasks</strong>: RAG models may struggle with complex tasks that require a deep understanding of the topic or domain, as they rely on the quality of the retrieved text.</li><li><strong>High-level abstractions</strong>: RAG models may not be able to generate high-level abstractions or creative ideas, as they are limited by the retrieved text.</li><li><strong>Domain-specific knowledge</strong>: RAG models may not be able to capture domain-specific knowledge or nuances, as they rely on the retrieved text.</li></ul><p>In conclusion, RAG models offer a promising approach to language generation, but it’s essential to consider the limitations and potential biases of the retrieved text and the generation algorithm used. By acknowledging these limitations, we can effectively apply RAG models to specific use cases and achieve better results.</p><h3>Architecture</h3><p>The diagram below provides a general overview of Retrieval-Augmented Generative (RAG) models. In essence, RAGs can operate similarly to traditional Language Language Models (LLMs), where users pose questions and the LLM generates responses. The key distinction lies in the context that the RAG controller sends to the LLM, which is informed by the knowledge database.</p><p>This architecture gives rise to three distinct sections:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cN8MwsmXRqTPfRbR4FXmvw.png" /></figure><p><strong>Large Language Model (LLM)</strong></p><p>I utilize ollama to access a variety of pre-trained models, including open-source options such as LLMA 2 and LLAMA 3. These models can be leveraged to meet specific requirements.</p><p><strong>Knowledge Database (Context DB)</strong></p><p>In my implementation, I have chosen to use Chroma DB, an open-source and production-ready database, for storing and retrieving knowledge. However, this is not the only option available. Other vector databases, such as NeoGraph, can also be utilized to support vector-based queries. For the purposes of this article, I will focus on Chroma DB for simplicity.</p><p><strong>RAG Controller</strong></p><p>The RAG Controller is the core component responsible for retrieving similar contexts from the Knowledge Database, sending them to the Large Language Model for processing, and then returning the response to users.</p><h3>Prototype</h3><p>In this prototype, I will demonstrate how to bring the architecture to life by setting up the code and making it functional. The first step is to store our knowledge context in a vector database, which in this case is Chroma DB. I refer to this phase as “Data Feeding”.</p><p><strong>OLLAMA Installation and Setup</strong></p><p>To run OLLAMA on your local system, use the following command:</p><p>docker run --name ollama -d -p 11434:11434 ollama/ollama</p><p>Once the container is up and running on port 11434, it exposes four endpoints:</p><ul><li>http://localhost:11434/api/embeddings: This endpoint is used for embedding and will be explained in more detail in the &quot;Data Feeding&quot; section.</li><li>http://localhost:11434/api/pull: This endpoint is used to pull a model.</li><li>http://localhost:11434/api/generate: This endpoint is used to generate a response based on a given prompt.</li><li>http://localhost:11434/api/tags: This endpoint returns a list of all pulled and ready-to-use models.</li></ul><p><strong>Pulling a Model</strong></p><p>After the container is up and running, you need to pull a model. OLLAMA offers various models, such as llama3.1, gemma2, mistal-nemo, and mistral-large. You can find the list of available models on the OLLAMA website at <a href="https://ollama.com/library">https://ollama.com/library</a>. For this setup, I chose to pull llama2, which is smaller than llama3.1.</p><p>To pull a model, use the following command:</p><pre>curl --location &#39;http://localhost:11434/api/pull&#39; \<br> --header &#39;Content-Type: application/json&#39; \<br> --data &#39;{ &quot;model&quot;: &quot;llama2&quot; }&#39;</pre><p>After running this command, you should see the log of downloading in the Docker Desktop container logs section</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*xGZUrXHbyprwGVTurLpzjQ.png" /></figure><p><strong>Verifying the Pulled Model</strong></p><p>After successfully pulling the model, you can verify that it has been downloaded by checking the response of the http://localhost:11434/api/tags endpoint. This endpoint returns a list of all pulled and ready-to-use models. Once the model is downloaded, you should be able to see the pulled model, such as &quot;llama2&quot;, in the response of this endpoint</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*dXEUzb1IIFNXrs3HMqV1TA.png" /></figure><p><strong>Data Feeding</strong></p><p>As previously mentioned, I will use a vector database like Chroma DB to store and retrieve our context (documents). In the following sections, I will describe how to install and store data in our database.</p><p><strong>Chroma DB Installation</strong></p><p>I can run Chroma DB using Docker by executing the following command:</p><p>docker run --name vectordb -d -p 8000:8000 chromadb/chroma:latest</p><p>This command exposes port 8000, which is the default port for Chroma DB.</p><p><strong>Generating Data</strong></p><p>In this prototype, I used the Faker library to generate a fake document (knowledge). In a real scenario, you can use real data such as documents, pages, texts, etc. For simplicity, I skipped partitioning of data, which may be required in a production environment.</p><p><strong>Transforming Data into Vectors</strong></p><p>Before storing data in our vector database, I need to transform it into a vector. To achieve this, I need to call our Large Language Model (LLM) for embedding. Embedding is the process of transforming a text into a vector.</p><p>The code snippet below demonstrates how to generate a vector using the LLM:</p><pre>const embedder: IEmbeddingFunction = {<br>  async generate(text: string[]) {<br>    const data = await axios.post(&quot;http://localhost:11434/api/embeddings&quot;, {<br>      model: embeddingModel,<br>      prompt: text[0],<br>    });<br>    const response = data.data[&quot;embedding&quot;];<br>    return [response];<br>  },<br>};</pre><p>This code sends a POST request to the http://localhost:11434/api/embeddings endpoint, passing the embeddingModel and the first element of the text array as parameters. The response from the LLM is then stored in the response variable and returned as a vector.</p><h4>Data Generation</h4><pre>async function addData(<br>  client: ChromaClient,<br>  collectionName: string,<br>  start = 0,<br>  length = 100<br>) {<br>  const collection = await client.getOrCreateCollection({<br>    name: collectionName,<br>    embeddingFunction: embedder,<br>  });<br><br>  for (let i = start; i &lt; start + length; i++) {<br>    const faker = new Faker({ locale: [en] });<br>    const fakeName = faker.company.name();<br>    const doc = `fullname:${faker.person.fullName()}, department:${faker.commerce.department()}, company:${faker.company.name()}, job:${faker.person.jobTitle()}, address:${faker.location.streetAddress(<br>      { useFullAddress: true }<br>    )}, country:${faker.location.country()}, salary:${faker.finance.amount({<br>      min: 500,<br>      max: 6000,<br>    })}, sex:${faker.person.sex()}, loan:${faker.commerce.price({<br>      min: 1000000,<br>      max: 9900000,<br>    })}`;<br>    console.log(`adding ${fakeName}`);<br>    await collection.add({<br>      ids: [&quot;comp-&quot; + i.toString()],<br>      documents: [doc],<br>      metadatas: {<br>        id: i,<br>      },<br>    });<br>  }<br>  console.log(&quot;done.&quot;);<br>}</pre><p><strong>Parameters:</strong></p><ul><li>client: An instance of the ChromaClient class, which is used to interact with the Chroma DB.</li><li>collectionName: The name of the collection where the data will be added.</li><li>start (optional): The starting index for generating fake data. Defaults to 0.</li><li>length (optional): The number of fake data records to generate. Defaults to 100.</li></ul><p><strong>Functionality:</strong></p><ol><li>The function first gets or creates a collection with the specified collectionName using the client.getOrCreateCollection method. The embeddingFunction parameter is set to embedder, which is likely an instance of an embedding function (e.g., a language model).</li><li>The function then loops through a range of indices from start to start + length using a for loop.</li><li>Inside the loop, the function generates a fake document using the Faker library. The document contains various fields such as:</li></ol><ul><li>fullname: A fake full name generated using the Faker library.</li><li>department, company, job, address, country, salary, sex, and loan: Various fake values generated using the Faker library.</li></ul><ol><li>The function then adds the fake document to the collection using the collection.add method. The ids parameter is set to an array containing a single ID, comp-&lt;index&gt;, where &lt;index&gt; is the current index. The documents parameter is set to an array containing the fake document, and the metadatas parameter is set to an object with a single property, id, set to the current index.</li><li>After adding all the fake data records, the function logs a message indicating that the process is complete.</li></ol><p>In summary, this function generates fake data using the Faker library and adds it to a Chroma DB collection using the ChromaClient instance.</p><h3><strong>Retrieving Similar Contexts and Crafting a Prompt</strong></h3><p>This section consists of two crucial steps in our knowledge retrieval system. Firstly, I need to identify similar contexts or documents from our vector database that are relevant to the user’s query. This involves querying the vector database to retrieve a set of documents that are closest to the user’s input.</p><p>Once I have retrieved the similar contexts, I need to craft a prompt template that will be sent to our Large Language Model (LLM). The prompt template serves as a guide for the LLM to generate a response that is tailored to the user’s question. The LLM will analyze the context and generate a response based on the information it has learned from the training data.</p><p>The prompt template is a critical component in our system, as it enables the LLM to provide accurate and relevant responses to the user’s queries. By carefully crafting the prompt template, I can ensure that the LLM generates responses that are not only informative but also engaging and helpful to the user.</p><p>In the next section, I will explore how to integrate the LLM with our knowledge retrieval system and how to fine-tune the prompt template to achieve optimal results.</p><h4>Find Similar Contexts</h4><pre>async function querySimilarContexts(<br>  client: ChromaClient,<br>  collectionName: string,<br>  search: string,<br>  resultCount = 3<br>) {<br>  const collection = await client.getCollection({<br>    name: collectionName,<br>    embeddingFunction: embedder,<br>  });<br>  console.log(`generate embedding.... for ${search}`);<br>  const queryVetex = await embedder.generate([search]);<br>  const query = await collection.query({<br>    queryEmbeddings: queryVetex,<br>    nResults: resultCount,<br>  });<br>  console.log(`query data have been populated for ${search}`);<br>  console.log({ doc: query.documents });<br>  return query;<br>}</pre><p>This is an asynchronous function named querySimilarContexts that queries a Chroma DB collection using a search query and returns similar results. Here&#39;s a breakdown of the function:</p><p><strong>Parameters:</strong></p><ul><li>client: An instance of the ChromaClient class, which is used to interact with the Chroma DB.</li><li>collectionName: The name of the collection to query.</li><li>search: The search query to use for querying the collection.</li><li>resultCount (optional): The number of results to return. Defaults to 3.</li></ul><p><strong>Functionality:</strong></p><ol><li>The function first gets the collection with the specified collectionName using the client.getCollection method. The embeddingFunction parameter is set to embedder, which is likely an instance of an embedding function (e.g., a language model).</li><li>The function logs a message indicating that it’s generating an embedding for the search query.</li><li>The function uses the embedder to generate an embedding for the search query. The generate method takes an array of strings as input, and in this case, it&#39;s an array containing the search query.</li><li>The function logs a message indicating that the query data has been populated for the search query.</li><li>The function uses the collection.query method to query the collection using the generated embedding. The queryEmbeddings parameter is set to the generated embedding, and the nResults parameter is set to the resultCount parameter (or 3 if not specified).</li><li>The function logs the query results to the console and returns the results as an object containing the documents property.</li></ol><p>In summary, this function generates an embedding for a search query, queries a Chroma DB collection using the embedding, and returns the results.</p><h3>Send Prompt to LLM</h3><pre>async function askAI(<br>  client: ChromaClient,<br>  collectionName: string,<br>  question: string<br>) {<br>  console.log(`thinking about....${question}`);<br>  const data = await querySimilarContexts(client, collectionName, question, 10);<br>  const context = data.documents[0];<br><br>  const contextText = context.join(&quot;\n----\n&quot;).replace(/\&quot;/g, &quot;&quot;).toString();<br>  const promptTemplate = `Answer the question based only on the following context:<br>  <br>  ${contextText}<br>  <br>  ---<br>  <br>  Answer the question based on the above context: ${question}<br>  NOTE: in case u couldnt answer the question u have to answer: &#39;I DONT KNOW&#39;.<br>  `;<br><br>  console.log({ promptTemplate: promptTemplate.toString() });<br>  console.log(<br>    `asking from llama for \&quot;${question}\&quot; to generate the response.`<br>  );<br>  const response = await axios.post(&quot;http://localhost:11434/api/generate&quot;, {<br>    model: llamaModel,<br>    prompt: promptTemplate,<br>    stream: false,<br>    context: [],<br>  });<br>  console.log({ response: response.data[&quot;response&quot;] });<br>}</pre><p>This is an asynchronous function named askAI that asks an AI (LLaMA) a question and retrieves a response based on a context. Here&#39;s a breakdown of the function:</p><p><strong>Parameters:</strong></p><ul><li>client: An instance of the ChromaClient class, which is used to interact with the Chroma DB.</li><li>collectionName: The name of the collection to query.</li><li>question: The question to ask the AI.</li></ul><p><strong>Functionality:</strong></p><ol><li>The function queries the Chroma DB collection using the querySimilarContexts function to retrieve similar contexts to the question. The querySimilarContexts retrieves a list of documents from the collection that are similar to the question.</li><li>The function <strong>creates a prompt template</strong> using the extracted text and the question. The prompt template is a string that provides context to the AI and asks it to answer the question based on that context. The prompt template includes the following:</li></ol><ul><li>A header indicating that the AI should answer the question based only on the provided context.</li><li>The extracted text from the similar context document.</li><li>The question itself.</li><li>A note indicating that if the AI cannot answer the question, it should respond with “I DONT KNOW”.</li></ul><p><strong>Prompt Template:</strong></p><p>The prompt template is a critical component of this function, as it provides context to the AI and asks it to answer the question based on that context. The prompt template is carefully crafted to include the following elements:</p><ul><li>A header indicating that the AI should answer the question based only on the provided context.</li><li>The extracted text from the similar context document, which provides additional information to help the AI answer the question.</li><li>A separator line (---) to separate the context from the question.</li><li>A header indicating that the AI should answer the question based on the provided context.</li><li>The question itself, which is the primary input to the AI.</li><li>A note indicating that if the AI cannot answer the question, it should respond with “I DONT KNOW”.</li></ul><p>By carefully crafting the prompt template, the function ensures that the AI receives the necessary context and guidance to generate a relevant and accurate response to the question.</p><h3><strong>Improvement Areas:</strong></h3><p>To further enhance the performance and effectiveness of our knowledge retrieval system, I have identified several areas for improvement:</p><ol><li><strong>Faster and More Accurate Embeddings:</strong> I aim to optimize our embedding process to be faster and more accurate, tailored to our specific needs and requirements. This will enable us to retrieve more relevant and precise results from our vector database.</li><li><strong>Enhancing Similar Context Retrieval:</strong> Finding similar contexts plays a vital role in our system, as our Large Language Model (LLM) relies on these contexts to generate responses. I will focus on improving the efficiency and accuracy of this process to ensure that our LLM receives the most relevant and useful context information.</li><li><strong>Optimizing LLM Model Selection:</strong> I will explore and evaluate various LLM models to determine the best fit for our specific needs and requirements. This will enable us to leverage the strengths of each model and optimize our system’s performance.</li><li><strong>Exploring Alternative Database Options:</strong> Instead of relying solely on vector databases, I will investigate the use of graph databases that support vector operations. This may provide additional benefits, such as improved query performance and enhanced data relationships.</li></ol><p>By addressing these improvement areas, i can further refine my knowledge retrieval system, enabling it to provide more accurate and relevant responses to user queries.</p><h4>Reference:</h4><p><a href="https://github.com/amirofy2002/rag-controller/tree/main">GitHub - amirofy2002/rag-controller: rag prototype</a></p><p>#ai #rag #chromadb #docker #nodejs #typescript</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=729d58118f9b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>