<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Lazy Techie - Medium]]></title>
        <description><![CDATA[The first virtue of good programmers - Medium]]></description>
        <link>https://medium.com/lazy-techie?source=rss----2f582ba13401---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Lazy Techie - Medium</title>
            <link>https://medium.com/lazy-techie?source=rss----2f582ba13401---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 14:11:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/lazy-techie" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Simple Aggregation Metrics using DynamoDB]]></title>
            <link>https://medium.com/lazy-techie/simple-aggregation-metrics-using-dynamodb-285b6f158042?source=rss----2f582ba13401---4</link>
            <guid isPermaLink="false">https://medium.com/p/285b6f158042</guid>
            <dc:creator><![CDATA[Ketan Khairnar]]></dc:creator>
            <pubDate>Mon, 06 Sep 2021 06:09:14 GMT</pubDate>
            <atom:updated>2021-09-05T17:11:36.655Z</atom:updated>
            <content:encoded><![CDATA[<p>When you need quick access to some metrics typically grouped by some attribute hierarchy and one of that is time, day, month etc; you don’t have to get into full blown OLAP or time series/columnar database solution.</p><blockquote>We need to fall back to default heuristic — what’s immediately doable and satisfies current need without focusing too much on what’s the most ideal solution or exploring all potential solutions — what libraries, data-stores or tech on needs to do this.</blockquote><p>In this post; am going to talk about use case and then propose a very simple pre-aggregation so that we’re calculating metrics in near-real time for all potential questions.</p><p><strong>Use Case</strong>: You’re running SaaS platform where your customers can issue zero to many physical or virtual credit cards to their employees. Employees would use these cards for their assigned expenses. Within each organization there could be multiple departments/teams. One need to track expenses for customer organization, specific user/team or specific card across time line such as days, weeks or months</p><p><strong>More on Transaction Data: </strong>Transaction does hold day, time, merchant, amount information by default. Each transaction although identified by specific card; we do have association of that card to user and whole organization hierarchy. Transactions are stored in relational database and for performance reason only records primary transaction information specifically some identifier for card and nothing about user, department, customer etc. One more thing — transaction amounts can be positive or negative depending upon purchases, cancellations and refunds etc.</p><p><strong>Proposed Solution</strong></p><p>There are three steps to end goal.</p><blockquote>1. Original transaction data needs to be enriched so that it is linked to specific user, team. organization in addition to card</blockquote><blockquote>2. Determine data access patterns and response payload expectations for all queries.</blockquote><blockquote>3.Cost effective and optimal performance — for storage and retrieval APIs</blockquote><p><strong>Data Enrichment</strong></p><p>As soon as transaction is completed it can be handed off to another worker to enrich transaction data with other useful information. I delegated it to akka worker which fetched this data from cache and db lookups which is additional 10–20 ms in processing. Hence it is decoupled from the actual OLTP system</p><p><strong>Data Access Patterns</strong></p><p>Here are few examples</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*Rc4I7TzKgZW0YIEggeMHpA.png" /><figcaption>questions on aggregations for diff attributes of transactions.</figcaption></figure><p><strong>Storage and Retrieval API</strong></p><p>Since we know all possible queries in advance and data is not going to vary much in terms of number of attributes; it boils down to simple sub problems.</p><ol><li>Taking care of range queries — for different Time Units e.g. Day, Week, Month etc</li><li>Coming up with overloaded field which can act as partition key or shard identifier.</li></ol><p>DynamoDB has exactly same concepts — Parition Key and Sort Key or Range Key.</p><p>Since this is pre-aggregation; each transction results into multiple writes/updates.</p><ol><li>Org records for Day, Week, Month and Total</li><li>User records for Day, Week, Month and Total</li><li>Team records for Day, Week, Month and Total</li><li>Card records for Day, Week, Month and Total</li></ol><p>Doing these 16–20 updates is kind of database transaction and/or batch-write depending upon the use case and this could take easily additional 10–50 ms.</p><blockquote>I do want to emphasize that this is one time computation cost and for all future reads it acts as pre-aggregated results. It’s upto end user to select which time range one needs to look for specific data.</blockquote><blockquote>One can also render this in in effectively in visualizations —</blockquote><blockquote>1. metric trends over specified time range</blockquote><blockquote>2. metric distributions across time ranges — Month vs Weeks or days, Total vs Months</blockquote><blockquote>3. and also each day stat can account to multiple metrics such as spend, count, withdrawals vs refunds, international vs local and we can show them across same time range as trend or distribution</blockquote><p>This is very simple but effective strategy to address lot of similar use cases. Note that though this precomputation though hinted as near-real time that is not actually necessary condition. One can run this as EOD job and have these pre-aggregated computations stored/updated at leisure or at demand.</p><p>In the next sections; I will go in bit more detail that how we do leverage DynamoDB sort key and partition keys to achieve the same. And how one can do the same in postgres/mysql or clickhouse in the follow up posts.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=285b6f158042" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lazy-techie/simple-aggregation-metrics-using-dynamodb-285b6f158042">Simple Aggregation Metrics using DynamoDB</a> was originally published in <a href="https://medium.com/lazy-techie">Lazy Techie</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>