<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Joseph Sciarrino on Medium]]></title>
        <description><![CDATA[Stories by Joseph Sciarrino on Medium]]></description>
        <link>https://medium.com/@jsciarrino12?source=rss-b960d5224ae6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*WOAsi5ruBtSjnzFHeI16SQ.png</url>
            <title>Stories by Joseph Sciarrino on Medium</title>
            <link>https://medium.com/@jsciarrino12?source=rss-b960d5224ae6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 06:44:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jsciarrino12/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Outgrowing Postgres? Keep using Postgres!]]></title>
            <link>https://medium.com/@jsciarrino12/outgrowing-postgres-keep-using-postgres-eaf8a24d3241?source=rss-b960d5224ae6------2</link>
            <guid isPermaLink="false">https://medium.com/p/eaf8a24d3241</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Joseph Sciarrino]]></dc:creator>
            <pubDate>Wed, 02 Feb 2022 00:34:28 GMT</pubDate>
            <atom:updated>2022-02-02T00:37:10.758Z</atom:updated>
            <content:encoded><![CDATA[<p>When data and indexes no longer fit in memory Postgres slows down. That’s expected — no single database will solve all problems.</p><p>Databases have trade-offs. Postgres is great with high-throughput transactions, but analytics in row-stores are <em>*generally*</em> slower. Like most things, weaknesses become clearer at scale.</p><p>Engineering organizations pick the right tools for the right jobs. This approach eventually collides with another great design principle: “Keep it simple as long as you can.” At scale, using many ‘right tools’ breed nostalgia for running a good ol’ Postgres monolith.</p><p>Cue the data warehouses. Snowflake, Redshift, BigQuery, etc have become crucial to most corporate data journeys. They’re a ‘source of truth,’ aggregating data with many data pipelines from SaaS. Data warehouses are fairly complimentary with Postgres, as they compensate for tasks Postgres isn’t designed for.</p><p>You should keep it simple (KISS) <strong>AND</strong> choose the right tool: Outgrowing Postgres? Keep using Postgres!</p><h3><strong>How?</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*3FKwM2MgrJSyKjwWohhOQA.png" /><figcaption>Query Postgres, route to Snowflake with Hydra!</figcaption></figure><h4><a href="https://www.hydras.io/">Hydra</a>: a clever database program that automatically routes your queries to the correct (and fastest!) database always. 🐘 Postgres-compliant SQL is all you need to access any database. Let Hydra intelligently handle database trade-offs so you don’t have to wrestle them in the application layer.</h4><ul><li>Postgres-compliant SQL for every database</li><li>Fast queries: routes to fastest database always</li><li>JOIN between databases, cross-database HTAP workloads</li><li>No query re-writes to adopt new databases</li></ul><p>Hydra makes it easy to merge operational and analytical workloads. Use a single endpoint for Postgres (OLTP) and Snowflake (OLAP) to simplify how apps access data. Data movement is expensive and slow at scale, so use Hydra to access the data where it lives.</p><p><a href="https://en.wikipedia.org/wiki/Hybrid_transactional/analytical_processing">HTAP</a> Snowflake!?</p><p>Snowflake is our first integration because its fast analytics are complimentary to Postgres. Hydra appears like a single database with Snowflake’s analytical speeds and Postgres’ transactional throughput. Keep using Postgres-compliant SQL over every database.</p><h4>Want to scale with Postgres? Get to a better data model quicker by adopting Hydra. Don’t re-write your queries when you transition from PG monolith to (PG + whatever other DB you need).</h4><p>Thanks for reading! — the Hydra team</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eaf8a24d3241" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Importance of Incremental MapReduce to the ‘Intelligence Layer’]]></title>
            <link>https://medium.com/@jsciarrino12/the-importance-of-incremental-mapreduce-to-the-intelligence-layer-628243a4471f?source=rss-b960d5224ae6------2</link>
            <guid isPermaLink="false">https://medium.com/p/628243a4471f</guid>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[mapreduce]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[newsql]]></category>
            <dc:creator><![CDATA[Joseph Sciarrino]]></dc:creator>
            <pubDate>Tue, 15 Oct 2019 17:46:01 GMT</pubDate>
            <atom:updated>2022-07-18T23:42:53.010Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A review of hybrid transactional/analytical processing and ingestion micro-batching in an MPP architecture</em></p><p>There’s an estimated 2.5 <em>quintillion</em> bytes of data being created per day. The ‘intelligence layer’ is responsible for processing and analyzing that massive data footprint through Machine Learning and Deep Learning algorithms. The accuracy and usefulness of insights-powered user facing applications is dependent on ingesting more data points, a wider variety of correlated variables, and using ‘progressive learning’ through which models self-improve. As one would expect, more advanced and comprehensive models necessitate greater data collection and so the cycle compounds.</p><p>Unsurprisingly, this compounding cycle poses a scale problem, outlined as follows: The intelligence layer sits above the database or ‘data layer’ and both need to operate harmoniously. To feed the intelligence layer our databases must perform equally (if not better) with 10X the data volume. Moore’s law predicts processing power will double every two years, but if our data capture quintuples — what happens? It doesn’t work.</p><p>Massively Parallel Processing (MPP) architectures offer a solution by employing MapReduce: a framework that splits up data (a logical concept), concurrently executes parallel tasks against the resulting smaller data subsets, then recombines and returns the results. MapReduce presents two efficiencies. First, running queries in parallel maximizes the resource utilization (CPU + RAM) of a server. Second, MapReduce enables sharded data and indexes to more easily fit in-memory. Processing in-memory makes Online transactional processing (OLTP) and analytic processing (OLAP) possible at much higher throughputs and high read +write concurrency. As a result, with equivalent hardware, MapReduce accelerates data ingestion and can return analytic queries 100X faster than previously possible.</p><p>Processing in-memory with an MPP (multi-node) architecture allows emergence of combining transactional and analytics workloads into one hybrid system called HTAP. HTAP is critical to intelligence layer solutions as value is created during the transaction real-time, rather than after an ETL process, thereby delivering on A.I’s promise to immediately respond to human events in real time.</p><p>Data recency and ingestion is managed by <strong>Incremental</strong> MapReduce: a concept where data is first ingested to a raw table. From there, levering the power of parallelized bulk inserts, you can ‘micro-batch’ to an aggregated ‘Rollup’ table. In the Postgres world, micro-batch means copy is running against a small segment of the ingested data on a frequent and recurring increment. The time increment can be a second, five seconds, a minute, or an hour depending on the ‘real-time’ requirements. By micro-batching the copy command, Postgres is able to ingest 2–5 million transactions per second — the ingestion speeds necessary to process insights at the edge from aggregated and raw data.</p><p>By leveraging pre-aggregation, micro-batch ingestion, in-memory processing, and parallelized architecture, AI solutions can execute in immediate response to human real-time events. Rather than ‘lifting and shifting’ between databases, consider combining into a merged transactional and analytical workload. HTAP provides the unique opportunity to create AI driven value at the time of transaction. The technical requirements for functional apps, infrastructure, data collection and processing, and AI-powered insights has never been larger. For AI-powered insights to be possible, we must nail scalability of data capture, processing, and analytics at the data layer.</p><p><em>A thank you to </em><a href="https://www.linkedin.com/in/andrewmontalenti/"><em>Andrew Montalenti</em></a><em> for their input and edits.</em></p><p>Feel free to reach out on <a href="https://www.linkedin.com/in/jsciarrino12/">LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=628243a4471f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>