<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Max Kremer on Medium]]></title>
        <description><![CDATA[Stories by Max Kremer on Medium]]></description>
        <link>https://medium.com/@mkremer_75412?source=rss-10737e34fb8b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*kLORkOA8n_kpMDwJ.jpg</url>
            <title>Stories by Max Kremer on Medium</title>
            <link>https://medium.com/@mkremer_75412?source=rss-10737e34fb8b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 07:59:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mkremer_75412/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to replicate Postgres RDS functionality with a few scripts and a Cloudformation template]]></title>
            <link>https://medium.com/@mkremer_75412/how-to-replicate-postgres-rds-functionality-with-a-few-scripts-and-a-cloudformation-template-748c391fce51?source=rss-10737e34fb8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/748c391fce51</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloudformation]]></category>
            <dc:creator><![CDATA[Max Kremer]]></dc:creator>
            <pubDate>Wed, 06 Mar 2024 19:45:41 GMT</pubDate>
            <atom:updated>2024-03-08T02:31:18.833Z</atom:updated>
            <content:encoded><![CDATA[<h3>or How to build a Postgres rocket ship in your garage</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mkocGhkypuIQzNOI" /></figure><p>Greetings tech world denizens. In my last post <a href="https://medium.com/@mkremer_75412/why-postgres-rds-didnt-work-for-us-and-why-it-won-t-work-for-you-if-you-re-implementing-a-big-6c4fff5a8644">Why Postgres RDS didn’t work for us</a> I discussed some of the gotchas with running Postgres on Amazon RDS. I work with an application that collects a ton of online user behavior. Every click, every field input, every action. Data is stored in postgres and end users, via our UI or raw SQL access, run models and reports that crunch the above (sometimes very large) data sets. Our application is of the multi-tenant SAAS variety, so data is being ingested and stored per tenant across hundreds of tenants. It needs to be scalable and performant and not cost an arm and a leg.</p><p>In this post, I’ll guide you through constructing your own high-performance PostgreSQL setup on EC2 based off of our production environment. We’ll replicate 80% of RDS functionality — crafting both single instance clusters and scalable read-write replicas — optimized for performance and cost. This method not only offers a deeper dive into the granular control of your database environment but also unveils the potential for significant savings and performance gains.</p><p>Let’s start with the basics, RDS is slick but pricey. However, it gives turn-key ability to spin up PostgreSQL DBs and not worry about infrastructure. It comes with charts, monitoring, performance graphs, click to backup, click to restore. It’s a complete DB as a service solution. But that comes <a href="https://www.infoq.com/articles/managed-relational-databases-costs/">at a cost</a>, and <a href="https://www.infoq.com/articles/managed-relational-databases-costs/">I’m not just talking moola</a>. RDS offers very poor performance (and I/O transparency) compared with what Postgres is actually capable of given the right hardware (read my <a href="https://medium.com/@mkremer_75412/why-postgres-rds-didnt-work-for-us-and-why-it-won-t-work-for-you-if-you-re-implementing-a-big-6c4fff5a8644">previous post</a> for details on the issues we faced with RDS). So if you want to run high-performance PostgreSQL databases that scale with predictable cost, or require more than the limited control RDS gives you (e.g. you want to deploy Postgres extensions of your choice) then read on.</p><h3>What is Cloudformation</h3><p>At its core, AWS CloudFormation is an automation and infrastructure-as-code service provided by Amazon Web Services (AWS). It allows users to define, deploy, and manage a collection of AWS resources (like EC2 instances, VPCs, EBS volumes, etc…) using simple JSON/YAML files or “templates”. These templates describe the desired state of your <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stacks.html">AWS “Stack”</a> as Cloudformation calls it. This enables you to automate the provisioning and updating processes in a predictable and repeatable way. Think Terraform’s clumsy little brother (I have some beefs with Cloudformation but I’ll save those for another post).</p><p>Bottom line is that instead of manually setting up each component of your Postgres environment on EC2 — like configuring storage, network settings, security groups, and replication — CloudFormation lets you encapsulate all these steps within a template that groups a bunch of resources together into a “Stack”. In our case each stack will be a read/write Postgres primary with an optional read-only replica. Along with that we can automate the creation of the bells and whistles that are included with RDS such as metrics, alerts, access control and security, logging, replication and backups.</p><h3>Inventory of moving parts</h3><p>Let’s start by mapping out an inventory of moving parts.</p><h4>AMI</h4><p>Our AMI or base image is based off of the AWS Linux2 image with a few added yum installs:</p><pre>sudo amazon-linux-extras install epel<br>sudo yum install -y kernel-develsudo yum install -y epel-release<br>sudo yum install https://zfsonlinux.org/epel/zfs-release-2-2.el7.noarch.rpm<br>sudo sed -i ‘s/\$releasever/7/g’ /etc/yum.repos.d/zfs.repo**<br>reboot<br>sudo yum install -y zfs<br>sudo /sbin/modprobe zfs</pre><p>The above installs the ZFS filesystem and its dependencies. ZFS allows us to use compression, in essence trading a bit of CPU time for higher I/O. ZFS, along with HugePages which we’ll discuss later are the parts of this recipe that give your postgres instances an added performance boost. Spicey!</p><pre>sudo amazon-linux-extras install postgresql14 <br>sudo yum install -y postgresql-server postgresql-contrib postgresql-server-devel.x86_64 <br>sudo yum update</pre><p>Of Course we’ll need to install PostgreSQL version 14 and some of its libs.</p><pre>sudo yum install -y perl-Switch \<br> perl-DateTime \<br> perl-Sys-Syslog \<br> perl-LWP-Protocol-https \<br> perl-Digest-SHA.x86_64<br><br>curl https://aws-cloudwatch.s3.amazonaws.com/downloads/CloudWatchMonitoringScripts-1.2.2.zip -O<br>unzip CloudWatchMonitoringScripts-1.2.2.zip &amp;&amp; \<br> rm CloudWatchMonitoringScripts-1.2.2.zip &amp;&amp; \<br> cd aws-scripts-mon</pre><p>Some handy Perl utils and the AWS Cloudwatch monitoring scripts. Remember we’re starting with an Amazon Linux2 image so it will come with the AWS CLI which makes accessing other AWS resources via command line a breeze.</p><pre>wget https://github.com/wal-g/wal-g/releases/download/v2.0.1/wal-g-pg-ubuntu-18.04-amd64.tar.gz<br>tar -xvf wal-g-pg-ubuntu-18.04-amd64.tar.gz <br>mv wal-g-pg-ubuntu-18.04-amd64 wal-g<br>sudo chown root:root wal-g<br>sudo mv wal-g /usr/local/bin/<br>sudo yum install -y lzo<br>sudo yum install nvme-cli</pre><p>Here we’re installing WAL-G and a utility for querying the metadata of local NVMe drives (more on that below in EC2 Instances). For EC2 instance types that use the Nitro Hypervisor, volumes that use the NVMe driver appear as nvmeXnX volumes. We need to read those names dynamically so we’ll call the nvme cli in a script below.</p><p>WAL archiving is performed by WAL-G via PostgreSQL’s <strong>archive_command</strong>, and streaming replication is supported by PostgreSQL’s protocol. These functions are distinct, but closely related. WAL-G handles backups and point-in-time-restore (PITR), while streaming replication supports the hot standby server. However, if the standby falls behind the primary and misses a WAL segment it can also retrieve it from S3 using WAL-G.</p><h3>EC2 Instances</h3><p>We are going to be working with 2 types of instances here and 3 sizes. This does not mean that you are limited to the instance types I use here, just that these are the ones I’ve found work well for our analytics workload.</p><p>For our read-write primary our Cloudformation script will be configured to use one of 3 instance types:</p><ul><li>t3a.medium — general purpose instance, great for one or smaller tenants as well as QA workloads. We don’t pair this instance type with read replica but that doesn’t stop you from doing so</li><li>r5a.xlarge — Memory optimized instance so we can give postgres more memory</li><li>r5a.2xlarge — Same as above but more! For our enterprise tenants</li></ul><p>Note: r5b instances are the newest memory optimized instances that increase EBS performance by 3x compared to same-sized R5 instances so you may want to explore those.</p><p>For our read replicas we use i3en instance type for their lightning fast NVMe drives.</p><ul><li>i3en.xlarge — pairs with the r5a.xl master</li><li>i3en.2xlarge — pairs with the r5a.2xl master</li></ul><p>But we can’t have our cake and eat it too… i3en instances come with up to 60TB of NVMe storage however the catch is the drives are ephemeral — Ephemeral storage is transient; data is lost if the instance is stopped, terminated, or if the underlying physical drive fails. This contrasts with EBS volumes, where data persists independently of the instance’s life. So the high I/O for our instance-attached NVMe drives comes with potential data volatility. We’re using this for our read replicas so it’s fine. If a drive fails or an instance terminates we can easily recreate it using data from the master.</p><h4>Wal-G</h4><p>WAL-G is an archival restoration tool for PostgreSQL. It’s going to push Postgres transaction log files (WALs) to S3 from the primary and from the replica it’s going to read those WALs from S3 to bring the replica up to date. It also functions as our backup utility for taking scheduled full backups via cron.</p><h4>S3</h4><p>Amazon’s simple storage service we all know and love. We are going to be using S3 for 3 key purposes:</p><ul><li>Each instance type is going to pull its postgresql.conf file from an s3 path based on the instance type. For example for our primary r/w instance type the config file for r5a.2xlarge instance types will be in s3 under the following URL in my cf-pg-configs bucket: <a href="https://cf-pg-configs.s3.us-west-2.amazonaws.com/rw/r5a.2xlarge.conf">https://cf-pg-configs.s3.us-west-2.amazonaws.com/rw/r5a.2xlarge.conf</a></li><li>S3 will also be used by Wal-G to push/pull WAL files</li><li>WAL-G will also push nightly backups to s3 via shell script and cron job</li></ul><h4>Cloudwatch</h4><p>Cloudwatch is Amazon’s monitoring service. Remember that CloudWatchMonitoringScripts utility we installed on the AMI? It will enable us to send metrics to CloudWatch so we can put them on dashboards and create alarms. We’ll be looking at the free space available on drives and setting alarms to alert us when they reach 80% capacity</p><h4>Log forwarding agent</h4><p>It’s very handy to be able to access Postgres cluster logs in one central place so you’re not SSHing into every server. Here we use Splunk which includes a small daemon that runs on each instance and forwards specified log data (I’ve omitted that installation from my base AMI image instructions in Github). You can use any number of tools to accomplish this, for example Elastic Stack (ELK Stack): Comprising Elasticsearch, Logstash, and Kibana, the ELK Stack is a popular open-source choice for searching, analyzing, and visualizing log data in real-time. Splunk is not free but I do love it.</p><h4>Shell Scripts</h4><p>We are going to be using 3 short scripts to wire things up. Some of the scripts are generated dynamically because the Cloudformation Stack Name needs to be part of the script (although there are other ways around this via environment variables). All three of the scripts listed below are embedded within the Cloudformation template so look for the content of these scripts there (see Github repo link at the end). The scripts are all so short as not to warrant being version-controlled assets, separate from the Cloudformation template. From a devops perspective I felt confident including them within.</p><p><strong>push-backup.sh </strong>As the name suggests this script invokes the wal-g backup-push command to push a full backup to s3. We will be invoking this script via cron and at the time of postgres primary instance creation so that the replica has something to restore.</p><p><strong>poll-pg_hba_update.sh </strong>The purpose of this script is to poll Cloudformation from the postgres primary (via AWS CLI) to get the internal IP of replica. It’s a shame that Cloudformation doesn’t provide any facility for this — there is no way to create resources and then use their details when creating other resources later in your template — also circular dependencies are not allowed. Nonetheless we need the IP of the replica on the primary because the <a href="https://www.postgresql.org/docs/14/auth-pg-hba-conf.html">pg_hba.conf</a> on the primary needs a line to authenticate the replica like so:</p><pre>host replication repuser [REPLICA_IP]/32 scram-sha-256</pre><p>Now if you want to play fast and loose with security and rely on your VPC security groups to keep your instance safe then you don’t need to specify the replica’s ip and can eliminate this script. You can do this instead:</p><pre>host replication repuser 0.0.0.0/0 scram-sha-256</pre><p><strong>zpool-nvme.sh </strong>When creating the replica instance using i3en instance types the NVMe drives aren’t predictably named, furthermore depending on the instance size we get a different number of drives. We need the names of the attached NVMe drives to create our ZFS file system which will span the drives. To do that we use the NVMe cli utility we installed on the AMI to query the NVMe drive names and create our ZFS file system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*NmdHehnp_LVUG65f" /></figure><p>We end with something like this. Splunk can be replaced with any other log monitoring solution</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*t4K5F87k88fkfk7o" /></figure><p>The read only is similar only that WALs and backups move in the opposite direction (from S3)</p><h3>Putting it all together</h3><p>The Cloudformation template file is a YAML or JSON text file that lets you define resources to create a Stack. Your template can contain parameters where the user of your template can define inputs via an AWS screen at stack launch time, in our case the inputs are the AMI (there’s only one but I decided to make it a parameter for flexibility), the KeyPair name for instance security, the AV Zone, the Primary (or RW) instance type, the replica (or RO) instance type, the IP you wish to SSH from and the DB password for the replica user. The stack name itself is another implicit and required parameter, it used as a namespace to when creating other things, references to the stack name can be found throughout the template. The Parameters section looks like this:</p><pre> &quot;Parameters&quot;: {<br>    &quot;BaseImage&quot;: {<br>      &quot;Type&quot;: &quot;String&quot;,<br>      &quot;Default&quot;: &quot;ami-005973cfb88e3d3af&quot;,<br>      &quot;AllowedValues&quot;: [<br>        &quot;ami-005973cfb88e3d3af&quot;<br>      ]<br>    },<br>    &quot;KeyName&quot;: {<br>      &quot;Description&quot;: &quot;&quot;,<br>      &quot;Type&quot;: &quot;AWS::EC2::KeyPair::KeyName&quot;,<br>      &quot;ConstraintDescription&quot;: &quot;The name of an existing EC2 KeyPair.&quot;<br>    },<br>    &quot;AvailabilityZone&quot;: {<br>      &quot;Description&quot;: &quot;&quot;,<br>      &quot;Type&quot;: &quot;AWS::EC2::AvailabilityZone::Name&quot;<br>    },<br>    &quot;InstanceTypeRW&quot;: {<br>      &quot;Description&quot;: &quot;PostgreSQL EC2 instance type&quot;,<br>      &quot;Type&quot;: &quot;String&quot;,<br>      &quot;AllowedValues&quot;: [<br>        &quot;r5a.2xlarge&quot;,<br>        &quot;r5a.xlarge&quot;,<br>        &quot;t3a.medium&quot;<br>      ],<br>      &quot;ConstraintDescription&quot;: &quot;must be a valid EC2 instance type.&quot;<br>    },<br>    &quot;InstanceTypeRO&quot;: {<br>      &quot;Description&quot;: &quot;PostgreSQL EC2 instance type&quot;,<br>      &quot;Type&quot;: &quot;String&quot;,<br>      &quot;Default&quot;: &quot;not_set&quot;,<br>      &quot;AllowedValues&quot;: [<br>        &quot;not_set&quot;,<br>        &quot;i3en.2xlarge&quot;,<br>        &quot;i3en.xlarge&quot;<br>      ],<br>      &quot;ConstraintDescription&quot;: &quot;must be a valid EC2 instance type.&quot;<br>    },<br>    &quot;SSHLocation&quot;: {<br>      &quot;Description&quot;: &quot; The IP address range that can be used to SSH to the EC2 instances&quot;,<br>      &quot;Type&quot;: &quot;String&quot;,<br>      &quot;MinLength&quot;: &quot;9&quot;,<br>      &quot;MaxLength&quot;: &quot;18&quot;,<br>      &quot;Default&quot;: &quot;0.0.0.0/0&quot;,<br>      &quot;AllowedPattern&quot;: &quot;(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})&quot;,<br>      &quot;ConstraintDescription&quot;: &quot;must be a valid IP CIDR range of the form x.x.x.x/x.&quot;<br>    },<br>    &quot;DBPassword&quot;: {<br>      &quot;NoEcho&quot;: &quot;true&quot;,<br>      &quot;Description&quot;: &quot;Password for database access&quot;,<br>      &quot;Type&quot;: &quot;String&quot;,<br>      &quot;MinLength&quot;: &quot;1&quot;,<br>      &quot;MaxLength&quot;: &quot;41&quot;,<br>      &quot;AllowedPattern&quot;: &quot;[a-zA-Z0–9]*&quot;,<br>      &quot;ConstraintDescription&quot;: &quot;must contain only alphanumeric characters.&quot;<br>    }<br>  }</pre><p>The template then goes on to create the resources, in our case these are:</p><p><strong>InstanceProfile</strong> — To pass an IAM role to the EC2 Instances we create</p><p><strong>SG</strong> — The VPC security group we are creating for this stack (you can use existing security groups if you want)</p><p><strong>SGinSelfPG, SGinPG, SGinPG</strong> — a bunch of ingress rules for the above SG</p><p><strong>PsqlServer </strong>— this is our PostgreSQL RW primary</p><p><strong>PsqlServerPgTankAlarm</strong> — Our cloudwatch alarm for the ZFS zpool named “pgtank”. Will create an alarm when pgtank is over 80% full</p><p><strong>PsqlServerRO </strong>— Our read-only replica (this optional, we can run a single r/w instance)</p><p><strong>PsqlServerROPgTankAlarm </strong>— a similar cloudwatch alarm for the replica</p><p><strong>disk1, disk2 </strong>— EBS volumes for the RW primary</p><p><strong>PrimaryDNSRecord, ReadOnlyDNSRecord</strong> — DNS records in .local hosted zone. This is handy so that the application can know about the cluster without referring to IPs but is not strictly necessary</p><p>Rather than paste the entire Cloudformation template here I will provide a Github repo. Having to annotate the template line by line is painful so here is the gist of the gist of the recipe that spins up the Postgres cluster — you’ll find the details in the template:</p><h3>PostgreSQL Read/Write Primary</h3><ol><li><a href="https://docs.oracle.com/cd/E19253-01/819-5461/gaynr/index.html">Create</a> the zfs zpool called pgtank</li><li>Set some details on the zpool (compression etc…)</li><li>Do some housekeeping (directories, permissions, users)</li><li>Run postgre initdb to initialize the database</li><li>Update sysctl.conf to set the number of huge pages. Huge pages are a way for a process to make more efficient use of memory on your system. If your interested in the details of how this applies to postgres read <a href="https://pganalyze.com/blog/5mins-postgres-tuning-huge-pages">this</a>, but in a nutshell it allows postgres to work with larger chunks of memory</li><li>Setup a bunch of Cloudformation scripts for Cloudformation hooks</li><li>Write a pg_hba.conf file</li><li>Create a postgresql.conf file by pulling the right one from S3 (I’ve already done all the tuning so that memory settings, etc match the instance type — they are in the Github repo)</li><li>Create a postgresql.service file so that Postgres can be run by the systemd service daemon</li><li>Create the push-backup.sh and poll-pg_hba_update.sh scripts I mentioned above</li><li>Call psql and do an ALTER USER on postgres to set the password given in the params</li><li>Run systemctl restart postgresql to restart the service</li><li>Add the <strong>archive_command</strong> to postgresql.conf</li><li>Add cron jobs for push-backup.sh and the cloudwatch monitoring scripts (to send metrics)</li><li>Add cron job for poll-pg_hba_update.sh to begin polling for replica’s IP</li><li>Restart and run the push-backup.sh script to push the 1st backup to s3 so it will be available to the replica</li></ol><h3>PostgreSQL Read-only Replica</h3><ol><li>Run zpool-nvme.sh to query nvme drive names and create a zpool</li><li>Do some housekeeping (directories, permissions, users)</li><li>Perform a WAL-G backup-fetch to get the latest backup from s3 (pushed by primary via push-backup.sh)</li><li>Create a postgresql.conf file by pulling the right one from S3</li><li>Append the <strong>restore_command </strong>to postgresql.conf to begin restore</li><li>touch /pgtank/db/postgres/standby.signal to let Postgres know this is a replica</li><li>Add the details of the primary to postgresql.conf by appending <strong>primary_conninfo</strong></li><li>Update sysctl.conf to set the number of huge pages (just like the primary)</li><li>Setup a bunch of Cloudformation scripts for Cloudformation hooks</li><li>Create a postgresql.service file so that Postgres can be run by the systemd service daemon</li><li>Run systemctl restart postgresql to restart</li></ol><h3>Conclusion</h3><p>And there you have it, folks — your very own Postgres rocket ship, built from scratch in the comfort of your digital garage. You’ve just strapped a turbocharger onto Postgres, giving it the freedom to tear through data on the open road of EC2, unshackled from the terrestrial limitations of RDS.</p><p>What’s next on this wild ride? Well, you’re now (or at least after you’ve read the READMEs in the repo below) the proud pilot of a setup that’s not just a cost-effective alternative to RDS but a performance beast ready to scale to the moon and back.</p><p>Remember, with great power comes great responsibility. Keep an eye on those CloudWatch alarms, and don’t let your backups gather dust.</p><p>So, go ahead, give it a spin, and let me know how it goes. Did your Postgres setup fly high, or did it need a few tweaks to get off the ground? Share your tales of triumph or troubleshooting in the comments. I’m all ears and always up for a good cloud adventure story.</p><p>Github Repo: <a href="https://github.com/maxkremer/cloudformation_pg_cluster">https://github.com/maxkremer/cloudformation_pg_cluster</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=748c391fce51" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Postgres RDS didn’t work for us (and why it won’t work for you if you’re implementing a big…]]></title>
            <link>https://medium.com/@mkremer_75412/why-postgres-rds-didnt-work-for-us-and-why-it-won-t-work-for-you-if-you-re-implementing-a-big-6c4fff5a8644?source=rss-10737e34fb8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c4fff5a8644</guid>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-rds]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[postgresql]]></category>
            <dc:creator><![CDATA[Max Kremer]]></dc:creator>
            <pubDate>Mon, 15 Jan 2024 21:14:11 GMT</pubDate>
            <atom:updated>2024-03-20T23:40:20.507Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why Postgres RDS didn’t work for us (and why it won’t work for you if you’re implementing a big data solution)</h3><h3>Background</h3><p>We were building a startup that measured marketing effectivness via a statistical attribution model. For this to work we had to collect a ton of online user behavior. Every click, every field input, every action. Data was stored in postgres and end users, via our UI, ran dashboards and reports that crunched the above (sometimes very large) data sets. Important to note that we were building a multi-tenant SAAS application, so data was being ingested and stored per tenant across hundreds of tenants. It needed to be scalable and performant. We started with AWS RDS managed postgreSQL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*c8ns6q4Q5iYcBCqy" /></figure><h3>The big problem is with EBS: throughput opacity</h3><p>It’s almost impossible to accurately measure throughput on EBS. The metrics AWS gives you are latency and IOPS. Neither of these provide a measure of throughput. When you’re storing a large time-series table (say 20 million rows) and the size of the table on disk is 10GB and you want to scan the table (for whatever reason — say a sequential scan) then you need an accurate measure of throughput to understand how long it will take to scan the data from disk.</p><h3>Bursting and throttled IOPS — the slippery slope</h3><p>Queries like the one above quickly use up IO credits. We want more IOPS so what do we do? Naturally we increase the storage volume so we can get a higher baseline of 10000 IOPS from the measly 3000 you get with smaller volumes. But then backup costs increase because you’re paying for the entire volume, not just what’s used. Costs increase, performance doesn’t get much better — it snowballs from there.</p><p>We also kept uping our instance size to get more memory for caching. It’s easy to make the DB fast if it can fit it all in memory. There was also the promise of greater parallelization that came with PG v10, but sadly that brought us very little benefit due to the IO throttling.</p><p>The problems with measuring throughput are described in more detail here:</p><p><a href="https://www.datadoghq.com/blog/aws-ebs-latency-and-iops-the-surprising-truth/">https://www.datadoghq.com/blog/aws-ebs-latency-and-iops-the-surprising-truth/</a></p><h3>Aurora</h3><p>So then we migrated to Aurora. Promises of 5–7 X performance increases over standard postgres are hard to ignore. However the caveat is that these performance metrics are based on pgbench (in the case of postgresql). That’s great if you have an OLTP system and want to measure the speed (or latency) of transactions with a large number of parallel clients but is not a good performance indicator for large data-sets and data-mining operations.</p><p>Costs went up instead of down.</p><p>This thread covers IOPS and billable IOPS <a href="https://forums.aws.amazon.com/message.jspa?messageID=835303#835303">https://forums.aws.amazon.com/message.jspa?messageID=835303#835303</a></p><p>Again the biggest frustration was getting accurate measurements of throughput. Simply dividing the amount of data scanned by the time it took (using tack_io_timing) we saw that performance for a large dataset was nowhere near SSD speeds, more like magnetic disks.</p><p>Adding to the frustration was the endless hours dealing with support. While support staff were always very knowledgeable and eager to assist it always came down to tuning the query or tuning the DB. Again this came down to throughput — I would’ve loved to hear something along the lines of “This service is not a good choice given your workloads”. But eventually we figured that out for ourselves.</p><h3>The Solution: Roll your own</h3><p>For us the solution was fairly simple: don’t use a managed database services and roll our own infrastructure on EC2.</p><p>We did this using relatively low-grade EC2 instances (We were running 4XL on RDS) xl and 2xls.</p><p>The key was to separate READ and WRITE workloads between a master and standby replica that was kept up to date using WAL-G (<a href="https://github.com/wal-g/wal-g">https://github.com/wal-g/wal-g</a>)</p><p>The WRITE node could be slow, it’s only ingesting data, so EBS was good enough but we were able to <strong>DOUBLE</strong> EBS throughput using ZFS with compression.</p><p>For the READ node we had a setup with ephemeral NVMe drives, and again ZFS to boost performance.</p><p>The results speak for themselves:</p><p>$11K bill down to a $2100 (projected for September which will the first full month we’re clear of any RDS baggage)</p><p>Queries that would take hours or timeout altogether run in <strong>seconds</strong>.</p><p>We are able to measure actual throughput using command line tools on the unix instances (zpool iostat)</p><h3>Conclusion</h3><p>Sure it’s easy to hit a button and spin up an instance and never worry about backup and recovery. But that comes at a cost, both monetary and performance in our case. If you need REAL speed for crunching TBs or GBs of data then use bare metal, and if you can’t use bare metal then the next best thing is EC2 with NVMe drives. Read <a href="https://medium.com/p/748c391fce51">part two</a> where I provide the recipe for how to spin up your own Postgres Cluster on AWS using instances with NVMe drives.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c4fff5a8644" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Streamlining Behavioural Data Integration: Lassoo Meets Snowflake via Airbyte]]></title>
            <link>https://medium.com/@mkremer_75412/streamlining-behavioural-data-integration-lassoo-meets-snowflake-via-airbyte-50a0748a334a?source=rss-10737e34fb8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/50a0748a334a</guid>
            <category><![CDATA[airbyte]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Max Kremer]]></dc:creator>
            <pubDate>Tue, 12 Dec 2023 23:10:30 GMT</pubDate>
            <atom:updated>2023-12-12T23:10:30.167Z</atom:updated>
            <content:encoded><![CDATA[<p>Dive into the world of advanced analytics with our latest ETL installment on connecting Lassoo’s headless analytics with Snowflake via Airbyte. Discover how this powerful combination unlocks real-time insights, scales with your data needs, and drives informed business.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GEwRh5IH_WE4XZsD.png" /></figure><h3>Harness the Full Potential of Your Behavioral Data</h3><p>In today’s data-driven landscape, the ability to seamlessly integrate and analyze vast amounts of data is paramount. Lassoo.io, with its innovative approach to behavioral analytics, pairs exceptionally well with Airbyte, an open-source data integration platform. This guide will walk you through integrating Lassoo’s PostgreSQL-based headless analytics data with Snowflake using Airbyte, showcasing the simplicity and power of this integration.</p><h3>What is headless analytics?</h3><p>Headless Analytics is an architectural approach that decouples data collection from reporting. It focuses on the efficient ingestion, curation, reconciliation, and enrichment of data. Data teams can then use their preferred tools for reporting, BI and activation, directly on the collected data, or after loading/streaming it to their data warehouse.</p><h3>Why Lassoo and Airbyte?</h3><p>Lassoo.io offers a groundbreaking approach to behavioral analytics. Our headless analytics solution auto-tracks all customer behavior, providing granular first-party data directly via SQL. This data is not just comprehensive; it’s ready for immediate use, thanks to built-in features, classification, and rollups. Lassoo can power a wide variety of advanced use cases, allowing organizations to drive significant business value with behavioral data.</p><p>Airbyte stands out in the data integration space for its open-source nature, flexibility, and rapidly growing community. It supports a wide range of connectors, including PostgreSQL and Snowflake, making it a versatile choice for various data integration needs.</p><h3>Integrating Lassoo with Snowflake Using Airbyte</h3><h3>Step 1: Preparing Your Lassoo Data</h3><p>Ensure your Lassoo account is active, and you have access to the Lassoo db credentials for your behavioral data insance. Lassoo’s platform offers an intuitive interface for managing and understanding your data, connecting via SQL and generating SQL queries you can use downstream in BI tools, ML tools, reverse ETL etc…</p><h3>Step 2: Setting Up Airbyte</h3><p>If you’re new to Airbyte, setting up an instance is straightforward. Once you have Airbyte running:</p><ol><li><strong>Select PostgreSQL as the Source</strong>: In Airbyte, configure PostgreSQL as your source connector and input your Lassoo database credentials (host, port, database name, user, and password).</li></ol><p>You will also need to select the update method. CDC is idea, it is most lightweight therefore will incur the lowest processing charges. This type of CDC is built into PostgreSQL via the WAL (write ahead log AKA transaction log) which PostgreSQL publishes to via its native wire protocol which it uses for replication so you know its going to be fast and efficient.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/953/0*_te50kBfraASGz_P.png" /></figure><p>The one beef I have with Airbyte is in the sheer number of IP addresses they make you put in your white list for accessing Lassoo/PostgreSQL. 5 or 6 is understandable, 8 and up and its too much. Figure out your networking Airbyte! However I do give them props for supporting SSH, this is a nice-to-have feature that not ETL tools support. Currently Lassoo does not support SSH for ETL but it is in the works.</p><ol><li><strong>Configure Your Source Settings</strong>: Choose the tables or schemas you want to replicate from Lassoo to Snowflake.</li></ol><h3>Step 3: Configuring Snowflake as the Destination</h3><p>Next, set up Snowflake within Airbyte:</p><ol><li><strong>Choose Snowflake as the Destination</strong>: In Airbyte, select Snowflake from the destination options.</li><li><strong>Enter Your Snowflake Credentials</strong>: Provide details like account identifier, warehouse, database, schema, username, and password.</li><li><strong>Customize Your Data Sync</strong>: Airbyte allows you to customize replication frequency and other settings to align with your data strategy.</li></ol><h3>Step 4: Initiating the Data Sync</h3><p>With both source and destination configured, start the replication process. Airbyte will begin transferring data from Lassoo to Snowflake, making it ready for advanced analytics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zGgh4OfFL9x3bdnV.png" /></figure><p>I can now see my core tables <strong>person_</strong> and <strong>session_</strong> and all my event tables, event_page, event_track, event_ecommerce etc..</p><h3>Benefits of This Integration</h3><ul><li><strong>Real-Time Insights</strong>: Analyze customer behavior in real-time with Lassoo’s data now in Snowflake.</li><li><strong>Flexibility and Customization</strong>: Airbyte’s open-source nature allows for extensive customization and flexibility in your data integration processes.</li><li><strong>Scalability</strong>: As your data grows, this integration scales with you, accommodating increasing data volumes effortlessly.</li><li><strong>Enhanced Data Analysis</strong>: Combine Lassoo’s detailed behavioral data with Snowflake’s powerful analytics capabilities for deeper insights. \</li><li><strong>Power your Composable CDP</strong>: Leverage your data for marketing automation.</li><li><strong>Power ML models</strong> : Lassoo’s simple schema with built in features and classification makes training machine learning models that much easier.</li></ul><h3>Conclusion</h3><p>Integrating Lassoo with Snowflake via Airbyte represents a significant advancement in data integration and analytics. This combination empowers businesses to leverage their behavioral data fully and in real-time enabling more informed decisions and strategies. This data also powers downstream systems which consume data from Snowflake.</p><p>Stay tuned as we continue to explore more integrations with leading data tools.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=50a0748a334a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Analytics Awakening: Breaking free of BigQuery and Embracing Elegance in Behavioral Data]]></title>
            <link>https://medium.com/@mkremer_75412/the-analytics-awakening-4b7b20e57711?source=rss-10737e34fb8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b7b20e57711</guid>
            <category><![CDATA[google-analytics]]></category>
            <category><![CDATA[behavioral-data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[bigquery]]></category>
            <dc:creator><![CDATA[Max Kremer]]></dc:creator>
            <pubDate>Fri, 01 Dec 2023 22:18:37 GMT</pubDate>
            <atom:updated>2023-12-01T22:22:51.201Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/0*W4kcH71vqYA5TW1z" /></figure><p>Picture this: you’re tasked with developing an analytics tool for capturing user behavior. The big question pops up: How would you architect the data schema? If you’re thinking, “Well, it depends on how the data’s gonna be used,” you’re spot on. A set of specific reports/metrics ad-hoc queries, etc? This would definitely affect the design choices. What if I said the design intent is for ad-hoc querying using a language with wide adoption. Why reinvent the wheel, SQL is still often the best tool for the job when doing any kind of data analysis. It’s terse, simple to read, easy to grasp syntax without funky language constructs. So, what does an intuitively simple schema for web or, let’s go a bit broader, behavioral analytics look like?</p><p>It would probably look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xrGoEmCMpktVRWb1" /></figure><p>It would be helpful to introduce the artificial concept of a “visit” or session. They don’t exist, not really, the way <strong>humans</strong> and <strong>events</strong> do but they’re a useful artificial construct because it captures a lot about a particular long-lived interaction with our user. Sessions are useful for measuring “intent” or telling a more relatable, qualitative story than replying on granular events alone. Sessions also are important when considering our use of multiple devices (it’s not unusual for customers to have multiple, active sessions with interleaved events).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ktxqcqS5IoisMcZN" /></figure><p>Such a simple schema would be the most useful because relational data is easy to reason about. A lot of what a company is interested in are the humans not the events, sessions, etc… Those are just a means to an end so we can better measure how/what we’re doing and its effect on our humans (customers).</p><p>Now, I’m going to throw you a curveball — the schema must support multiple tenants. And not just a few or a few thousand but millions. That alone changes everything, now the schema has to handle millions of data points across millions of tenants. Will that force you to rethink how you store and manage data? Of course. Now, what if I say we also have to honor legacy data model contracts (for example the way GA must) . Well now you’d tear up your design and say ok show me ALL the requirements.</p><p>But, from the POV of an analyst or data engineer, they want to work with the simplest, most naive, most intuitive model of the data. Humans and events (and sessions).</p><p>Many of us who rely on GA behavioral data to power BI, ML models, dashboards, CRM, CDPs and more are forced to reckon with its’ quirks and intricacies and its departure from our perfect relational model. It’s no wonder that the GA schema and BigQuery itself exists in the form it is in now due to Google’s complex requirements of managing a free analytics tool for the internet. Google Analytics must handle vast amounts of data generated by millions of websites. The schema is optimized for scalability, capable of storing and processing huge volumes of event data efficiently. BigQuery’s nested and repeated fields are ideal for this because they allow for a more compact data representation and can reduce the number of I/O operations required for reading large datasets. Indeed BigQuery is a marvel of scalability, but would your company benefit from this level of scale? Unlikely. Even at over 100k visits a day it wouldn’t generate THAT much data (less than a TB). Few e-commerce brands generate the amount of data necessary to justify that level of scale. So you end up paying for scale you don’t need with complexity you don’t want.</p><p>In the data world, simplicity is the ultimate sophistication. Unfortunately, when it comes to querying Google Analytics data in BigQuery, simplicity often goes out the window. What should be straightforward insights require a labyrinth of specialized SQL extension clauses, making one pine for the clarity of traditional relational databases. Let’s unravel some examples that demonstrate how something as basic as understanding user behavior becomes a Herculean task in the GA on BigQuery environment.</p><p>Consider a simple mission: counting users visiting a specific page. With a traditional relational schema:</p><pre>SELECT COUNT(DISTINCT user_id) <br>FROM page_visits <br>WHERE page_path = &#39;/home&#39;;</pre><p>Clear and concise, mission accomplished. Now, let’s turn to BigQuery’s GA schema. The equivalent information is buried within arrays of nested records:</p><pre>SELECT COUNT(DISTINCT user_pseudo_id) <br>FROM `bigquery-public-data.ga_sessions_20210101`, <br>UNNEST(hits) AS hit <br>WHERE hit.page.pagePath = &#39;/home&#39;;</pre><p>Notice the need to UNNEST hits — because user interactions are encapsulated in nested structures, a simple count becomes a multi-step process.</p><p>This is where the love affair with window functions and arrays takes a toll on your SQL purity.</p><h3>The Cost of Complexity</h3><p>BigQuery’s pricing model is based on the amount of data crunched. Every UNNEST, every subquery, and every analytical function adds to the bill. The irony is palpable — you pay more for the complexity you never asked for. Let’s look at a few more examples to illustrate.</p><h4>Funnel Analysis in BigQuery with GA Data</h4><p>A funnel analysis tracks the series of steps that users take, often leading towards a goal, like making a purchase. In GA data within BigQuery, each step might correspond to a different event, and each event can have multiple parameters.</p><p>Let’s consider a simplified funnel with three steps:</p><ul><li>User visits a product page (view_item event).</li><li>User adds the product to the cart (add_to_cart event).</li><li>User completes the purchase (purchase event).</li></ul><p>BigQuery SQL for Funnel Analysis:</p><pre>WITH steps AS (<br> SELECT<br> user_pseudo_id,<br> event_name,<br> event_timestamp,<br> -- Use UNNEST to flatten the event_params and extract the product_id for each event<br> (SELECT value.int_value FROM UNNEST(event_params) WHERE key = &#39;product_id&#39;) AS product_id<br> FROM `your_project.your_dataset.ga_sessions_*`,<br> UNNEST(events) AS event<br> WHERE event_name IN (&#39;view_item&#39;, &#39;add_to_cart&#39;, &#39;purchase&#39;)<br> AND _TABLE_SUFFIX BETWEEN &#39;20210101&#39; AND &#39;20210131&#39;<br>)<br><br>, funnel AS (<br> SELECT<br> user_pseudo_id,<br> -- Aggregate steps for each user to track their path through the funnel<br> ARRAY_AGG(STRUCT(event_name, event_timestamp) ORDER BY event_timestamp) AS user_funnel<br> FROM steps<br> GROUP BY user_pseudo_id<br>)<br><br>SELECT<br> user_pseudo_id,<br> user_funnel,<br> -- Check the sequence of events for each user to confirm if they completed the funnel<br> IF(<br> ARRAY_LENGTH(user_funnel) = 3<br> AND user_funnel[OFFSET(0)].event_name = &#39;view_item&#39;<br> AND user_funnel[OFFSET(1)].event_name = &#39;add_to_cart&#39;<br> AND user_funnel[OFFSET(2)].event_name = &#39;purchase&#39;,<br> &#39;Completed Funnel&#39;,<br> &#39;Did Not Complete Funnel&#39;<br> ) AS funnel_status<br>FROM  funnel<br><br></pre><p>In this example, the WITH clause is used to create a Common Table Expression (CTE) that first identifies the relevant events for each user and flattens the event_params to extract a product_id. Then, it aggregates these events into an array ordered by event_timestamp to understand the sequence of steps each user has taken.</p><p>The final SELECT statement checks the sequence of events for each user to determine if they completed the funnel. This check relies on the aggregated array of events, which is made possible by the nested structure of GA data in BigQuery.</p><p>Nested structures enable the representation of complex, multi-step interactions within a user’s session, which would be much more challenging to model in a flat relational table without extensive join operations or additional processing. The power of BigQuery’s nested structures shines in analyses like this, where the relationship and order of events are crucial to the analysis.</p><p>Let’s look at one more example: calculating average session duration:</p><p>In BigQuery, you might find yourself writing a query like:</p><pre>WITH sessions AS (<br> SELECT<br> user_pseudo_id,<br> TIMESTAMP_MICROS(event_timestamp) as event_time,<br> LEAD(TIMESTAMP_MICROS(event_timestamp)) OVER (PARTITION BY user_pseudo_id ORDER BY event_timestamp) as next_event_time<br> FROM `your_project.your_dataset.ga_sessions_*`<br> WHERE event_name = &#39;session_start&#39;<br>)<br>SELECT<br> user_pseudo_id,<br> AVG(TIMESTAMP_DIFF(next_event_time, event_time, SECOND)) as avg_session_length<br>FROM sessions<br>GROUP BY user_pseudo_id</pre><p>Whereas with our ideal relational schema it may look something like this:</p><pre>SELECT person_id, AVG(session_duration) as avg_session_length<br>FROM session<br>GROUP BY person_id</pre><p>So, what would an ideal schema look like? At its foundation it should be user-friendly and accurately model the real-world while also offering built-in features akin to materialized views so that data that is often required is already rolled up on the entity where I need it. For example, don’t make me join the event table every time I need to figure out if a person is a customer or not, just give me a boolean <strong>is_customer</strong> boolean on the person table, better yet, give me a <strong>total_spent </strong>column. Imagine having key metrics like total spent, days since last purchase, or customer status directly accessible within your user table — this is the kind of customer-centric, efficient design that modern data teams need. In the new world of data pipelines and warehouses, analytics should be built with the analyst in mind. Today’s analysts have become data wranglers. Artificial complexity we never asked for compiles the time needed to get basic data analysis/data science work done.</p><h3>How organizations deal with GA data when importing from BigQuery to their data warehouse</h3><p>When organizations import Google Analytics (GA) data from BigQuery into their data warehouse, they generally follow one of these strategies:</p><h4>1. Flatten the Data During ETL:</h4><p>Most organizations choose to flatten the data during the Extract, Transform, Load (ETL) process. This means they will extract the nested and repeated fields from BigQuery and transform them into a tabular format that fits a relational model. This process often involves:</p><ul><li>Extracting nested fields: Using UNNEST() to turn nested fields into separate rows.</li><li>De-duplicating: Ensuring that the flattening process doesn’t introduce duplicate rows.</li><li>Creating new tables: Making new tables in the data warehouse to hold data that was nested in BigQuery.</li></ul><p>Flattening the data simplifies querying and allows the use of standard SQL without complex UNNEST operations. It also makes it easier to use the data with various BI tools that may not support nested schemas. This is by far my favorite option</p><h4>2. Import Nested Data As-Is:</h4><p>Some organizations may choose to import the GA data into their data warehouse without flattening, keeping the nested and repeated fields. This strategy might be used when:</p><p>The data warehouse supports nested structures: If the target data warehouse (like Snowflake or another instance of BigQuery) supports similar data types, the data might be imported as-is to maintain the structure. But then you have to support Google’s data structure in all of your queries.</p><p>Minimal transformation requirement: If the organization wants to minimize ETL complexity and processing time, they might opt to keep the data in its original format. Generally this seems like a bad idea as you’ll have to pay for this complexity down the road and may infect a plethora of downstream systems.</p><h4>3. Hybrid Approach:</h4><p>In some cases, organizations might adopt a hybrid strategy, where they:</p><p>Flatten critical data: Flatten only the most frequently accessed data or the data needed for specific high-priority analyses.</p><p>Keep detailed data nested: Retain the detail of less frequently accessed data in a nested format to save on processing time and costs.</p><h4>4. Custom Transformations for Specific Use Cases:</h4><p>Organizations may also perform custom transformations tailored to their specific analytical needs. This can involve:</p><p>Creating summary tables: For high-level reporting and dashboards, where detailed event-level data isn’t necessary.</p><p>Materialized views: For databases that support them, creating materialized views to provide a flattened view of the data without physically restructuring the underlying data. Of course these need to be maintained or “refreshed” to stay current.</p><p>In all of the above cases you’re maintaining system configs, code or a combination of both to deal with GA BigQuery complexity.</p><p>In the quest for data-driven insights, the elegance and sophistication lie in simplicity. The GA schema in BigQuery, necessitated by Google’s vast scale needs, often overshoots this mark for most businesses. It’s a data Goliath, most businesses seek a David — nimble, swift, and effective. The vast majority of companies, even with high web traffic, will find more value in a schema that prioritizes ease of use, simplicity in querying, and aligns with the natural structure of human interactions and events. A well-designed relational model, like the one you’d find in traditional SQL databases, respects these principles. It empowers analysts and data engineers to focus on deriving insights rather than unraveling the complexities of their tools. As we advance into the era of big data, let’s not forget the power of simplicity and the value of intuitive data models. These were the two driving forces behind our Postgres-first “headless analytics” solution. We started with the goal of delivering a user friendly schema that made sense and was easy to work with. Then we added the classification, features and binning to the data to make 80% of data use cases that much easier. When it comes to data, time-to-insight and time-to-action have a direct effect on the bottom line. I invite you to take a look at Lassoo if your business is craving 1st party behavioral data that’s simple to work with.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b7b20e57711" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unraveling the True Cost of Data Pipelines]]></title>
            <link>https://medium.com/@mkremer_75412/unraveling-the-true-cost-of-data-pipelines-7d853f570ddb?source=rss-10737e34fb8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/7d853f570ddb</guid>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Max Kremer]]></dc:creator>
            <pubDate>Mon, 27 Nov 2023 15:45:49 GMT</pubDate>
            <atom:updated>2023-11-27T15:45:49.663Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Deep Dive with Zapatería La Bailarina</h4><h3>Setting the Stage</h3><p>Welcome to the thrilling world of Zapatería La Bailarina, where high heels meet high data!</p><p>Zapatería La Bailarina, an emerging e-commerce brand in the shoe industry, is navigating the complexities of leveraging data-driven strategies in a competitive market. Establishing an efficient data pipeline, crucial for synthesizing e-commerce transactions, customer interactions, and social media insights, involves intricate complexities. In the bustling e-commerce bazaar, Zapatería La Bailarina is dancing its way through the tango of data-driven strategies. But let’s not sugarcoat it — setting up an efficient data pipeline is like preparing for a grand ball. It’s exciting, essential, and yes, expensive.</p><p>This exploration delves into what these challenges mean for Zapatería La Bailarina (let’s call them ZB for short). Let’s peel back the curtain and see what this dance really costs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*el3L5CQ2OOS0Aarl" /></figure><h3>Tackling Complexity Head-On</h3><p>ZB’s data pipeline is built to manage diverse data sources:</p><p><strong>Data Warehousing: </strong>Opting for a solution like Snowflake, ZB incurs costs based on storage and computing needs, averaging around $3,000 to $5,000 per month.</p><p><strong>ETL Tools: </strong>Integrating data from their e-commerce platform, customer interactions, and social media analytics with Fivetran adds about $2,000 to $4,000 monthly.</p><h3>Operational Costs: Navigating Complexity</h3><p>The operational costs for ZB’s pipeline are influenced by several factors:</p><p><strong>Maintenance and Monitoring: </strong>A small team of data engineers is necessary for pipeline management, adding roughly $10,000 to $15,000 per month.</p><p><strong>Data Integration Tools:</strong> ZB uses tools like Informatica for complex data mappings and Stitch for simpler integrations. This mix adds an additional $2,000 to $3,000 to their monthly operational budget.</p><h3>Understanding the Complexity of Data Tool Pricing</h3><p>The pricing for ZB’s data tools includes multiple facets:</p><p><strong>Monthly Active Rows (MAR): </strong>Pricing depends on the number of rows processed each month, fluctuating with seasonal data volume.</p><p><strong>Computation and Transformation Costs: </strong>These costs arise from processing and transforming data from various sources like online sales, customer feedback, and marketing campaigns.</p><p><strong>Loading and Storage Fees: </strong>The amount and duration of data storage also impact costs.</p><p><strong>Payment Options:</strong> ZB weighs between On-Demand (month-to-month) payments for flexibility and Pre-Purchased plans for cost savings but less scalability.</p><h3>The Impact of Speed and Time-to-Use</h3><p>Speed in data processing significantly benefits ZB. As Gartner states, “Faster data leads to 2x better marketing outcomes.” This translates to:</p><p><strong>Enhanced Customer Experiences:</strong> Quick analysis of customer behavior enables personalized interactions.</p><p><strong>Agile Decision Making:</strong> Rapid data processing allows ZB to swiftly adapt to market trends.</p><p><strong>Optimized Marketing Spend: </strong>Real-time insights facilitate immediate adjustments in marketing strategies.</p><p>Combining these factors, the bottom line for ZB’s monthly data costs ranges from approximately $18,000 to $29,000 per month. This total encompasses warehousing, ETL processing, maintenance, integration tools, and other operational expenses like paying the data team’s salary.</p><h3>Conclusion: Weighing Costs Against Benefits</h3><p>Despite significant costs, the investment in a sophisticated data pipeline is justified for ZB by the enhanced marketing effectiveness and operational agility it brings. Understanding the intricacies of data tool pricing, including payment options, and the value of fast data processing is key to making sure ZB stays en vogue in the data-driven world.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7d853f570ddb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decoding the Difference: Headless Analytics Versus Headless BI]]></title>
            <link>https://medium.com/@mkremer_75412/decoding-the-difference-headless-analytics-versus-headless-bi-da554cedc28e?source=rss-10737e34fb8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/da554cedc28e</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Max Kremer]]></dc:creator>
            <pubDate>Thu, 23 Nov 2023 19:00:17 GMT</pubDate>
            <atom:updated>2023-11-23T19:02:42.030Z</atom:updated>
            <content:encoded><![CDATA[<p>The other day I got a message from a colleague in response to a Lassoo launch post. He said “headless analytics is the same as headless BI, prove me wrong”. While there are some similarities there is also a world of difference and I felt it was a good opportunity to describe the differences in a post. So what are the differences between headless BI and headless analytics?</p><p>While both share a common philosophical foundation (i.e. “headless“), they are applied in slightly different contexts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zjiab-93Txg9q0Fv.jpg" /></figure><h3>Headless BI</h3><p><strong>Focus</strong>: Primarily centered around business intelligence, which often involves aggregating, analyzing, and presenting data to inform business decisions. It typically deals with historical data.</p><p><strong>Architecture</strong>: Separates the data processing and storage aspects from the presentation and visualization layer. This allows businesses to use any front-end tool or custom application for data visualization. <a href="https://www.gooddata.com/headless-bi/">Headless BI</a> typically has a “semantic layer” to unify data from multiple sources and provides an abstraction over complex and disparate data sources.</p><p><strong>User Base:</strong> Often used by business analysts and decision-makers who need comprehensive reports and dashboards to make informed decisions.</p><p><strong>Customization and Flexibility:</strong> Provides a high degree of flexibility in how data is presented and reported, allowing for bespoke solutions.</p><h3>Lassoo’s Headless Analytics</h3><p><strong>Focus</strong>: Concentrates on collecting and processing behavioral data, especially customer interactions, and making it accessible for various uses, including but not limited to BI.</p><p><strong>Architecture</strong>: Similar to Headless BI, Lassoo’s approach decouples data collection (backend) from data utilization (front-end). It emphasizes direct SQL access to a well-structured database, enabling use with any analytical tool and/or easy transport into any data warehouse.</p><p><strong>User Base</strong>: Targeted more towards data teams, including data scientists, data engineers, and analysts who need granular data for diverse applications ranging from AI/ML modeling to detailed customer behavior analysis and activation.</p><p><strong>Broad Application:</strong> While Headless BI primarily focuses on driving business decisions through aggregated data insights, the realm of Headless Analytics, as championed by Lassoo, stretches further into dynamic and complex data science domains. This includes real-time data processing and analytics, which enable businesses to react instantly to user interactions and trends. Additionally, Lassoo facilitates advanced predictive modeling, allowing for sophisticated forecasting and trend analysis based on user behavior. The platform also supports intricate data science tasks such as segmentation analysis, churn prediction, customer lifetime value estimation, and complex event processing. These tasks involve deep analysis of large datasets to uncover patterns, correlations, and insights that can drive highly targeted, data-driven strategies. In essence, Lassoo’s Headless Analytics equips data teams with the tools to delve into more nuanced and sophisticated challenges, transforming raw behavioral data into action. We use PostgreSQL as our data layer to maximize compatibility with the data ecosystem; ETL (Fivetran, Hevo, etc…), warehouses (Snowflake, Redshift, Databricks), BI &amp; Dashboards(Tableau, etc), modeling tools (Python, etc…)</p><p>In summary, while both Headless BI and Lassoo’s Headless Analytics advocate for a decoupled, flexible approach to data handling, they differ in their primary focus and application. Headless BI is about empowering business intelligence with more flexibility in data presentation, whereas Lassoo’s Headless Analytics provides a robust, flexible foundation for a wide range of data-driven use-cases, including but not limited to BI, with a strong emphasis on real-time, granular behavioral data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da554cedc28e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Taking the Reins on Customer Data: The Imperative of First-Party Behavioral Analytics]]></title>
            <link>https://medium.com/@mkremer_75412/taking-the-reins-on-customer-data-the-imperative-of-first-party-behavioral-analytics-d6dcb9f066e7?source=rss-10737e34fb8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6dcb9f066e7</guid>
            <category><![CDATA[digital-marketing]]></category>
            <category><![CDATA[cto]]></category>
            <category><![CDATA[customer-data-platform]]></category>
            <category><![CDATA[data-management]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Max Kremer]]></dc:creator>
            <pubDate>Thu, 23 Nov 2023 18:37:40 GMT</pubDate>
            <atom:updated>2023-11-23T19:03:49.068Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In the realm of data-driven decision-making, owning your customer behavioral data isn’t just an advantage; it’s essential. For CxOs, VPs, and Directors steering their organization’s data journey, relying on third-party tools for customer insights is no longer sufficient. This post explores why first-party control over behavioral data is crucial and how it empowers your organization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*-z-RIXKO6MHLU1rE.png" /></figure><h3>The Pitfalls of the Traditional Approach</h3><p>Traditionally, businesses have relied on a mix of third-party tools and first-party systems, leading to a ‘data exhaust’ approach. This method is fraught with challenges:</p><p>Complexity and Errors: Juggling multiple data sources leads to complexity and increases the likelihood of errors.<br>Resource-Intensive Data Wrangling: Vast amounts of time and resources are consumed in data wrangling, distracting from more strategic tasks.<br>Delayed Insights: Without real-time data, businesses can lag in timely decision-making.</p><h3>The New Way: Capturing Your Own First-Party Data</h3><p>Shifting to first-party behavioral analytics brings transformative change:</p><p>Real-Time, Structured, and Clean Data: Direct data capture means data is not only available faster but is also of higher quality and more reliable.<br>Total Ownership and Control: Full control over the data stream allows for better compliance with regulations and governance policies.<br>Empowering Data Teams: With first-party data, teams spend less time wrangling and more time on analysis and building AI/ML models.</p><p>Real-World Benefits: Supercharging Activation and Personalization Tools<br>Consider an e-commerce company transitioning to first-party data capture. The marketing team can now trigger real-time campaigns based on the latest customer behaviors. This higher fidelity data supercharges other activation and personalization tools. Personalization engines become more adept at tailoring user experiences, marketing automation tools segment and target audiences with greater accuracy, and CRM systems provide richer insights.</p><p>The data team, freed from data cleaning, now focuses on predictive models to forecast trends. Their work enhances the AI/ML models, leading to more accurate predictions and smarter business strategies. Every tool in the tech stack becomes more powerful, fueled by the best quality data.</p><h3>Conclusion: A Strategic Imperative</h3><p>For leaders in data technology, embracing first-party behavioral analytics is a strategic imperative. This approach not only streamlines and economizes your data infrastructure but also unlocks unprecedented levels of insight, responsiveness, and operational efficiency. It’s about breaking away from the constraints of traditional data collection methods and seizing full control of your data narrative. In doing so, you’re not just enhancing your customer engagement strategies; you’re transforming them into dynamic, data-driven interactions that resonate with every customer touchpoint.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6dcb9f066e7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Headless Analytics is the Game-Changer We’ve Been Waiting For]]></title>
            <link>https://medium.com/@mkremer_75412/why-headless-analytics-is-the-game-changer-weve-been-waiting-for-62e49a89b5b9?source=rss-10737e34fb8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/62e49a89b5b9</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <dc:creator><![CDATA[Max Kremer]]></dc:creator>
            <pubDate>Tue, 08 Aug 2023 00:51:08 GMT</pubDate>
            <atom:updated>2023-08-08T18:30:05.487Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1XXkQ3BrbgfyWAQr.png" /></figure><h3>Headless Analytics: Why Now?</h3><p>In the heart of Silicon Valley, nestled among start-ups and tech giants, Sarah, a seasoned data scientist, found herself spending the majority of her days wrangling data. It was like asking a virtuoso to tune pianos all day, every day. But Sarah’s story isn’t unique; it’s a narrative echoed in countless cubicles, open spaces, and home offices worldwide. What happens when industries evolve, but the tools lag behind?</p><h3>The State of Today’s Tech Landscape</h3><p>Remember the early days of the internet, when dial-up modems sang their robotic lullabies? If we trace back to the late 90s, the realm of data seemed simpler. We were just starting to grasp its potential. But as we moved into the 21st century, our ambition grew. Snowflake, Redshift, BigQuery became the giants in warehousing. Cloud advancements, particularly in ETL and data-pipelines, made it possible to do more than we ever dreamed. We weren’t just looking at data; we started predicting the future with machine learning, modeling, and even artificial intelligence. Our dreamscape had become a galaxy of possibilities, and we were shooting for the stars.</p><h3>The Evolution of Data Teams and Their Tools</h3><p>Our data teams evolved; they no longer sat in dimly lit rooms with green monochrome screens. Today’s data teams produce artifacts consumed by business people in tools like Tableau, PowerBI, and Looker. But they’re not just creating reports; they’re training machine learning models, predicting market trends, and influencing boardroom decisions. And while these tools offered beauty in data visualization, they often seemed shackled to outdated ETL processes — cumbersome systems that, at best, acted as sticky tape holding the intricate machine together.</p><h3>The Rise of Headless Systems</h3><p>In recent years, the concept of “headless” has taken root in several technological domains, including most recently <a href="https://www.google.com/search?q=headless+ecommerce">ecommerce</a> , symbolizing a profound shift in how we approach software design. At its core, going “headless” means breaking away from traditional monolithic systems, allowing components to function independently of one another. Why give you a UI if all you need is an API? The headless CMS, for example, emerged as a response to the evolving nature of content delivery. No longer confined to websites alone, content began to find its place in mobile apps, IoT devices, and beyond. Headless CMS let content be content. These successes underscore a core benefit of the headless approach: By focusing on doing one thing exceptionally well, systems can offer far greater efficiency, performance, and adaptability.</p><h3>Why Headless Analytics Makes Sense Now</h3><p>As AI modeling and machine learning take center stage, the demand for rich, curated, and clean data is higher than ever. But amidst this, do we really want our brightest minds mired in mundane data cleansing? SQL, with its universal appeal, emerges as the bridge between data and its myriad applications. And this is where headless analytics fits in — focusing on efficient data ingestion, curation, reconciliation, and enrichment, and letting visualization tools do what they do best. By decoupling data gathering from processing, from warehousing, from visualization, we create a playground for specialists. One where the focus is on refining, reconciling, and enriching data. A world where data is not just seen, but also felt, trained, and integrated using a myriad of tools.</p><h3>In Conclusion</h3><p>The world stands at an inflection point. Just as Malcolm Gladwell’s ‘tipping point’ phenomena suggest, there comes a time when a series of small changes become significant enough to create a larger, more important change. Headless analytics isn’t just another buzzword; it’s the future. It’s the culmination of decades of tech evolution, a response to our changing needs, and most importantly, it’s the recognition that our data scientists, like Sarah, deserve better.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=62e49a89b5b9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Do we really need data pipelines?]]></title>
            <link>https://medium.com/@mkremer_75412/do-we-really-need-data-pipelines-d0921b2e9b90?source=rss-10737e34fb8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0921b2e9b90</guid>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Max Kremer]]></dc:creator>
            <pubDate>Thu, 27 Jul 2023 21:50:33 GMT</pubDate>
            <atom:updated>2023-07-29T02:09:53.426Z</atom:updated>
            <content:encoded><![CDATA[<p>This past year yours truly volunteered to teach at a high school in Costa Rica. The subject was of course computer science, but more simply “the basics of programming”. Now, let me tell ya, engaging these young minds is like trying to tame a flock of wild flamingos with marshmallows. It’s an adventure, I’ll give you that!</p><p>Now, I assure you, my friends it was a learning experience for me, I’d never taught in a class setting before, nor had I worked with grade 9s before. BTW for all the teachers out there, much respect! Putting together a curriculum and trying to get students to actually learn is hard work. You are all grossly underpaid. Anyway, back to my programming students, their engagement was lackluster from the beginning and it wasn’t till I stepped back and thought for a moment: what I’m missing. Then, it hit me like a freight train, that in the midst of my pedagogical endeavors I had taken a core concept for granted: abstraction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*-tDaqXavEyqi3pEq" /></figure><p>As programmers we live in the abstract, we operate in the realm of objects and functions, variables and I/O. Being able to think this way allows us to assemble multiple moving parts into solutions without worrying too much about the parts themselves and their internals. So, my amigos, I set out explaining the concept of abstraction to my 9ers and how these things: variables, objects, functions were abstractions and we made their implementation concrete through code. I then had them design these Rube-Goldberg style machines to perform a given task like taking a pineapple and outputting edible bits. The students realized they needed to put these abstract notions together like a peeler, a cutter, a mover of things etc… and sketch out <em>implementations</em> and assemble them on paper. After this exercise we went back to the world of coding and lo and behold things began to click!</p><p>Abstraction is so powerful a concept that it permeates our language and almost everything we set out to build and do. We take it for granted because it’s hard wired into our human brains, we rarely give it a second thought. But sometimes getting stuck with one abstraction or another can be a limiting trap.</p><p>This, my tech veterans, finally brings me to the idea of pipelines. Data pipelines are all the rage right now. ETL, Kafka, queues, transformations. In the connected enterprise where data is produced at astonishing rates, pipelines play an important role in turning all that data into useful information and ultimately into insights and actions.</p><p>The notion of a pipeline is a natural one when we want to move data from <em>here</em> to go <em>there</em>. Verbs like flow and ingest make sense in this context. These abstractions help us visualize and build a mental model. We can even borrow language from physical pipelines like “flow-control” and “<a href="https://medium.com/@beeindian04/back-pressure-in-data-pipeline-bdc25c6c1d79">backpressure</a>” .</p><p>The problem with pipelines, both in the physical world and the data world is complexity. Pipelines need operational controls, orchestration and management. These things are not trivial especially when talking about data at scale and the demands of modern data-science applications. Also consider that modern data pipelines are increasingly pushing data out for actioning in other systems, called “reverseETL” (hate that term) . Data pipelines don’t just feed records into lakes and warehouses anymore.</p><p>Really it’s just a matter of getting data from <em>here to there.</em> What if we could do that and not be so concerned with the plumbing (plumbing — it’s so hard to escape the “pipelines” abstraction). It could be that I’m descending into a pit of meaningless nihilism and oversimplifying the problem. After all, half of all IT problems revolve around getting data in and out, we call it I/O. But I can’t help thinking that the pipeline abstraction over data confines us. It begs the question, if not pipelines, what is a better paradigm?</p><p>Think for a moment about your favorite streaming service (mine used to be Neflix but lately I’m all about the Disney+). You’re watching a movie and hundreds if not thousands of devices play a role in bringing you that stream. Netflix is pulling it out of some cache in their infrastructure (maybe disk if you found something deep within the unwatched vault). Then it makes its way, via the internet and some of its nodes to your ISP who then sends it to your home router and finally to your TV. All of this is made possible through protocols, agreements on how to send and receive data. Contracts between devices and operating systems. None of it do we have to orchestrate ourselves, it all just works.</p><p>To my fellow compatriots over the age of 45, those who remember the first PCs. You may recall that to run a game it took a few steps, you had to grab a floppy disks or two, insert them, run a few commands to load the disk, then run the game exe (I tried to search for an example command sequence but alas came up empty. You can watch someone do it here<a href="https://www.youtube.com/watch?v=e6BXzBF0_uc"> https://www.youtube.com/watch?v=e6BXzBF0_uc</a> yes, obsolete PCs are a thing now and they go for big money on eBay). Today you just click a button, we came up with various new abstractions to make life easier, a graphical user interface to name one. And so, dear tech denizens, it begs the question, can we use a better paradigm than pipes and plumbing?</p><p>As I alluded to in my <a href="https://medium.com/@mkremer_75412/beyond-databases-postgres-as-the-ultimate-data-protocol-164320ce3463">previous article</a>, I believe that it is the network+protocol paradigm which can replace the data pipeline and be our salvation. Ultimately it’s better suited to what we are trying to achieve, which is, at the end of the day, the sharing of information between systems and people. The nodes of our big-data network are the engines that do the work, the computation, the enrichment, the transformation. With one or more protocols in place to standardize communication, we have more efficient and seamless data movement and processing, thereby reducing the burden of managing complex pipelines.</p><p>Now I know, dear reader, that the devil is in the details, and a communication protocol for big-data-systems is no easy feat, let alone the adoption. But I do believe that by looking beyond traditional data pipelines and exploring alternative paradigms, data engineers and scientists can open up possibilities for simplification, automation, and enhanced data-driven applications, all at lower cost. And so my compatriots, I urge you to embrace the spirit of open-mindedness, challenge established abstractions and seek a path less traveled where possibility knows no bounds.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0921b2e9b90" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Crafting Connections: Building a Composable CDP]]></title>
            <link>https://medium.com/@mkremer_75412/crafting-connections-building-a-composable-cdp-840a568e6301?source=rss-10737e34fb8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/840a568e6301</guid>
            <category><![CDATA[customer-data-platform]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[composable-architecture]]></category>
            <dc:creator><![CDATA[Max Kremer]]></dc:creator>
            <pubDate>Fri, 21 Jul 2023 00:06:36 GMT</pubDate>
            <atom:updated>2023-07-21T00:06:36.722Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In the dynamic realm of customer data management, organizations seek ways to harness the power of data for personalized experiences. Today, we embark on a journey to explore the construction of a composable Customer Data Platform (CDP) solution using Lassoo.io as the source of customer behavioral data. By embracing the composable approach and leveraging Lassoo.io’s capabilities, organizations can unlock valuable insights and orchestrate exceptional customer experiences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FiijUYKPSvC-ZGug" /></figure><h3>Understanding Composable CDPs</h3><p>Composable CDPs represent a paradigm shift in customer data management, offering organizations a flexible and modular framework to integrate various tools and data sources. With the ability to assemble custom ecosystems, composable CDPs enable a comprehensive view of customer data for informed decision-making.</p><h3>Lassoo.io: Unleashing the Potential of Customer Behavioral Data:</h3><p>Lassoo.io emerges as a powerful platform for capturing and harnessing customer behavioral data. Acting as a central hub, Lassoo.io allows organizations to collect and unify data from diverse touchpoints, such as websites, mobile apps, and server-side interactions. Its capabilities align perfectly with our quest to build a composable CDP solution.</p><h3>Constructing the Composable CDP Solution</h3><p>a. Framework Design: Begin by designing a flexible framework that accommodates the integration of Lassoo.io. This framework should facilitate seamless connectivity between Lassoo.io and other components, ensuring a cohesive data management ecosystem.</p><p>b. Data Integration and Transformation: Integrate Lassoo.io’s behavioral data with other relevant data sources within the composable CDP solution. Employ Lassoo.io’s capabilities to standardize, cleanse, and enhance the data, ensuring consistency and quality throughout the ecosystem.</p><p>c. Tool Integration: Embrace the modular nature of composable CDPs, allowing for the integration of various tools that complement Lassoo.io’s capabilities. Select tools for customer segmentation, predictive analytics, marketing automation, and other functionalities based on your specific requirements. Integrate these tools into the composable CDP solution to amplify the power of Lassoo.io’s behavioral insights.</p><p>4. Activating Insights and Personalization:</p><p>With our composable CDP solution taking shape, we can leverage the consolidated and enhanced data to activate powerful insights. Utilize integrated tools to segment customers, derive actionable analytics, and deliver personalized experiences. Lassoo.io acts as a conduit for empowering data-driven decision-making and customer-centric strategies.</p><p>5. Continuous Optimization:</p><p>Building a composable CDP solution is an iterative process. Continuously analyze performance metrics, refine segmentation strategies, and optimize campaigns based on the insights derived from Lassoo.io and the integrated tools. Embrace a culture of continuous improvement to ensure the solution remains effective and adaptable to evolving customer needs.</p><p>By crafting a composable CDP solution using Lassoo.io as the source of customer behavioral data, organizations can unlock the true potential of their customer insights. Lassoo.io’s capabilities empower organizations to capture, unify, and leverage customer data from various touchpoints. Embracing the composable approach enables personalized experiences, data-driven decision-making, and a deeper understanding of customers. Let us embark on this journey, leveraging Lassoo.io’s capabilities to weave connections, orchestrate exceptional experiences, and create lasting customer relationships.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=840a568e6301" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>