<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nico Acosta on Medium]]></title>
        <description><![CDATA[Stories by Nico Acosta on Medium]]></description>
        <link>https://medium.com/@acossta?source=rss-1dc258c172fb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*ZHC4-QKWzX1e7kIU.jpg</url>
            <title>Stories by Nico Acosta on Medium</title>
            <link>https://medium.com/@acossta?source=rss-1dc258c172fb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 11:18:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@acossta/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Introducing Usage Metering for ChatGPT-powered LLM Apps]]></title>
            <link>https://medium.com/propel-data-analytics-blog/introducing-usage-metering-for-chatgpt-powered-llm-apps-e90b2c02b826?source=rss-1dc258c172fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/e90b2c02b826</guid>
            <category><![CDATA[generative-ai]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[api]]></category>
            <dc:creator><![CDATA[Nico Acosta]]></dc:creator>
            <pubDate>Tue, 30 May 2023 14:27:10 GMT</pubDate>
            <atom:updated>2023-05-30T14:27:10.452Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Add usage metering to your ChatGPT-powered products with five lines of code. Quickly implement functionality to enforce limits, detect abuse, act on user feedback, and visualize usage insights and get to market faster.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kvgdbmu8ENgJpwTa7w8YIw.png" /></figure><p>Today, we are excited to announce the preview of Propel’s usage metering solution for ChatGPT-powered applications. It enables developers to streamline building and operating usage metering functionality, such as token usage tracking, latency monitoring, and user feedback collection, through a highly scalable analytics backend and a blazing-fast data API.</p><p>With Propel, developers can quickly implement functionality to enforce limits, detect abuse, act on user feedback, and visualize usage insights in their user-facing product or internal tools. The preview supports OpenAI’s ChatGPT API (GPT-3.5 and GPT-4). We plan to add support for more APIs and custom-built LLMs based on customer demand in the future. To get access, <a href="https://www.propeldata.com/llm-app-analytics-and-usage-metering">join our waitlist</a>.</p><p>Large Language Models (LLMs), such as ChatGPT, have been the biggest advance in computing since the graphical user interface. They fundamentally change how we interact with computers. LLM apps are being built into every consumer or B2B product experience. However, taking ChatGPT-powered products to market is not an easy task.</p><h3>The Problem: Generative AI Apps Need to Track Usage</h3><p>Each ChatGPT API call consumes tokens that cost money. Therefore, usage needs to be tracked and communicated to end users. It must be designed and built into the product experience. With all the excitement around LLMs, prompt design and tuning, preventing prompt injections, and model benchmarking, tracking usage is often an afterthought. However, it’s a critical aspect that can cause delays in bringing generative AI products to market.</p><p>There are different ways to productize GPT-powered features. Some products will include GPT-powered features as part of their current offerings. In such cases, they need to keep track of usage and enforce limits to avoid runaway costs. Other products will pass the cost onto the end user. In such cases, they need to keep track of usage to be able to charge at the end of the month. In either case, usage metrics must be available via APIs for developers to surface to end-users, and to integrate into the rest of their application logic that enforces limits, bills, and sends user notifications.</p><h3>Why Is It Hard to Track Usage?</h3><p>Tracking usage is harder than it seems. Why not just store it in the database and read it from there? There are three main challenges:</p><ol><li><strong>Aggregates and query latency:</strong> Usage queries are analytical in nature, meaning they do not retrieve individual records from a database. Instead, they retrieve a set of records for a given time range based on a given aggregation. For example, your customers may want to know their token consumption by day, broken down by user, for the last 30 days. This query requires a SUM aggregation over the last 30 days, grouped by user, and filtered by customer ID. As usage data grows, your transactional database will become increasingly slower at serving these queries, which can slow down your entire application.</li><li><strong>Ever-evolving metadata:</strong> Your product is constantly evolving, with new GPT-powered features being shipped to different parts of the product. This means that the metadata contextualizing usage data is constantly changing. The nature of this data is unstructured, so you will need the ability quickly add new metadata and query your usage data with the ability to slice and dice by the metadata you provided.</li><li><strong>Custom metrics:</strong> Every product is unique, and how you account for usage and productize your GPT features will depend on your product, customer base, and the experience you want to provide. Therefore, you need the ability to define custom metrics to track your primary usage unit. Some products track usage based on requests, while others track usage on input words or input tokens. Whatever method you choose, you need a centralized metric definition that you define once and use everywhere.</li></ol><h3>Propel’s Usage Metering for LLM Apps</h3><p>Propel’s LLM usage metering solution equips generative AI developers with a powerful analytics backend, low-latency data API, and a React component library to easily capture usage data, model it into metrics, and expose usage data via a fast and responsive GraphQL API. This API can be used to drive product behaviors, such as enforcing limits or usage accounting or building in-product usage visualizations like reports and dashboards.</p><h3>Key Features</h3><p>The solution consists of the following features:</p><p><strong>Data collection endpoint</strong></p><p>The data collection endpoint enables developers to send the response of the OpenAI API directly to Propel with a simple HTTP POST.</p><pre>const openAICompletion = await openai.createChatCompletion({<br>  model: &quot;gpt-4&quot;,<br>  messages: [{role: &quot;user&quot;, content: &quot;Hello world&quot;}],<br>});<br><br>// Log response with Propel<br>const logResponse = await fetch(&quot;PROPEL_URL&quot;, {<br>    method: &quot;POST&quot;, <br>    headers: { &quot;Authorization&quot;: &quot;Bearer YOUR_TOKEN&quot;},<br>    body: JSON.stringify(openAICompletion),<br>});</pre><p><strong>Define custom metrics</strong></p><p>Once the data is in Propel, developers can define custom metrics that give meaning to the data. The example below illustrates how to configure a TokenUsage metric.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7rGaPMmo-IGvkwk9t67Y3w.png" /><figcaption>Propel’s custom Metric creation interface</figcaption></figure><p><strong>Query data via a GraphQL API</strong></p><p>Once the Metric is defined, developers can query the data with a simple GraphQL API request. The example below queries the token usage metric when the gpt-4 was used over the last 30 days.</p><pre>query TokenUsage {<br>  metricByName (uniqueName: &quot;TokenUsage&quot;) {<br>    counter ({<br>      timeRange: { relative: &quot;LAST_N_DAYS&quot;  n:&quot;30&quot;}<br>      filters: [{<br>        column: &quot;model&quot;<br>        operator: EQUALS<br>        value: &quot;gpt-4&quot;      <br>      }]<br>    }) {<br>      values<br>      labels<br>    }<br>  }<br>}</pre><p><strong>Slice and dice on your own metadata</strong></p><p>If you added custom metadata to the payload, you can slice and dice the metric values by your own metadata. The example below adds a filter based on a customerId value provided.</p><pre>query TokenUsage {<br>  sales: metricByName (uniqueName: &quot;tokenUsage&quot;) {<br>    timeSeries ({<br>      timeRange: { relative: &quot;LAST_N_DAYS&quot;  n:&quot;30&quot;}<br>      granularity: &quot;DAY&quot;<br>      filters: [{<br>        column: &quot;metadata.customerId&quot;<br>        operator: EQUALS<br>        value: &quot;ac5bc654-6449-4003-ac72-6f07e761b152&quot;      <br>      }]<br>    }) {<br>      values<br>    }<br>  }<br>}</pre><p><strong>Build user-facing usage dashboards</strong></p><p>By querying the API and using our <a href="https://github.com/propeldata/ui-kit">UI components</a> (optional), you can quickly build dashboards, log interfaces, and in-product metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yLcROuFgKbve-b2fM4qYrA.png" /><figcaption>A example Chat-GPT-powered application</figcaption></figure><p><strong>Integrate’s with the observability tools you already use</strong></p><p>Lastly, you can easily integrate with your existing observability and internal tools stack, such as <a href="https://grafana.com/grafana/plugins/propeldata-graphql-datasource/">Grafana</a> and <a href="https://docs.retool.com/docs/connect-to-propel">Retool</a> to manage and monitor your ChatGPT-powered apps. No need to reinvent the wheel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*sT-YwOwaTjTxbJZJ8FvM8A.png" /><figcaption>An example chart that can be displayed in Grafana or Retool</figcaption></figure><h3>Final thoughts</h3><p>With Propel&#39;s usage metering solution, developers can focus on what they do best - building great ChatGPT-powered products - without having to worry about the complexities of usage tracking. By streamlining the development and operation of usage metering functionality, Propel is helping to bring ChatGPT-powered products to market faster and more efficiently than ever before.</p><p><a href="https://www.propeldata.com/llm-app-analytics-and-usage-metering">Join our waitlist now</a> and start getting the benefits of Propel&#39;s usage metering solution for ChatGPT-powered applications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e90b2c02b826" width="1" height="1" alt=""><hr><p><a href="https://medium.com/propel-data-analytics-blog/introducing-usage-metering-for-chatgpt-powered-llm-apps-e90b2c02b826">Introducing Usage Metering for ChatGPT-powered LLM Apps</a> was originally published in <a href="https://medium.com/propel-data-analytics-blog">Propel Customer-Facing Analytics Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing Propellers: Easily select the optimal cost and query speed for each use case]]></title>
            <link>https://medium.com/propel-data-analytics-blog/introducing-propellers-easily-select-the-optimal-cost-and-query-speed-for-each-use-case-415c578581bc?source=rss-1dc258c172fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/415c578581bc</guid>
            <category><![CDATA[customer-facing-analytics]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Nico Acosta]]></dc:creator>
            <pubDate>Mon, 05 Dec 2022 14:56:38 GMT</pubDate>
            <atom:updated>2022-12-05T14:56:38.583Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Propellers equip developers with an easy way to select the optimal cost and query speed for each customer-facing analytics use case.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*InqNkg6-aerYkRMEjuwrAA.png" /></figure><p>Today, we are thrilled to introduce Propellers, an easy way for product development teams to select the optimal cost and query speed for their customer-facing analytics use cases.</p><p>In some use cases, such as analytics on a core product workflow, you might want to prioritize speed over cost. In others, like an internal app, you might want to prioritize cost over speed.</p><p>Propel’s architecture <a href="https://www.propeldata.com/blog/what-is-the-separation-of-storage-and-compute-in-data-platforms-and-why-does-it-matter">separates storage and compute</a>, allowing each to be consumed, scaled, and priced interdependently. This enables you to select the optimal levels of query speed and cost for your use case.</p><h3>What are Propellers?</h3><p>Propellers are the unit of compute in Propel. The larger the Propeller, the faster the queries and the higher the cost. Every Propel Application (and therefore every set of API credentials) has a Propeller that determines the speed and cost of queries.</p><p>You can think of a Propeller like a car engine. The size of the engine (Propeller) determines the speed at which the car can go. But just as importantly, it also affects how much fuel (costs) the car uses.</p><p>A small Propeller can be thought of as a “compact” car engine. It’s great for getting around town, and it’s very fuel-efficient. But if you need to go on the highway, or if you have a lot of people or cargo, you’re going to want a bigger engine.</p><p>Propellers determine the maximum number of records a query can process per second. Queries that are very complex or that scan a larger time range will require the processing of more records.</p><p>The table below describes the different Propellers, their capacity, and their price. The Propeller’s price is determined by the maximum number of records it can process per second.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*_USX7YGEq093kHCn-8X0uQ.png" /></figure><h3>How do Propellers work?</h3><p>In order to query data from Propel, you first create a Propel Application. Your Propel Application includes settings, like OAuth 2.0 credentials, as well as the Propeller to use when executing queries.</p><p>You can choose a Propeller ranging from P1_X_SMALL through P1_X_LARGE when creating your Application, and you can resize it at any time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/0*A8RWvB3EVyNituI2.gif" /><figcaption><em>An animated screen capture showing how to create a Propel Application with a P1_MEDIUM Propeller.</em></figcaption></figure><p>Next, when you query data from Propel, you include your Propel Application’s access token (see <a href="https://www.propeldata.com/docs/api/authentication">authentication</a>). Propel checks your access token, identifies your Application, and executes your query with your Application’s configured Propeller.</p><h3>How do I choose a Propeller size?</h3><p>If you’re not sure what size Propeller you need, don’t worry! The Propel Console can help you select the optimal Propeller size for your use case. Here’s how:</p><ol><li>Go to the Metric Playground in Console.</li><li>Create a representative query for your Metric.</li><li>This should be a query your Application will actually execute.</li><li>Starting with the smallest Propeller, run the query and check the query statistics.</li><li>If the query was too slow, choose the next highest Propeller from the drop-down.</li><li>Repeat until you get the desired cost-to-performance.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/0*j-gFOO06CSKld-Cy.gif" /><figcaption><em>An animated screen capture showing how to use the Metric Playground to test the different Propellers.</em></figcaption></figure><p>Here are a few rules of thumb to follow when choosing Propellers:</p><ul><li>Start small and move to bigger Propellers as your Data Pool grows.</li><li>Choose bigger Propellers when speed is important or your Data Pool is large.</li><li>Don’t automatically choose the biggest Propeller just because you have billions of records. Filtering by time range and tenant ID significantly cuts down on the number of records, meaning even medium-sized Propellers can perform well on larger Data Pools.</li></ul><h3>What are the use cases for Propellers?</h3><p>Propel’s customers use Propellers for many different things, including the following:</p><h4>Optimizing for fast response times on large datasets</h4><p>Imagine the main dashboard of a SaaS product that has to query a ton of data and is core to every workflow in the product. Customers have no tolerance for slow products, so you’ll want that dashboard to load blazingly fast. A large Propeller lets you optimize for speed here in the customer-facing product.</p><h4>Optimizing for cost on small datasets and non-core workflows</h4><p>Not all customer-facing workflows handle massive amounts of data or need sub-second response times. For these non-core workflows that query small datasets, you could use a smaller Propeller and avoid paying for performance that isn’t needed.</p><p>Similarly, employees might be OK waiting 5‒10 seconds for a query to load in an internal tool. Here, you might want to keep costs lower and go with a smaller Propeller.</p><h4>Optimizing for development and production environments</h4><p>Production environments that serve customers might want to use large Propellers, while development and staging environments can get away with smaller Propellers. Again, Propellers enable you to pay for performance where you need it.</p><p>💡 Not all use cases are created equal, and that’s where Propellers come in. With Propellers, you can easily select the optimal cost and query speed for each customer-facing analytics use case.</p><h3>How do I get started with Propellers?</h3><p>You can use the Propel Console to create an Application and select the size of Propeller. You can learn more about <a href="https://www.propeldata.com/docs/applications">Propellers by reading the documentation</a>.</p><p>If you don’t have your Propel account yet <a href="https://www.propeldata.com/join-waitlist?__hstc=17958374.7add73943c8f554c1c478c947e376657.1661881287613.1667148400062.1667167207872.21&amp;__hssc=17958374.2.1667167207872&amp;__hsfp=2429260968">Join our waitlist!</a></p><p>We are onboarding users first as fast as we can. We can’t wait to see what you can build with Propel!</p><p>Follow us on Twitter <a href="https://twitter.com/@propeldatacloud">@propeldatacloud</a> or <a href="https://www.propeldata.com/newsletter?__hstc=17958374.7add73943c8f554c1c478c947e376657.1661881287613.1667148400062.1667167207872.21&amp;__hssc=17958374.2.1667167207872&amp;__hsfp=2429260968">subscribe to our email newsletter</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=415c578581bc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/propel-data-analytics-blog/introducing-propellers-easily-select-the-optimal-cost-and-query-speed-for-each-use-case-415c578581bc">Introducing Propellers: Easily select the optimal cost and query speed for each use case</a> was originally published in <a href="https://medium.com/propel-data-analytics-blog">Propel Customer-Facing Analytics Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is the separation of storage and compute in data platforms and why does it matter?]]></title>
            <link>https://medium.com/propel-data-analytics-blog/what-is-the-separation-of-storage-and-compute-in-data-platforms-and-why-does-it-matter-468dfc7f4d12?source=rss-1dc258c172fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/468dfc7f4d12</guid>
            <category><![CDATA[product-data]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[customer-facing-analytics]]></category>
            <category><![CDATA[data-platforms]]></category>
            <dc:creator><![CDATA[Nico Acosta]]></dc:creator>
            <pubDate>Mon, 21 Nov 2022 19:26:58 GMT</pubDate>
            <atom:updated>2022-11-21T19:26:58.589Z</atom:updated>
            <content:encoded><![CDATA[<h4>As businesses strive to become data-driven, the separation of storage and compute has become a critical factor in data platforms. By separating these two components, businesses can take further advantage of the advances in cloud computing for their data analytics.</h4><figure><img alt="Illustration by Propel" src="https://cdn-images-1.medium.com/max/1024/1*UTLP7fDWLfpd_OD-IWwU2g.png" /></figure><p>The separation of storage and compute is a new paradigm in data platforms. By separating storage and compute, they can be consumed, scaled, and priced independently. This allows businesses to pay for what they use and nothing more. As customers, we want to avoid waste in all forms, including unused capacity. Let’s take a closer look at how this separation works and the benefits it brings.</p><p>Before this separation was possible, many companies were running systems like Hadoop or HDFS where the dominant paradigm was that compute and storage were tightly coupled 1:1. For example, if your queries got more complicated, you had to resize the HDFS cluster; if the amount of data you had increased, you had to resize the HDFS cluster. And this always had to be done for the peak queries and peak storage. It resulted in a very inelastic data architecture with a lot of waste, but this was state-of-the-art in 2006 when it was first released and is used still today.</p><p>In this post, we will cover how the separation of storage and compute works, how you can take advantage of it, what are some of the key technologies that make it possible, and some examples of data platforms that implement the separation of storage and compute.</p><h3>How does the separation of storage and compute work?</h3><p>When you buy a computer or rent an AWS instance, it comes with one or more CPUs and one or more hard drives, the compute and storage. Typically, more powerful CPUs are paired with larger hard drives. So if you only need the more powerful compute, the extra storage goes unused, and vice versa.</p><p>With the advances in serverless computing and cloud storage, data platforms can now combine whatever storage or compute they need on-demand vs relying on predefined instance configurations. The “infinite storage” offered by services like AWS S3 makes it so that data platforms don’t even need to provision more storage as customers use more space. Similarly, elastic computing enables data platforms to offer different levels of compute capacity on demand that customers can choose to meet their query processing requirements.</p><p>If a customer on a data platform needs a lot more storage, but a small amount of compute processing power, the platform can offer this by allocating more storage to that specific customer and a smaller processor for compute because they are not coupled 1:1.</p><h3>What are the benefits of separating storage and compute?</h3><p>The are many benefits of separating storage and compute in data platforms. Here are some of the main ones:</p><ul><li><strong>You only pay for what you use:</strong> With the separation of storage and compute, you only pay for the resources you actually use. If you have a lot of data but don’t need to do much processing on it, you won’t be paying for expensive compute resources that you don’t need.</li><li><strong>Flexible Scaling:</strong> You can scale your storage and compute independently to match your needs, potentially unlocking new use cases. If you need to do more processing, you can simply add more compute resources.</li><li><strong>Better Utilization of Resources:</strong> By being able to scale storage and compute independently, data platforms can make better use of their resources. For example, if you have a lot of data but don’t need to do much processing on it, you can use less expensive storage options and save money.</li></ul><h3>How can you take advantage of the separation of storage and compute?</h3><p>There are a few steps businesses can take advantage of the separation of storage and compute in data platforms:</p><ol><li><strong>Identify your workloads:</strong> Identify what are the discrete data workloads you have. For example, internal reporting could be one workload and customer-facing analytics for your product could be another.</li><li><strong>Allocate resources to each workload:</strong> Once you have identified the workloads you need, the next step is to allocate resources. Following our example, internal employees might be OK waiting 30 seconds for reports to load, while customers have no tolerance for slow dashboards. In this case, you could allocate more resources to your customer-facing analytics workload.</li><li><strong>Optimize your resources:</strong> Finally, make sure to optimize your resources to get the most out of them. This is going to depend on the platform, especially if these resources need to be provisioned versus offered on demand.</li></ol><h3>What are some examples of data platforms that separate storage and compute?</h3><p>There are a few data platforms that have adopted this new paradigm:</p><ul><li><a href="https://www.snowflake.com/">Snowflake</a> — In Snowflake you pay for storage and compute independently. Compute is provisioned with “<a href="https://www.propeldata.com/blog/what-are-warehouses-in-snowflake-data-analytics-platform">Warehouses</a>” that have different computing power.</li><li><a href="https://www.dremio.com/">Dremio</a> — Dremio is a data lakehouse based on the open-source Apache Iceberg table format. It offers different compute instances to process data that lives in your S3 bucket. You pay for S3 storage independently.</li><li><a href="https://www.propeldata.com/">Propel</a> — is an Analytics API platform to build customer-facing analytics. In Propel you can customize the compute assigned to your different web or mobile apps. Storage and compute are priced independently.</li></ul><h3>What are the key technologies that make the separation of compute and storage possible?</h3><p>Every data platform that takes advantage of the separation of storage and compute needs a storage layer (an object store or network attached storage), table and file formats, and a query engine. Some, like Snowflake, use proprietary technologies for each. The examples below are open source technologies for each of these categories (except the storage services).</p><h3>Object storage and network attached storage: AWS S3, GCP Cloud Storage, or Azure Blob Store</h3><p>Object Storage: Services like <a href="https://aws.amazon.com/s3/">AWS S3</a> can be used to store files on-demand with “infinite storage”. There is no need to provision or manage capacity and it is completely decoupled from the instance or serverless that is doing the processing.</p><p>Network Attached Storage (NAS): NAS is a storage architecture where data is stored on a dedicated server that is attached to a network. This allows users to access the data over the network using protocols like SMB (Server Message Block) or NFS (Network File System). NAS allows to swap or sometimes resize the storage capacity without changing the processing.</p><h3>Open table and file formats: Apache Iceberge and Apache Parquet</h3><p><a href="https://iceberg.apache.org/">Apache Iceberg</a> is an open table format designed to store vast amounts of data for analytics. Iceberg tables can be used with different data processing engines such as Spark, Trino, Flink, or Hive. It works with Parquet files, giving them SQL table-like semantics through its metadata.</p><p><a href="https://parquet.apache.org/">Apache Parquet</a> is a file format that is used to store data in a columnar format. This allows for faster reads and reduced disk space. Not only is the Parquet format open source, but it also has an entire ecosystem of tools to help you create, read and transform data.</p><h3>Query engines: DuckDB, Trino and Spark</h3><p>The query engine actually executes queries against your tables, so it needs to support your underlying table format, file format, and object storage.</p><p><a href="https://duckdb.org/">DuckDB</a> is an in-process SQL OLAP database with a highly optimized query engine. It supports reading from object stores like S3, and it can perform predicate and projection pushdown when scanning Parquet files. These optimizations mean queries that might have previously required a cluster to complete can be executed on a single host, due to better utilization of compute.</p><p>However, once your data reaches a certain size or you reach the limits of vertical scaling, it may be necessary to distribute your queries across a cluster, or scale horizontally. This is where distributed query engines like <a href="https://trino.io/">Trino</a> and <a href="https://spark.apache.org/">Spark</a> come in. Distributed query engines make use of a coordinator to plan the query and multiple worker nodes to execute them in parallel.</p><h3>Final thoughts on the separation of storage and compute</h3><p>The separation of storage and compute is a new paradigm in data platforms that offers many benefits, including flexible scaling, better utilization of resources, and only paying for what you use. Not all data platforms take advantage of this new paradigm. There are key open-source technologies that not only make it possible but also accelerate its adoption. Snowflake, Dremio, and Propel are examples of data platforms that take advantage of this new paradigm. To take advantage of the separation of storage and compute in your own application, make sure you identify your workloads and allocate resources accordingly.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=468dfc7f4d12" width="1" height="1" alt=""><hr><p><a href="https://medium.com/propel-data-analytics-blog/what-is-the-separation-of-storage-and-compute-in-data-platforms-and-why-does-it-matter-468dfc7f4d12">What is the separation of storage and compute in data platforms and why does it matter?</a> was originally published in <a href="https://medium.com/propel-data-analytics-blog">Propel Customer-Facing Analytics Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why do product and data teams struggle to work together?]]></title>
            <link>https://medium.com/propel-data-analytics-blog/why-do-product-and-data-teams-struggle-to-work-together-99b438b366d1?source=rss-1dc258c172fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/99b438b366d1</guid>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[work]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Nico Acosta]]></dc:creator>
            <pubDate>Tue, 23 Aug 2022 15:19:59 GMT</pubDate>
            <atom:updated>2022-08-23T15:19:59.292Z</atom:updated>
            <content:encoded><![CDATA[<h4>Product and data teams struggle to work together because there is a fundamental tradeoff in data between flexibility, performance, and cost-effectiveness. Here’s the solution.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aCsTTCRQcX8Sup5b" /><figcaption>Photo by <a href="https://unsplash.com/@dj_johns1?utm_source=medium&amp;utm_medium=referral">DJ Johnson</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Data and product teams <strong>always</strong> work smoothly; yeah, sure.</p><p>Why is it so hard for data and product teams to work together?</p><p>Their conversations go something like this:</p><blockquote><strong>Product team:</strong> We need to build this insights dashboard in our console and mobile app for customers to see the outcome of XYZ by the hour and be able to slice and dice by A, B, C, D, and E.</blockquote><blockquote><strong>Data team:</strong> We only have that data monthly, and you can only slice and dice by C, D, <em>or</em> E. How important is this? How fast do you need to query it?</blockquote><blockquote><strong>Product team:</strong> We need to be able to query super-fast (sub-second), and it is crucial to meeting our goal, so it is super important.</blockquote><blockquote><strong>Data team:</strong> We can’t do it. File a ticket with the requirements, and we can see where it falls in the priorities.</blockquote><p>The product team is frustrated because they can’t build the product they envisioned.</p><p>The data team is frustrated because they are constantly bombarded with high-priority tasks that are impossible to prioritize against each other.</p><h3>What’s the problem?</h3><p>It is not that the product team is unreasonable or the data team is incapable. The product team is looking to build what’s best for their customer. The data team seeks to inform decisions as effectively as possible across the organization.</p><p>To understand the root of the problem, we must dig deeper. There is a fundamental tradeoff in data between flexibility, performance, and cost-effectiveness.</p><ul><li><strong>Flexibility</strong>: refers to the different ways to slice and dice the data. What filters do you have? Can you see it by minute, hour, day, or month (its granularity), and how far back in time does it go?</li><li><strong>Performance</strong>: refers primarily to the query speed. How fast is a given query?</li><li><strong>Cost-effectiveness:</strong> refers to the total cost of implementing the desired outcome, with compute and storage being the primary factors driving the cost.</li></ul><p>You can have two data characteristics above at the same time, but not all three.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*gnG65B2jqbzKyans.png" /><figcaption>A triangle illustrating the tradeoffs when dealing with large-scale data between flexibility, performance, and cost-effectiveness. Image courtesy of the author</figcaption></figure><p>You can <strong>pre-aggregate</strong> data which will give you performance and cost-effectiveness, but you will have tragic flexibility limitations.</p><p>You can <strong>beef up compute</strong> to query as fast as possible in order to keep the performance and flexibility, but it will come at a steep cost.</p><p>You can have all the flexibility in the world and be cost-effective, but it will likely get very <strong>slow queries</strong>, even on a small scale.</p><h3>So what’s the problem?</h3><p>The root of the problem is not the tradeoff itself but who makes the tradeoff.</p><p>In the previous conversation, the data team made a tradeoff that sacrificed flexibility. At the time, it was the right tradeoff. They are also the team empowered and equipped to make such a tradeoff.</p><p>Unfortunately, that tradeoff did not work for what the product team needed. It would be surprising if it did, as it was made a long time ago with other requirements in mind.</p><p>Now imagine the product team has a huge budget. Their big budget can’t get them the performant and flexible product they want because it is blocked by the data team’s ability to prioritize it. The data team’s resources are very constrained, and the organization has many other priorities which make the product’s team ask fall below the line every quarter and therefore it never gets done.</p><p>Unlike the data team, the product team is not empowered or equipped to make this tradeoff.</p><p>The root of the problem is that the team that is closest to the customer is not empowered to make and execute the tradeoff.</p><p>The data team gets pulled in a million different directions and is not in a position to prioritize what’s best for an individual product team.</p><h3>What now?</h3><p>The solution is for product teams, who are closer to the customer experience, to be empowered and equipped to make the tradeoff. How does an organization get there without each product development team having data engineering resources?</p><p>Our goal at <a href="https://www.propeldata.com/">Propel</a> is to empower product development teams to build in-product analytics quickly and easily. We want product teams to be able to make the tradeoff they need to build their product. Propel does this by:</p><ul><li><strong>Simplifying data requirements.</strong> There’s no need to pre-aggregate, roll-up data, or create complicated indexes; just point us at your data.</li><li><strong>Easily scaling compute.</strong> Product development teams can authorize applications with different levels of computing power with different levels of computing power that enable them to prioritize flexibility and performance.</li><li><strong>Optimizing common queries.</strong> When a team builds a dashboard, there are a common set of queries that power it. Once developers know what those common queries are, they can optimize them for blazing-fast response times with just one click.</li></ul><p>If you are interested in what we are doing, <a href="https://www.propeldata.com/newsletter">sign up for monthly product updates</a>, follow us on Twitter <a href="https://twitter.com/@propeldatacloud">@propeldatacloud</a>, or <a href="https://www.propeldata.com/join-waitlist">join our waitlist</a> to kick the tires on the product.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99b438b366d1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/propel-data-analytics-blog/why-do-product-and-data-teams-struggle-to-work-together-99b438b366d1">Why do product and data teams struggle to work together?</a> was originally published in <a href="https://medium.com/propel-data-analytics-blog">Propel Customer-Facing Analytics Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to build Snowflake data apps with GraphQL]]></title>
            <link>https://medium.com/propel-data-analytics-blog/how-to-build-snowflake-data-apps-with-graphql-8c1bdc4635bb?source=rss-1dc258c172fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c1bdc4635bb</guid>
            <category><![CDATA[graphql]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Nico Acosta]]></dc:creator>
            <pubDate>Fri, 19 Aug 2022 19:51:33 GMT</pubDate>
            <atom:updated>2022-08-19T19:51:33.538Z</atom:updated>
            <content:encoded><![CDATA[<h4>In this feature demo, you’ll see exactly what it takes to create and query a Metric on top of your Snowflake data warehouse using Propel’s blazing fast GraphQL API. In just five easy steps, you’ll connect to Snowflake, design a Metric, and query it with our GraphQL API.</h4><figure><img alt="Propel Data offers a blazing fast GraphQL API for building data apps from data that organizations already have in Snowflake data warehouse, as illustrated by this photograph of a laptop showing a real-time product dashboard with analytical reports and data visualization." src="https://cdn-images-1.medium.com/max/1024/1*fi2uHjAqZCUJFX2QO2XGdA.jpeg" /><figcaption>Cover photo by <a href="https://unsplash.com/@kmuza?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Carlos Muza</a> on <a href="https://unsplash.com/s/photos/data-app?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>We just launched our latest feature: <a href="https://www.propeldata.com/blog/how-to-build-in-product-analytics-with-snowflake-and-graphql">Snowflake as a Data Source</a>. Now, let’s demo how to connect Snowflake and Propel via a GraphQL API that’s perfect for building in-product analytics in minutes, instead of months.</p><p>In this short tutorial, we’ll be helping you build a Snowflake data app with GraphQL. Data apps are customer-facing web or mobile apps that include native, large-scale, aggregate data analysis into their product functionality and core workflows.</p><p>Before you get started, you’ll need to have your data already loaded into Snowflake data platform. (If you’re interested in learning more about using Snowflake for data analytics, consider <a href="https://propeldata.com/newsletter">subscribing to our newsletter</a> for free educational content.)</p><h3>How do you build a high-performance data app with Snowflake as a Data Source?</h3><p>To start building a data app inside Propel with Snowflake as your Data Source, you just need to complete these five easy steps:</p><h3>1. Connect to Snowflake data warehouse</h3><p>First, you will need to create a user, role, and warehouse for Propel in your Snowflake account. (If you need a Propel account, <a href="https://www.propeldata.com/join-waitlist">join our waitlist</a>, and we’ll onboard you as soon as we can!) Then, you can create the Snowflake Data Source inside Propel’s Console by entering your Snowflake connection details.</p><p>Propel will connect and run a series of checks to validate that the connection has the correct permissions.</p><figure><img alt="An animated screen capture of the Propel Console, showing how to create a Data Source and the numerous checks that confirm the connection is working." src="https://cdn-images-1.medium.com/max/1024/0*veCZQgej0fu6NbuT.gif" /><figcaption>An animated screen capture of the Propel Console, showing how to create a Data Source and the numerous checks that confirm the connection is working.</figcaption></figure><p>An animated screen capture of the Propel Console, showing how to create a Data Source and the numerous checks that confirm the connection is working.</p><h3>2. Sync a table from Snowflake</h3><p>Once connected, you can operationalize a Snowflake table by creating a Data Pool inside Propel, which will start the sync process. When using Propel, a Data Pool automatically creates an optimized mirror of your table. There’s no need for any pipelines or data engineering.</p><p>While creating the Data Pool, you will need to indicate the table you want to operationalize and its primary timestamp column in the Console.</p><figure><img alt="An animated screen capture of the Propel Console, showing how to create a Data Pool from a table within a Data Source." src="https://cdn-images-1.medium.com/max/1024/0*DWYJLEcTTVVAclh4.gif" /><figcaption>An animated screen capture of the Propel Console, showing how to create a Data Pool from a table within a Data Source.</figcaption></figure><p>An animated screen capture of the Propel Console, showing how to create a Data Pool from a table within a Data Source.</p><h3>3. Define Metrics inside Propel</h3><p>With your data operationalized, you can define the Metrics you want to use in your data app. Metrics are the analytics measures you want to track, and they can be of different types: SUM, COUNT, COUNT_DISTINCT, MIN, MAX, and AVERAGE.</p><p>Once you configure your Metric settings in Propel, you can select the columns you want to expose as Dimensions. Dimensions are the different ways you can filter Metric data to uncover insights. When defining a customer-facing Metric, you can control which columns are exposed to the public via Dimensions.</p><figure><img alt="An animated screen capture of the Propel Console, showing how to create a Metric from a Data Pool, how to choose the Metric type, and how to add Dimensions for filtering." src="https://cdn-images-1.medium.com/max/1024/0*5mDwA1fmlw3dySp1.gif" /><figcaption>An animated screen capture of the Propel Console, showing how to create a Metric from a Data Pool, how to choose the Metric type, and how to add Dimensions for filtering.</figcaption></figure><p>An animated screen capture of the Propel Console, showing how to create a Metric from a Data Pool, how to choose the Metric type, and how to add Dimensions for filtering.</p><h3>4. Query your Metrics using GraphQL</h3><p>Propel gives you a GraphQL API that is always up and always performant in just a few easy steps. Once you have your Metric defined, you can try Propel’s GraphQL Explorer to query the Metric data in the format you are going to use: a Time Series, a Counter, or a Leaderboard.</p><p>Looking at the top-right of the GraphQL Explorer, you’ll notice the fast response times you get with Propel’s high-performance GraphQL API. In this example, the response time is about 500 milliseconds (end to end) when querying hundreds of thousands of records.</p><figure><img alt="An animated screen capture of Propel’s GraphQL Explorer, showing how to query a Metric using the GraphQL API with various time granularities and filters." src="https://cdn-images-1.medium.com/max/1024/0*xWyY-G8wBufNExX2.gif" /><figcaption>An animated screen capture of Propel’s GraphQL Explorer, showing how to query a Metric using the GraphQL API with various time granularities and filters.</figcaption></figure><p>An animated screen capture of Propel’s GraphQL Explorer, showing how to query a Metric using the GraphQL API with various time granularities and filters.</p><h3>5. Build in-product analytics using GraphQL</h3><p>The Metric you defined previously is now available via the GraphQL API. Using the technology stack of your choice, you can build in-product analytics into your app. You didn’t have to do any data engineering, beyond loading your data to Snowflake.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*33DQxUnZePKUBZDs.gif" /><figcaption>An animated screen capture of Visual Studio Code, showing how to use Propel’s GraphQL API, React, and ECharts to query and visualize Metric data from Snowflake.</figcaption></figure><p>An animated screen capture of Visual Studio Code, showing how to use Propel’s GraphQL API, React, and ECharts to query and visualize Metric data from Snowflake.</p><h3>How do I get started building data apps with Propel and Snowflake?</h3><p><a href="https://www.propeldata.com/join-waitlist">Join our waitlist!</a></p><p>We are now onboarding Snowflake users first as fast as we can. We are excited to see what you build with Propel!</p><p>Follow Propel Data on Twitter <a href="https://twitter.com/@propeldatacloud">@propeldatacloud</a> or <a href="https://www.propeldata.com/newsletter">subscribe to our product updates</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c1bdc4635bb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/propel-data-analytics-blog/how-to-build-snowflake-data-apps-with-graphql-8c1bdc4635bb">How to build Snowflake data apps with GraphQL</a> was originally published in <a href="https://medium.com/propel-data-analytics-blog">Propel Customer-Facing Analytics Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to build in-product analytics with Snowflake and GraphQL]]></title>
            <link>https://medium.com/propel-data-analytics-blog/how-to-build-in-product-analytics-with-snowflake-and-graphql-8457e167e9ee?source=rss-1dc258c172fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/8457e167e9ee</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[graphql]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Nico Acosta]]></dc:creator>
            <pubDate>Tue, 16 Aug 2022 18:33:04 GMT</pubDate>
            <atom:updated>2022-08-16T18:33:04.311Z</atom:updated>
            <content:encoded><![CDATA[<h4>Propel Data is excited to announce support for Snowflake as a Data Source. Developers are now able to build on top of GraphQL APIs powered by their Snowflake data warehouse, meaning it’s easier than ever for companies to build in-product analytics with their existing teams.</h4><figure><img alt="Propel Data’s now supports Snowflake warehouse, meaning developers can build data apps on top of GraphQL APIs powered by their Snowflake data, as illustrated by this single snowflake on a dark blue background consistent with Propel’s brand colors." src="https://cdn-images-1.medium.com/max/1024/0*1SVzbaMEPG35Ugs0" /><figcaption>Photo by <a href="https://unsplash.com/@dariuscotoi?utm_source=medium&amp;utm_medium=referral">Darius Cotoi</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>When running any business, especially software as a service (SaaS) applications, companies generate a lot of data. In the modern data stack, developers typically clean, test, and transform this data with tools such as <a href="https://www.getdbt.com/">dbt</a>, <a href="https://airbyte.com/">Airbyte</a>, or <a href="https://www.fivetran.com/">Fivetran</a>, and then they store this data in cloud <a href="https://www.propeldata.com/blog/is-snowflake-a-data-warehouse-for-analytics-and-insights">data warehouses like Snowflake</a>.</p><p>Data warehouses centralize data from different data silos, including internal databases or SaaS applications, with the ultimate goal being data visualization and analysis. Analytics engineers can also use data warehouses for creating event-driven product features, like automated reporting emails and product notifications.</p><p>The data stored in data warehouses like Snowflake is frequently used for internal purposes like business intelligence, but Snowflake is not often used for building customer-facing applications, namely in-product analytics or customer insights.</p><h3>Why doesn’t Snowflake get used for building in-product analytics?</h3><p>Querying the warehouse directly from customer-facing applications is slow; costs can get out of control; and building a GraphQL API requires substantial backend infrastructure to serve this data rapidly and securely to the frontend.</p><p>In other words, because of the complexity involved in producing a high-performance GraphQL API from a data warehouse like Snowflake, customer-facing applications often can’t leverage the organization’s data warehouse at all — until now.</p><p>Today, we are thrilled to announce Propel’s Snowflake Data Source. Teams can now use Snowflake data in customer-facing products without costly data engineering.</p><p>Propel offers low-latency response times, easy-to-use GraphQL API, and cost-effectiveness at scale. These features allow developers to build complex products in minutes, including analytics products that serve millions of users.</p><figure><img alt="Diagram illustrating the modern data stack, which shows how Propel helps you generate GraphQL APIs from Snowflake data warehouse." src="https://cdn-images-1.medium.com/max/1024/1*z-h7H14RddRH1-6Hw975uQ.jpeg" /><figcaption><em>The modern data stack, showing how Propel helps you build from Snowflake data via a GraphQL API. Image courtesy of the author</em></figcaption></figure><p>Propel manages all the caching, optimization, authorization, and APIs so that teams can focus on the product experience, without the overhead of managing highly-available data infrastructure. Our powerful Snowflake integration reduces the level of effort needed to produce scalable experiences when building in-product analytics.</p><p>With Propel, organizations are able to build the analytics they’ve always wanted with the teams they already have. Developers just need to connect Snowflake data platform to Propel, and Propel provides a high-performance GraphQL API.</p><h3>When should you connect Snowflake as a Data Source for analytics?</h3><p>There are many use cases where developers can leverage their Snowflake data using Propel, including:</p><ul><li><strong>Self-service analytics</strong> — SaaS products need to report on usage and operational metrics that help end-users manage and use the product. Placing this data in Snowflake allows developers to run usage and operational queries, and then Propel provides a fast GraphQL API without further data engineering.</li><li><strong>Automated email reports</strong> — Product companies often find themselves manually pulling CSV files and reports for their users. Manually generating reports is a bad experience both for the customer and the company. With Propel, teams can build the ideal experience, where users automatically receive these reports via email.</li><li><strong>Analytics-driven workflows</strong> — Many product experiences can be triggered by data, such as an account suspension or making a special offer. With Propel, developers can easily build data-driven workflows that enable automated product features that don’t involve managing additional complex data infrastructure.</li><li><strong>Analytics APIs</strong> — Many SaaS companies provide programmatic access to their end users via custom APIs so customers can build their own reports and product experiences. With Propel, developers can easily build customer-facing analytics APIs that expose data metrics to end users while securing permissions such that each end user can only see their own data.</li></ul><h3>Propel helps developers build in-product analytics from Snowflake</h3><p>Many organizations use Snowflake as a data warehouse in order to support business intelligence, event-driven architecture, and other data science projects. However, it’s always been tricky to use data warehouses for anything other than internal manual reporting.</p><p>With Propel, developers can easily build in-product analytics for web or mobile apps directly from the data warehouse without data engineering. Our mission is to empower customers with amazing data experiences that always load quickly, whether the application needs to query 30 days or 5 years of data at once.</p><p>Additionally, Propel offers powerful filters to help teams easily uncover analytical insights. These filters are fast to build and iterate for the product development teams, improving the developer experience compared to hiring an entire data engineering team.</p><p>Of course, analytical data can be useful for more than just visualizations and reporting. In addition to in-product analytics, organizations are able to use Propel to easily automate reporting emails, pulling the necessary analytical data via the GraphQL API and then sending the emails with their existing email stack.</p><p>Finally, the ability to use Snowflake as a Data Source means that organizations can provide end customers with direct API access to build their own reports and product experiences. Behind the scenes, Propel handles security, performance, and reliability, while providing a cost-effective and user-friendly interface for software engineers.</p><h3>How do I get started building analytics with Snowflake as a Data Source?</h3><p><a href="https://www.propeldata.com/join-waitlist">Join our waitlist!</a></p><p>We are onboarding Snowflake users first as fast as we can. We can’t wait to see what you can build with Propel!</p><p>Follow us on Twitter <a href="https://twitter.com/@propeldatacloud">@propeldatacloud</a> or <a href="https://www.propeldata.com/newsletter">subscribe to our email newsletter</a>.</p><p>If you’re ready for more, check out our feature demo on <a href="https://www.propeldata.com/blog/how-to-build-snowflake-data-apps-with-graphql">how to build Snowflake data apps with GraphQL</a> using Propel.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8457e167e9ee" width="1" height="1" alt=""><hr><p><a href="https://medium.com/propel-data-analytics-blog/how-to-build-in-product-analytics-with-snowflake-and-graphql-8457e167e9ee">How to build in-product analytics with Snowflake and GraphQL</a> was originally published in <a href="https://medium.com/propel-data-analytics-blog">Propel Customer-Facing Analytics Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why we built Propel Data]]></title>
            <link>https://medium.com/propel-data-analytics-blog/why-we-built-propel-data-ed2b3277b237?source=rss-1dc258c172fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed2b3277b237</guid>
            <category><![CDATA[entrepreneurship]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Nico Acosta]]></dc:creator>
            <pubDate>Mon, 15 Aug 2022 21:09:25 GMT</pubDate>
            <atom:updated>2022-08-15T21:32:55.994Z</atom:updated>
            <content:encoded><![CDATA[<h4>A letter from the co-founders</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rf2GN_q1fH8VF_ey_QO2Xw.jpeg" /><figcaption>The three co-founders (from left): Mark Roberts, Tyler Wells, and Nico Acosta. Image courtesy of the author</figcaption></figure><p>Today, we are thrilled to announce <a href="https://www.propeldata.com/">Propel Data</a> — an API Platform for developers to easily build in-product analytics with large-scale data. Propel Data enables developers to quickly build usage metering, insights dashboards, product metrics, and advanced reporting without the overhead of managing complex data infrastructure. We are excited to launch our API platform to the market and also to announce a $4.6M funding round led by Matrix Partners with notable angels including founders and executives from Twilio, Databricks, RapidAPI, FaunaDB, Labelbox, and Calixa.</p><p>Building in-product analytics should be fast and easy for developers and <strong>that’s why we built Propel Data</strong>. Empowering your customers with data shouldn’t require a multi-month or year-long project.</p><p>In-product analytics are a frequent feature request businesses make to their SaaS vendors. It’s easy to understand why — running a software product without analytics is like running a website without Google Analytics. You wouldn’t know how many people visit your site, where the traffic is coming from, and if visitors are converting to customers — you’d be flying blind. <strong>Yet, most SaaS products provide no analytics and they are often pushed off and delayed on product roadmaps — despite being a top feature request — leaving their customers flying blind. </strong>That’s because building data products is really <strong>HARD</strong>, until Propel Data.</p><h3>Why? What’s so hard about it? 🤔</h3><p>It’s taken leading companies like Twilio, LinkedIn, Shopify, and Uber years to build their customer-facing analytic products despite having huge teams behind them. Why is it so hard? Why is it so common for analytics to get punted in product roadmaps or take forever to build? Can’t you just query the database?</p><p>There are good reasons for this:</p><ol><li><strong>Data chaos</strong> — Data is often siloed in different data stores and transactional databases don’t perform well with analytical queries.</li><li><strong>Scarce engineering resources</strong> — Data engineering is a very scarce resource and analytics infrastructure and pipelines require dedicated teams to stand up and operate.</li><li><strong>Inadequate tooling</strong> — Business intelligence dashboards that work well for internal use cases lack the customizability required to feel fully native when applied to a customer-facing data analytics use case.</li></ol><p>My co-founders and I faced these challenges at Twilio where we were early product and engineering leaders. Customers loved our APIs but needed insight into usage, spend, and deliverability in order to troubleshoot with autonomy and give them the confidence to accelerate adoption. In the early days, we had grand ideas to visualize ‘Google Analytics’-like features in the Twilio Console. Unfortunately, these fell flat when we realized that we needed new data infrastructure that we quickly learned was out of our reach.</p><p>It wasn’t until years later that Tyler Wells and Mark Roberts (my co-founders) built the first<a href="https://www.twilio.com/voice/insights"> Voice Insights </a>product for WebRTC at Twilio. Voice Insights is a call quality analytics product that gives customers a complete picture of call performance as calls move through the customer’s networks, Twilio’s infrastructure, and carrier networks. It started by collecting telemetry from the WebRTC endpoints, processed it in real-time, and gave customers metric dashboards to understand what networks or individual agents had connectivity issues. Today, you’d be irresponsible to run a sizable Twilio Voice deployment without Voice Insights. But the biggest learning — it turned customer frustration into customer empowerment.</p><p>Even for the best companies like Twilio, building data products that use large-scale analytical data requires a significant engineering investment and takes years to build. When we left Twilio at the end of 2020, hundreds of engineers were working on the data products across the company. This is not unique to Twilio.</p><p>We asked ourselves — does it have to be this hard? Could the complexity be delivered as an API so that every company can deliver in-product analytics without fleets of engineers and investment?</p><h3>Introducing Propel Data! 🎉</h3><p>Propel Data is an Analytics API Platform for developers to easily build data products powered by large-scale analytical data. It enables developers to quickly build usage metering, insights dashboards, product metrics, and advanced reporting without the overhead of managing additional complex data infrastructure or creating new data silos.</p><p>Propel’s GraphQL API and serverless infrastructure make building and running data products seamless. It leverages the data you already have in your data warehouse, operationalizes it, and serves it via highly available low-latency APIs.</p><p>Building large-scale data analytics into the product features requires sophisticated data infrastructure because customer-facing products have unique requirements:</p><ul><li>⚡️ <strong>Low latency →</strong> Customers expect fast, snappy product experiences. While internal employees may be okay waiting 45 seconds or even minutes for a query to run, this latency is a no-go for customer-facing products where the data analysis is part of the core workflow that the product enables.</li><li>🌊 <strong>High concurrency →</strong> Products need to support thousands or millions of users. Because data products ultimately serve customers rather than employees, they must support a dramatically higher number of concurrent requests than what internal data tools are designed to handle. And they must do so seamlessly and cost-effectively.</li><li>📜 <strong>APIs</strong> <strong>→</strong> A contract in code between product and data teams. Shipping a modern data application requires collaboration between data and the product development teams (frontend &amp; backend). APIs codify the contract and SLAs that allow the separation of concerns that keep the product development teams focused on solving customer problems.</li><li>🔐 <strong>App-centric security layer →</strong> Each end-user can only see their own data. In SaaS and consumer products, end-users access their own data from the web and mobile applications. This requires an app-centric security layer for multi-tenant or consumer environments, not just the employee-centric role-based access control.</li></ul><p>With Propel you can build in days what has taken leading companies years and hundreds of engineers to build.</p><h3>What are customers saying? ❤️</h3><p>Over the last year, we have been working with around a dozen design partners to build Propel. We are thrilled to share that <a href="https://www.courier.com/">Courier</a>, the leading developer notifications platform, will be powering their notifications insights with Propel.</p><blockquote>“Analytics was one of the top feature requests that always got punted due to the complexity it involved. With Propel, we were able to ship the analytics we always wanted without having to staff new teams or manage additional infrastructure, all in a couple of weeks.” <em>— Seth Carney, CTO at Courier</em></blockquote><h3>How do I get started? 🚀</h3><p><a href="https://www.propeldata.com/join-waitlist">Join our waitlist!</a> We are onboarding customers as fast as we can. We are starting with Snowflake customers and will be adding support for AWS Redshift, GPC BigQuery, and Databricks in the coming quarters.</p><p>Stay tuned! Follow us on <a href="https://twitter.com/propeldatacloud">Twitter</a> or subscribe to our <a href="https://www.propeldata.com/newsletter">product updates</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed2b3277b237" width="1" height="1" alt=""><hr><p><a href="https://medium.com/propel-data-analytics-blog/why-we-built-propel-data-ed2b3277b237">Why we built Propel Data</a> was originally published in <a href="https://medium.com/propel-data-analytics-blog">Propel Customer-Facing Analytics Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>