<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by ajo on Medium]]></title>
        <description><![CDATA[Stories by ajo on Medium]]></description>
        <link>https://medium.com/@ajoabraham?source=rss-870610477712------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*pw0sRvs29KcvMdcUFLLQhg.jpeg</url>
            <title>Stories by ajo on Medium</title>
            <link>https://medium.com/@ajoabraham?source=rss-870610477712------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 11:43:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ajoabraham/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Chapter 1: Revealing the Secret Language of Data Engineers]]></title>
            <link>https://medium.com/@ajoabraham/chapter-1-revealing-the-secret-language-of-data-engineers-bd99e8b7a973?source=rss-870610477712------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd99e8b7a973</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[ajo]]></dc:creator>
            <pubDate>Fri, 06 Sep 2019 01:01:33 GMT</pubDate>
            <atom:updated>2019-09-06T01:01:33.873Z</atom:updated>
            <content:encoded><![CDATA[<h4>Untitled Book on SQL for Analysts</h4><p><em>This is the first chapter of a book I’m thinking about writing. Not sure if the world needs yet another book on SQL. Let me know in the comments if its worth pursuing. Scroll to the bottom to learn more about the content I plan for the book.</em></p><h3>Overview</h3><p>First things first, you need to learn the lingo. Sure, we could jump right into syntax like most books, but this isn’t one of those books. Syntax is easy, you’ll see. Pro’s don’t get paid the big bucks because they know basic SQL syntax. They get paid for their ability to wrangle data and produce answers.</p><p>In this chapter you’ll learn core architectural concepts that underpins a professional data platform. Most of the concepts described here will make their way into a data driven culture. If you find yourself in a poorly run data environment, these concepts should help you influence your partners in the right direction. Or, if you’re in leadership this book will help you build the right team.</p><h3>The Who&#39;s Who</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MOecfcOhOWf3qJLmGNq0lg.png" /></figure><p>Your data team might look like the structure above. Or, it might be just one guy: You. Team shape, size, and structure varies across companies. Some companies may have additional roles like Quality Assurance or may collapse Business Intelligence (BI) Engineer and Visualization Developer into a single role. Having a sense of key roles and responsibilities will help you seek out the right people for the right help.</p><h4>Data Engineers</h4><p>Data Engineers deal with raw source data. It’s their job to write code and build pipelines to move data from source systems into data warehouses. When you need to understand business logic and no one can answer your question, seek out the data engineer. At the very least, they’ll be able to look at their code and explain the logic applied.</p><h4>Data Warehouse (DWH) Architects</h4><p>The modern big data lake based Data Warehousing strategy utterly lacks any craftsmanship in design and usability. Our craftsmen were the DWH Architects. They planned the overall layout of the DWH and how various data domains can seamlessly interact. They tend to be people with the broadest knowledge of the data warehouse from a business and technical POV. However, since the advent of big data (running queries across 100’s of cheap computers), the importance of these roles have diminished. We sort of threw out the baby out with the bath water. The good news is that people are recognizing the limits to the kitchen sink approach to DWH design. And I predict a resurgence of these roles. While performance is no longer a big challenge, usability still prevents data access to everyone.</p><p>If you are lucky enough to have architects in your company, leverage them to understand core tables, how to join tables, and the business rules you should be aware of when running your analysis.</p><h4>Data Scientists</h4><p>If you’re in the business of data, you already know that the Data Scientist title is thrown around liberally. In some companies it could be held by Analysts or Business Intelligence (BI) Engineers. And sometimes its justifiable. Essentially a Data Scientist is someone with advanced statistics and mathematics skills. They maybe running A/B tests (simplest case) or Machine Learning algorithms (advanced case) for a variety of scenarios.</p><h4>Business Intelligence (BI) Engineers</h4><p>This role tends to have some overlap with Data Engineers, Visualization developers, and Analysts. Fundamentally, someone with this title will be an expert on one or more Business Intelligence tools (e.g. Tableau, MicroStrategy, or Looker). They will likely have some data knowledge across one or more data domains. You’ll frequently find them pulling data with direct SQL queries, developing Reports, and building Dashboards in a BI tool. As an analyst, BI engineers will be your key technical partner. You will work with them to automate reports and dashboards.</p><h4>Visualization Developers</h4><p>In recent years, data driven companies have turned to Custom Visualizations for their executive dashboards. As the data industry tries to further democratize data, the need for purpose built apps are growing. Custom Visualizations deploy the latest Javascript frameworks on the frontend to create an intuitive and rich experience with data. This role is quite rare and you are unlikely to encounter it unless you work in the Tech industry. Now, if you do have viz developers on your team then your in luck. Visualization developers, or rather the dashboards they’ve developed, will be your first stop for getting at the core numbers and vetting your own numbers. You might even partner with them to build out core dashboards for your domain.</p><h3>Components of a Data Pipeline</h3><p>Where does all this big data come from and how does it get to you? This section will cover the origin of data and the processes it goes through before getting to you. You’ll finally know why it takes so long to get data you need in the way you need it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CkmO2kkFGCInTgawuqAgRQ.png" /><figcaption>Simplified ETL Pipeline</figcaption></figure><h4>Source Systems</h4><p>It all starts with systems that generate data. Typically, a data record is triggered by a real world event. When you buy something, ship something, watch something, or even when you visit a website a record is created and stored in a database. Data engineers call these databases Source Systems.</p><p>A source system (in the context of analytics) is the database layer of an applications like your company’s website, HR system, or inventory management tool. Any database that captures or creates original data could be used as a source system. Data Engineers will interchangeably use words like Source System, Operational System, or “prod DB” to refer to places where important data reside.</p><p>In many organizations there are important Excel and CSV files that are also considered source systems. These files may come from third parties or they could be generated by other teams within your company. Technically speaking, any data source be it a database or a file that is an input to an analytics pipeline is a source system.</p><h4>Extract Transform and Load (ETL)</h4><p>The suite of code, queries, and tools used by your Data Engineers are for the explicit purpose of building Extract Transform and Load (ETL) pipelines. Unless you’re working in a small startup or on a non-critical application, you never want to run analytical queries on source systems. The reasons are many fold including the fact that analytical queries are extremely resource intensive on operational data models (more on this later). One bad query could bring a database to a screeching halt. No database equals no transaction equals no revenue! Angry DBA (Database Admin/IT Guy) means “no data for you.” For this reason, a basic data environment will at least include the ‘E’ and ‘L’ of ETL. In the ‘E’ and ‘L’ scenario, source system data is Extracted and Loaded into a separate database for analytics.</p><p><em>In small data environments this E and L could simply be a read replica of the source database. A read replica is a snapshot of a source database taken at fixed periods throughout the day.</em></p><p>The Transform (T) in ETL is where much of the value in Data Warehousing can be exploited. In a typical data team, Data Engineers will take an operational data model and <strong><em>artfully</em></strong> transform it into an analytical data model. Data from source systems is extracted into a staging area (computers and databases from where complex transformation can be run in isolation), transformed, and then loaded into a separate database usually referred to as the Data Warehouse (DWH). Notice I said <strong><em>artfully</em></strong>. It takes many iterations of reviewing business rules and writing code to maintain data for a domain.</p><p>The transformation (E<strong><em>T</em></strong>L) step could be used to fix problems with the source data, enrich data with more information, convert data from one type to another, or create new data. Lets look at a few examples to drive this home.</p><h4><strong>Fixing Data.</strong></h4><ol><li>The inventory database includes products that were never sold by your company. Perhaps these were entered as samples but never removed. These should be eliminated in the transform step so we get an accurate product count.</li><li>The customer service agent data is sometimes missing the agent hire date. This is manually entered and service centers sometimes forgets to enter the field. The transform step can derive the hire date based on when the agent first took a call and subtract 14 days from it. Rules like this are developed by working closely with the business teams who would know that an Agent can only take calls 14 days after hiring.</li></ol><h4>Enriching Data.</h4><ol><li>Multiple source systems capture product description data with varying levels of quality. The transform step can asses the quality of the description field based on description length, source, and person who created the description. Based on these factors the best description can be chosen.</li><li>Attach demographic data to customer records. These may include estimated income, gender, net worth, home value, debt levels, etc.</li></ol><h4>Converting Data.</h4><ol><li>Convert IP addresses into city, state, and country columns. This might require the acquisition of 3rd party data.</li><li>Survey data may come in as 0 for true and 1 for false. For better business context these could be converted to True/False or Yes/No.</li></ol><h4>Creating New Data.</h4><ol><li>Defining a new metric based on business rules to track performance. For example, in a CRM context a Lead Follow Up Rate could be created as an additional metric for Sales efficiency. Here valid leads might mean those with a valid phone number or email on file. This would likely have more rules requiring the metric to be generated in the ETL layer rather than in visualization tools.</li></ol><h4>Databases</h4><p>Technically anything that stores records could be considered a database. Your Microsoft Excel workbook, a csv file, or even a paper ledger could be a database. What makes a Database Management System (DBMS) though is its ability to scale to 100’s of concurrent queries, control user access, and deliver data from 1000’s of tables. Moreover, the most popular databases in the world for analytics are RDBMS’s where the “R” stands for Relational. In addition to the capabilities of a DBMS, Relational databases support SQL queries, table joins, and other Set operations. The theoretical foundation of these operations are given by <a href="https://en.wikipedia.org/wiki/Relational_algebra">Relational Algebra</a>. Hence the term relational.</p><p>One final concept that is useful to be aware of is the existence of NoSQL databases. Examples include MongoDB, Cassandra, CouchDB, and many more. These databases have become very popular for source systems and are designed to scale horizontally across many machines. They excel at single record write and read over massive data sets. In our world of data analysis, we are dealing with aggregations across many records and in the case of big data that could mean trillions of rows. Data analysis also tend to span across multiple tables. Relational databases that support SQL and table joins are still the best technology for analytical data. Furthermore, with advancements in database technology we have relational databases today that can scale across many machines and can physically store data in ways optimized for analytics.</p><h3>The Data Model</h3><p>Data models describe how collections of tables are organized and related to one another in a database. You can think of a data model as a map that describes how you can get data from point A to point B (i.e how to get sales numbers combined with customer location and product inventory count). Source system data models are highly optimized for single record read and write. In Data Engineer (DE) speak, this is referred to as a ‘highly normalized’ model.</p><p>A normalized table represents a collection of real world concepts. For example, in a normalized <strong><em>customers</em></strong><em> </em>table each row represents one and only one customer entity per row. Each row in this table would be unique and have a unique column like a <strong><em>customer_id</em></strong>. This type of column is called a <strong><em>primary key</em></strong>. A simple website that sells shoes may only require a unique id, first name, last name, email, zip code, and password to represent a real world customer. When a customer buys a shoe, a record will be created in the <strong><em>orders </em></strong>with reference back to the customers table via the <strong><em>customer_id </em></strong>column<em>. </em>This type of reference is called a <strong><em>foreign key</em></strong>. Now when ever you need to find the hottest zip code for shoes you have to join the <strong><em>orders </em></strong>table with the <strong><em>customers </em></strong>table on <strong><em>customer_id.</em></strong></p><p>On the flipside, a Data Warehouse (DWH) would employ a denormalized model to speed up analytical queries. Instead of leaving the zip code in the <strong><em>customers </em></strong>table it would be faster and better to push it into the <strong><em>transactions</em></strong> table for analytics. Its faster because we no longer have to join two massive tables and its better because now even if the customer moves zip codes we can attribute the sale to the correct zip code.</p><p>All of the pro’s have a solid understanding of data warehouse data models. You should too! We just scratched the surface here. Things are about to get interesting.</p><p><em>That’s it for my first draft of the chapter 1. Here’s a look at some of the planned chapters (not exhaustive):</em></p><ul><li><em>Unraveling the Data Warehouse</em></li><li><em>Essential Syntax</em></li><li><em>Thinking in Sets with Practical Examples</em></li><li><em>Blending Data Across Fact Tables</em></li><li><em>Debugging Data</em></li><li><em>Loading CSV Files</em></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd99e8b7a973" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What are Custom Visualizations?]]></title>
            <link>https://medium.com/@ajoabraham/what-are-custom-visualizations-f70681891943?source=rss-870610477712------2</link>
            <guid isPermaLink="false">https://medium.com/p/f70681891943</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[custom-visualization]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[ajo]]></dc:creator>
            <pubDate>Tue, 02 Jan 2018 14:30:25 GMT</pubDate>
            <atom:updated>2018-01-02T14:30:25.758Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sBut3YBPBX3M0MmlduZndQ.png" /><figcaption>Mock up of a Generic Call Center Operation</figcaption></figure><p>Custom Visualizations are purpose built dashboards implemented as a web or mobile application. We build them to provide a better user experience, handle ungodly volumes of data, or to dramatically reduce cost for certain use cases. They may be embedded in other applications or stand alone. Generally, they do not rely on commercial products but is rather built on open source platforms.</p><p>I personally don’t think Custom Visualization is the best term to encapsulate this concept, but it is the term currently in use. The problem with it is you imagine a fancy chart type rather than a complete app. I prefer the term Dashboard App or Big Data App. Even these are not prefect because you might imagine mobile specific use cases.</p><h4>Components of a Custom Visualization</h4><ol><li>A web development platform (even in mobile apps you need a web backend). This could be Node.js + Express, Ruby on Rails, Flask, Java Struts, Groovy on Rails etc.</li><li>Source Data. Typically this is your data warehouse or big data platform like Hive or Presto.</li><li>Cache Storage Layer. One of the reasons we build Custom Visualizations is for a better user experience. This means snappier UI’s. <a href="https://medium.com/@ajoabraham/cache-storage-options-for-your-next-custom-visualization-818ee0ee93cc">See my earlier post on cache storage options.</a></li><li>Data source connectors for both your Cache Layer and your source data (ie data warehouse).</li></ol><p>The above are the essential building blocks of a custom viz, however, you may have other components. The additional layers could be a metadata manager, cache loader, mailer, and background services.</p><p>Read these for more on the Custom Visualization wave sweeping Silicon Valley and beyond:</p><ul><li><a href="https://medium.com/@ajoabraham/forget-build-vs-buy-its-build-and-buy-dfd6dcd28d0a">Forget Build vs Buy — Its Build and Buy</a></li><li><a href="https://medium.com/@ajoabraham/scaling-bi-with-custom-visualizations-5a3f4479d73">Scaling BI with Custom Visualizations</a></li><li><a href="https://medium.com/@ajoabraham/custom-visualizations-skeptic-to-champion-da4a4c65364d">Custom Visualizations: Skeptic to Champion</a></li><li><a href="https://medium.com/@ajoabraham/viz-engineers-think-like-a-product-manager-6903080ce5d9">Viz Engineers: Think like Product Managers</a></li><li><a href="https://medium.com/@ajoabraham/product-managing-a-custom-visualization-41f150c75f13">Product Managing a Custom Visualization</a></li><li><a href="https://medium.com/@ajoabraham/a-custom-visualization-is-much-more-e8267417603a">A Custom Visualization is Much More</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f70681891943" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ugly Dashboards Destroy Credibility]]></title>
            <link>https://medium.com/@ajoabraham/ugly-dashboards-destroy-credibility-efa5f39d5846?source=rss-870610477712------2</link>
            <guid isPermaLink="false">https://medium.com/p/efa5f39d5846</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[dashboard]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <dc:creator><![CDATA[ajo]]></dc:creator>
            <pubDate>Wed, 27 Dec 2017 16:44:01 GMT</pubDate>
            <atom:updated>2017-12-27T16:44:01.841Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ANs_0vvyi8D-VSyPYGTD-w.png" /></figure><p>This fact should be obvious to most Business Intelligence (BI) and Data Visualization (DV) professionals by now, but sadly its at best an afterthought. The ideas of design, user experience, and craftsmanship are rarely considered in BI organizations. When these topics come up they are quickly dismissed as “ just eye candy.” As if that isn’t half the problem you’re trying to solve when delivering well adopted data products.</p><p>For the most part this should be common sense. We all have an innate ability to detect when time and energy was put into something. Whether a pencil or a dashboard the quality of craftsmanship is obvious to most. You don’t even need to know much about big data to make basic judgments about craftsmanship. System 1 thinking, according to Daniel Kahneman, helps us make quick autopilot judgements. Unfortunately ugly dashboards triggers our System 1 thinking to doubt the veracity of the data we present. Our judgement on quality and craftsmanship is instant and a poorly designed dashboard is sure to be met with skepticism. Even if the data you present are precise and useful, it lacks gravitas when delivered on a poorly crafted canvas. This type of design makes it easy for decision makers to lean more on gut feeling when good data is available.</p><p>A well crafted dashboard would at minimum consider the following 7 components of design :</p><ol><li>General use of whitespace</li><li>Padding</li><li>Font styles</li><li>Color Scheme</li><li>User interaction and flow (navigating through data)</li><li>Visualization design (matching data to the best viz)</li><li>Contemporary design styles</li></ol><p>Many of the ideas above are obvious except number 7. This is one of the biggest issues with off the shelf BI tools. Their overall design style is fixed in time to the last pivotal release. This often doesn’t correspond to major release numbers, but to the last paradigm shift. For example, product A decides to embrace in memory OLAP engines and release a revamped product at version 9. You will see the essential design philosophy carried out for the next 10 versions there after. This means when we are debating material design vs flat design, and the latest font styles, they are still pondering 3d buttons with gradients. The problem here is that your well thought out dashboard is stuck inside the outdated chrome of a previous style era. Like fashion, user interface styles and design philosophy evolve over time. In this context, your dashboard is still ugly compared to other apps used by your users.</p><p>The screen shot above is a dashboard likely from the early 90’s. At that time this must have wowed many an executive. But since then, we’ve gotten better at creating beautiful interfaces and our design philosophies have evolved. Today this same design looks horrible to our eyes only because we are surrounded by well crafted user interfaces.</p><p>Your stakeholders are unlikely to come out and complain about the look and feel of your dashboards. Instead, you’ll see poor adoption, lack of trust in the data quality, and nit picking of seemingly irrelevant details. Custom visualizations always have an advantage when it comes to design. Its a good idea for your team to consider adding at least one custom viz expert to work on high impact use cases.</p><p>Read these for more on the Custom Visualization wave sweeping Silicon Valley and beyond:</p><ul><li><a href="https://medium.com/@ajoabraham/forget-build-vs-buy-its-build-and-buy-dfd6dcd28d0a">Forget Build vs Buy — Its Build and Buy</a></li><li><a href="https://medium.com/@ajoabraham/scaling-bi-with-custom-visualizations-5a3f4479d73">Scaling BI with Custom Visualizations</a></li><li><a href="https://medium.com/@ajoabraham/custom-visualizations-skeptic-to-champion-da4a4c65364d">Custom Visualizations: Skeptic to Champion</a></li><li><a href="https://medium.com/@ajoabraham/viz-engineers-think-like-a-product-manager-6903080ce5d9">Viz Engineers: Think like Product Managers</a></li><li><a href="https://medium.com/@ajoabraham/product-managing-a-custom-visualization-41f150c75f13">Product Managing a Custom Visualization</a></li><li><a href="https://medium.com/@ajoabraham/a-custom-visualization-is-much-more-e8267417603a">A Custom Visualization is Much More</a></li></ul><p><em>Ajo Abraham is a big data expert known for building beautiful, fast, and high impact custom viz apps. For consulting requests you can email him here</em><a href="/service/mailto:ajo@veroanalytics.com"><em>ajo@veroanalytics.com</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=efa5f39d5846" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cache Storage Options for your Next Custom Visualization]]></title>
            <link>https://medium.com/@ajoabraham/cache-storage-options-for-your-next-custom-visualization-818ee0ee93cc?source=rss-870610477712------2</link>
            <guid isPermaLink="false">https://medium.com/p/818ee0ee93cc</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[ajo]]></dc:creator>
            <pubDate>Wed, 20 Dec 2017 17:48:27 GMT</pubDate>
            <atom:updated>2017-12-20T17:48:27.526Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gdLXJltiYAMP9WZZFEQI1Q.jpeg" /><figcaption>Files, RDBMS, Elasticsearch, Druid, MapD</figcaption></figure><p>There are so many options for supporting fast interactive Custom Viz apps. I’m going to cover some that I’ve looked into or actually used in my projects.</p><h4>Precomputed Files</h4><p>If the set of dimensions and measures are small and the number of rows are small, you can reasonably pre calculate several mini cubes into files (often JSON format). For a personal project, I also developed a client side OLAP library to further filter/pivot the data in the browser. You have to consider bandwidth and file size in this case. Larger files will take longer to download.</p><h4>Pros</h4><ul><li>No external setup dependencies</li><li>Super snappy for small data scenarios</li><li>Quick to get up and running</li><li>Free. No additional software expenses</li></ul><h4>Cons</h4><ul><li>File size has to be limited for fast downloads which leads to the generation of numerous files. This could become difficult to manage.</li><li>Harder to debug data issues within files without moving it to some queryable interface</li><li>OLAP functions such as filtering, grouping, windowing etc has to be hand coded</li></ul><p><em>I plan to refactor my client side OLAP lib and open source it someday soon. Stay tuned for that by following me on medium or twitter.</em></p><h4>Relational DB (Postgres)</h4><p>When we are talking about dashboards or Custom Visualizations we are talking about a subset of your data. This typically ranges from 2GB to 10GB and sometimes more. Relational DB’s can provide sub second response times if architected for the dashboard in question. One of the gotchas are going to be table width. The wider your table the more IO will be needed to process your data. An RDBMS like Postgres stores data in a row oriented fashion and as such will retrieve the entire row from disk to complete the request.</p><h4>Pros</h4><ul><li>Well known technology and ecosystem (drivers, connectors, and automation tooling)</li><li>SQL</li><li>Builtin analytical functions (windowing, stats etc)</li><li>Can join across many tables</li><li>Performance can be sub second for well architected systems</li><li>Battle tested for decades</li></ul><h4>Cons</h4><ul><li>You must spend time planning, tuning, and optimizing your DB for your Viz use case. It can be done, but it takes some effort.</li><li>RDBMS are built for transactional use cases and are typically row oriented storage. As a result, analytical workloads are rarely sub second for bigger data volumes.</li><li>Poorly planned and architected systems will be very slow</li></ul><h4>Elasticsearch</h4><p>If you can afford it, Elasticsearch is one of the fastest options out there. Some of our indexes gets sub 100 millisecond response times. While it will perform well in many scenarios, it has a steep learning curve.</p><h4>Pros</h4><ul><li>Superfast!</li><li>HTTP endpoints make it easy to query from any programming language</li><li>Great documentation and well known in the big data community</li><li>Schema-less design makes it easy manage and update</li></ul><h4>Cons</h4><ul><li>Analytical workloads often involve aggregations. Elasticsearch aggregation syntax is convoluted</li><li>Aggregated results are not simple array of arrays. They are deeply nested JSON objects where the depth is dependent on the number of fields in the “group by”</li><li>Limited aggregation functions</li><li>Designed to deployed on a distributed cluster adding to infrastructure maintenance and management</li><li>Expensive</li></ul><h4>Druid</h4><p><a href="http://druid.io/">Druid</a> is rapidly emerging as the go to OLAP solution for major big data operations. It is built for distributed storage of data in memory but optimized for OLAP workloads. This means fast ingestion and aggregations. Druid claims Petabyte scale performance and is deployed at companies like Netflix, Alibaba, PayPal, and more.</p><h4>Pros</h4><ul><li>Designed for OLAP workloads</li><li>Distributed in memory architecture makes it possible to scale to Petabytes of data</li><li>Well adopted by major companies including Netflix etc.</li><li>Extremely fast queries</li><li>Runs on commodity hardware</li><li>Can use Javascript to transform and filter data</li></ul><h4>Cons</h4><ul><li>Complex query syntax</li><li>Very minimal join capabilities</li><li>Only basic aggregation functions ( sum,avg,count,max,min )</li></ul><h4>MapD</h4><p><a href="https://www.mapd.com/">MapD</a> is an in memory columnar DB that can utilize GPU for even faster query processing. It even supports full on SQL! I found it extremely promising as a storage layer than can <a href="https://medium.com/@ajoabraham/scaling-bi-with-custom-visualizations-5a3f4479d73">ride with your Custom Visualization</a>. You can use the community edition or upgrade to an enterprise option.</p><h4>Pros</h4><ul><li>Full SQL support including windowing functions!</li><li>Very fast for certain data sizes making it optimal for Custom Viz workloads</li><li>Super easy to setup and load data from your existing data warehouse (SQL Importer tool is great)</li></ul><h4>Cons</h4><ul><li>New kid on the block with not many notable customers yet</li><li>Connector ecosystems needs improvement</li><li>A few rough edges that will surely improve with time</li><li>Poor documentation regarding HTTP api</li></ul><p>Read these for more on the Custom Visualization wave sweeping Silicon Valley and beyond:</p><ul><li><a href="https://medium.com/@ajoabraham/forget-build-vs-buy-its-build-and-buy-dfd6dcd28d0a">Forget Build vs Buy — Its Build and Buy</a></li><li><a href="https://medium.com/@ajoabraham/scaling-bi-with-custom-visualizations-5a3f4479d73">Scaling BI with Custom Visualizations</a></li><li><a href="https://medium.com/@ajoabraham/custom-visualizations-skeptic-to-champion-da4a4c65364d">Custom Visualizations: Skeptic to Champion</a></li><li><a href="https://medium.com/@ajoabraham/viz-engineers-think-like-a-product-manager-6903080ce5d9">Viz Engineers: Think like Product Managers</a></li><li><a href="https://medium.com/@ajoabraham/product-managing-a-custom-visualization-41f150c75f13">Product Managing a Custom Visualization</a></li><li><a href="https://medium.com/@ajoabraham/a-custom-visualization-is-much-more-e8267417603a">A Custom Visualization is Much More</a></li></ul><p><em>Ajo Abraham is a big data expert known for building beautiful, fast, and high impact custom viz apps. For consulting requests you can email him here </em><a href="/service/mailto:ajo@veroanalytics.com"><em>ajo@veroanalytics.com</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=818ee0ee93cc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why In Memory BI Sucks]]></title>
            <link>https://medium.com/@ajoabraham/why-in-memory-bi-sucks-a14d7c648bbe?source=rss-870610477712------2</link>
            <guid isPermaLink="false">https://medium.com/p/a14d7c648bbe</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[custom-visualization]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[ajo]]></dc:creator>
            <pubDate>Tue, 12 Dec 2017 15:06:44 GMT</pubDate>
            <atom:updated>2017-12-12T15:06:44.760Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0rohIrz2sL7hRyYlOFvFjw.jpeg" /></figure><p>In a <a href="https://medium.com/@ajoabraham/scaling-bi-with-custom-visualizations-5a3f4479d73">previous post</a>, I discussed how you can scale BI with Custom Visualizations. Now, let’s take a look at why in memory cubes end up slow.</p><p>This is how your proof of concept architecture looked like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*3My_AKXPHelhG4PFfAqCDw.png" /></figure><p>Your dashboards and reports were snappy and your company happily put down some serious dough. However, this is what you ultimately ended up with as demand for dashboards and visualization grew:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*PqVkDnqBnUq5yhjMegusrw.png" /></figure><p>We have all done this. More and more cubes get built into our BI servers making our dashboards go from snappy to tortuous. As the number of in memory cubes increase, the average load time for dashboards (and the cubes backing them) increases.</p><p>The max throughput of an AWS (Amazon Web Services) SSD based storage device is 500 MB/S. A 10 GB cube will take 20 seconds to load from disk to memory:</p><pre>( 10 GB * 1024 MB/GB ) / 500 MB/S = 20.48 Seconds</pre><p>Twenty seconds doesn’t even include the time it takes to find and deallocate stale cubes, run any post processing, and finally generate the data for the current requested view. I personally have had 5GB cubes take over a minute to load and render the initial view of a dashboard.</p><p>Now you know why your in memory BI platform sucks. To be fair the problem is really our monolithic BI architectures. It would be great to deploy certain dashboards independently, but its not easy with the current set of tools. We’d have to decipher complicated license models, customize the navigation layers of the BI tool, and figure out how to automate deployments of developer unfriendly platforms. This is why I’m encouraging BI teams to build some muscle around Custom Visualizations.</p><p>Read these for more on the Custom Visualization wave sweeping Silicon Valley and beyond:</p><ul><li><a href="https://medium.com/@ajoabraham/scaling-bi-with-custom-visualizations-5a3f4479d73">Scaling BI with Custom Visualizations</a></li><li><a href="https://medium.com/@ajoabraham/custom-visualizations-skeptic-to-champion-da4a4c65364d">Custom Visualizations: Skeptic to Champion</a></li><li><a href="https://medium.com/@ajoabraham/viz-engineers-think-like-a-product-manager-6903080ce5d9">Viz Engineers: Think like Product Managers</a></li><li><a href="https://medium.com/@ajoabraham/product-managing-a-custom-visualization-41f150c75f13">Product Managing a Custom Visualization</a></li><li><a href="https://medium.com/@ajoabraham/a-custom-visualization-is-much-more-e8267417603a">A Custom Visualization is Much More</a></li><li><a href="https://medium.com/@ajoabraham/make-it-rain-save-millions-per-year-on-bi-with-this-one-weird-trick-9d5427aaf647">Make it Rain! Save Millions per Year on BI with this one Weird Trick</a></li></ul><p><em>Ajo Abraham is a big data expert known for building beautiful, fast, and high impact custom viz apps. For consulting requests you can email him here </em><a href="/service/mailto:ajo@veroanalytics.com"><em>ajo@veroanalytics.com</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a14d7c648bbe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Whats up with Bitcoin?]]></title>
            <link>https://medium.com/@ajoabraham/whats-up-with-bitcoin-e6c2d49c797b?source=rss-870610477712------2</link>
            <guid isPermaLink="false">https://medium.com/p/e6c2d49c797b</guid>
            <category><![CDATA[bitcoin]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <dc:creator><![CDATA[ajo]]></dc:creator>
            <pubDate>Fri, 08 Dec 2017 20:03:09 GMT</pubDate>
            <atom:updated>2017-12-08T20:03:09.952Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve been casually thinking and learning about Bitcoin, Cryptocurrency, and the Blockchain for more than two years now. Friends and acquaintances a like have been throwing questions my way at a feverish pace recently. The chart below probably has something to do with it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QBZXSmqOwJtQytnBYvhOfQ.png" /><figcaption>Coinbase BTC chart on 12/7/17 @ 7:29PM PST</figcaption></figure><p>Bitcoin is up more than 2000% since last year. This chart will likely be irrelevant by the time this article is posted. We are witnessing the fastest appreciation of an asset class ever. As a college graduate of 2003 who missed both the Dot Com bubble and the Real Estate bubble, I finally get my shot at a bubble. Are we at the tail end of the mania or the beginning? Who knows and who cares, am I right? Sadly, I only dipped half a pinky toe into crypto mania and only recently. I’ve been meaning to buy Bitcoin for two years and like everyone else I procrastinated until recently.</p><p>All kidding aside, whether Bitcoin succeeds or not, we are talking about a world changing phenomenon. Bitcoin might be the heir to Gold in the internet age, it might the ultimate medium of value exchange, or it might be nothing at all. But two important ideas have burst onto the scene and we ain’t ever going back: 1) a decentralized, independent, and purely digital currency, and 2) a distributed trust-less ledger aka the blockchain. Much ink has been spilled on the glories of the blockchain while simultaneously throwing shade at Bitcoin and Cryptocurrency in general. While the blockchain is a technologic leap the idea of a independent currency is revolutionary.</p><p>One reason people buy Gold and invest in Real Estate is to protect their hard earned money against financial mismanagement by Governments. Central authorities can deflate the value of your savings simply by printing more money (<a href="https://www.forbes.com/sites/realspin/2017/02/03/why-venezuelas-currency-crisis-is-a-case-study-for-bitcoin/#1f9ef2eb19b2">See Venezuela</a>). This forces you to either invest or spend. I personally believe a mathematically limited digital asset like Bitcoin will eventually displace Gold as a store of value. Digital currency is easier to transport, will be easier to liquidate, and can evade Authoritarian governments. In any case, money is simply an agreement between me and you regarding the value of something. It has been sea shells, sticks, silver, gold, paper, and now digital cryptocurrency.</p><p>Right now, the frenzy is driven by the Fear of Missing Out (FOMO). However, I’m still bullish on Bitcoin in the very long run. It is still difficult to acquire and there is a massive learning curve. In the past few weeks, the most popular US exchange, Coinbase, has experienced several periods of outage. As awareness of Bitcoin builds worldwide it is only likely to continue upwards until the bubble officially pops. Be vigilante and start learning.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e6c2d49c797b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Forget Build vs Buy — Its Build and Buy]]></title>
            <link>https://medium.com/@ajoabraham/forget-build-vs-buy-its-build-and-buy-dfd6dcd28d0a?source=rss-870610477712------2</link>
            <guid isPermaLink="false">https://medium.com/p/dfd6dcd28d0a</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[ajo]]></dc:creator>
            <pubDate>Fri, 08 Dec 2017 03:20:22 GMT</pubDate>
            <atom:updated>2017-12-08T03:22:03.394Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AYHE6ZEPMQTo96mS_W0ITA.png" /></figure><p>The perennial topic in BI circles and enterprise software in general is whether one should build or buy. While most of the arguments I present here is applicable to enterprise software in general, I’m going to focus on my area of expertise: Business Intelligence.</p><p>In 2017, the question of build vs buy is simple minded. With the rapid commoditization of Visualizations, the wide spread availability of web development frameworks, and the automation capabilities of the cloud — we are asking the wrong question. Instead we need to be looking at what is the most efficient way to deliver <strong>persuasive insights</strong> to the target audience. This means for some use cases we buy and for others we build.</p><p>For rapid iteration and data discovery, without a question, tools like Tableau are productivity boons. But when it comes to widespread dissemination of critical insights, generic out of the box solutions are subpar. We all know of decision makers leading fortune 500 companies that frequently avoid the confusing clutter of modern BI tools. Who could blame them for not figuring out which subfolder contains the dashboard with that one metric and dimension?</p><p>On the other hand, companies small enough to effectively communicate and collaborate should rarely build. Even if their execs are resistant to BI tools, you can be sure information will get to them. In the small company case, building will only make sense when BI is part of a product.</p><p>So when should you augment your BI stack with a build?</p><h4>BI is core feature of your product.</h4><p>This is a no brainer. Out of the box solutions will be clunky and slower compared to a purpose built data product. You don’t want your product to be held hostage to a third party vendor.</p><h4>You have thousands of End of Line Business Users.</h4><p>In all likelihood your classic BI team will push back on this. They will say that a Custom Visualization app will be slower to develop, costlier to maintain, and outside the capabilities of the team. I thoroughly dispatch these arguments <a href="https://medium.com/@ajoabraham/make-it-rain-save-millions-per-year-on-bi-with-this-one-weird-trick-9d5427aaf647">here</a>, <a href="https://medium.com/@ajoabraham/hammer-boys-and-custom-visualizations-f31b6a93ede3">here</a>, and <a href="https://medium.com/@ajoabraham/custom-visualizations-skeptic-to-champion-da4a4c65364d">here</a>. In the aforementioned posts I explain how you can save millions of dollars per year with a Custom Viz app and deliver a more effective product — one that will actually lead to data driven decision making.</p><h4>Big Data. The Biggest Data, Believe me!</h4><p>Are you a Google, Netflix, Uber, AirBnB, or some other Silicon Valley unicorn? Then you are likely to have massive amounts of data beyond the capabilities of most of the shelf BI tools. These companies are literally minting money by expertly utilizing the troves of data they’ve collected. Okay, maybe your company doesn’t have the same name recognition, but has amassed equally impressive volumes of data. Whats the point if exploring said data is dreadfully slow? It will just cause smart Product Managers and Analysts to avoid it like the plague. To best leverage data at this scale you must constantly explore new and better ways to store, organize, and deliver this data.</p><h4>Maximize data driven decision making at the Leadership level.</h4><p>Out of the box BI solutions are designed for generic use cases and users. When your company is angling towards critical objectives, it is important to get everyone on the same page. At exec levels politics and gut instincts runs wilder than people realize. To cut through the crap and align the executive team, its important to have a well crafted dashboard. Here you will carefully plan navigation, layout, and interaction such that no one can claim ignorance. Let data actually be the arbitrator of disputes.</p><p>I hope you see now that it is simply not about build or buy. You certainly buy and occasionally build.</p><p><em>Ajo Abraham is a big data expert known for building beautiful, fast, and high impact custom viz apps. For consulting requests you can email him here </em><a href="/service/mailto:ajo@veroanalytics.com"><em>ajo@veroanalytics.com</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dfd6dcd28d0a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Make it Rain! Save Millions per Year on BI with this One Weird Trick]]></title>
            <link>https://medium.com/@ajoabraham/make-it-rain-save-millions-per-year-on-bi-with-this-one-weird-trick-9d5427aaf647?source=rss-870610477712------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d5427aaf647</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[ajo]]></dc:creator>
            <pubDate>Sat, 02 Dec 2017 16:12:40 GMT</pubDate>
            <atom:updated>2017-12-02T16:12:40.264Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9ItOTFP11kSqWwMEhk4q0A.png" /></figure><p>Okay, the title is a little click baity but I’m actually going to show you how to save a million bucks on BI. This will only work for teams that deliver end of line business users with dashboards and visualizations. These users are typically retail staff, call center employees, field workers etc. For this example let&#39;s suppose we have 3,000 such users and we are comparing against an equivalent Tableau deployment. Many companies using older BI tools might pay much more than the estimates below.</p><h4><strong>Annual Tableau Costs</strong></h4><ul><li>3000 (Users) x $420 (Tableau Server per Year) = $1,260,000</li><li>$840 (Tableau Desktop per Year)</li><li>$150,000 (Sr. BI Developer per Year) * 25% (Time per year) = $37,500</li><li><strong>Total = $1,298,340 per year</strong></li></ul><h4>Annual Cost for Replacement Custom Visualization</h4><ul><li>$250,000 (Sr. Visualization Engineer) * 30% (Time per Year) =<strong> $75,000 per Year</strong></li></ul><p><em>A Sr. Visualization Engineer is a full stack developer with big data experience, basic UI/UX sensibilities, and some BI experience. Hard to find, but worth it when you do.</em></p><p>Look Ma, we just saved our company <strong>$1.2M per year! </strong>Custom visualization apps, when deployed for the right use cases, can deliver better user adoption and save us money!</p><p>Build vs Buys is a more complicated question now. You don’t want to “buy” for everyone, and you don’t want to “build” for every use case. If you have large isolated populations, you should consider building.</p><p>Read these for more on the Custom Visualization wave sweeping Silicon Valley and beyond:</p><ul><li><a href="https://medium.com/@ajoabraham/scaling-bi-with-custom-visualizations-5a3f4479d73">Scaling BI with Custom Visualizations</a></li><li><a href="https://medium.com/@ajoabraham/custom-visualizations-skeptic-to-champion-da4a4c65364d">Custom Visualizations: Skeptic to Champion</a></li><li><a href="https://medium.com/@ajoabraham/viz-engineers-think-like-a-product-manager-6903080ce5d9">Viz Engineers: Think like Product Managers</a></li><li><a href="https://medium.com/@ajoabraham/product-managing-a-custom-visualization-41f150c75f13">Product Managing a Custom Visualization</a></li><li><a href="https://medium.com/@ajoabraham/a-custom-visualization-is-much-more-e8267417603a">A Custom Visualization is Much More</a></li></ul><p><em>Ajo Abraham is a big data expert known for building beautiful, fast, and high impact custom viz apps. For consulting requests you can email him here </em><a href="/service/mailto:ajo@veroanalytics.com"><em>ajo@veroanalytics.com</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d5427aaf647" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scaling BI with Custom Visualizations]]></title>
            <link>https://medium.com/@ajoabraham/scaling-bi-with-custom-visualizations-5a3f4479d73?source=rss-870610477712------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a3f4479d73</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[ajo]]></dc:creator>
            <pubDate>Fri, 01 Dec 2017 14:01:11 GMT</pubDate>
            <atom:updated>2017-12-01T14:01:11.471Z</atom:updated>
            <content:encoded><![CDATA[<p>Even with the latest in memory cube technology, your dashboards aren’t snappy like Netflix.com or Google.com. Of course during the proof of concept stage, the BI vendor showcased his product in the best light. You were smart though and built real world dashboards to stress test the product. Everything looked great at first but your dashboards soon degraded to slow load times and the exact opposite of a snappy user experience.</p><p>This happens to every BI team. Time degradation of performance is really difficult to anticipate.</p><p>In the original test you had anticipated a handful of dashboards driven by even fewer in memory cubes. Your test may have looked something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*7oCMp4boVLJy_kg6aehhag.png" /><figcaption>In Memory BI Architecture Proof of Concept</figcaption></figure><p>However, overtime as you succeeded demand also increased. Your production environment now looks something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ojmplkT8o8foxUL_5uIRpQ.png" /><figcaption>Successful deployment of In Memory BI Architecture</figcaption></figure><p>Many dozens of cubes now power hundreds of dashboards. It’s wonderful to have such high utilization, but you will eventually see your <a href="https://medium.com/@ajoabraham/product-managing-a-custom-visualization-41f150c75f13">MAU % (Monthly active users as a percent of total possible users) declining</a>. Your platform goes from sort of snappy to several minutes to load.</p><p>At some point you will have more data volume than available memory space. Memory is cheaper but it&#39;s not free or even as inexpensive as disk. This means for many users, their cube won’t be loaded into memory at request time. The time to load a large cube into memory could take tens of seconds to minutes. We recently replaced an in memory powered dashboard with a Custom Viz. The in memory, off the shelf, dashboard used to take 1 to 2 minutes to load. The custom viz on the other hand is instantaneous.</p><p>Custom Visualizations gives you many architectural options. You could store your viz data in a separate data store, it can ride in the same instance as the viz, or it can use newer open source in memory stores like <a href="http://druid.io/blog/2011/04/30/introducing-druid.html">Druid</a>. Each option has variable costs and benefits. I’ll go into them in detail in another post. Here I’m going to propose the mode that is easiest to deploy and will work in the majority of cases: the data store rides with the viz on the same machine instance. Our new architecture might look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-3Bc63mXTOUdcS3AoWH5cA.png" /><figcaption>BI Architecture with Custom Visualizations</figcaption></figure><p>Notice I kept the traditional BI server in place but with fewer cubes. As I’ve argued before, custom visualizations might only represent about 30% of total visualizations/reports served, but it will command higher MAU’s and MAU %.</p><p>So, does this cost more?</p><p>Let&#39;s play out the extreme scenario where everything is a custom viz, the total volume of in memory data is 400GB, and the average cube is 10GB. Forty constantly running machine instances ( 400GB / 10GB per Viz = 40 ) should cost a lot more, right?</p><p>The best AWS option to house 400GB of data in memory on a single BI server is the memory optimized r4.16xlarge instance with 195 ecu’s and 488GB of total memory. <strong>At $4.742 per hour and 750 hours per month that comes out to a monthly total of $3,556.5</strong>.</p><p>For our Custom Viz, I can certainly go with very small instances with 2 ecu’s, but we should allocate equivalent compute power. As a result, I’m choosing the r4.large instance with 15.25GB of memory and 7 ecu’s. <strong>At $0.148 per hour and 750 hours per month times 40 instances we arrive at $4440.0 per month. </strong>It only costs 25% more in this extreme scenario. In reality I would choose much smaller instances for my Custom Viz with fewer ecu’s resulting in lower monthly costs. Even still, my 40 instance scenario affords me 610 GB of total memory capacity which is 25% more than the single instance scenario. <strong>On a per GB of memory basis the price is practically the same ($7.28 per GB of memory).</strong></p><p>You&#39;re probably thinking that managing 40 instances will be insanely costly or difficult. In 2017, you can run 1000’s of instances with a few clicks. I would be surprised if you weren’t already running 100’s of machines to power your Big Data infrastructure alone. This would mean you already have the necessary automation and orchestration capabilities. Congratulations, if that describes you. If not, now would be a good time to think about moving your infrastructure to the cloud.</p><p>Finally, you can certainly deploy this architectural strategy with classic BI tools, but I would imagine the licensing costs would make it cost prohibitive.</p><p>Remember to share this post or give me a clap when you celebrate your first Custom Viz launch!</p><p><em>Ajo Abraham is a big data expert known for building beautiful, fast, and high impact visualizations. For consulting requests you can email him here </em><a href="/service/mailto:ajo@veroanalytics.com"><em>ajo@veroanalytics.com</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a3f4479d73" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Product Managing a Custom Visualization]]></title>
            <link>https://becominghuman.ai/product-managing-a-custom-visualization-41f150c75f13?source=rss-870610477712------2</link>
            <guid isPermaLink="false">https://medium.com/p/41f150c75f13</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[ajo]]></dc:creator>
            <pubDate>Tue, 28 Nov 2017 16:31:01 GMT</pubDate>
            <atom:updated>2017-12-11T16:34:36.281Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KVXMKCvlnt5OgOUk00MKXQ.jpeg" /></figure><p>In my <a href="https://medium.com/@ajoabraham/viz-engineers-think-like-a-product-manager-6903080ce5d9">last post</a> I made a strong case for why Viz Engineers should think like Product Managers. This is likely a new frame on the world for many Viz and Business Intelligence engineers. I know, because as a Big Data developer of more than 10 years, I’ve only recently started using product management techniques for my dashboards and visualizations. While this advice is critical for Custom Visualizations, the same principles will help generic Dashboards succeed as well.</p><h4><strong>Step 1: Where’s the Beef</strong></h4><p>One of the first things a Product Manager thinks about is Value Proposition and Positioning. What problem does your Viz solve? A Viz can surface business opportunities, prescribe decisions, save time, identify problems, save money etc etc. Another way to think about it is what are the costs for you company if this Viz did not exist. A common example is the time saved by automating the aggregation of data into a single place. Without the Viz your FP&amp;A team spends nearly a quarter manually assembling data into a spreadsheet. In this case, decision makers can only make data driven decisions once a quarter. A Custom Viz or Dashboard will automate these insights and make the data available in near realtime.</p><p>Positioning is about how your solution fits into the current mode of getting at the same thing. Many times you are <strong>competing against</strong> manually aggregated spreadsheets. Off the shelf Dashboards could perform well in these cases. The new kid on the block is Custom Visualizations. Here we are<strong> positioning against</strong> manually aggregated spreadsheets and off the shelf generic dashboards. The case for Custom Viz comes down to Usability, Scalability, and Cost. For a detailed discussion of each of these read my first post on the topic:</p><p><a href="https://medium.com/@ajoabraham/custom-visualizations-skeptic-to-champion-da4a4c65364d">Custom Visualizations: Skeptic to Champion</a></p><p>At a primal level beautiful things are imbued with credibility and authority. Its human nature. Seven out of ten times a generic Dashboard is the way to go. However, if the data is critical to great outcomes, special treatment is warranted even if it misses on the need for Usability, Scalability, and Cost. <em>(More on this in a future post).</em></p><h4>Step 2: Whose the Boss</h4><p>Understanding your audience is critical to the success of any product. Buckets of ink has been spilled on this topic. I’m going to discuss some of the non-obvious aspects relevant to Viz Engineers.</p><p>Before you get started with the Viz, whether a generic dashboard or a Custom Viz, you want to meet with potential users. I usually run the meetings in an ad hoc focus group sort of way. By the time you get to such a meeting you should have developed a theory of product in your mind. At the very least you want to go in with a rough idea of the key metrics and any proposed new metrics. This will help move the conversation along.</p><p><strong>Key takeaways:</strong></p><ol><li>How do they gather this insight today <em>(positioning)</em>? Let them show you their workflow if possible.</li><li>Identify the most relevant dimensions and metrics and rank them if possible.</li><li>Discuss any proposed new metrics or views of data they have not seen before.</li><li>Develop more context around how the data will be used. What sort of decisions will this Viz power?</li></ol><p>This exercise will not only help you develop a great product but prepare you for a successful product roll out. For maximum impact, incorporate some of the best feedback into the UI. Then at launch, call out the feedback and mention the critical importance of getting the stakeholders involved early on. You just created champions for your Viz. If done correctly, they will promote it and accelerate adoption.</p><h4>Step 3: Lifecycle Management</h4><p>In the past, we developed generic dashboards, threw the kitchen sink at, and called it done. Maybe we’d occasionally update it, but for the most part it was one and done. It’s time to reframe that thinking. With a defined problem domain, value prop, and user feedback, we now have a roadmap. This means we can think of our Custom Visualization in terms of Versions and Release Dates. Naturally, the first launch of our Viz would be the MVP (minimum viable product). This should comprise of enough features from our roadmap to satisfy the essential value prop of the whole project. For Custom Visualizations and generic Dashboards, this means having views of data along the most important metrics and dimensions.</p><p>With the launch of Version 1 (aka MVP), feedback is likely to pour in. At this point, it would be a good idea to publish a public roadmap with tentative release dates and versions. This will keep your users and stakeholders invested in the success of your Viz.</p><p>Off the shelf Dashboards and Custom Visualizations will significantly diverge at say version 2 or 3. With generic Dashboards, we will hit a wall where adding capabilities will no longer be possible or make sense. I’ve spent a good portion of my career customizing generic BI tools. Often we reach a point where it would have been easier to develop a custom viz than hack poorly documented API’s of legacy platforms.</p><p>Finally, thinking in terms of Versions, Release Dates, and Roadmaps will reduce overall anxiety. All of your stakeholders can monitor progress and anticipate when their requests will be fulfilled. And you don’t have to throw the whole kitchen sink at your Viz in version 1.</p><h4>Step 4: Track the MAU %</h4><p>Typical SaaS applications focus on Conversion Rates, Lifetime Values, and Customer Acquisition Costs (CAC) to determine success.</p><p>How do you tell if your Custom Viz is succeeding?</p><p><strong>The most important metric is Monthly Active Users (MAU) as a percent of total possible users. </strong>We are dealing with a captive audience of a known size. Your competition is no data, manually aggregated data, or a generic Dashboards. A successful Viz should have a MAU % of 75% or higher and a great Viz should be in the 90% range.</p><p>A MAU % of 50 or lower means a Custom Visualization was probably not worth the effort. Of course one month is not determinant. You should monitor which way usage is trending.</p><p>If you started strong but ended with a declining MAU %, it might be time to rethink this project. Get back in a room with your stakeholders and top users and conduct another user study. Perhaps the data is no longer needed and the project should be sunsetted, or it’s time for a rebuild with a new perspective.</p><h4>Step 5: The Launch</h4><p>Getting the launch right is important especially when competing against status quo. If possible, the launch announcement should be made by the most influential stakeholder, i.e. an executive in charge of the problem domain. This will lend credibility of an authority figure and further invest key stakeholders in the projects success.</p><p>Finally, celebrate victories with your team. Pop the cork, you just launched Version 1!</p><p><em>Ajo Abraham is a big data expert known for building beautiful, fast, and high impact visualizations. For consulting requests you can email him here </em><a href="/service/mailto:ajo@veroanalytics.com"><em>ajo@veroanalytics.com</em></a></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2F8f5f8b%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2F8f5f8b&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/c43026df6fee7cdb1aab8aaf916125ea/href">https://medium.com/media/c43026df6fee7cdb1aab8aaf916125ea/href</a></iframe><figure><a href="https://becominghuman.ai/artificial-intelligence-communities-c305f28e674c"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*2f7OqE2AJK1KSrhkmD9ZMw.png" /></a></figure><figure><a href="https://upscri.be/8f5f8b"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*v-PpfkSWHbvlWWamSVHHWg.png" /></a></figure><figure><a href="https://becominghuman.ai/write-for-us-48270209de63"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*Wt2auqISiEAOZxJ-I7brDQ.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41f150c75f13" width="1" height="1" alt=""><hr><p><a href="https://becominghuman.ai/product-managing-a-custom-visualization-41f150c75f13">Product Managing a Custom Visualization</a> was originally published in <a href="https://becominghuman.ai/">Becoming Human: Artificial Intelligence Magazine</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>