<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Transparent Analytics - Medium]]></title>
        <description><![CDATA[Musings on data engineering &amp; Big Data analytics within digital marketing. - Medium]]></description>
        <link>https://medium.com/transparent-analytics?source=rss----8a4d28b70e9f---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Transparent Analytics - Medium</title>
            <link>https://medium.com/transparent-analytics?source=rss----8a4d28b70e9f---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 10:58:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/transparent-analytics" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Working with Docker for the first time]]></title>
            <link>https://medium.com/transparent-analytics/working-with-docker-for-the-first-time-68769f078607?source=rss----8a4d28b70e9f---4</link>
            <guid isPermaLink="false">https://medium.com/p/68769f078607</guid>
            <dc:creator><![CDATA[Bryan Dieudonne]]></dc:creator>
            <pubDate>Fri, 04 May 2018 01:25:16 GMT</pubDate>
            <atom:updated>2018-05-04T18:39:04.408Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*op11vRA6Z3jCRmdWXwxPkw.jpeg" /><figcaption>Image Credit: Paola Aguilar</figcaption></figure><h4>Introduction</h4><p>Docker is the future of DevOps. For those of you who don’t know Docker is the best tool to store &amp; run any application in any runtime environment. As far as DevOps is concerned, Docker expedites the process of running new software onto any server. Therefore, we decided to package the AntiMoz project into Docker</p><h4>What’s Docker?</h4><p>The learning curve for Docker is a bit cumbersome but not terrible.At Antimoz, it took us approximately 45 days after the approval of the proposal to grasp docker with functional understanding. Once we got off <br>for automating the packaging &amp; deployment applications inside containers. Containers can be thought analogously as shipping containers: self sufficient, standardized, &amp; easy to transport.</p><p>For Antimoz, we saved hours of DevOps time for setting up the Hadoop environment on our CentOS servers by downloading Docker images.</p><p>Given that we have Docker installed, we used our command line to retrieve the latest docker images:</p><pre>docker search hadoop</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xVSHpS3l8VTuBbj7dDzrxA.png" /></figure><p>Docker images are like class templates for creating individual instances, docker containers. A docker image is considered the blueprint for a docker container, a single instance of a docker image.</p><h4>Dockerfiles</h4><p>Using docker, we can customize our linux containers by laying more applications into the source Dockerfiles are text files readable by humans and the <a href="https://nickjanetakis.com/blog/understanding-how-the-docker-daemon-and-docker-cli-work-together">docker daemon</a>. They’re used to define an image to build.</p><p>In the following I’ll demonstrate how I added Hive and Pig to our Hadoop base image. These steps are also generally used in dockerizing an application :</p><ul><li>Select a base image</li></ul><pre>#Build pseudo-distrubuted Hadoop 2.3</pre><pre>FROM harisekhon/hadoop:2.3</pre><p>With the commands above, we build our base Hadoop system with a CentOS Linux distribution with a Java SDK pre-installed.</p><ul><li>Copy your application</li></ul><pre># Add pig layer MR Jobs <br><strong>RUN</strong> curl -s http://apache.mirror.anlx.net/pig/pig-${PIG_VERSION}/pig-${PIG_VERSION}.tar.gz | tar -xz -C /usr/local <br><strong>ENV</strong> PIG_HOME /usr/local/pig-${PIG_VERSION}/ <br><strong>RUN</strong> ln -s $PIG_HOME /usr/local/pig <br><strong>ENV</strong> PATH $PATH:$PIG_HOME/bin</pre><ul><li>Install your dependencies</li></ul><p>In this dockerfile, our dependency was the base image containing the Hadoop Distributed File System.</p><p>Docker is versatile &amp; a godsend to DevOps. There are plenty of other feautres available that are currently outside the scope of this blog post such as Docker Compose &amp; Docker Machine.</p><h3><strong>References</strong>:</h3><p>[What is Docker? | Microsoft Docs](<a href="https://docs.microsoft.com/en-us/dotnet/standard/microservices-architecture/container-docker-introduction/docker-defined">https://docs.microsoft.com/en-us/dotnet/standard/microservices-architecture/container-docker-introduction/docker-defined</a>)</p><p>[The American Scholar: The 300-Word Challenge — William Zinsser](<a href="https://theamericanscholar.org/the-300-word-challenge/#">https://theamericanscholar.org/the-300-word-challenge/#</a>)</p><p>[Copy of Docker Talk by Richie Young](<a href="https://slides.com/richieyoung/docker-9/#/5/8">https://slides.com/richieyoung/docker-9/#/5/8</a>) `</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68769f078607" width="1" height="1" alt=""><hr><p><a href="https://medium.com/transparent-analytics/working-with-docker-for-the-first-time-68769f078607">Working with Docker for the first time</a> was originally published in <a href="https://medium.com/transparent-analytics">Transparent Analytics</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting up your first Postgres database using Docker]]></title>
            <link>https://medium.com/transparent-analytics/setting-up-your-first-postgres-database-using-docker-fb4682422f59?source=rss----8a4d28b70e9f---4</link>
            <guid isPermaLink="false">https://medium.com/p/fb4682422f59</guid>
            <dc:creator><![CDATA[Bryan Dieudonne]]></dc:creator>
            <pubDate>Fri, 04 May 2018 01:20:31 GMT</pubDate>
            <atom:updated>2018-05-04T19:03:14.673Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OqpCleIhSb1U9LyT-eY8hQ.jpeg" /></figure><h4>Why use Postgres ?</h4><p>Postgres is an open source object oriented database. Within the data engineering ecosystem, it is similar to a MySQL datastore. Its generally used for smaller datasets, CSV files less than 1GB.</p><p>For AntiMoz, we decided to incorporate Postgres as a datastore for the CSV file dumps pulled from the Google Search Console &amp; Google Analytics API. Our Hadoop Distributed File System (HDFS) is well suited for significantly larger data.</p><h4>Setting up Postgres</h4><p>Assuming that you have docker set up on your server, pull the latest version of Postgres from DockerHub</p><pre>docker pull postgres:latest</pre><p>Once you have postgres downloaded, run the container with the following commands:</p><pre>docker run — name post_db -d postgres:latest</pre><p>With theses commands we give a custom name for the container</p><pre>— name db        # let’s make a custom name for our container<br>-d               # detach the container process from the terminal <br>-postgres:latest # launch a Docker instance</pre><p>Once we run the postgres instance we can access the database with the following docker commands:</p><h4>Integrating Postgres into the AntiMoz data pipeline</h4><p>We realized the beauty of Docker when we noticed how easy it was to integrate it into our data pipeline with a simple Dockerfile.</p><p>Here is our original Dockerfile for our batch process:</p><pre><br># Author: Bryan Dieudonne</pre><pre># Build pseudo-distrubuted Hadoop 2.9<br>FROM harisekhon/hadoop:2.9<br>MAINTAINER bdieu178</pre><pre>ARG PIG_VERSION=0.16.0<br>ARG HIVE_VERSION=2.3.3</pre><pre># hadoop-common fixes<br>ENV PATH $PATH:$HADOOP_PREFIX/bin</pre><pre># pig<br>RUN curl -s <a href="http://apache.mirror.anlx.net/pig/pig-$%7BPIG_VERSION%7D/pig-$%7BPIG_VERSION%7D.tar.gz">http://apache.mirror.anlx.net/pig/pig-${PIG_VERSION}/pig-${PIG_VERSION}.tar.gz</a> | tar -xz -C /usr/local<br>ENV PIG_HOME /usr/local/pig-${PIG_VERSION}/<br>RUN ln -s $PIG_HOME /usr/local/pig<br>ENV PATH $PATH:$PIG_HOME/bin</pre><pre># hive<br>RUN curl -s <a href="http://apache.mirror.anlx.net/hive/hive-$%7BHIVE_VERSION%7D/apache-hive-$%7BHIVE_VERSION%7D-bin.tar.gz">http://apache.mirror.anlx.net/hive/hive-${HIVE_VERSION}/apache-hive-${HIVE_VERSION}-bin.tar.gz</a> | tar -xz -C /usr/local<br>ENV HIVE_HOME /usr/local/apache-hive-${HIVE_VERSION}-bin/<br>RUN ln -s $HIVE_HOME /usr/local/hive<br>ENV PATH $PATH:$HIVE_HOME/bin</pre><p>From the Dockerfile, we have Pig &amp; Hive as add-ons to our Hadoop system. To add Postgres we simply add the following lines:</p><pre>#add PostGres datastore &amp; Python <br>RUN yum -y update<br>RUN yum -y install postgresql-server postgresql-contrib</pre><pre>#RUN postgresql-setup initdb<br>RUN yum -y install yum-utils<br>RUN yum -y groupinstall development<br>RUN yum -y install <a href="https://centos7.iuscommunity.org/ius-release.rpm">https://centos7.iuscommunity.org/ius-release.rpm</a><br>RUN yum -y install python36u python36u-pip python36u-devel</pre><pre>#add Psycopg library ← processing csv files into PostGres.<br>RUN pip install -U pip<br>RUN pip install psycopg2-binary<br></pre><p>First we’ll update our package manager, in this case we’re using yum on the CentOS Linux distribution. Then we’ll</p><p>#antimoz</p><p>===<br>References:<br>[Docker Talk by Richie Young](<a href="https://slides.com/richieyoung/docker-9/#/1/1">https://slides.com/richieyoung/docker-9/#/1/1</a>)<br><a href="http://xyz.insightdataengineering.com/blog/pipeline_map/">http://xyz.insightdataengineering.com/blog/pipeline_map/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb4682422f59" width="1" height="1" alt=""><hr><p><a href="https://medium.com/transparent-analytics/setting-up-your-first-postgres-database-using-docker-fb4682422f59">Setting up your first Postgres database using Docker</a> was originally published in <a href="https://medium.com/transparent-analytics">Transparent Analytics</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Progress and Status Update]]></title>
            <link>https://medium.com/transparent-analytics/progress-and-status-update-895c3e613f66?source=rss----8a4d28b70e9f---4</link>
            <guid isPermaLink="false">https://medium.com/p/895c3e613f66</guid>
            <dc:creator><![CDATA[Akhil Jacob]]></dc:creator>
            <pubDate>Fri, 04 May 2018 01:15:36 GMT</pubDate>
            <atom:updated>2018-05-04T01:57:42.116Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NpngzUUbR0tNDogA." /><figcaption>Photo by <a href="https://unsplash.com/@jonathanroger?utm_source=medium&amp;utm_medium=referral">Jonathan Roger</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This semester there was a lot that was done in design and implementation. We thought through and decided several of the project requirements and end goals. Starting a project of this magnitude was difficult especially with the heavy student workloads, as both members of the current team are full-time college students.</p><p>We have so far setup several of docker containers required to get the project of the ground. In referencing our proposal, we are halfway through our major milestones. In hindsight, we should have focused on creating 4–5 moderate milestones rather than 3 major milestones. Development speed was hindered by technology learning curves. This was the first time our team used Kafka, Divolte, Hadoop, and Zookeeper.</p><p>Future plans are to add a searchable UI to obtain the information collected by Hadoop and Kafka, this will probably be done using Plotly and a combination of Flask.</p><p>We just wanted to thank everyone this semester especially the RCOS team( Rensselaer Center for Open Source) for mentoring us and encouraging us to learn and improve.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=895c3e613f66" width="1" height="1" alt=""><hr><p><a href="https://medium.com/transparent-analytics/progress-and-status-update-895c3e613f66">Progress and Status Update</a> was originally published in <a href="https://medium.com/transparent-analytics">Transparent Analytics</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Collection Methods and why they matter]]></title>
            <link>https://medium.com/transparent-analytics/data-collection-methods-and-why-they-matter-91987d5d339b?source=rss----8a4d28b70e9f---4</link>
            <guid isPermaLink="false">https://medium.com/p/91987d5d339b</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[web-analytics]]></category>
            <dc:creator><![CDATA[Akhil Jacob]]></dc:creator>
            <pubDate>Sat, 28 Apr 2018 04:49:45 GMT</pubDate>
            <atom:updated>2018-04-28T04:49:44.876Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OcvKu9cppbIXW6bb." /><figcaption>Photo by <a href="https://unsplash.com/@cdr6934?utm_source=medium&amp;utm_medium=referral">Chris Ried</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Getting usage data is a key portion of modern-day analytics, understanding what the user is doing, what works and what doesn’t. It helps site creators understand and learn from their customers. This is essential for any business. There are two main types of data collection used today, <strong>Javascript Tracking </strong>and <strong>Log File Analysis</strong>. Each with it’s Pros and Cons.</p><p>Javascript Tracking is the far more commonly used one (and the one used by Google Analytics), the reason being it is far easier to setup and provides far greater detail about what each user is doing. Since JavaScript is an event-driven language using tags allow the code to be triggered each time that a user hovers, clicks, or drags something on the website. This is extremely useful, the data provides an in-depth user-flow of someone using your website. However, there is a major con with Javascript tracking it can easily be blocked by Ad-Blockers and script blockers. This is a huge problem because the number of people using Ad-Blockers increase by the day. A large portion of user data is lost because of this and will skew the overall numbers.</p><p>The alternative is to use the Log Collection analysis. This type of Analysis gives a better picture of the number of users, especially since there is no way for a user to block it because the data collection is done server side. In addition through this method we can find if there are broken links or page loading error. We can also measure the bandwidth of the connected devices and use it see there are issues with loading. However that is the only data we can collect from log file analysis and since it is so limited it is often neglected as an option. Furthermore logs are often very complicated to read so it would be more difficult for analysts.</p><p>In conclusion both methods have their pros and cons. Javascript provides a lot more useful data about usage but is often blocked by clients. The alternative is Web Log though the data is often times more reliable, especially to find the number of users on your site, actually doing the analysis is often a far more difficult task. So there is no perfect way to measure the user usage data, but using a combined approach will most likely result in the most fruitful results.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=91987d5d339b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/transparent-analytics/data-collection-methods-and-why-they-matter-91987d5d339b">Data Collection Methods and why they matter</a> was originally published in <a href="https://medium.com/transparent-analytics">Transparent Analytics</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Defining the Pipeline]]></title>
            <link>https://medium.com/transparent-analytics/defining-the-platforms-15eeef3ea204?source=rss----8a4d28b70e9f---4</link>
            <guid isPermaLink="false">https://medium.com/p/15eeef3ea204</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Akhil Jacob]]></dc:creator>
            <pubDate>Tue, 03 Apr 2018 05:08:35 GMT</pubDate>
            <atom:updated>2018-04-03T05:09:04.169Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*y1sIMr1bZTE7efB8." /><figcaption>Photo by <a href="https://unsplash.com/@badashproducts?utm_source=medium&amp;utm_medium=referral">Ash Edmonds</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>We are attempting to build a better solution to for web analytics data. To learn more about why and what we are building checkout our previous blog post <a href="https://medium.com/transparent-analytics/defining-anti-moz-725fd1163fec">here</a>. A large portion of this project began with researching the technology that would power our project. After several discussions we came up with the frameworks that would power the platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZD7zxMIuaicoi49F5Gp8XA.png" /></figure><p>Through several hours of research and testing we began to put together an a basic setup about how our code would be laid out. We also defined how each component would communicate with the other. One of the main goals in the early stage was to stick to open sourced technologies. This makes it easier for components to swapped out with leisure and also makes the code more accessible.</p><p><strong>The technologies and reasonings:</strong></p><p><strong>Divolte Collector: </strong>A server designed to collect &amp; send click-stream data into the Hadoop Distributed File System (HDFS) and Kafka topics. It utilizes client-side JavaScript to collect on user behavior on web pages.</p><p><strong>Hadoop &amp; Hive: </strong>The HDFS &amp; Hive components will help us to leverage batch processing for report generation &amp; data analysis on web data gathered from Google and from our Divolte Connector.</p><p><strong>Kafka: </strong>This is real time pipeline store that allows to take real-time data from Divolte and store the theme using it’s Consumer API.</p><p><strong>Druid:</strong> A high-performance data store designed for interactive analytical queries. It’s designed to work with real-time streams and/or batch data sets.</p><p><strong>Superset &amp; Plotly Dash: </strong>Both are highly configurable plotting libraries designed to communicate with Druid. We plan on using this to create a dashboard to view and filter the web data.</p><p><strong>PostgreSQL &amp; Google Data Studio: </strong>A short-term testing area for BI analytics.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15eeef3ea204" width="1" height="1" alt=""><hr><p><a href="https://medium.com/transparent-analytics/defining-the-platforms-15eeef3ea204">Defining the Pipeline</a> was originally published in <a href="https://medium.com/transparent-analytics">Transparent Analytics</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Defining AntiMoz]]></title>
            <link>https://medium.com/transparent-analytics/defining-anti-moz-725fd1163fec?source=rss----8a4d28b70e9f---4</link>
            <guid isPermaLink="false">https://medium.com/p/725fd1163fec</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[marketing]]></category>
            <category><![CDATA[data-management]]></category>
            <dc:creator><![CDATA[Bryan Dieudonne]]></dc:creator>
            <pubDate>Sat, 10 Mar 2018 01:09:19 GMT</pubDate>
            <atom:updated>2018-05-03T16:49:44.419Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ItzunfD61b1t0L9asdQqDg.jpeg" /><figcaption>Every successful project relies on the planning phase</figcaption></figure><p>Marketing agencies lack modern data management systems.I’ve worked at an agency once where a relational database was non-existent, let alone data governance. The staff heavily relied on their Mac filesystems &amp; Excel to store &amp; process their data. This usually led to operational issues such as Excel taking 30 minutes to process a sort function on a 18 MB CSV file.</p><h4>Open Source Backend</h4><p>So to fix this problem, our team decided to create an open source data pipeline for marketing agencies to plug in &amp; visualize their data anyway they want. In the early stages of the internet, backend infrastructures for companies were propriety. However as the internet &amp; its use cases expanded, the adoption &amp; use of open source grew significantly since the first internet browser. Currently, 78% of companies implement open source software into their information systems.</p><p>The difference between our product &amp; other web analytics solutions is the intent. We are building AntiMoz to (1) provide digital marketers a free solution to the storage &amp; processing of un-sampled, granular web analytics data, (2) simple deployment of a data management infrastructure onto a Linux server &amp; (3) access to search engine optimization data not administered by any Google products.</p><h4>Our mission</h4><p>We want to make data management &amp; access to Big Data processing accessible &amp; feasible for digital marketers. We’re creating Anti-Moz with the end-user in mind. By implementing design thinking in our development process we hope to build a tool that would fit all possible business use cases at various marketing agencies.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=725fd1163fec" width="1" height="1" alt=""><hr><p><a href="https://medium.com/transparent-analytics/defining-anti-moz-725fd1163fec">Defining AntiMoz</a> was originally published in <a href="https://medium.com/transparent-analytics">Transparent Analytics</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>