<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Regina Ryan on Medium]]></title>
        <description><![CDATA[Stories by Regina Ryan on Medium]]></description>
        <link>https://medium.com/@reginaryann?source=rss-7275383a55dd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*CWb68x2Ydljivs2m.jpg</url>
            <title>Stories by Regina Ryan on Medium</title>
            <link>https://medium.com/@reginaryann?source=rss-7275383a55dd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 03:22:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@reginaryann/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Better Faster EDA]]></title>
            <link>https://medium.com/@reginaryann/better-faster-eda-d335472ebf88?source=rss-7275383a55dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/d335472ebf88</guid>
            <dc:creator><![CDATA[Regina Ryan]]></dc:creator>
            <pubDate>Wed, 04 Dec 2019 18:25:20 GMT</pubDate>
            <atom:updated>2019-12-04T18:25:20.921Z</atom:updated>
            <content:encoded><![CDATA[<p><em>An analysis is only as good as the data on which it is based.</em></p><p>In the last few projects I’ve worked on I found that finding and selecting the “right” data has taken me way more time that I would’ve liked it to. This becomes especially time consuming if you are searching for ideas for a project starting with the data (as opposed to finding data to implement your idea), i.e. you are searching for data to implement a specific model on, rather than starting with the data and then selecting a model that would work best on it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*-oiyhgp6C9YPziPkCeyiOg.png" /></figure><p>Different models are sensitive to different inconsistencies in data and without sufficient practice it may be hard to gauge the fitness of your data to your pre-selected model and quickly determine whether you want to continue working with it or look for something more suitable.</p><p>This means that you have to be able to examine different datasets quickly and efficiently. This step in the data science process is called Exploratory Data Analysis and it allows you to get a good idea of what kind of data you’re looking at, get a sense of its shortcomings, issues, and what kind of analysis can be performed.</p><p>Pandas and Matplotlib provide numerous tools for data exploration. Some of the most useful ones are: .info(), .describe(), value_counts(), etc.</p><p>But even with these tools, getting a good gauge on all your variables is time consuming. To make the process faster (and less painful) I’ve used some additional tools and 2 of them that I wanted to share in this blog are pandas_profiling and bamboolib.</p><h3>pandas_profiling</h3><p>Pandas Profiling as an awesome open-source python package that automates a lot of these initial steps and outputs a detailed interactive HTML report on each variable and the dataset as a whole.</p><p><strong>First things first — Installation:</strong></p><p>With pip:</p><pre>pip install pandas-profiling</pre><p>With conda:</p><pre>conda install -c conda-forge pandas-profiling</pre><p><strong>Now that thats done let’s see it in action:</strong> I used a dataset from my last project that that contains US County data on health metrics like High Blood Pressure, Obesity and Diabetes and population metrics like ethnicity and age.</p><pre>import pandas as pd<br>import pandas_profiling</pre><pre>df = pd.read_csv(&#39;df_final.csv&#39;)</pre><pre>df.profile_report(style={&#39;full_width&#39;:True})</pre><p>With just one line of code you get this beautiful thing in return:</p><p><strong>Profile Report Link</strong></p><p>It provides the following statistics:</p><ul><li>Essentials: type, unique values, missing values</li><li>Quantile statistics like minimum value, Q1, median, Q3, maximum, range, interquartile range</li><li>Descriptive statistics like mean, mode, standard deviation, sum, median absolute deviation, coefficient of variation, kurtosis, skewness</li><li>Most frequent values</li><li>Histogram</li><li>Correlations highlighting of highly correlated variables, Spearman, Pearson and Kendall matrices</li><li>Missing values matrix, count, heatmap and dendrogram of missing values</li></ul><p>You might notice by default the report greys out values that are highly correlated. To retrieve those variables use this:</p><pre>profile = df.profile_report()<br>rejected_variables = profile.get_rejected_variables(threshold=0.9)</pre><p>This will give you a list of rejected variables. If you want to include that list into a report pass on an option ‘correlation_overrides’ argument to the profile_report function:</p><pre>profile_all = df.profile_report(title=&#39;Pandas Profiling Report All&#39;,          correlation_overrides=rejected_variables)</pre><p>And if you want to save the report to an html output, simply run this:</p><pre>profile = df.profile_report(title=&#39;Pandas Profiling Report&#39;)<br>profile.to_file(output_file=&quot;output.html&quot;)</pre><h3><strong>bamboolib</strong></h3><p>Bamboolib is another tool that found really cool. It provides a GUI for data exploration in pandas and works with Jupyter notebooks and most other editors. It is not free to use locally, but you can try it out using Binder on publicly available data.</p><p>There are a couple of steps to create a binder and you can follow the steps outlined on the packages github page (link at the end of the post)</p><p>Follow this link to view a bamboolib demo:</p><p><a href="https://mybinder.org/v2/gh/8080labs/bamboolib_binder_template/master?filepath=bamboolib_demo_titanic.ipynb">GitHub: 8080labs/bamboolib_binder_template/master</a></p><p>Once imported, it will display a “Show Bamboolib UI” button when you run df</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YwkTsaT0u4ZH6MffwNRNvg.png" /></figure><p>The UI is very intuitive and provides a lot of functionality on data cleaning and restructuring:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qz4NMSHbAUlIee8G6gl0oA.png" /></figure><p>There are other packages worth mentioning, as well as full applications developed specifically for data exploration like Tableau, and I will explore them more in a later blog post.</p><p>Resources:</p><ul><li><a href="https://pandas-profiling.github.io/pandas-profiling/docs/">pandas_profiling API documentation</a></li><li><a href="https://github.com/8080labs/bamboolib_binder_template/blob/master/create_your_own_binder.md">8080labs/bamboolib_binder_template</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d335472ebf88" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding the ETL process and considering best practices]]></title>
            <link>https://medium.com/@reginaryann/understanding-the-etl-process-and-considering-best-practices-68d9d6f5dec3?source=rss-7275383a55dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/68d9d6f5dec3</guid>
            <dc:creator><![CDATA[Regina Ryan]]></dc:creator>
            <pubDate>Tue, 29 Oct 2019 14:59:50 GMT</pubDate>
            <atom:updated>2019-10-29T16:59:24.832Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9O5Lu20Zd_EFL6MSIdt5eA.jpeg" /></figure><p>What is an ETL?</p><p>ETL is an integration process referring to three distinct but interrelated steps (Extract, Transform and Load) that enables the collection of data from various sources into one data store, ready for analysis. This process is at the core of every organization’s data management strategy and ensuring that your ETL flow is well-designed is cruicial. A properly structured ETL process should be: Reliable, Resilient, Reusable, Maintainable, Well-performing and Secure.</p><p>ETLs can be implemented manually with code or with dedicated ETL tools.</p><p>Two approaches to data processing are Batch Processing and Stream Processing. Batch processing is when processing happens of blocks of data that have already been stored over a period of time. It’s a good option in situations where you don’t need real-time analytics and are working with large amounts of data. Stream processing (a.k.a. real-time or streaming analytics) is when processing happens continuously and allows us to process data in real time and quickly detect changes and trends. A combination of both is often used.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fyx5W8FunIjurDxZ3pOFjQ.png" /></figure><h3>ETL Process and Best Practices</h3><h3>1. Extraction</h3><p>Before we start working with the data, it’s important to know and understand your data source. Data can come come from structured sources (Traditional data sources for BI applications include Oracle, SQL Server, MySql, DB2, Han, etc) or unstructured sources e.g. text, emails and web pages and in some cases custom apps.</p><p>In this step we assess the data quality and utility for each specific purpose. This means looking at key metrics, such as quantity and distribution and identifying the properties of the data set. Designing the extraction process with care is vital to ensuring the accuracy of your final analysis as well as avoiding adverse effects on the source system in terms of performance, response time, and locking.</p><p>One thing to consider is whether your database application supports CDC. CDC stands for Change Data Capture and it is a process of capturing and preserving the state of data across time, and its one of the core functions of a data warehouse. When you’re working with big data, sometimes you need a dedicated second database(Business Intelligence or BI database) from which you can make queries for analytics and data visualization. The need for the 2nd database is so that the queries do not over load your main database.</p><p>Keeping these two databses synced is a problem when you’re working with big data. Dumping <strong>ALL</strong> the data into your BI database from your main database can be computationally expensive. This is where CDC becomes useful, by only grabbing the <strong>NEW</strong> data that’s being stored in the main database and putting it into your BI database, it is much more efficient.</p><p>CDC can be a piece of code that recieves notifications from the main databse when it has new data, and captures the data, then stores it into the BI database.</p><p>Some of the challenges in this step include dealing with:</p><ul><li>Change in data formats over time</li><li>Increase in data velocity and volume</li><li>Rapid changes on data source credentials</li><li>Null issues</li><li>Change requests for new columns, dimensions, derivatives and features</li><li>Using source specific code can create problems down the line and create overhead to future maintenance of ETL flows</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*q5hILy_YJswr02kd2K4x7w.jpeg" /></figure><h3><strong>2. Data Cleaning and Transformation</strong></h3><p>This step should include designing a workflow to execute all data cleansing and transformation steps for multiple sources and large data sets in a reliable and efficient way. Data cleaning involves <strong>detection and separation of invalid, duplicate, or inconsistent data</strong> to improve the quality and utility of data that is extracted before it is transferred to a target database or Data Warehouse.</p><ul><li>Detection and removal of errors and inconsistencies</li><li>Ensuring columns are in the same order and data is in the same format.</li><li>Enriching data by merging in additional information</li><li>Mapping functions for data cleaning should be specified in a declarative way and be reusable</li></ul><h4>Potential Problems with Data from a Single Source</h4><ul><li>Uniqueness</li><li>Misspelling</li><li>Redundancy/Duplicates</li><li>Outside domain range</li><li>Data entry errors</li><li>Referential integrity</li><li>Contradictory values</li></ul><h4>Potential Problems with Data from Multiple Sources</h4><ul><li>Naming conflicts at the schema level</li><li>Structural conflicts</li><li>Inconsistent aggregating</li><li>Inconsistent timing</li></ul><p>For data analysis, metadata can be analyzed that will provide insight into the data properties and help detect data quality problems. There are two related approaches:</p><h4>Data Profiling</h4><p>Data profiling, data assessment, data discovery, data quality analysis is a process through which data is examined from an existing data source in order to collect statistics and information about it. A systematic up-front analysis of the content of the data sources is required to assess scope of the data, variation of data patterns and formats in the database.</p><h4>Data Mining</h4><p>Data mining, data discovery, or knowledge discovery (KDD) is a systematic and sequential process of identifying and discovering hidden patterns and information in a large dataset. It also refers to the nontrivial extraction of implicit, previously unknown, and potentially useful information from data in databases.</p><p>Once data cleansing is complete, the data needs to be moved to a target system or to an intermediate system for further processing.</p><h4><strong>Aspects of transformation</strong></h4><ul><li>Format Standardization</li><li>Mapping of values to some derived/short meaning</li><li>Removing duplicate values</li><li>Establishment of key relationships across tables.</li><li>Merging of similar information(from multiple sources)</li><li>Splitting single fields</li></ul><h3><strong>3. Loading</strong></h3><p>In this phase, extracted and transformed data is loaded into the end target source which may be a simple delimited flat file or a Data Warehouse depending on the requirement of the organization.</p><h4>Referential Integrity Constraints</h4><p>Referential integrity constraints will check if a value for a foreign key column is present in the parent table from which the foreign key is derived. This constraint is applied when new rows are inserted or the foreign key column is updated. Can slow down the process when working with large data — in this case it’s important that referential integrity is maintained by a well-designed ETL porcess itself.</p><p>Some of the challenges of this step are:</p><ul><li><strong>Ordering</strong>: Data may be processed in a different order than when it was received. If data is deleted or updated, then processing in the wrong order will lead to data errors, therefore maintaining and ordering is crucial for keeping data accurate.</li><li><strong>Monitoring Capability: </strong>Data coming from a variety of sources presents complexities, and potentially failures due to an API being unavailable, network congestion or failure, API credential expiration, data incompleteness or inaccuracy — monitoring is critical as recovering from these issues can be complex.</li></ul><h3>Conclusions</h3><p>ETL pipeline are a core building block of data warehousing and analysis. Understanding your data is key when designing an ETL process. Ensuring that your ETL process is structurally sound will improve operational resilience, reduce issues and mistakes and ultimately lead to better business intelligence insights.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68d9d6f5dec3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>