<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jacob Stellon on Medium]]></title>
        <description><![CDATA[Stories by Jacob Stellon on Medium]]></description>
        <link>https://medium.com/@jacobstellon?source=rss-a23d2fafe744------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*STjCbtimvZv8eTfhhH6Wng.jpeg</url>
            <title>Stories by Jacob Stellon on Medium</title>
            <link>https://medium.com/@jacobstellon?source=rss-a23d2fafe744------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 07:51:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jacobstellon/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data Threads: Automated Data Lineage for a Complete Picture]]></title>
            <link>https://medium.com/ibm-data-ai/data-threads-automated-data-lineage-for-a-complete-picture-42ef0d700be6?source=rss-a23d2fafe744------2</link>
            <guid isPermaLink="false">https://medium.com/p/42ef0d700be6</guid>
            <category><![CDATA[ibm]]></category>
            <category><![CDATA[data-lineage]]></category>
            <category><![CDATA[manta]]></category>
            <category><![CDATA[data-privacy]]></category>
            <category><![CDATA[data-governance]]></category>
            <dc:creator><![CDATA[Jacob Stellon]]></dc:creator>
            <pubDate>Mon, 22 Aug 2022 16:11:41 GMT</pubDate>
            <atom:updated>2022-08-23T14:29:50.274Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DbKJeh2rT4vSJ9cagqBzIg.png" /></figure><p>An executive’s worst, and far too common, nightmare. Days before submitting quarterly earnings, the revenue analysis from the sales team and the product team are at odds. The analyst from each team insists their dashboard paints the accurate picture as they made no mistakes when compiling the earnings numbers into their BI software of choice, but if this is true, then where did the data diverge?</p><p>Organizations have more data, more data tools, and more people involved in data consumption than ever before. As data proliferates throughout an organization, it’s undeniably more challenging to trust, govern, and secure privacy all while meeting the demands of increasingly stringent regulations. Organizations need a map that simplifies a dataset’s journey from its origin to its end use with specific detail on how it has transformed and by whom along the way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_gLQOHiKPQlSSz_iFaYjFw.png" /><figcaption>Data Lineage in Watson Knowledge Catalog</figcaption></figure><p><strong>Data lineage explained</strong></p><p>Data lineage is a visual representation of a data set’s journey from its origin to end use. It has evolved into the main enterprise tool to understand the flow of data and the contribution of each person and program along data’s lifecycle.</p><p>Prior to automated scanners, some enterprises spent hours creating a manual data lineage. A manual lineage usually involves documenting the knowledge of application owners, data stewards, and integration specialists. This approach is obviously time intensive, can contain contradictory or missing information, and leads to teams relying on unsound lineage to make critical decisions. Moreover, manually examining code, comparing column names, and reviewing tables by hand is tedious. Due to code volumes, complexity, and the rate of change, this method quickly becomes unsustainable. Sooner or later, a manually managed lineage will fall out of sync with the actual state of the data, and the enterprise will have a lineage that cannot be trusted.</p><p>As a product of IBM’s partnership with MANTA, data engineers can scan data connections into <a href="https://www.ibm.com/products/cloud-pak-for-data">IBM Cloud Pak for Data</a> to automatically retrieve a complete technical lineage and a summarized view including information on data quality and business terms. IBM’s collaboration with industry leader MANTA enables users to capture data lineage consistently, accurately, and often in a matter of seconds. Simply put, every operation whether in a database, integration, or an analysis tool is reverse engineered to create a map telling the story from data source to end application. Data lineage in Cloud Pak for Data provides the deep technical lineage that data engineers need, historical versioning to view lineage changes over time, and indirect lineage to record even the most specific data operations.</p><p><strong>Use cases</strong></p><p>As mentioned above, data lineage is an excellent tool for pinpointing the root cause of a data issue in an end use application like a BI report. However, users also rely on data lineage for compliance, impact analysis, and architecture migrations. Data privacy regulations like<a href="https://www.ibm.com/data-responsibility/gdpr/"> GDPR</a> continue to expand in scope and new regulations are on the rise. Many require data lineage as a first step in compliance reports. Enterprises are constantly implementing changes to their data architecture and pipelines. Without a data lineage it can be difficult or impossible to assess the impact of planned changes. <a href="https://www.researchgate.net/figure/IBM-System-Science-Institute-Relative-Cost-of-Fixing-Defects_fig1_255965523">Research from IBM</a> shows that fixing a bug in production is 15x more expensive than fixing it during the implementation phase. Data lineage gives teams insight into the downstream impacts of these changes before potentially costly bugs are introduced. Customers also use data lineage to speed up the migration process while undergoing a digital transformation. Data lineage gives engineers visibility into which architectural components must be migrated at once and which need not be migrated at all. To summarize, data lineage builds trust in data when it is needed most and increases the efficiency of data engineers.</p><p>If data lineage is ignored or mapped inaccurately, decision makers lose faith in reports and models. Analysts and data scientists deserve data that inspires accurate, timely, and confident decision making. Only when one has a complete understanding of data can they rely on it.</p><p><strong>Data lineage and a data fabric architecture</strong></p><p>Although an automated data lineage is crucial to understand and build trust in an organization’s data, it must be used in combination with other approaches to protect, integrate, and democratize data across the enterprise. A data fabric architecture automates data discovery, governance, and consumption by simplifying data access in an organization to facilitate self-service data consumption. Recognized as a leader in enterprise data fabric solutions by <a href="https://www.ibm.com/account/reg/us-en/signup?formid=urx-51607">Forrester</a>, IBM’s approach to designing and implementing a data fabric is agnostic to data environments, processes, utility, and geography.</p><p>A data fabric ensures end users have access to the data they need at the exact moment it is needed regardless of where the data lives and what tool is being used. Data lineage allows organizations to understand how their data moves, transforms, and is consumed across multiple sources and tools. For this reason, data lineage is a key tool in a data fabric architecture. With this approach, the next time an executive receives conflicting reports and an engineer is tasked with tracking down the root cause of a discrepancy, the executive or data engineer can take advantage of the data lineage provided with a data fabric architecture to quickly understand and regain trust in data.</p><p>Learn more about<a href="https://www.ibm.com/resources/the-data-differentiator/data-architecture"> IBM’s approach to creating a data fabric solution</a> or how <a href="https://www.ibm.com/products/cloud-pak-for-data">IBM Cloud Pak for Data</a> makes this a reality.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42ef0d700be6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ibm-data-ai/data-threads-automated-data-lineage-for-a-complete-picture-42ef0d700be6">Data Threads: Automated Data Lineage for a Complete Picture</a> was originally published in <a href="https://medium.com/ibm-data-ai">IBM Data Science in Practice</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Combat the Great Resignation Using Machine Learning (Without Code!)]]></title>
            <link>https://medium.com/@jacobstellon/combat-the-great-resignation-using-machine-learning-without-code-ca3a3b174851?source=rss-a23d2fafe744------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca3a3b174851</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[ibm]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Jacob Stellon]]></dc:creator>
            <pubDate>Thu, 31 Mar 2022 21:20:14 GMT</pubDate>
            <atom:updated>2022-04-01T16:37:04.826Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Introduction</strong></h3><p>In 2021, an average of 3.98 million Americans quit their jobs each month, marking the highest Bureau of Labor Statistics employee turnover record since the organization began tracking this data in 2000. This highly volatile labor market has many referring to the last two years as <em>The Great Resignation</em>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fdatawrapper.dwcdn.net%2FrRQKJ%2F4%2F&amp;display_name=Datawrapper&amp;url=https%3A%2F%2Fdatawrapper.dwcdn.net%2FrRQKJ%2F4%2F&amp;image=https%3A%2F%2Fdatawrapper.dwcdn.net%2FrRQKJ%2Fplain-s.png%3Fv%3D4&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=dwcdn" width="730" height="482" frameborder="0" scrolling="no"><a href="https://medium.com/media/89314be83717c7d3f53a3cb030ba7854/href">https://medium.com/media/89314be83717c7d3f53a3cb030ba7854/href</a></iframe><p>Although it’s difficult to generalize the cost of labor turnover to a business, the expenses of hiring, onboarding, and developing employees can be massive. When corporations include the revenue lost due to an unfilled role, the cost of losing personnel can range from five to six figures for each case of employee turnover. Before implementing retention plans to mitigate workforce attrition, it’s critical for business leaders to pinpoint the root causes of turnover among their staff. A business armed with historical data can look towards machine learning to help answer the following questions:</p><p><em>What factors are key indicators of attrition?</em></p><p><em>Which employees are most likely to leave?</em></p><p>To some, machine learning seems like a specialized field only available to those with advanced degrees and many years of experience. However, visual tools like <a href="https://www.ibm.com/products/spss-modeler">IBM’s SPSS Modeler</a> allow anyone to extract insights from machine learning models through a visual drag-and-drop canvas. Even better, these tools have made great strides to incorporate advanced capabilities so that even the most experienced data scientists benefit from adopting visual model building. I’ll demonstrate how anyone can build a model to predict employee churn in SPSS Modeler on Cloud Pak for Data and compare this to the traditional method of building the model in a Python notebook.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rUdsQXrEIrokAbR3QkJD3A.png" /><figcaption>SPSS Modeler stream used in this project</figcaption></figure><p>A business looking to answer the aforementioned questions will start by gathering historical employee data labeled with a binary classification variable; in this case we’ll look at the variable <em>Attrition</em> which is labeled either <em>Yes </em>or <em>No</em>. I’ll use a fictitious data set created by IBM containing over 1000 records with information like an employee’s salary, commute time, and data from employee satisfaction surveys. The full data set can be downloaded <a href="https://www.kaggle.com/datasets/pavansubhasht/ibm-hr-analytics-attrition-dataset">here</a>.</p><h3><strong>Data Exploration</strong></h3><p>To better understand the variables and recognize areas where data cleansing may be required, a typical first step in the data science process is data exploration utilizing samples and visualizations. To do this, Python programmers often take advantage of open-source libraries like Pandas, but SPSS Modeler users can gain the same insight into their data by simply running a Data Audit node. The screenshots below show the Python functions used to search for missing values, explore basic statistics, and view distributions for each variable. Comparatively, the Data Audit node in SPSS Modeler produces the same information without the need for code.</p><p><strong>Python:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fOJT8ki66WSzLG5eqWUQmA.png" /><figcaption>View basic statistics</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gpnIgua_S6mOwzEXh6kqzA.png" /><figcaption>Plot histograms</figcaption></figure><p><strong>Data Audit Node in SPSS Modeler:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IQV_6Sf5SJmTCvYEMFMp-w.png" /><figcaption>View statistics, histograms, metadata, and missing values in one screen</figcaption></figure><h3><strong>Data Preprocessing</strong></h3><p>Before using this data to train a machine learning algorithm, some preprocessing is needed. A data scientist working on this project in a Python notebook manually takes steps like label encoding and feature scaling to prepare the data set. Someone who finds these terms unfamiliar or who struggles to remember when a preprocessing step is required shouldn’t be alarmed because the Auto Data Prep (ADP) node in SPSS Modeler will automatically handle data preparation.</p><p><strong>Python:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dCWGfwjq04VzecpCF-_vmA.png" /><figcaption>Label encoding in Scikit-learn</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EizNwgvtSHCPUmc7y1XA2Q.png" /><figcaption>Feature scaling</figcaption></figure><p><strong>SPSS Modeler:</strong></p><p>With a few clicks, the ADP node analyzes data and identifies fixes, screens out fields that are problematic or not likely to be useful, derives new attributes when appropriate, and improves performance through intelligent screening techniques.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/1*scrLryQ8_mo6DG4KoMFALA.png" /><figcaption>Auto Data Prep Settings</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6J98yfZDJ1jaJ_PB0aH9Eg.png" /><figcaption>Output preview post ADP</figcaption></figure><h3><strong>Partitioning Data</strong></h3><p>A necessary step of model building is splitting historical data into training and testing sets. Typically, a percentage of the data will be randomly assigned as training data and will be used to build a machine learning model. Once the model is trained, the remaining testing percentage is used to score the model for accuracy. Sometimes, the data will also include a validation set which allows for further fine tuning after a model is scored, but for this example only a training and testing set are used.</p><p><strong>Python:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*87l7pOK053-GK5yE_C88cQ.png" /></figure><p><strong>SPSS Modeler:</strong></p><p>In SPSS Modeler this is quickly done using the Partition node. The result is an additional column in your dataset which identifies each record as either training or testing data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*w5sFDz7qj-WVc9w4mBxTEg.png" /><figcaption>Partition settings</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cq38ri0llr9SafwbPKYidA.png" /><figcaption>Partition output preview</figcaption></figure><h3><strong>Building Models</strong></h3><p>Building machine learning models can be an incredibly iterative process partly because it’s nearly impossible to know which algorithm will create the best model at the outset of a project. Due to this, it’s often best to build and test multiple models before homing in on a solution. Programmatically, this means manually selecting and training models from a library like Scikit-learn as shown below.</p><p><strong>Python:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*on6xhA1MaXyIW5R7szvm-Q.png" /><figcaption>Creating a list of models to build</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TVSUK-ywSiwyiibp0Ns8tw.png" /><figcaption>Training models</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/1*kTySA3IF7-rkqxET7pUm4w.png" /><figcaption>Model scoring metrics</figcaption></figure><p><strong>SPSS Modeler:</strong></p><p>Selecting, building, and evaluating models can be tedious and requires some knowledge of popular machine learning algorithms. To quickly test many models, SPSS Modeler users can take advantage of the Auto Classifier node. The Auto Classifier node estimates and compares models using a number of different methods, enabling users to try out a variety of approaches in a single run. For this project, I used the Auto Classifier to build and rank 15 models with only a few button clicks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cYsVC4RYGWF2xvdjmoL3Wg.png" /><figcaption>Auto Classifier output</figcaption></figure><h3><strong>Evaluation</strong></h3><p>After building and scoring models using the training data set, it’s important to evaluate models on the testing data set. A confusion matrix is a common visual means of model evaluation as it succinctly displays model accuracy. It’s also important to look at metrics like area under the curve (AUC), precision, and recall.</p><p><strong>Python:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VKcsTKQiKmkySLbRzoFaHA.png" /><figcaption>Creating confusion matrix</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WvakbsZ7Gdddr37N4QhPfA.png" /><figcaption>Displaying evaluation metrics</figcaption></figure><p><strong>SPSS Modeler:</strong></p><p>In SPSS Modeler these same evaluation metrics are displayed using the Analysis node. Running this node attached to the Auto Classifier node can output the evaluation metrics for each model individually and on the group of models as an ensemble.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uX3KtcAdJgxpao2VeurjPQ.png" /><figcaption>Analysis node output</figcaption></figure><p>Studying the analysis output from the models I built shows that the logistic regression model had the best AUC value for the testing data set, so I will choose this model to make my predictions and inform my decision making around feature importance. For reference, the model built in Python had an AUC value of 0.85 compared to 0.88 for the model built in SPSS Modeler.</p><h3><strong>Informing Business Decisions</strong></h3><p>Now, it’s time to answer the two questions posed at the beginning of this project.</p><p><em>What factors are important in determining attrition?</em></p><p>Some ML models output the features that are most important in the prediction of a target like attrition. In the employee attrition model, this information conveys the key factors that led an employee to quit their job. The code used to plot the feature importance from a model built in Python is shown below. Alternatively, a SPSS Modeler user can view this information as part of the model viewer output that is automatically generated upon training a model without extra steps.</p><p><strong>Python:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vsFC378_wE1ABd2_0JNxBQ.png" /><figcaption>Plotting feature importance</figcaption></figure><p><strong>SPSS Modeler:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Xg1rKZyCMm9b9edkcniMA.png" /><figcaption>Model viewer feature importance summary</figcaption></figure><p><em>(Slight differences in feature importance are due to different parameters used when training the models)</em></p><p>A business looking to implement a retention plan could take the three most important features and build a plan that addresses these critical issues. The three most important features in this model are stock option level, job satisfaction, and income. If a company went through this process and saw the results shown above, the company could take the following actions.</p><ol><li>Provide stock options to employees who have been historically left out of company ownership programs.</li><li>Survey employees about their satisfaction at work, have conversations to understand feedback, and address the top concerns that arise.</li><li>Evaluate the external market around compensation data and strive to meet or exceed median pay across the industry.</li></ol><p>Building a retention plan around these three areas would likely be more effective than a plan built without understanding the root causes of attrition at a specific workplace. Although granting stock options and increasing salaries may seem expensive, it’s critical to consider these items in the context of the cost of employee attrition. In many cases, spending to address these issues can be a good financial decision.</p><p>The second question I looked to answer at the outset of this project is:</p><p><em>Which employees are most likely to leave?</em></p><p>The machine learning model built in this project makes a yes or no prediction on employee attrition and assigns a confidence value associated with that prediction. After building an employee attrition model on historical data, a company can input new data and understand how likely each employee is to leave their job.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c48sCBFHIVgSdO4pnnW7nA.png" /><figcaption>Prediction output with confidence</figcaption></figure><p>This information can be an incredibly valuable aspect of a retention plan. If the first part of a company’s plan is to make changes based on the most important features, the second aspect should be to address the employees who are most likely to leave. Based on the model’s prediction and its associated confidence, a company could establish a threshold for high risk of attrition. For example, high risk could mean the model predicting yes with a confidence of 0.8 or greater. The company can ensure that high risk employees receive support from managers and HR including conversations about their work environment, satisfaction, and long-term goals.</p><h3><strong>Conclusions</strong></h3><p>Machine learning is not only for those with years of experience and knowledge of programming languages. Anyone with a curious mind and data can utilize a visual tool like SPSS Modeler to build models and create high value business results.</p><h4>Disclaimers and Acknowledgments</h4><p>The goal of this project was to demonstrate that inexperienced users can quickly create a model to predict employee churn without getting into the minutiae of data science. With that said, an experienced data scientist or someone looking to productionalize a similar model could easily expand this work using SPSS Modeler in Cloud Pak for Data. Some areas to consider would be hyperparameter optimization of the models, expanding the flow with R, and deploying the model in Watson Machine Learning.</p><p>There are also some important limitations to this project. First, a fictional data set was used. Therefore, the results of this project should not be used outside of a learning context. If a model like this is to be used to predict future employee attrition it is important to retrain the model overtime to prevent drift.</p><p>The Python code shown in this post was borrowed from a kaggle user whose profile can be seen <a href="https://www.kaggle.com/hamzaben">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca3a3b174851" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>