<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jonas Dieckmann on Medium]]></title>
        <description><![CDATA[Stories by Jonas Dieckmann on Medium]]></description>
        <link>https://medium.com/@jonas_dieckmann?source=rss-1c8d1cf684f2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*mql_rVqbJfGV44cWo8IazA.jpeg</url>
            <title>Stories by Jonas Dieckmann on Medium</title>
            <link>https://medium.com/@jonas_dieckmann?source=rss-1c8d1cf684f2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 23:54:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jonas_dieckmann/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Innovations in Analytics: Elevating Data Quality with GenAI]]></title>
            <link>https://pub.towardsai.net/innovations-in-analytics-elevating-data-quality-with-genai-d3a8d5cc5ba9?source=rss-1c8d1cf684f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/d3a8d5cc5ba9</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[genai]]></category>
            <dc:creator><![CDATA[Jonas Dieckmann]]></dc:creator>
            <pubDate>Thu, 31 Oct 2024 12:02:04 GMT</pubDate>
            <atom:updated>2024-11-10T13:29:53.786Z</atom:updated>
            <content:encoded><![CDATA[<p>Data analytics has become a key driver of commercial success in recent years. The ability to turn large data sets into actionable insights can mean the difference between a successful campaign and missed opportunities. However, data quality is still a major challenge: if the data that is fed into a model lacks quality/consistency, the resulting output will also be of low quality. This is well exemplified by the popular saying “garbage-in, garbage-out”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*llLzpx6M7rTM04VJ4dLRAg.jpeg" /><figcaption>Image Credits: <a href="https://pixabay.com/photos/brick-construction-work-7754908/">Pixabay</a></figcaption></figure><p>Although AI is often in the spotlight, the focus on strong data foundations and effective data strategies is often overlooked. In this article, we’ll explore how AI can directly improve these foundations through:</p><ol><li>Automating data harmonization</li><li>Dynamic labeling and classification</li><li>Generating synthetic data</li></ol><p>Rather than dealing with flawed data, we’re using GenAI to enhance data quality from the start. This approach also sets the stage for more effective AI applications later on.</p><h4>The rise of (Generative) AI</h4><p>Many industries are undergoing significant changes thanks to AI technologies. In marketing, for example, AI helps organizations extract actionable insights from vast data sets, leading to targeted campaigns and better customer engagement. According to Gartner’s Hype Cycle, GenAI is at the peak, showcasing its potential to transform analytics.¹</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bgnnghbrvEDMAw2V_zOlBA.png" /><figcaption>Hype Cycle for Emerging Technologies 2023 (source: Gartner)</figcaption></figure><p>Despite AI’s potential, the quality of input data remains crucial. Inaccurate or incomplete data can distort results and undermine AI-driven initiatives, emphasizing the need for clean data. For marketers and digital innovators, handling inconsistent data from various sources can be a major barrier to unlocking AI’s full potential.</p><h4>Flipping the paradigm: Using AI to enhance data quality</h4><p>What if we could change the way we think about data quality? Instead of seeing it as a prerequisite for using AI, we could use AI to improve data quality itself. By leveraging GenAI, we can streamline and automate data-cleaning processes:</p><blockquote>Clean data to use AI? Clean data through GenAI!</blockquote><h3>Three ways to use GenAI for better data</h3><p>Improving data quality can make it easier to apply machine learning and AI to analytics projects and answer business questions. Here are three ways to use ChatGPT² to enhance data foundations:</p><h4>#1 Harmonize: Making data cleaner through AI</h4><p>A core challenge in analytics is maintaining data quality and integrity. Algorithms can automatically clean and preprocess data using techniques like outlier and anomaly detection. GenAI can now assist in direct data mapping and cleaning by identifying and fixing inconsistencies.</p><p>For example, a healthcare organization aggregating market data from different sources might face issues with varying naming conventions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jhP4taoIlBNks0b3Ge64cA.png" /><figcaption>Example prompt use case #1. Image by author</figcaption></figure><p>GenAI can automatically detect and correct these discrepancies, resulting in a clean and reliable mapping dataset. This not only saves analysts significant time on manual data checks but also obviates the need for complex regular expressions with traditional methods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R86f5nmiUr1Nw9TiD27GYw.png" /><figcaption>GPT-4o mini response use case #1. Image by author</figcaption></figure><h4>#2 Label: Enabling the use of previously unusable data</h4><p>Organizations often have large amounts of data that are unused due to low quality or lack of labeling. GenAI can help by automatically clustering similar data points and inferring labels from unlabeled data, obtaining valuable insights from previously unusable sources.</p><p>Natural Language Processing (NLP) is an example of where traditional methods can struggle with complex text data. For instance, extracting numeric details from clinical sector articles can be misleading if the numbers don’t refer to actual quantities. GenAI prompts can address such challenges effectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t6W7yECHLTz-t1P58EewTA.png" /><figcaption>Example prompt use case #2. Image by author</figcaption></figure><p>The result is straightforward but accurate in this case. Numeric extraction is just one example of how labeling can be powerful. Clearly, GenAI is a strong tool for extracting precise details or classifications from text data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*88Y1wyuPM1cbmH2BWiG5sQ.png" /><figcaption>GPT-4o mini response use case #2. Image by author</figcaption></figure><h4>#3 Generate: Use of LLMs to generate sample data</h4><p>GenAI can also generate synthetic data to train AI models. Large Language Models (LLMs) can produce realistic sample data, helping address data scarcity in fields where data availability is limited.</p><p>For example, a pharmaceutical company developing a drug for a niche market can use LLMs to create synthetic patient profiles, medical histories, and treatment outcomes. This approach not only enhances data diversity but also alleviates privacy concerns related to sensitive patient data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n0PV0F2cTwmk8FOgRCE5Cg.png" /><figcaption>Example prompt use case #3. Image by author</figcaption></figure><p>This approach not only increases data diversity but also addresses privacy concerns related to sharing sensitive patient information. It can also be extended to other applications, such as targeting audiences for marketing campaigns, creating examples for fraud detection, and more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SoZY0QWnwVGcmhFqb375yg.png" /><figcaption>GPT-4o mini response use case #3. Image by author</figcaption></figure><h3>Automating Data Quality Enhancement via APIs</h3><p>To fully harness GenAI’s potential for improving data quality, it’s crucial to integrate this technology in an automated and seamless way. Manually copying datasets into prompts and processing responses is not practical.</p><p>Using APIs, like ChatGPT’s API, within your coding environment can streamline this process by incorporating AI-driven data quality enhancements directly into your workflows. For guidance on using OpenAI’s API with Colab or Databricks, <a href="https://medium.com/towards-artificial-intelligence/how-to-use-chatgpt-api-for-direct-interaction-from-colab-or-databricks-39969a0ead5f">you can refer to my other article.</a> An even more practical way is to request a JSON file as response. The results from these automated requests can be directly written back to your data storage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t4bCj4outYtyQE0tE5QwIQ.png" /><figcaption>Example processing flow: utilizing databricks to communicate with APIs to improve data. Image by author</figcaption></figure><h4><strong>Automated harmonization, labeling, and data generation</strong></h4><p>By establishing data pipelines, organizations can utilize GenAI as new data enters their systems. For instance, when new datasets come in, the API can automatically apply data harmonization algorithms or identify patterns to infer labels. This removes the need for manual data cleaning and preprocessing, freeing up data engineers to concentrate on more valuable tasks. Although GenAI shows great promise, it’s important to recognize data privacy issues with public APIs.</p><p>Integrating the API into your data pipeline enables you to generate diverse and realistic datasets directly in your training notebook. The API can also create synthetic data to fill gaps in existing datasets, supporting more robust AI model development. This automated data generation not only speeds up research but also minimizes privacy concerns.</p><h3>Conclusion</h3><p>Integrating GenAI APIs into data quality workflows offers a powerful way to automate data cleaning, labeling, and generation. This seamless integration helps organizations fully leverage GenAI’s capabilities without manual intervention, making data management more efficient and improving overall data quality.</p><p>In summary, the intersection of AI and data quality marks a significant turning point in analytics. GenAI’s ability to boost data quality and provide actionable insights has the potential to transform the industry. By rethinking traditional approaches and using AI to enhance data, organizations can unlock new opportunities for innovation and growth. As we move forward, it’s clear that the future of analytics will be shaped by those who embrace the power of GenAI.</p><p><a href="https://medium.com/@jonas_dieckmann">Jonas Dieckmann - Medium</a></p><p>I hope you find it useful. Let me know your thoughts! And feel free to connect on LinkedIn <a href="https://www.linkedin.com/in/jonas-dieckmann/">https://www.linkedin.com/in/jonas-dieckmann/</a> and/or to follow me here on Medium.</p><h4>References</h4><p>[1] Gartner (2023): Hype Cycle for Emerging Technologies<br><a href="https://www.gartner.com/en/newsroom/press-releases/2023-08-16-gartner-places-generative-ai-on-the-peak-of-inflated-expectations-on-the-2023-hype-cycle-for-emerging-technologies">https://www.gartner.com/en/newsroom/press-releases/2023-08-16-gartner-places-generative-ai-on-the-peak-of-inflated-expectations-on-the-2023-hype-cycle-for-emerging-technologies</a></p><p>[2] OpenAI — ChatGPT: <a href="https://chatgpt.com/">https://chatgpt.com/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d3a8d5cc5ba9" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/innovations-in-analytics-elevating-data-quality-with-genai-d3a8d5cc5ba9">Innovations in Analytics: Elevating Data Quality with GenAI</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI in healthcare: Addressing ethical bias for better outcomes]]></title>
            <link>https://medium.com/philips-technology-blog/ai-in-healthcare-addressing-ethical-bias-for-better-outcomes-31a1da961acf?source=rss-1c8d1cf684f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/31a1da961acf</guid>
            <category><![CDATA[healthcare]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[philips]]></category>
            <category><![CDATA[ethics]]></category>
            <dc:creator><![CDATA[Jonas Dieckmann]]></dc:creator>
            <pubDate>Thu, 03 Oct 2024 12:24:51 GMT</pubDate>
            <atom:updated>2024-10-03T12:24:51.773Z</atom:updated>
            <content:encoded><![CDATA[<p>As the integration of artificial intelligence (AI) in healthcare accelerates, so does the urgency to address ethical concerns, particularly regarding biased algorithms. While blaming developers or data analysts for biased outcomes is convenient, a deeper examination reveals that ethical data abuse may be the root cause of societal challenges. Examining the intersection of AI ethics and healthcare reveals that bias in algorithms is mainly rooted in the biases of our society, which are reflected in the datasets used to train our models.</p><p>However, bias in algorithms extends beyond just the data input side. It can manifest at various stages of the AI lifecycle, including data processing, algorithm development, and deployment. At Philips, we understand the multifaceted nature of bias and take a holistic approach to address it throughout the entire AI lifecycle. While this article primarily focuses on the biases present in the data input side, it’s crucial to acknowledge that bias can infiltrate every step of the AI development process, resulting in our commitment to tackling bias comprehensively at Philips.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8ANrZGHsoKbB6Rz5rA9IcA.png" /></figure><p>The promise of AI in healthcare is substantial, offering improved diagnostics, treatment personalization, and overall patient care. However, alongside these advancements come ethical considerations that must not be overlooked. Data ethics, a branch of ethics focused on moral issues surrounding data use, algorithms, and associated practices, plays a pivotal role in ensuring AI applications in healthcare adhere to ethical standards.¹ We at Philips recognize the importance of navigating these challenges responsibly.</p><blockquote>A better world won’t come about simply because we use data; data has its dark underside. ²</blockquote><h4>Biased Algorithms in Healthcare</h4><p>Today, discussions of data science are often associated with significant opportunities in industry and healthcare to deliver more accurate predictive and classification solutions and improve people’s lives.³ ⁴ Perhaps as important as the benefits are the data ethics challenges that should be considered when evaluating new solutions and approaches in data analytics. Overall, there seems to be a fine line between the use and misuse of data. It is common for companies to collect data not only to increase their profits but also to provide a tailored and targeted experience to their customers.⁵ However, when companies start to externally exploit their data for purposes other than those for which it was collected, ethical questions arise.</p><p>In healthcare, the implications of biased algorithms can be profound, affecting diagnoses, treatment recommendations, and patient outcomes. One notable concern is the potential for racial or gender bias in predictive models. For instance, algorithms used to predict patient outcomes may inadvertently perpetuate disparities if historical data reflect existing biases in healthcare delivery. Such biases can result in differential treatment recommendations or resource allocation, further exacerbating healthcare disparities.</p><h3>Well-known examples of algorithmic discrimination</h3><p>The issues are not theoretical — they lead to real concerns and cause serious discussions as described in several case studies:</p><ul><li><strong>Racial bias in medical recommendation models</strong><br>A study by Vyas et al. found racial bias in an algorithm, that was used to help assess the suitability of Vaginal Birth after Cesarean Delivery (VBAC) for subsequent births. The model considers many things, such as the woman’s age, the reason for the previous C-section, and how long ago it happened. The study has shown racial bias in the algorithm, resulting in disproportionately higher rates of recommending (non-required) C-sections for Black/African American and Hispanic/Latino women compared to other demographic groups.⁶</li><li><strong>Discrimination in Crime Prediction</strong><br>Further problematic examples of biased data science approaches can be found within a risk assessment algorithm used in the United States to predict the likelihood of re-offending (repeated criminal behavior) by arrested individuals. It turned out that the underlying algorithm tended to predict the likelihood of recidivism of black defendants twice as high as that of white defendants, while white defendants tended to be incorrectly classified as low-risk when in fact they were recidivists. This bias stemmed from historical imbalances in the criminal justice system, where factors such as over-policing in minority communities and disparities in sentencing contributed to unequal outcomes in arrest records and thus biased training data for the algorithm.⁷</li><li><strong>Gender discrimination in recruitment</strong><br>Another area that is affected by biased algorithms is recruitment. In 2014, Amazon used machine learning-based software to evaluate and rank the CVs of potential candidates to find new top talent. In 2015, the company discovered that its new system was not evaluating applicants in a gender-neutral way, especially for software development jobs. Based on hiring over the last ten years, the system penalized all applications that included the word “women” in their CVs. This bias was rooted in historical hiring patterns within the company, where male-dominated roles in the tech industry were favored, leading to imbalanced training data that perpetuated gender bias in the algorithm.⁸</li></ul><h4>Importance of data ethics in healthcare</h4><p>Recognizing the critical importance of addressing bias in healthcare, Philips has a clear commitment to fairness, transparency, and accountability throughout the AI lifecycle. Bias in AI applications is one of the most ethical problems of Big Data, as many people assume that machines are neutral by definition. In reality, this is not the case, and the potential risk can be very high.⁹ But what are possible reasons for the lack of neutrality? It is hard to imagine that the best data scientists in the world have developed their algorithms with this goal in mind.</p><h3>Root causes analysis</h3><p>There are many popular examples where algorithms learn and act on the wrong causes, which results in skewed outcomes. Therefore, algorithmic bias should not be viewed solely through the prism of neutrality but also with a focus on (missing) causes.</p><p>The following analysis, therefore, aims to further explore issues related to bias and how today’s ethical pressure on developers could be understood as a collective responsibility. It is crucial to underline the importance of eliminating discrimination in the both technical and non-technical world, as the two are directly linked and influence each other:</p><ol><li><strong>A biased society leads to biased data<br></strong>By design, <strong>discrimination takes place before the data is even collected, as datasets are historical collections that have been influenced by society itself</strong>. To think about the technical aspects of this issue, one needs to examine the possible causes of biased systems. In many papers and studies, the authors state that the algorithms themselves are biased and cause discrimination.¹⁰ ¹¹ However, it is difficult to ascribe the very cause of discrimination to algorithms themselves, since algorithms perform decision rules based on a given input and the interactions of given parameters in an automated way. ¹² The rationale behind machine learning algorithms is usually an optimization problem where rules are applied to parameters based on their past effect on the outcome.¹³ The presence of bias in the interaction between a given variable (e.g., gender) and the target (e.g., treatment recommendation) is therefore largely influenced by the underlying data set. Looking at the VBAC recommendation example, the biased outcome (i.e., higher C-section rates for Black/African American and Hispanic/Latino women) can be attributed to the training dataset initially fed to the algorithm. Assuming that the majority of patient records contained white women, the algorithm got skewed. Hence, discrimination does not begin with the algorithm or its developer as an individual, but rather with the differences in healthcare systems between races, which has been perpetrated for decades. However, not every bias is bad bias, and given our great diverse society, there will always be differences in human behavior. Nevertheless, the aim should be to take out those indicators that can potentially lead to discrimination.</li><li><strong>Hidden bias between variables<br></strong>Provocatively, <strong>one could argue that every dataset is biased to some extent since it originated in a likely biased environment</strong>. An algorithm can only factor in what is present in the (biased) data, and data usually only contains what is present in the (biased) non-technical world. A reasonable and common-sense approach would be to remove such discriminatory variables from the dataset before beginning to model. However, there is no guarantee that removing a variable will remove the presence or influence of that variable on the rest of the dataset. Consider a fictitious dataset containing the variables <em>gender</em> and <em>physical activity</em>. Both variables could have an interaction, such as a “correlation between men and football and women and riding”, although this blunt example may not fully represent our society, since there may be women who prefer football to ride and vice versa. However, due to an unbalanced (e.g., male-heavy) dataset, it is likely that the algorithm indirectly favors candidates with football as physical activity, as the training process is based on biased information. This example shows that removing critical variables can reduce the risk of discrimination, but is not sufficient to prevent it. Moreover, it is of great importance how balanced the data sets represent the different expressions of a variable. A sensitive (discriminatory) item may be less problematic if the expressions are evenly distributed.¹⁴</li><li><strong>Technical decisions vs. human decisions</strong><br><strong>There is no guarantee that humans make less biased decisions than algorithms</strong>. In addition to the idea that any dataset can be biased and lead to biased results in an algorithmic process, another key indicator of unfair predictions/classifications is the lack of information, in the form of incomplete data.¹⁵ It is difficult to guarantee that a given dataset possesses full visibility to explain a target behavior. Common reasons for the lack of data may be limited access, active exclusion of information, or lack of knowledge about the marginal impact of additional information. Another risk for discrimination does not only result from the technical implementation but can also be rooted in the existing structures of our society. The advantage of Big Data in combination with machine learning algorithms is the possibility to apply human rules and decisions to a large amount of information. If algorithms are considered biased, one might wonder how e.g., doctors would have processed the analysis of given data. Algorithms process automated and standardized decision rules based on general rather than individual patterns. Ultimately, it can be assumed that <strong>algorithms support collectivism </strong>rather than individualism.¹⁶</li></ol><h3>Conclusion</h3><p>The analysis has shown that the bias in algorithms is mainly a result of bias in our society. The latter supports the collection of biased datasets, which may contain (hidden) relationships, ultimately leading to biased results. However, it’s important to recognize that bias can also be introduced at various stages beyond just the data collection phase, such as data processing, algorithm development, and deployment. Furthermore, we question whether human decision-making is less biased compared to algorithmic rules.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ESBnycwULog666pNudsOKA.png" /></figure><h4><strong>Mitigation strategy</strong></h4><p>In discussions about the responsibility for biased results of machine learning algorithms, the developer or the data science team is often blamed for discriminatory behavior. On one hand, this argument is overly simplistic, and the apportionment of the blame needs to be further differentiated. On the other, any developer working with machine learning algorithms should be aware of the sensitivity of their task and their respective responsibilities.</p><p>Since algorithms support collectivism, it is important to investigate the fairness of these collective algorithms that make decisions or suggestions for individuals. Literature suggests other techniques to reduce bias, such as a system design that randomizes a small sample of queries or the use of propensity weighting techniques.¹⁷ For bias in human-based content, there are several ideas for quantifying discrimination and assessing the fairness of algorithms.¹⁸</p><p><strong>Fairness as a shared responsibility: both in the technical and real world<br></strong>In tackling biased algorithms, Philips emphasizes a collaborative approach, recognizing that responsibility extends beyond data scientists to encompass healthcare practitioners, policymakers, and society at large. It is the responsibility of data teams to identify and minimize biased behavior in their data models. However, looking at the discriminatory factors examined in this article, it is a global responsibility to ensure that no one experiences such injustices, either through technical systems or human actions:</p><ol><li>Humans must treat humans equally and without discrimination</li><li>Algorithms must treat humans equally and without discrimination</li><li>We should not hide behind algorithms to justify biased and discriminatory human behavior</li></ol><h4><strong>Outlook: now what?</strong></h4><p>As AI continues to transform healthcare, addressing bias in algorithms is paramount to ensuring equitable and high-quality patient care. By integrating ethical principles into its AI development processes, Philips demonstrates its commitment to responsible innovation in healthcare. Moving forward, collaborative efforts and ethical vigilance will be essential in navigating the complex intersection of AI and healthcare ethics.</p><p>It is important to evaluate decisions or predictions not only based on the outcome itself but also on the ethical considerations involved in working out the solution. We also need to rethink systems that involve biased behavior. However, there is no evidence that the alternative (human-based decision-making) leads to more equality in general. Consequently, we should not only focus on the problems with bias <em>within</em> technology but rather broaden the discussion to include bias <em>outside</em> technology. Nevertheless, <em>precisely because of this</em>, <strong>unbiased systems should help improve our current social weaknesses</strong> and possibly lead to <strong>more equality in the world to come. </strong>People could use AI to more easily detect discrimination in the non-technological world. Because <strong>wherever we find bias in the data, there is also bias in society</strong>.</p><p>Consequently, Philips advocates for ongoing monitoring and evaluation of AI systems to detect and address bias proactively. Through continuous refinement and validation, we endeavor to uphold the highest ethical standards in healthcare AI initiatives.</p><p><strong>Curious about working in tech at Philips?<em> </em></strong><a href="https://www.careers.philips.com/global/en/philips-tech-careers"><strong><em>Find out more here</em></strong></a></p><p><a href="https://www.careers.philips.com/global/en/philips-tech-careers?source=post_page-----d47698235094--------------------------------">Tech Careers | Philips</a></p><p><strong>Make sure to follow our Tech Blog in case you liked this article!</strong></p><p><a href="https://medium.com/philips-technology-blog">Philips Technology Blog</a></p><p>I hope you find it interesting. Let me know your thoughts and feel free to connect on LinkedIn <a href="https://www.linkedin.com/in/jonas-dieckmann/">https://www.linkedin.com/in/jonas-dieckmann</a> and/or to follow me here on medium.</p><p><a href="https://medium.com/@jonas_dieckmann">Jonas Dieckmann - Medium</a></p><h3>References</h3><p>[1]: Mittelstadt, B., &amp; Floridi, L. (2015). The Ethics of Big Data: Current and Foreseeable Issues in Biomedical Contexts. Science And Engineering Ethics, 22(2), 303–341. DOI: 10.1007/s11948–015- 9652–2</p><p>[2]: Loukides, M., Mason, H., &amp; Patil, D. (2018). Ethics and Data Science.</p><p>[3]: Floridi, L., &amp; Taddeo, M. (2016). What is data ethics? Philosophical Transactions Of The Royal Society A: Mathematical, Physical, And Engineering Sciences, 374(2083), 20160360. doi: 10.1098/rsta.2016.0360</p><p>[4]: Google (2021). AI FOR SOCIAL GOOD, Applying AI to some of the world’s biggest challenges Retrieved 10 March 2021, from <a href="https://ai.google/social-good">https://ai.google/social-good</a></p><p>[5] Hand, D. (2018). Aspects of Data Ethics in a Changing World: Where Are We Now? Big Data, 6(3), 176–190. DOI: 10.1089/big.2018.0083</p><p>[6] Vyas DA, Jones DS, Meadows AR, Diouf K, Nour NM, Schantz-Dunn J. Challenging the Use of Race in the Vaginal Birth after Cesarean Section Calculator. Womens Health Issues. 2019 May-Jun;29(3):201–204. doi: 10.1016/j.whi.2019.04.007. Epub 2019 May 6. PMID: 31072754.</p><p>[7] Angwin, J., Larson, J., Mattu, S., &amp; Kirchner, L. (2016). Machine Bias. Retrieved 28 February 2021, from <a href="https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing">https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing</a></p><p>[8] Dastin, J. (2018). Amazon scraps secret AI recruiting tool that showed bias against women. Retrieved 28 February 2021, from <a href="https://www.reuters.com/article/us-amazon-com-jobsautomation-insight-idUSKCN1MK08G">https://www.reuters.com/article/us-amazon-com-jobsautomation-insight-idUSKCN1MK08G</a></p><p>[9] European Economic and Social Committee. (2017). The ethics of Big Data.</p><p>[10] Eslami, M., Vaccaro, K., Karahalios, K. and Hamilton, K. (2017) ‘Be Careful; Things Can Be Worse than They Appear’: Understanding Biased Algorithms and Users’ Behavior Around Them in Rating Platforms. Proceedings of the International AAAI Conference on Web and Social Media, 11(1).</p><p>[11] Kim, P. (2017) Auditing Algorithms for Discrimination, 166 U. Pa. L. Rev. Retrieved 10 March 2021, from scholarship.law.upenn.edu/penn_law_review_online/vol166/iss1/10</p><p>[12] Mittelstadt, B., Allo, P., Taddeo, M., Wachter, S., &amp; Floridi, L. (2016). The ethics of algorithms: Mapping the debate. Big Data &amp; Society, 3(2), 205395171667967. DOI: 10.1177/2053951716679679</p><p>[13] Lehr, D., Ohm, P. (2017). Playing with the Data: What Legal Scholars Should Learn About Machine Learning, UCDL Review 51: 653, 671.</p><p>[14] Calders, T., &amp; Žliobaitė, I. (2013). Why Unbiased Computational Processes Can Lead to Discriminative Decision Procedures. Studies In Applied Philosophy, Epistemology, And Rational Ethics, 43–57. DOI: 10.1007/978–3–642–30487–3_3</p><p>[15] Rosen, R. (2019). Missing Data and Imputation. Retrieved 10 March 2021, from <a href="https://towardsdatascience.com/missing-data-and-imputation-89e9889268c8">https://towardsdatascience.com/missing-data-and-imputation-89e9889268c8</a></p><p>[16] Saake. I., Nassehi, A. (2004). Die Kulturalisierung der Ethik. Eine zeitdiagnostische Anwendung des Luhmannschen Kulturbegriffs. Frankfurt/M: 102, 135</p><p>[17] Ridgeway, G., Kovalchik, S., Griffin, B., &amp; Kabeto, M. (2015). Propensity Score Analysis with Survey Weighted Data. Journal Of Causal Inference, 3(2), 237–249. doi: 10.1515/jci-2014–0039</p><p>[18] Floridi, L. (2016). Faultless responsibility: on the nature and allocation of moral responsibility for distributed moral actions. Philosophical Transactions Of The Royal Society A: Mathematical, Physical, And Engineering Sciences, 374(2083), 20160112. doi: 10.1098/rsta.2016.0112</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=31a1da961acf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/philips-technology-blog/ai-in-healthcare-addressing-ethical-bias-for-better-outcomes-31a1da961acf">AI in healthcare: Addressing ethical bias for better outcomes</a> was originally published in <a href="https://medium.com/philips-technology-blog">Philips Technology Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Focusing on Impact: Pitfalls and Maturity Levels in Data Analytics]]></title>
            <link>https://medium.com/philips-technology-blog/focusing-on-impact-pitfalls-and-maturity-levels-in-data-analytics-d47698235094?source=rss-1c8d1cf684f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/d47698235094</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[maturity]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-strategy]]></category>
            <dc:creator><![CDATA[Jonas Dieckmann]]></dc:creator>
            <pubDate>Mon, 03 Jun 2024 18:09:59 GMT</pubDate>
            <atom:updated>2024-07-02T11:30:13.234Z</atom:updated>
            <content:encoded><![CDATA[<p>A myriad of challenges arise each day for data and AI professionals. With the increasing impact of technology, analytics teams must continue to prove their impact on businesses. Working in data analytics at Philips, I want to share the most common observations of commercially focused data teams and offer three practical recommendations to increase trust within a company:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*poPGEXYOqKzVmLsv.png" /></figure><ol><li>Ensure that every<strong> data problem is treated as a business problem</strong></li><li>Focus on relevant KPI<strong> impact and</strong> <strong>outcome </strong>rather than output</li><li>Grow the teams’ <strong>maturity over time</strong> and don’t skip the basics</li></ol><p>Let’s explore these underlying problem areas in more detail.</p><h3>Domain Knowledge, Business Relevance and Expectations</h3><p>One of the most common pitfalls I have encountered is the disconnect between the objectives of a data team and the overarching business goals. Without a clear understanding of the business context and requirements, data initiatives often fall short of delivering meaningful value. This lack of alignment leads to frustration on both ends with stakeholders feeling underwhelmed by the outputs and data teams grappling with vague directives. In most cases, when data products or teams fail, they don’t fail for technical reasons; they fail because they delivered something that did not lead to real outcomes for stakeholders.</p><p>As Leandro Carvalho summarized in his <a href="https://medium.com/@leandroscarvalho/data-product-canvas-a-practical-framework-for-building-high-performance-data-products-7a1717f79f0">Data Product Canvas article</a>: <br>“The success of a data-driven culture depends on the definition and implementation of strategies, not technologies. That’s why it’s important to make it clear that data products are a business domain problem, not a technology one.”</p><p>In data analytics, the true essence of business needs is sometimes obscured beneath surface-level requests. It’s crucial to dive deeper, fostering a culture of inquiry to unearth the root causes behind these needs. A tested methodology involves continuously asking “why” until the underlying issue is revealed. For instance, when someone requests a new data product, probing further with questions like “why do you need/want/can’t X?” can lead to valuable insights. Through this iterative process, we can uncover the real challenges and requirements faced by different business units or product groups. By understanding these nuances, data teams can tailor their solutions to address the specific needs of the business effectively.</p><p>Another hurdle that can impede progress is if there is an absence of domain expertise within data teams. Whether it’s due to organizational silos or a failure to integrate domain knowledge into the analytics process, this deficiency can hinder a team’s ability to generate actionable insights. Without a deep understanding of the industry and business dynamics, data analyses may overlook critical factors or misinterpret trends. This need has led to the rise of data-mesh/hub-and-spoke architectures in various industries, such as <a href="https://thecuberesearch.com/breaking-analysis-how-jp-morgan-is-implementing-a-data-mesh-on-the-aws-cloud/">JP Morgan</a>, <a href="https://tech.deliveryhero.com/meet-data-hub-delivery-heros-data-mesh-platform/">Delivery Hero</a>, and <a href="https://medium.com/intuit-engineering/intuits-data-mesh-strategy-778e3edaa017">Intuit</a>. The idea is simple: while providing infrastructure and data platforms centrally (hub), bringing business knowledge into data products can only work by making these business teams responsible for them in a decentralized way (spokes). This promotes collaboration and reliability.</p><p>Numbers only make sense in context. When numbers don’t have proper context, even the best data scientists in the world can treat them just as numbers in the end. Hence, there is no way around making this the top priority when working in analytics: whatever you work on has to deliver relevant outcomes for (and together with) your stakeholders to solve a problem. Nobody claps for nice-to-haves.</p><h3>Data Access &amp; Data Quality</h3><p>The second area of trouble is centered around the data itself. One of the challenges analysts often face is ensuring the reliability of the data they use. Access to accurate data is vital for meaningful analysis, yet many analysts face obstacles related to data quality and accessibility. The bigger the environment, the higher the risk of not knowing which data exists or who to ask for it, and the higher the risk of the data not being in good shape to analyze.</p><p>Trust is key. When raw data is seen as unreliable, it casts doubt on analysis results, leading to skepticism from stakeholders and reducing the chance that data teams’ work leads toward meaningful outcomes. Therefore, establishing clear data governance is crucial: data teams have to know who owns the data and where they can find it. Clear ownership can also help with data quality. When analysts are unsure of which data they can trust most for their project, they should ask stakeholders for help. They likely have an opinion on which data makes sense for their domain and which data doesn’t.</p><h3>Maturity &amp; Stakeholder Buy-In</h3><p>Thirdly, let’s consider stakeholder buy-in. Every data person loves nice dashboards, fancy machine learning models, and (recently) the exploration of GenAI use cases. That resonates with an analyst’s intrinsic perspective, but illustrates the risk of focusing on the wrong things. In the rush to showcase sophistication, data teams tend to bypass the foundational stages of maturity. Without a solid groundwork in place, attempts to tackle complex problems often result in inefficiencies and missed opportunities for incremental improvements. There is no point in building AI models when even simple reporting is not in place.</p><p>While the Gartner maturity framework is a well-known model (from descriptive to prescriptive), it focuses more on the methodologies of data analytics, rather than the process of being a good business partner to business stakeholders. Therefore, we would summarize the maturity level of data teams as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*JoDNY_zOdyHJr2nJ.png" /></figure><ul><li><strong>Level 1: Accurate Reporting<br></strong>As the basic foundation, data teams need to be recognized as the ‘go-to’ organization for data needs. This could result in running several well-known reports that act as the source of truth for a wider range of stakeholders.</li><li><strong>Level 2: Insights Facilitator</strong><br>Once the data is reliable, analysts can act as insights facilitators to solve analytical and business problems. They don’t <em>just</em> report the numbers anymore, but try to put them in a meaningful context, which eventually leads to data-driven decision-making.</li><li><strong>Level 3: Break the Bottleneck</strong><br>The downside of good data and smart analysts is that they become popular and get too many requests. That is a good moment to scale capabilities, invest in self-service functionality, and consider organizational changes as well as better infrastructure.</li><li><strong>Level 4: Proactive Business Partner</strong><br>The last stage turns the data team from reactive to proactive. It is time to start generating hypotheses to challenge and support stakeholders, rather than letting them dictate what matters.</li></ul><p>As data teams grow over time, it is important to stick to the basics: starting small and focusing on trust. There is a smaller chance that self-service functionality will be appreciated if the underlying data is not trustworthy, or if a team is not seen as a reliable data partner. Reacting to the needs of the organization and turning into a proactive data organization over time is the last step.</p><h3>Conclusion</h3><p>Trying to solve a company’s analytics problems can be overwhelming. Especially in large organizations, it is impossible to solve data quality problems in isolation. Teams should focus more on what is possible to directly influence rather than tackling things that are too dependent on external factors.</p><p>It’s also necessary to ensure that every data problem is treated as a <strong>business problem </strong>and not just a technology problem. Data products should be related to concrete business questions and embedded in related business processes.</p><p>And teams must show the impact KPIs and <strong>outcomes </strong>rather than just output. The value of data products should never be measured by output alone. Instead, aim to give it a proper price tag: make visible what additional revenue was generated or influenced. And highlight the costs and time saved.</p><p>Lastly, it is pivotal to ensure that teams’ <strong>maturity grows over time</strong> and the basics are solidly in place. Successful (AI) applications depend on solid data foundations and good data quality.</p><p><strong>Curious about working in tech at Philips?<em> </em></strong><a href="https://www.careers.philips.com/global/en/philips-tech-careers"><strong><em>Find out more here</em></strong></a></p><p><a href="https://www.careers.philips.com/global/en/philips-tech-careers?source=post_page-----409cdd23bf28--------------------------------">Tech Careers | Philips</a></p><p><strong>Make sure to follow our Tech Blog in case you liked this article!</strong></p><p><a href="https://medium.com/philips-technology-blog">Philips Technology Blog</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d47698235094" width="1" height="1" alt=""><hr><p><a href="https://medium.com/philips-technology-blog/focusing-on-impact-pitfalls-and-maturity-levels-in-data-analytics-d47698235094">Focusing on Impact: Pitfalls and Maturity Levels in Data Analytics</a> was originally published in <a href="https://medium.com/philips-technology-blog">Philips Technology Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Started Predicting Time Series Data with Facebook Prophet]]></title>
            <link>https://medium.com/data-science/getting-started-predicting-time-series-data-with-facebook-prophet-c74ad3040525?source=rss-1c8d1cf684f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/c74ad3040525</guid>
            <category><![CDATA[prophet]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Jonas Dieckmann]]></dc:creator>
            <pubDate>Tue, 30 Jan 2024 17:57:24 GMT</pubDate>
            <atom:updated>2024-05-28T14:39:14.629Z</atom:updated>
            <content:encoded><![CDATA[<h4>This article aims to take away the entry barriers to get started with time series analysis in a hands-on tutorial using Prophet</h4><p>When getting started with data science, time series analysis is a common thing people would love to try themselves! The general idea here is to learn from historical patterns over time to predict the future. Typical use cases could be weather predictions or sales forecasting. But what does all this have to do with this wise prophet below?!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/0*gHRE9pon-unvA-sq.jpg" /><figcaption>AI-generated image by leonardo.ai</figcaption></figure><p>This article aims to take away the entry barriers to get started with time series analysis in a hands-on tutorial using one of the easiest tools called <a href="https://facebook.github.io/prophet/">Facebook Prophet</a> within <a href="https://colab.research.google.com/">Google Colab</a> (both are free!). In case you want to get started immediately, feel free to skip the next two chapters where I will give a short background on time series principles and also Facebook Prophet itself. Have fun!</p><p>This article is structured into three main sections:</p><p>#1 Brief introduction to Time Series Analysis principles<br>#2 An Introduction to Facebook Prophet <br>#3 Hands-on tutorial on how to use Prophet in Google Colab (for free)</p><h3>#1 General Principles of Time Series Analysis</h3><p>Imagine you are a store manager for consumer products and you want to predict the upcoming product demand to better manage the supply. A reasonable machine learning approach for this scenario is to run some time series analysis which involves understanding, modeling, and making predictions based on sequential data points. [1]</p><p>The below graphic illustrates an artificial development of historic product demand (dark-blue line) over time, which can be used to analyze a time series pattern. Our ultimate goal would be to predict (red-dotted line) the actual future demand (light-blue line) as precise as possible:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*viwKD0fddQEv5g0Y5pcUxg.png" /><figcaption>Example of a plotted time series. Image by author</figcaption></figure><p>A time series is typically decomposed into three main components:</p><ol><li><strong>Trend</strong>: the long-term movement or general direction in the data.</li><li><strong>Seasonality</strong>: fluctuations or patterns that repeat at regular intervals</li><li><strong>Residual/error</strong>: remainder or leftover variation in the data</li></ol><p>The decomposition of a time series into these three components, often referred to as additive or multiplicative decomposition, allows analysts to better understand the underlying structure and patterns. This understanding is essential for selecting appropriate forecasting models and making accurate predictions based on historical data. [2]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O_TTrIydOCuPt3OcnVb9gg.png" /><figcaption>Example of a decomposed time series into trend + seasonality + error. Image by author</figcaption></figure><h3>#2 What is Facebook Prophet?</h3><p>Prophet is an open-source tool released by Facebook&#39;s Data Science team that produces time series forecasting data based on an additive model where a non-linear trend fits with seasonality and holiday effects. The design principles allow parameter adjustments without much knowledge of the underlying model which makes the method applicable to teams with less statistical knowledge. [3]</p><p>Prophet is particularly well-suited for business forecasting applications, and it has gained popularity due to its ease of use and effectiveness in handling a wide range of time series data. As with every tool, keep in mind that while Prophet is powerful, the choice of forecasting method depends on the specific characteristics of the data and the goals of the analysis. In general, it is not granted that Prophet performs better than other models. However, Prophet comes with some useful features e.g., a reflection of seasonality change pre- and post-COVID or treating lockdowns as one-off holidays.</p><p>For a more in-depth introduction by Meta (Facebook) itself, look at the video below<a href="https://www.youtube.com/watch?v=eJrbKU09h-0&amp;ab_channel=MetaOpenSource"> on YouTube</a>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FeJrbKU09h-0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DeJrbKU09h-0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FeJrbKU09h-0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/947daf5bcaacdffeb9fe2b6b3182cfa1/href">https://medium.com/media/947daf5bcaacdffeb9fe2b6b3182cfa1/href</a></iframe><p>In the following tutorial, we will implement and use Prophet with Python. However, you are more than happy to run your analysis using R as well!</p><h3>#3 Hands-on tutorial on how to use Prophet</h3><p>In case you have limited experience with or no access to your coding environment, I recommend making use of <a href="https://colab.research.google.com/">Google Colaboratory (“Colab”)</a> which is somewhat like “<em>a</em> <em>free Jupyter notebook environment that requires no setup and runs entirely in the cloud</em>.” While this tutorial claims more about the simplicity and advantages of Colab, there are drawbacks as reduced computing power compared to proper cloud environments. However, I believe Colab might not be a bad service to take the first steps with Prophet.</p><p>To set up a basic environment for Time Series Analysis within Colab you can follow these two steps:</p><ol><li>Open <a href="https://colab.research.google.com/">https://colab.research.google.com/</a> and register for a free account</li><li>Create a new <em>notebook </em>within Colab</li><li>Install &amp; use the <em>prophet </em>package:</li></ol><pre>pip install prophet</pre><pre>from prophet import Prophet</pre><h4>Loading and preparing Data</h4><p>I uploaded a small dummy dataset representing the monthly amount of passengers for a local bus company (2012–2023). You can find the data <a href="https://raw.githubusercontent.com/jonasdieckmann/prophet_tutorial/main/passengers.csv">here on GitHub</a>.</p><p>As the first step, we will load the data using pandas and create two separate datasets: a training subset with the years 2012 to 2022 as well as a test subset with the year 2023. We will train our time series model with the first subset and aim to predict the passenger amount for 2023. With the second subset, we will be able to validate the accuracy later.</p><pre>import pandas as pd<br><br>df_data = pd.read_csv(&quot;https://raw.githubusercontent.com/jonasdieckmann/prophet_tutorial/main/passengers.csv&quot;)<br><br>df_data_train = df_data[df_data[&quot;Month&quot;] &lt; &quot;2023-01&quot;]<br>df_data_test = df_data[df_data[&quot;Month&quot;] &gt;= &quot;2023-01&quot;]<br><br>display(df_data_train)</pre><p>The output for the display command can be seen below. The dataset contains two columns: the indication of the year-month combination as well as a numeric column with the passenger amount in that month. Per default, Prophet is designed to work with daily (or even hourly) data, but we will make sure that the monthly pattern can be used as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/209/1*9G45IjOk8vx86oPZV-CalA.png" /><figcaption>Passenger dataset. Image by autor</figcaption></figure><h4>Decomposing training data</h4><p>To get a better understanding of the time series components within our dummy data, we will run a quick decomposing. For that, we import the method from <em>statsmodels </em>library and run the decomposing on our dataset. We decided on an additive model and indicated, that one period contains 12 elements (months) in our data. A daily dataset would be period=365.</p><pre>from statsmodels.tsa.seasonal import seasonal_decompose<br><br>decompose = seasonal_decompose(df_data_train.Passengers, model=&#39;additive&#39;, extrapolate_trend=&#39;freq&#39;, period=12)<br><br>decompose.plot().show()</pre><p>This short piece of code will give us a visual impression of time series itself, but especially about the trend, the seasonality, and the residuals over time:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*YiO-bCXIuFctJWuy4ALBCg.png" /><figcaption>Decomposed elements for the passenger dummy data. Image by author</figcaption></figure><p>We can now clearly see both, a significantly increasing trend over the past 10 years as well as a recognizable seasonality pattern every year. Following those indications, we would now expect the model to predict some further increasing amount of passengers, following the seasonality peaks in the summer of the future year. But let’s try it out — time to apply some machine learning!</p><h4>Model fitting with Facebook Prophet</h4><p>To fit models in Prophet, it is important to have at least a ‘ds’ (datestamp) and ‘y’ (value to be forecasted) column. We should make sure that our columns are renamed the reflect the same.</p><pre>df_train_prophet = df_data_train<br><br># date variable needs to be named &quot;ds&quot; for prophet<br>df_train_prophet = df_train_prophet.rename(columns={&quot;Month&quot;: &quot;ds&quot;})<br><br># target variable needs to be named &quot;y&quot; for prophet<br>df_train_prophet = df_train_prophet.rename(columns={&quot;Passengers&quot;: &quot;y&quot;})</pre><p>Now the magic can begin. The process to fit the model is fairly straightforward. However, please have a look at the <a href="https://facebook.github.io/prophet/docs/handling_shocks.html">documentation </a>to get an idea of the large amount of options and parameters we could adjust in this step. To keep things simple, we will fit a simple model without any further adjustments for now — but please keep in mind that real-world data is never perfect: you will definitely need parameter tuning in the future.</p><pre>model_prophet  = Prophet()<br>model_prophet.fit(df_train_prophet)</pre><p>That’s all we have to do to fit the model. Let’s make some predictions!</p><h4>Making predictions</h4><p>We have to make predictions on a table that has a ‘ds’ column with the dates you want predictions for. To set up this table, use the <em>make_future_dataframe</em> method, and it will automatically include historical dates. This way, you can see how well the model fits the past data and predicts the future. Since we handle monthly data, we will indicate the frequency with “freq=12&quot; and ask for a future horizon of 12 months (“periods=12”).</p><pre>df_future = model_prophet.make_future_dataframe(periods=12, freq=&#39;MS&#39;)<br>display(df_future)</pre><p>This new dataset then contains both, the training period as well as the additional 12 months we want to predict:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/153/1*OWeD3a-4nANeemRsFXWSTg.png" /><figcaption>Future dataset. Image by author</figcaption></figure><p>To make predictions, we simply call the <em>predict </em>method from Prophet and provide the future dataset. The prediction output will contain a large dataset with many different columns, but we will focus only on the predicted value <em>yhat </em>as well as the uncertainty intervals <em>yhat_lower </em>and <em>yhat_upper</em>.</p><pre>forecast_prophet = model_prophet.predict(df_future)<br>forecast_prophet[[&#39;ds&#39;, &#39;yhat&#39;, &#39;yhat_lower&#39;, &#39;yhat_upper&#39;]].round().tail()</pre><p>The table below gives us some idea about how the output is generated and stored. For August 2023, the model predicts a passenger amount of 532 people. The uncertainty interval (which is set by default to 80%) tells us in simple terms that we can expect most likely a passenger amount between 508 and 556 people in that month.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*2h6AfVHA5-DqM91sk0U5mA.png" /><figcaption>Prediction subset. Image by author</figcaption></figure><p>Finally, we want to visualize the output to better understand the predictions and the intervals.</p><h4>Visualizing results</h4><p>To plot the results, we can make use of Prophet’s built-in plotting tools. With the plot method, we can display the original time series data alongside the forecasted values.</p><pre>import matplotlib.pyplot as plt<br><br># plot the time series <br>forecast_plot = model_prophet.plot(forecast_prophet)<br><br># add a vertical line at the end of the training period<br>axes = forecast_plot.gca()<br>last_training_date = forecast_prophet[&#39;ds&#39;].iloc[-12]<br>axes.axvline(x=last_training_date, color=&#39;red&#39;, linestyle=&#39;--&#39;, label=&#39;Training End&#39;)<br><br># plot true test data for the period after the red line<br>df_data_test[&#39;Month&#39;] = pd.to_datetime(df_data_test[&#39;Month&#39;])<br>plt.plot(df_data_test[&#39;Month&#39;], df_data_test[&#39;Passengers&#39;],&#39;ro&#39;, markersize=3, label=&#39;True Test Data&#39;)<br><br># show the legend to distinguish between the lines<br>plt.legend()</pre><p>Besides the general time series plot, we also added a dotted line to indicate the end of the training period and hence the start of the prediction period. Further, we made use of the true test dataset that we had prepared in the beginning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*bWBuoSz6mZc_zbGqLQTHGg.png" /><figcaption>Plotted results for the time series analysis incl. true test data and the prediction. Image by author</figcaption></figure><p>It can be seen that our model isn’t too bad. Most of the true passenger values are actually within the predicted uncertainty intervals. However, the summer months seem to be too pessimistic still, which is a pattern we can see in previous years already. This is a good moment to start exploring the parameters and features we could use with Prophet.</p><p>In our example, the seasonality is not a constant additive factor but it grows with the trend over time. Hence, we might consider changing the <em>seasonality_mode </em>from “additive” to “multiplicative” during the model fit. [4]</p><p>Our tutorial will conclude here to give some time to explore the large number of possibilities that Prophet offers to us. To review the full code together, I consolidated the snippets in this <a href="https://raw.githubusercontent.com/jonasdieckmann/prophet_tutorial/main/prophet_tutorial.py">Python file</a>. Additionally, you could upload<a href="https://raw.githubusercontent.com/jonasdieckmann/prophet_tutorial/main/Prophet_Tutorial.ipynb"> this notebook</a> directly to Colab and run it yourself. Let me know how it worked out for you!</p><h3>Conclusion</h3><p>Prophet is a powerful tool for predicting future values in time series data, especially when your data has repeating patterns like monthly or yearly cycles. It’s user-friendly and can quickly provide accurate predictions for your specific data. However, it’s essential to be aware of its limitations. If your data doesn’t have a clear pattern or if there are significant changes that the model hasn’t seen before, Prophet may not perform optimally. Understanding these limitations is crucial for using the tool wisely.</p><p>The good news is that experimenting with Prophet on your datasets is highly recommended! Every dataset is unique, and tweaking settings and trying different approaches can help you discover what works best for your specific situation. So, dive in, explore, and see how Prophet can enhance your time series forecasting.</p><p><a href="https://medium.com/@jonas_dieckmann">Jonas Dieckmann - Medium</a></p><p>I hope you find it useful. Let me know your thoughts! And feel free to connect on LinkedIn <a href="https://www.linkedin.com/in/jonas-dieckmann/">https://www.linkedin.com/in/jonas-dieckmann/</a> and/or to follow me here on medium.</p><h4>See also some of my other articles:</h4><ul><li><a href="https://pub.towardsai.net/how-to-use-chatgpt-api-for-direct-interaction-from-colab-or-databricks-39969a0ead5f">How To Use ChatGPT API for Direct Interaction From Colab or Databricks</a></li><li><a href="https://towardsdatascience.com/how-to-set-started-with-tensorflow-using-keras-api-and-google-colab-5421e5e4ef56">How to get started with TensorFlow using Keras API and Google Colab</a></li></ul><h4>References</h4><p>[1] <strong>Shumway, Robert H.; Stoffer, David S. (2017)</strong>: Time Series Analysis and Its Applications. Cham: Springer International Publishing.</p><p>[2] <strong>Brownlee, Jason (2017)</strong>: Introduction to Time Series Forecasting With Python</p><p>[3] <strong>Rafferty, Greg (2021)</strong>: Forecasting Time Series Data with Facebook Prophet</p><p>[4] <a href="https://facebook.github.io/prophet/docs/quick_start.html">https://facebook.github.io/prophet/docs/quick_start.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c74ad3040525" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/getting-started-predicting-time-series-data-with-facebook-prophet-c74ad3040525">Getting Started Predicting Time Series Data with Facebook Prophet</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How To Use ChatGPT API for Direct Interaction From Colab or Databricks]]></title>
            <link>https://pub.towardsai.net/how-to-use-chatgpt-api-for-direct-interaction-from-colab-or-databricks-39969a0ead5f?source=rss-1c8d1cf684f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/39969a0ead5f</guid>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[google-colab]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Jonas Dieckmann]]></dc:creator>
            <pubDate>Wed, 05 Apr 2023 14:01:50 GMT</pubDate>
            <atom:updated>2023-04-05T14:01:50.152Z</atom:updated>
            <content:encoded><![CDATA[<p>Have you wondered how you can use OpenAI’s API to interact directly with GPT algorithms? It’s easy, free, and also more powerful than the “classic” web interface at <a href="http://www.openai.com/">www.openai.com</a>. In the following tutorial, I will guide you through some simple steps that will allow you to use GPT for <strong>text generation</strong>, <strong>image creation</strong>, or <strong>debugging </strong>your code!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*o2SPkUX9IXyyYdqw" /><figcaption>Image by <a href="https://unsplash.com/">Unsplash</a></figcaption></figure><p>As the use of ChatGPT and other natural language processing (NLP) solutions increases, so does the number of tools and platforms that allow users to interact with these cutting-edge features. One of the most popular options is the OpenAI web interface, which has received widespread acclaim for its ability to handle complex NLP tasks.</p><p>Today, however, we will explore an alternative: the ChatGPT API. This article is divided into three main sections:</p><p>#1 Set up your OpenAI account &amp; create an API key<br>#2 Establish the general connection from Google Colab <br>#3 Try different requests: text generation, image creation &amp; bug fixing</p><p>Please note: <em>Although this tutorial was done in Google Colab (free), you may want to try other environments. For example, all the code was also applied in Databricks.</em></p><h3>#1 Set up your OpenAI account &amp; create an API key</h3><p>To interact with the GPT algorithms, you need to sign up for an OpenAI account (free of charge): <a href="https://platform.openai.com/signup/">https://platform.openai.com/signup/</a></p><p>Once you have registered and signed up, you will need to create an API key that will allow you to send requests to OpenAI from third-party services such as Google Colab or Databricks. Navigate to the “View API Key” section via the user menu, or use the following link: <a href="https://platform.openai.com/account/api-keys">https://platform.openai.com/account/api-keys</a></p><p>In this section, simply click on “<em>Create new secret key</em>” and save the created key somewhere on your computer (you will need it soon!).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0SHBnvz2WfknnwubbqIJ_g.png" /><figcaption>Screenshot by author</figcaption></figure><p>Please note that ChatGPT API is offering free trial usage (as of today) with limited requests and tokens per minute. See the rate limits below [1]:</p><ul><li><strong>Free trial users: 20 RPM 40000 TPM</strong></li><li>Pay-as-you-go users (first 48 hours): 60 RPM 60000 TPM</li><li>Pay-as-you-go users (after 48 hours): 3500 RPM 90000 TPM</li></ul><p><em>(RPM = requests per minute; TPM = tokens per minute)</em></p><h3>#2 Establish the general connection from Colab</h3><p>The easiest and most straightforward way to test the API is to use <a href="https://colab.research.google.com/">Google Colaboratory (“Colab”)</a>, which is something like “<em>a free Jupyter notebook environment that requires no setup and runs entirely in the cloud.</em>” Although there are many more professional environments you may want to explore (e.g. Databricks), I think Colab is not a bad service to take your first steps with the ChatGPT API.</p><p>To set up a basic environment for ChatGPT within Colab you can follow the next few steps:</p><ol><li>Open <a href="https://colab.research.google.com/">https://colab.research.google.com/</a> and register for a free account</li><li>Create a new <em>notebook </em>within Colab</li><li>Install &amp; use the <em>openai </em>package:</li></ol><pre>pip install openai</pre><p>To execute a simple chat request to the API using the <em>GPT 3.5 turbo model </em>(see other available models in their documentation linked at the end of this article), similar to what you know from the OpenAI web interface, you can simply execute the following lines of code in your notebook:</p><pre>import os<br>import openai<br><br>openai.api_key = &quot;please-paste-your-API-key-here&quot;<br><br>openai.ChatCompletion.create(<br>  model=&quot;gpt-3.5-turbo&quot;,<br>  messages=[<br>    {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Hello ChatGPT, does this work?&quot;}<br>  ]<br>  )</pre><p>As soon as you execute the command in Colab, you receive a JSON object as a response that contains the expected answer! (That was easy, wasn’t it?)</p><blockquote>Hello! As an AI language model, I don’t have the context of what \”this\” refers to. Could you please specify what you are referring to so I can assist you better?</blockquote><pre>&lt;OpenAIObject chat.completion id=chatcmpl-70ErnAfGGwU7GhMXzCcLGyUvr4hA2 at 0x7f097f0a5f40&gt; JSON: {<br>  &quot;choices&quot;: [<br>    {<br>      &quot;finish_reason&quot;: &quot;stop&quot;,<br>      &quot;index&quot;: 0,<br>      &quot;message&quot;: {<br>        &quot;content&quot;: &quot;Hello! As an AI language model, I don&#39;t have the context of what \&quot;this\&quot; refers to. Could you please specify what you are referring to so I can assist you better?&quot;,<br>        &quot;role&quot;: &quot;assistant&quot;<br>      }<br>    }<br>  ],<br>  &quot;created&quot;: 1680291503,<br>  &quot;id&quot;: &quot;chatcmpl-70ErnAfGGwU7GhMXzCcLGyUvr4hA2&quot;,<br>  &quot;model&quot;: &quot;gpt-3.5-turbo-0301&quot;,<br>  &quot;object&quot;: &quot;chat.completion&quot;,<br>  &quot;usage&quot;: {<br>    &quot;completion_tokens&quot;: 38,<br>    &quot;prompt_tokens&quot;: 17,<br>    &quot;total_tokens&quot;: 55<br>  }<br>}</pre><p>In addition, the JSON object provides information about the number of tokens used and the reason for the end of the request. If you only want to print the text response, you can access this element by slightly modifying your code:</p><pre>import os<br>import openai<br><br>openai.api_key = &quot;please-paste-your-API-key-here&quot;<br><br>response = openai.ChatCompletion.create(<br>  model=&quot;gpt-3.5-turbo&quot;,<br>  messages=[<br>    {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Hello ChatGPT, does this work?&quot;}<br>  ]<br>  )<br><br>print(response.choices[0].message.content)</pre><h3>#3 Try different requests: text generation, image creation &amp; bug fixing</h3><p>If you’re as excited as I was when I found this out, you could start sending lots of different requests to the API. A useful way to modularise your code is to create some helpful functions that you want to call for different purposes. Let me give you some ideas.</p><h4>Function to chat with ChatGPT</h4><p>The following code simply summarises the work done so far in a callable function that allows you to make any request to GPT and get only the text response as the result.</p><pre>import os<br>import openai<br><br>openai.api_key = &quot;please-paste-your-API-key-here&quot;<br><br>def chatWithGPT(prompt):<br>  completion = openai.ChatCompletion.create(<br>  model=&quot;gpt-3.5-turbo&quot;,<br>  messages=[<br>  {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt}<br>  ]<br>  )<br>  return print(completion.choices[0].message.content)</pre><p>Do you think it makes sense to learn python? Let’s ask GPT!</p><pre>chatWithGPT(&quot;is it a good idea to start learning python?&quot;)</pre><blockquote>As an AI language model, I cannot provide personal opinions, but I can say that Python is a popular and widely used programming language that is highly favored by beginners and experienced developers alike. It has a vast community support, a large number of libraries, and a simple syntax that makes it easy to grasp for those new to programming. It is useful for various applications, like data analysis, web development, machine learning, and more. Therefore, it could be a good idea to start learning Python if you want to pursue a career in programming or want to add another skill to your resume.</blockquote><h4>Function to fix bugs in your code</h4><p>Another use case for ChatGPT is to get ideas for fixing your code. Imagine your Python command returns an error and you want to get advice on what to do without using Google or StackOverflow:</p><pre>import os<br>import openai<br><br>openai.api_key = &quot;please-paste-your-API-key-here&quot;<br><br>def fixMyCode(code): <br>  completion = openai.ChatCompletion.create(<br>  model=&quot;gpt-3.5-turbo&quot;,<br>  messages=[<br>    {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;find error in my python script below and fix it: &quot; + code}<br>  ]<br>  )<br>  return print(completion.choices[0].message.content)</pre><p>See my python code has thrown an error, and I don’t know why…</p><pre>fixMyCode(&quot;&quot;&quot;<br><br>def some_function():<br>    print(&quot;I&#39;m going to sleep&quot;)<br>    time.sleep(10)<br>    print(&quot;I&#39;m awake again&quot;)<br><br>some_function()<br><br>&quot;&quot;&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*a6WgCozK9vDmzGOmuz6-tg.png" /><figcaption>Screenshot by author</figcaption></figure><p>No surprise, but ChatGPT immediately found out that I had forgotten to import the module before I used it. This can be very helpful in everyday life, especially when you can call for help directly from the programming environment.</p><h4>Function to create images</h4><p>The last use case I would like to present here is the creation of images. The request itself returns a hyperlink containing the picture. Using the IPhython library, you can display the picture directly in your notebook.</p><pre>import IPython<br>import os<br>import openai<br><br>openai.api_key = &quot;please-paste-your-API-key-here&quot;<br><br>def createImageWithGPT(prompt): <br>  completion = openai.Image.create(<br>  prompt=prompt,<br>  n=1,<br>  size=&quot;512x512&quot;<br>  )<br>  return IPython.display.HTML(&quot;&lt;img src =&quot; + completion.data[0].url + &quot;&gt;&quot;)</pre><p>Let’s be creative and ask for a cat driving a skateboard!</p><pre>createImageWithGPT(&quot;Cat driving a skateboard&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/1*vsm6KUSNep8fYqrf91mQIg.png" /><figcaption>Image created by ChatGPT, screenshot by author</figcaption></figure><h3>Summary</h3><p>With ChatGPT API, businesses and individuals can easily and affordably incorporate chatbots into their workflow without the technical knowledge or extensive resources typically required. The API can also be used to create virtual assistants, personal tutors, and more. I recommend the documentation made available by OpenAI for their API: <a href="https://platform.openai.com/docs/api-reference">https://platform.openai.com/docs/api-reference</a>.</p><p>In summary, it is easy to use the API in your programming environment. Not only might this be helpful for direct debugging of your code, but it has also shown more stable response rates compared to OpenAI’s (sometimes unavailable) web interface. With the ability to understand natural language and get smarter over time, ChatGPT has the potential to revolutionize the way businesses interact with their customers and streamline their workflows. Try it yourself and experience the future of chatbots!</p><p><a href="https://medium.com/@jonas_dieckmann">Jonas Dieckmann - Medium</a></p><p>I hope you find it useful. Let me know your thoughts! And feel free to connect on LinkedIn <a href="https://www.linkedin.com/in/jonas-dieckmann/">https://www.linkedin.com/in/jonas-dieckmann/</a> and/or to follow me here on medium.</p><h3>See also some of my other articles:</h3><ul><li><a href="https://towardsdatascience.com/how-to-set-started-with-tensorflow-using-keras-api-and-google-colab-5421e5e4ef56">How to get started with TensorFlow using Keras API and Google Colab</a></li><li><a href="https://towardsdatascience.com/introduction-to-ica-independent-component-analysis-b2c3c4720cd9">Introduction to ICA: Independent Component Analysis</a></li></ul><h3>References</h3><p>[1]: <a href="https://help.openai.com/en/articles/7039783-chatgpt-api-faq">https://help.openai.com/en/articles/7039783-chatgpt-api-faq</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39969a0ead5f" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/how-to-use-chatgpt-api-for-direct-interaction-from-colab-or-databricks-39969a0ead5f">How To Use ChatGPT API for Direct Interaction From Colab or Databricks</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Case Study: Applying a Data Science Process Model to a Real-World Scenario]]></title>
            <link>https://medium.com/data-science/case-study-applying-a-data-science-process-model-to-a-real-world-scenario-93ae57b682bf?source=rss-1c8d1cf684f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/93ae57b682bf</guid>
            <category><![CDATA[dasc-pm]]></category>
            <category><![CDATA[supply-chain]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-dives]]></category>
            <dc:creator><![CDATA[Jonas Dieckmann]]></dc:creator>
            <pubDate>Sat, 11 Mar 2023 03:55:15 GMT</pubDate>
            <atom:updated>2023-03-21T14:40:40.823Z</atom:updated>
            <content:encoded><![CDATA[<h4>Development of a machine learning model for materials planning in the supply chain</h4><p>In today’s rapidly changing environment, one of the most critical challenges facing companies is the ability to predict future demand accurately. This is especially true for supply chain teams, where accurate demand planning is vital for maintaining customer satisfaction and keeping costs under control.</p><p>In this <strong>case study</strong>, we will explore how a<strong> data science process model</strong> can help companies tackle this challenge hands-on<strong> </strong>by leveraging statistical forecasting methods. The goal of the fictitious company was to develop a more accurate demand planning process that reduced stock-outs, increased inventory turnover, and improve overall supply chain performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e-IGGGOxLJ1rKwqzLlfn7w.png" /><figcaption>Image by <a href="https://unsplash.com/">Unsplash</a></figcaption></figure><p>This project is a powerful example of how data science can transform a business by unlocking new insights, increasing efficiency, and improving decision-making. I hope that this case study will help you to think about the potential applications in your organization and showcase how you can apply the process model DASC-PM successfully.</p><blockquote><em>Please note that the entire article has also been published in the below publication and was written by </em><strong><em>Daniel Badura</em></strong><em> and </em><strong><em>Jonas Dieckmann</em></strong><em>:</em></blockquote><blockquote><strong>Chapter 3: “</strong>Development of a Machine Learning Model for Materials Planning in the Supply Chain” in: <strong>Schulz et al. (2023): </strong><em>DASC-PM v1.1 Case Studies</em>. Available from: <a href="https://www.researchgate.net/publication/368661660_DASC-PM_v11_Case_Studies">https://www.researchgate.net/publication/368661660_DASC-PM_v11_Case_Studies</a></blockquote><h3>1. Domain and project description</h3><p>SCHRAMME AG is a leading provider of dressings, band-aids, and bandages. The management thinks that there are qualitative optimization potential and savings opportunities in materials planning and the resulting production processes. Management assigns an internal project manager the task of developing a model based on machine learning to plan the materials and requirements in the supply chain. Due to negative experiences in previous data science projects, it is proposed that this project should initially be developed by using a process model.</p><p>The <a href="https://medium.com/towards-data-science/dasc-pm-a-novel-process-model-for-data-science-projects-9f872f2534b1">DASC-PM</a> is chosen to ensure a structured and scientific process for project management. To gain an overview of the project assignment, the project manager initially works out various use cases that are then checked for suitability and feasibility. The suitable use cases then serve as the basis for figuring out the specific problems and the design of the project. This design is then checked again for suitability and feasibility.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cK13B1uKNP4P7UPcnNIkaw.png" /><figcaption>Image by <a href="https://unsplash.com/">Unsplash</a></figcaption></figure><h4>Starting point and use case development</h4><p>The company manually plans and then produces over 2,500 different products at present. In the last few quarters, they increasingly had inventory shortages for some product series, while for individual products inventories exceeded storage capacities. While the controlling department complains about rising storage costs due to imprecise planning, the demand planners lament the insufficient amount of time for the planning. For some time, the head of the supply chain has criticized the fact that the planning is done solely manually, and the opportunities of digitalization appear not to be taken advantage of.</p><p><strong>Project goals</strong><br>One goal of the project is the development of a machine learning model where a large part of the product requirements should be planned automatically in the future, based on various influential factors. The demand planners should increasingly address the planning of important product groups and advertising. The system should take account of seasonality, trends, and market developments, and achieve planning accuracy of 75%. This means that the forecasts for quantities of each product should deviate from actual requirements by no more than 25%. Order histories, inventory and sales figures for customers, and internal advertising plans should be used as potential data sources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*ouS-hXzRNlUF0lHVpPORtw.png" /><figcaption>Phase 1: Project Order (Schulz et al. 2022)</figcaption></figure><p><strong>Current team set-up </strong><br>Along with the inclusion of the Supply Chain department, close collaboration with Sales and IT is also expected. The planning team in the Supply Chain department now consists of a global market demand planning team that deals with long-term planning (6–18 months) based on market developments, product life cycles, and strategic focus. In individual markets, there are local customer demand planning teams that implement short-term materials and advertising planning (0–6 months) for retail through the corresponding sales channels.</p><p>The data science model to be developed should support the monthly planning cycles and quantify the need for short-term and long-term materials. The projection is then loaded into the internal planning software and should be analyzed and, if need be, supplemented or corrected. The final planning quantity will ultimately be used by the factories for production planning. To take account of the customer- and product-specific expertise, seasonality, and experiences from the past, individual team members of the planning team should be included in the project, allocating up to 20% of their working hours to it.</p><p><strong>Suitability Check</strong><br>An important partial aspect during the use case selection is the suitability test. The project manager tries to examine whether the project can fundamentally be classified as feasible and whether the requirements can be carried out with the available resources. Expert interviews have shown that the problem in general is very well suited for the deployment of data science and corresponding projects have already been undertaken externally and also published. The data science team confirmed that there are a sufficient number of potentially suitable methods for this project and the required data sources are available.</p><p>Finally, the project manager analyzes feasibility. It is necessary to coordinate with the IT department to check the available infrastructure and the expertise of the involved employees. The available cloud infrastructure from Microsoft and the experience of the data science team withDatabricks software make the project appear fundamentally achievable. The project risk is classified as moderate in general since the planers assume a major role as controllers in the implementation phase and the results are checked.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*kKwpscTt3ZhOh1w13BkERw.png" /><figcaption>Data Science Process Model DASC-PM (Schulz et al. 2022)</figcaption></figure><h4>Project design</h4><p>Based on the problem and specific aspects of the domains, the project manager, the head of the supply chain, and a data scientist are now responsible for formally designing the project.</p><p>The project objective is assumed to be an improvement in planning accuracy and a reduction in the manual processes and is tied to the aim of developing an appropriate model for the project. According to an initial estimate, the cost framework totals EUR 650,000. A period of six months is proposed as the timeframe for the development, with an additional six months planned for process integration.</p><p>Since full planning and a description of the course of projects in the data science context are usually not possible in contrast to many other projects, the project manager solely prepares a project outline for this process with the basic cornerstones that were already indicated in the previous sections. The budget includes financial resources for 1 full-time project manager, 2 full-time data scientists, and 0.5 full-time data engineers. As already mentioned, the demand planners should allocate roughly 20% of the working hours to share their expertise and experience.</p><p>The project as a whole should be handled with an agile working method and based on the DASC-PM phases according to the Scrum methodology. The work is done iteratively in the areas of data procurement, analysis, utilization, and use, with the preceding and following phase moving into focus in each phase. The back-steps are especially important if gaps or problems are found in key areas and can only be solved by returning to the previous phase. The project outline is prepared visually and placed in a very visible area of the SCHRAMME AG office for all participants. Then the entire project description is checked for suitability and feasibility once again until the process moves on to the next phase.</p><h3>2. Data provision</h3><h4>Data preparation</h4><p>SCHRAMME AG has several data sources that can be included in automatic planning. Besides the historical sales data from the ERP system, order histories and customer data from the CRM system are options, along with inventories and marketing measures. Azure Data Factory is used to prepare a cloud-based pipeline that loads, transforms, and integrates the data from various source systems. The primary basis for the automatic forecasts should be the order histories: The remaining data is used either as background information for the planning teams or to carry out cluster analyses in advance if need be. In the initial phase of the project, the individual data sources still exhibit big differences regarding quality and structure. That is why adjustments are made together with the IT and technical departments to prepare the forecasts later on a solid basis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vmcEOSfnKy1gaIyq1njCoQ.png" /><figcaption>ELT data preparation process for analysis. Image by author</figcaption></figure><h4>Data management</h4><p>The data management process is automated by data engineers and done according to a daily schedule to always remain up to date. To keep the complexity reasonable, the most promising data sources are initially processed and the pipeline is then incrementally expanded with Continuous Integration / Continuous Deployment (CI/CD). After deployment, the processed data are stored in Azure Data Lake Storage where they can be used for future analysis with Azure Databricks. DataLake also stores the backups of the prepared data and analysis results as well as other data such as protocols, quality metrics, and credential structures. Writing and reading authorizations as well as plan versions also ensure that only the latest planning period can be processed so that the values from the past no longer change.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*DdYsablIXsWjAxbk_xhExw.png" /><figcaption>Phase 2: Data Provision (Schulz et al. 2022)</figcaption></figure><h4>Exploratory data analysis</h4><p>An important step in data preparation is the exploratory data analysis (EDA) where various statistics and visualizations are produced to start with. This results in an overview of the distributions, outliers, and correlations in the data. The results of the EDA provide insights into characteristics to be considered for the next phase of the analysis. In the second step, Feature Selection and Feature Engineering are used to select the relevant characteristics or produce new features. A dimension reduction method such as a principal component analysis is applied for data with high dimensionality. The EDA provides information about the existing demand histories of SCHRAMMEAG.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*0yjO3jfc3wfvrOWuh4EtZQ.png" /><figcaption>Example of results from exploratory data analysis. Image by author</figcaption></figure><h3>3. Analysis</h3><h4>Identification of suitable analysis methods</h4><p>The feasibility test at the beginning of the project made it clear that this project can and should be solved with data science methods. The two data science employees involved initially provide an overview of the existing methods that are well suited for the existing problem. This existing problem is part of the regression problem class in the supervised learning algorithms. Fundamentally, this is a type of time series analysis that can be expanded by additional factors or multiple regression.</p><p>In connection with the key area of scientificity, the latest developments in research on comparable problems were examined. This showed that XGBoost, ARIMA, FacebookProphet, and LightGBM are frequently named methods for the problem class. A data scientist documents the corresponding advantages and disadvantages of each method and sorts them according to the complexity and computational intensity. To receive the first indications on the model ability for products from SCHRAMME AG, simpler models are initially selected by the project team, which then adopts the classical exponential smoothing and ARIMA model family.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/659/1*RFieEV4L7CNIg_fAnjXlIg.png" /><figcaption>Phase 3: Analysis (Schulz et al. 2022)</figcaption></figure><h4>Application of analysis methods</h4><p>Since multiple users are involved in the analysis process for this project, the team initially relies on a suitable notebook-based development environment in Databricks. Along the typical machine learning workflow, the code for the import and data cleaning is initially implemented. To ensure validity, the underlying dataset is ultimately divided into training, validation, and test data by cross-validation. The selected methods are then applied to training and validation datasets to optimize the model. In this context, attempts are also repeatedly made to optimize the parameters of processes and sensibly reduce the number of available dimensions, if need be. The data scientists at SCHRAMME AG document the execution and validation results of the individual runs. The ARIMA family models fundamentally exhibit a better performance relative to the exponential smoothing, even if the target accuracy of 75% still cannot be achieved with a currently resulting value of 62.4%. The RMSE and MAPE metrics also show potential for optimization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dBrTA3popAjJBYJgBG4QHQ.png" /><figcaption>Comparison of the ARIMA forecast with actual need. Image by author</figcaption></figure><p>The parameter configurations and the basis for selecting the final model after the first application iteration are documented and prepared for the project manager and the head of the supply chain in a technically understandable way. What is seen in particular, is that some product groups have very unusual seasonality and certain products are generally very difficult to predict. Even if the product portfolio of SCHRAMME AG is affected somewhat less due to temporary closures (lockdowns) during the corona pandemic, a slight decline in demand for dressing products has been observed. It is assumed that less activity and transport, as well as fewer accidents and injuries, account for this drop.</p><p>The trend can be modeled quite well in the analysis method used. To improve the target accuracy, technically more complex methods are used in another experiment, with these methods proving to be relevant and applicable in the context of identifying suitable methods. After some iterations to optimize parameters and cross-validate, the Prophet and XGBoost methods demonstrated the highest validation results at 73.4% and 65.8%, respectively.</p><p>The data scientists consider Prophet to be the most suitable method among the applied processes and determine the planning accuracy relative to the test time series. Even if the accuracy is slightly below the target value of 73.4%, a significant improvement in planning accuracy is achieved. The MAPE is at 16.64% and the RMSE at 8,130, which implies a less absolute deviation in comparison to the RMSE in the XGBoost method (10,134). Similar to the first experiment, however, there are product groups that are very difficult to predict overall (37.2%) and negatively impact the cumulative accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*oyICo24Rln0Pt_Oe1tWzAg.png" /><figcaption>Performance comparison of various methods. Image by author</figcaption></figure><h4>Evaluation</h4><p>The results of the analyses are used as the basis for a logical evaluation and classification by the head of the supply chain and the analysts, which is organized and moderated by the project manager. The adopted metrics for evaluation are the cumulative planning accuracy of all products defined in advance together with the common RMSE and MAPE metrics. The department needs to have a realistic, trackable, and reliable basis for determining requirements on the product level.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/971/1*7iHScW9juDoQvDKRNyoW0A.png" /><figcaption>Evaluation of the three best models. Image by author</figcaption></figure><p>The benchmark for planning accuracy is assumed to be the current (manually planned) median accuracy of 58% over the last two years. The evaluation of results shows that many product groups overall can be planned with a high degree of accuracy by using the data science model and vastly exceed the benchmark. However, there are also product groups that reflect similar accuracy concerning manual planning. It is necessary to discuss above all the product area of drainage, which sees much worse results with the model than in the manual planning and appears to be unsuitable for a statistical calculation of requirements with the methods used to date.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*Rq8u59S5pDSCq2IYM8vIfQ.png" /><figcaption>Evaluation of the best model, distributed across product groups. Image by author</figcaption></figure><p>From a technical perspective, the head of the supply chain believes that it makes little sense to plan such product groups statistically since only limited planning accuracy is possible due to their specific seasonal and trend-based characteristics. She recommends the introduction of an error threshold value on a product basis to determine which products should be predicted with the model and which product groups will be removed from the modeling and still planned manually. A range slightly below the current benchmark seems to be a suitable threshold value since nearly as good accuracy with a less manual effort from the perspective of the department is always an improvement on the way to achieving the project objective. The project leader documents the results of the evaluation with the decisions and measures adopted.</p><p>The required quantities of all selected products for the next 18 months can be documented as the analysis result after the first real modeling. This can now be utilized and integrated into the planning process of the teams.</p><h3>4. Deployment</h3><p>The team now enters the utilization phase of the DASC-PM for integration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*hB2rXeT3Y12FaKFgo14MSQ.png" /><figcaption>Phase 4: Deployment (Schulz et al. 2022)</figcaption></figure><h4>Technical-methodological preparation</h4><p>It is possible to rely on the existing infrastructure for utilization. The forecasts are loaded in the planning software IBM Planning Analytics where they are tested and reprocessed. The so-called TurboIntegrator is used to automate the loading process that represents a central component of IBM Planning Analytics. The OLAP structure of Planning Analytics allows for the creation of flexible views where the users can personally choose their context (time reference, product groups, etc.)and adjust calculations in real-time. Furthermore, the reporting software QlikSense is also integrated for more in-depth analyses. Here, the components of the time series (trends, seasonality, noise) can be visualized on the one hand and additional information such as outliers and median values can be displayed on the other hand. The final plans are loaded into the Data Lake after processing by the planning teams so they can be referenced in the future.</p><h4>Ensuring technical feasibility</h4><p>The forecasts themselves are automatically regenerated at the beginning of the month. The planners can make their corrections during the first four working days of the month and view the results in the planning system in real-time. Since the algorithms work in a cloud environment, the computing power can be scaled, if need be. To get all processes to run automatically, changes in the data sources should be minimized. If there is a need for adjustment, the data engineer will be informed, and the interface document will be updated by recording all the information on data sources and connections. The planning and forecasting system is a mixture of the cloud (Microsoft Azure) and an on-premise system (Planning Analytics), with the planners only having active access to the on-premise structures. Credentials are awarded here so the local planners only have access to their areas, while the global planners can view all topics. After the end of the development phase, the support services are mainly handled by the IT department. In the case of complex problems, data scientists or data engineers are also consulted.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V5UtS79yIaJuautV8C7t7Q.png" /><figcaption>Image by <a href="https://unsplash.com/">Unsplash</a></figcaption></figure><h4>Ensuring applicability</h4><p>Users of the solution are the local and global planning teams. Since members of the teams have less of a technical orientation, training sessions are held to help them interpret the forecasts and classify their quality. The user interface is also designed with a focus on clarity and understandability. Simple line and bar charts for processes and benchmarks are used, along with tables reduced to what is most important. The users are included in the development from the beginning to ensure technical correctness and relevance and to ensure familiarity with the solution before the end of the development phase. In addition, complete documentation is drafted. The technical part of the documentation mostly builds on the interface document by demonstrating the data structures and connections, while the content part is jointly prepared with the users.</p><h4>Technical preparation</h4><p>To ensure that the new solution does not lose relevance or quality after a few months, work continues to be done on improvements after the completion of the first development phase, even if substantially less time is spent on it. The most important aspect of the ongoing improvement is the constant automated adjustment of the prediction model to new data. Other parts of the system still requiring manual work at the beginning are also automated over time. A change in various parameters such as the forecast horizon or threshold values for the accuracy of the prediction can be made by the planners themselves in Planning Analytics, with the model remaining flexible. Problems occurring after the release of the first version are entered via the IT ticket system and assigned to the data science area. At regular intervals, it is also checked whether the model still satisfies the expectations of the company or whether changes are necessary.</p><h3>5. (Application) Use and summary</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*H4-x7M-mphBXxovdlgd2Vg.png" /><figcaption>Phase 5: Application (Schulz et al. 2022)</figcaption></figure><p>The transition to the use of the developed model means that the Data Science Process Model(DASC-PM) enters its last phase. As a whole, SCHRAMME AG was able to achieve the objectives it had set in the supply chain area by using a structured and holistic approach. Additional or new projects can now be derived from here. The planning processes were largely automated and supported by machine learning algorithms. The relevant stakeholders in management, finance, and the supply chain were highly satisfied. After initial skepticism, the planning team itself is now also convinced by the reduction in workload and possible prioritization. However, it is also conceivable that weak points will surface during use and more iterations will be required in later phases.</p><blockquote>The case study as a whole showed that non-linear process models in particular are advantageous for the area of data science. The DASC-PM is a suitable novel process that can be transferred to numerous other domains and problems.</blockquote><h4>Conclusion</h4><p>In conclusion, data science plays an integral role in solving complex business problems by identifying hidden patterns and extracting actionable insights from data. Through this case study, we demonstrated how data science techniques can be used to develop predictive models to help businesses make informed decisions e.g., in the supply chain.</p><p>While this case study focuses on demand planning, the process model can be used in various ways, such as for building personalized recommendations on e-commerce websites, identifying fraud in financial transactions, or predicting customer churn in telecom or subscription-based businesses.</p><p>However, it’s essential to note that real-world data science projects pose several challenges, such as data quality issues, lack of domain expertise, and inadequate communication between stakeholders. In comparison, fictitious case studies provide an idealized environment with clean, well-labeled data and well-defined problem statements. Thus, real-world projects require a pragmatic approach that takes into account various factors such as business objectives, data quality, computational resources, and ethical considerations. I am pretty sure you know this from your own experience. Do not underestimate reality!</p><p>In summary, data science has immense potential to transform industries, and society and create new opportunities for businesses. The DASC-DM (or any) process model can help to structure the approach logically to ensure clear guidance for both, business stakeholders as well as the project team itself.</p><p>Please let me know about your experience with data science projects. How do you structure them &amp; what are the biggest challenges? Feel free to leave a comment!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EXpafdZWjEOxOEI5yIO-Jw.png" /><figcaption>Image by <a href="https://unsplash.com/">Unsplash</a></figcaption></figure><p><a href="https://medium.com/@jonas_dieckmann">Jonas Dieckmann - Medium</a></p><p>I hope you find it useful. Let me know your thoughts! And feel free to connect on LinkedIn at <a href="https://www.linkedin.com/in/jonas-dieckmann/">https://www.linkedin.com/in/jonas-dieckmann/</a> and/or to follow me here on medium.</p><h3>See also some of my other articles:</h3><ul><li><a href="https://towardsdatascience.com/ethics-in-ai-potential-root-causes-for-biased-algorithms-890091915aa3">Ethics in AI: Potential Root Causes for Biased Algorithms</a></li><li><a href="https://towardsdatascience.com/dasc-pm-a-novel-process-model-for-data-science-projects-9f872f2534b1">DASC-PM: a novel Process Model for Data Science Projects</a></li></ul><h3>References</h3><p><em>The whole case study has been published in:</em></p><p>[1] <strong>Schulz et al. (2023):</strong> “<em>DASC-PM v1.1 Case Studies”</em> Available from: <a href="https://www.researchgate.net/publication/368661660_DASC-PM_v11_Case_Studies">https://www.researchgate.net/publication/368661660_DASC-PM_v11_Case_Studies</a></p><p><em>Process images have been taken from:</em></p><p>[2] <strong>Schulz et al. (2022):<em> </em></strong><em>“</em><a href="https://www.researchgate.net/publication/361428426_DASC-PM_v11_-_A_Process_Model_for_Data_Science_Projects">DASC-PM v1.1 — A Process Model for Data Science Projects</a>”<strong> </strong>(2022), Publisher: NORDAKADEMIE gAG Hochschule der Wirtschaft, ISBN: 978–3–00–064898–4, DOI:<a href="http://dx.doi.org/10.25673/32872.2">10.25673/32872.2</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=93ae57b682bf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/case-study-applying-a-data-science-process-model-to-a-real-world-scenario-93ae57b682bf">Case Study: Applying a Data Science Process Model to a Real-World Scenario</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ensemble learning: Bagging and Boosting]]></title>
            <link>https://medium.com/data-science/ensemble-learning-bagging-and-boosting-23f9336d3cb0?source=rss-1c8d1cf684f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/23f9336d3cb0</guid>
            <category><![CDATA[boosting]]></category>
            <category><![CDATA[ensemble-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[bagging]]></category>
            <category><![CDATA[random-forest]]></category>
            <dc:creator><![CDATA[Jonas Dieckmann]]></dc:creator>
            <pubDate>Thu, 23 Feb 2023 17:23:28 GMT</pubDate>
            <atom:updated>2023-03-21T14:44:09.538Z</atom:updated>
            <content:encoded><![CDATA[<h3>Ensemble Learning: Bagging and Boosting</h3><p>Would you like to take your data science skills to the next level? Are you interested in improving the accuracy of your models and making more informed decisions based on your data? Then it’s time to explore the world of bagging and boosting. With these powerful techniques, you can improve the performance of your models, reduce errors and make more accurate predictions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XBELbnS76BAlbAxWL4g-RQ.png" /><figcaption>Image by <a href="https://unsplash.com/">Unsplash</a></figcaption></figure><p>Whether you are working on a classification problem, a regression analysis, or another data science project, bagging and boosting algorithms can play a crucial role. In this article, we #1 <strong>summarize the main idea of ensemble learning, </strong>introduce both,<strong> </strong>#2 <strong>bagging </strong>and #3 <strong>boosting, </strong>before we finally #4 <strong>compare both methods</strong> to highlight similarities and differences.</p><p>So let’s get ready for bagging and boosting to succeed!</p><h3>#1: Introduction and main idea: ensemble learning</h3><p>So when should we use it? Cleary, when we see overfitting or underfitting of our models. Let’s begin with the key concept of bagging and boosting, which both belong to the family of ensemble learning techniques:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8vznmPx1HXTvfW5QzbnsCg.png" /><figcaption>Image by author</figcaption></figure><p>The main idea behind ensemble learning is the usage of multiple algorithms and models that are used together for the same task. While single models use only one algorithm to create prediction models, bagging and boosting methods aim to combine several of those to achieve better prediction with higher consistency compared to individual learnings.</p><h4><strong>Example: Image classification</strong></h4><p>The essential concept is encapsulated by means of a didactic illustration involving image classification. Supposing a collection of images, each accompanied by a categorical label corresponding to the kind of animal, is available for the purpose of training a model. In a traditional modeling approach, we would try several techniques and calculate the accuracy to choose one over the other. Imagine we used logistic regression, decision tree, and support vector machines here that perform differently on the given data set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RS7jOPKFRPSvg5yW9daWTw.png" /><figcaption>Image by author</figcaption></figure><p>In the above example, it was observed that a specific record was predicted as a dog by the logistic regression and decision tree models, while a support vector machine identified it as a cat. As various models have their distinct advantages and disadvantages for particular records, it is the key idea of ensemble learning to combine all three models instead of selecting only one approach that showed the highest accuracy.</p><p>The procedure is called <em>aggregation </em>or <em>voting </em>and combines the predictions of all underlying models, to come up with one prediction that is assumed to be more precise than any sub-model that would stay alone.</p><h4><strong>Bias-Variance tradeoff</strong></h4><p>The next chart might be familiar to some of you, but it represents quite well the relationship and the tradeoff between bias and variance on the test error rate.</p><p>You might be familiar with the following concept, but I posit that it effectively illustrates the correlation and compromise between bias and variance with respect to the testing error rate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*Za83whww3ucVQ7bDJq93AQ.png" /><figcaption>Image by author</figcaption></figure><p>The relationship between the variance and bias of a model is such that a reduction in variance results in an increase in bias, and vice versa. To achieve optimal performance, the model must be positioned at an equilibrium point, where the test error rate is minimized, and the variance and bias are appropriately balanced.</p><p>Ensemble learning can help to balance both extreme cases to a more stable prediction. One method is called bagging and the other is called boosting.</p><h3>#2: Bagging (bootstrap aggregation)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*Y0zlL6ZKV5_u0x1S2CLgwQ.png" /><figcaption>A random bag. Image by <a href="https://unsplash.com/">Unsplash</a></figcaption></figure><p>Let us focus first on the Bagging technique called bootstrap aggregation. Bootstrap aggregation aims to solve the right extreme of the previous chart by <strong>reducing the variance</strong> of the model to avoid overfitting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9B7b8kDs9IrgfdcbWC0DLA.png" /><figcaption>Image by author</figcaption></figure><p>With this purpose, the idea is to have multiple models of the same learning algorithm that are trained by random subsets of the original training data. Those random subsets are called bags and can contain any combination of the data. Each of those datasets is then used to fit an individual model which produces individual predictions for the given data. Those predictions are then aggregated into one final classifier. The idea of this method is really close to our initial toy example with the cats and dogs.</p><p>Using random subsets of data, the risk of overfitting is reduced and flattened by averaging the results of the sub-models. All models are calculated in parallel and then aggregated together afterward.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oTtwV5r6Qg9bjepZtHSuGQ.png" /></figure><p>The calculation of the final ensemble aggregation uses either the simple average for regression problems or a simple majority vote for classification problems. For that, each model from each random sample produces a prediction for that given subset. For the average, those predictions are just summed up and divided by the number of created bags.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DG50MKHhODhkXVCiwFrjmg.png" /><figcaption>Image by author</figcaption></figure><p>A simple majority voting works similarly but uses the predicted classes instead of numeric values. The algorithm identifies the class with the most predictions and assumes that the majority is the final aggregation. This is again very similar to our toy example, where two out of three algorithms predicted a picture to be a dog and the final aggregation was therefore a dog prediction.</p><blockquote><strong>Random Forest</strong><br>A famous extension to the bagging method is the random forest algorithm, which uses the idea of bagging but uses also subsets of the features and not only subsets of the entries. Bagging, on the other hand, takes all given features into account.</blockquote><h4>Code example for bagging</h4><p>In the following, we will explore some useful python functions from the sklearn.ensemblelibrary. The function called BaggingClassifierhas a few parameters which can be looked up in the documentation, but the most important ones are <em>base_estimator</em>, <em>n_estimators</em>, and <em>max_samples</em>.</p><pre>from sklearn.ensemble import BaggingClassifier <br><br># define base estimator <br>est = LogisticRegression() # or est = SVC() or est = DecisionTreeClassifier<br><br># n_estimators defines the number of base estimators in the ensemble <br># max_samples defines number of samples to draw from X to train each base estimator <br><br>bag_model = BaggingClassifier(base_estimator= est, n_estimators = 10, max_samples=1.0) <br><br>bag_model = bag_model.fit(X_train, y_train) <br><br>Prediction = bag_model.predict(X_test)</pre><ul><li><em>base_estimator: </em>You have to provide the underlying algorithm that should be used by the random subsets in the bagging procedure in the first parameter. This could be for example Logistic Regression, Support Vector Classification, Decision trees, or many more.</li><li><em>n_estimators: </em>The number of estimators defines the number of bags you would like to create here and the default value for that is 10.</li><li><em>max_samples: </em>The maximum number of samples defines how many samples should be drawn from X to train each base estimator. The default value here is one point zero which means that the total number of existing entries should be used. You could also say that you want only 80% of the entries by setting it to 0.8.</li></ul><p>After setting the scenes, this model object works like many other models and can be trained using the fit()procedure including X and y data from the training set. The corresponding predictions on test data can be done using predict().</p><h3>#3: Boosting</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PhKOhnR0pXYIUKGQgVsLXw.png" /><figcaption>Boost your models! Image by <a href="https://unsplash.com/">Unsplash</a></figcaption></figure><p>Boosting is a little variation of the bagging algorithm and uses sequential processing instead of parallel calculations. While bagging aims to reduce the variance of the model, the boosting method tries aims to <strong>reduce the bias</strong> to avoid underfitting the data. With that idea in mind, boosting also uses a random subset of the data to create an average-performing model on that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dcSp9TmyJGHzlUPM45hgAQ.png" /><figcaption>Image by author</figcaption></figure><p>For that, it uses the miss-classified entries of the weak model with some other random data to create a new model. Therefore, the different models are not randomly chosen but are mainly influenced by wrong classified entries of the previous model. The steps for this technique are the following:</p><ol><li><strong>Train initial (weak) model</strong><br>You create a subset of the data and train a weak learning model which is assumed to be the final ensemble model at this stage. You then analyze the results on the given training data set and can identify those entries that were misclassified.</li><li><strong>Update weights and train a new model<br></strong>You create a new random subset of the original training data but weight those misclassified entries higher. This dataset is then used to train a new model.</li><li><strong>Aggregate the new model with the ensemble model</strong><br>The next model should perform better on the more difficult entries and will be combined (aggregated) with the previous one into the new final ensemble model.</li></ol><p>Essentially, we can repeat this process multiple times and continuously update the ensemble model until our prediction power is good enough. The key idea here is clearly to create models that are also able to predict the more difficult data entries. This can then lead to a better fit of the model and reduces the bias.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aEk9N5lK1VhdP1JKubAlFw.png" /><figcaption>Image by author</figcaption></figure><p>In comparison to Bagging, this technique uses weighted voting or weighted averaging based on the coefficients of the models that are considered together with their predictions. Therefore, this model can reduce underfitting, but might also tend to overfit sometimes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QEuoZpckXaKIGI3dzj61YA.png" /></figure><h4>Code example for boosting</h4><p>In the following, we will look at a similar code example but for boosting. Obviously, there exist multiple boosting algorithms. Besides the GradientDescent methodology, the AdaBoost is one of the most popular.</p><ul><li><em>base_estimator: </em>Similar to Bagging, you need to define which underlying algorithm you would like to use.</li><li><em>n_estimators: </em>The amount of estimators defines the maximum number of iterations at which the boosting is terminated. It is called the “maximum” number, because the algorithm will stop on its own, in case good performance is achieved earlier.</li><li><em>learning_rate</em>: Finally, the learning rate controls how much the new model is going to contribute to the previous one. Normally there is a trade-off between the number of iterations and the value of the learning rate. In other words: when taking smaller values of the learning rate, you should consider more estimators, so that your base model (the weak classifier) continues to improve.</li></ul><pre>from sklearn.ensemble import AdaBoostClassifier<br><br># define base estimator (requires support for sample weighting)<br>est = LogisticRegression() # or est = SVC() or est = DecisionTreeClassifier ….<br><br># n_estimators defines maximum number of estimators at which boosting is terminated<br># learning_rate defines the weight applied to each classifier at each boosting iteration<br>boost_model = AdaBoostClassifier(base_estimator= est, n_estimators = 10, learning_rate=1)<br><br>boost_model = boost_model.fit(X_train, y_train)<br><br>Prediction = boost_model.predict(X_test)</pre><p>The fit()and predict()procedures work similarly to the previous bagging example. As you can see, it is easy to use such functions from existing libraries. But of course, you can also implement your own algorithms to build both techniques.</p><h3>#4: Conclusion: differences &amp; similarities</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bz-M3Fd59LEuoEaKyaPXYA.png" /><figcaption>Image by <a href="https://unsplash.com/">Unsplash</a></figcaption></figure><p>Since we learned briefly how bagging and boosting work, I would like to put the focus now on comparing both methods against each other.</p><h4>Similarities</h4><ul><li><strong>Ensemble methods</strong><br>In a general view, the similarities between both techniques start with the fact that both are<strong> </strong>ensemble methods with the aim to use multiple learners over a single model to achieve better results.</li><li><strong>Multiple samples &amp; aggregation</strong><br>To do that, both methods generate random samples and multiple training data sets. It is also similar that Bagging and Boosting both arrive at the end decision by aggregation of the underlying models: either by calculating average results or by taking a voting rank.</li><li><strong>Purpose</strong><br>Finally, it is reasonable that both aim to produce higher stability and better prediction for the data.</li></ul><h4>Differences</h4><ul><li><strong>Data partition | </strong>whole data vs. bias<br>While bagging uses random bags out of the training data for all models independently, boosting puts higher importance on misclassified data of the upcoming models. Therefore, the data partition is different here.</li><li><strong>Models | </strong>independent vs. sequences<br>Bagging creates independent models that are aggregated together. However, boosting updates the existing model with the new ones in a sequence. Therefore, the models are affected by previous builds.</li><li><strong>Goal | </strong>variance vs. bias<strong><br></strong>Another difference is the fact that bagging aims to reduce the variance, but boosting tries to reduce the bias. Therefore, bagging can help to decrease overfitting, and boosting can reduce underfitting.</li><li><strong>Function | </strong>weighted vs. non-weighted<br>The final function to predict the outcome uses equally weighted average or equally weighted voting aggregations within the bagging technique. Boosting uses weighted majority vote or weighted average functions with more weight to those with better performance on training data.</li></ul><h4>Implications</h4><p>It was shown that the main idea of both methods is to use multiple models together to achieve better predictions compared so single learning models. However, there is <em>no one-over-the-other statement</em> to choose between bagging and boosting since both have advantages and disadvantages.</p><p>While bagging decreases the variance and reduces overfitting, it will only rarely produce better bias. Boosting on the other hand side decreases the bias but might be more overfitted that bagged models.</p><p>Coming back to the variance-bias tradeoff figure, I tried to visualize the extreme cases when each method seems appropriate. However, this does not mean that they achieve the results without any drawbacks. The aim should always be to keep bias and variance in a reasonable balance.</p><p>Bagging and boosting both uses all given features and select only the entries randomly. Random forest on the other side is an extension to bagging that creates also random subsets of the features. Therefore, random forest is used more often in practice than bagging.</p><p><a href="https://medium.com/@jonas_dieckmann">Jonas Dieckmann - Medium</a></p><p>I hope you find it useful. Let me know your thoughts! And feel free to connect on LinkedIn at <a href="https://www.linkedin.com/in/jonas-dieckmann/">https://www.linkedin.com/in/jonas-dieckmann/</a> and/or to follow me here on medium.</p><h3>See also some of my other articles:</h3><ul><li><a href="https://towardsdatascience.com/introduction-to-ica-independent-component-analysis-b2c3c4720cd9">Introduction to ICA: Independent Component Analysis</a></li><li><a href="https://towardsdatascience.com/how-to-set-started-with-tensorflow-using-keras-api-and-google-colab-5421e5e4ef56">How to get started with TensorFlow using Keras API and Google Colab</a></li></ul><h3>References</h3><p>[1]: Bühlmann, Peter. (2012). Bagging, Boosting and Ensemble Methods. Handbook of Computational Statistics. 10.1007/978–3–642–21551–3_33.</p><p>[2]: Machova, Kristina &amp; Puszta, Miroslav &amp; Barcák, Frantisek &amp; Bednár, Peter. (2006). A comparison of the bagging and the boosting methods using the decision trees classifiers. Comput. Sci. Inf. Syst.. 3. 57–72. 10.2298/CSIS0602057M.</p><p>[3]: Banerjee, Prashant. Bagging vs Boosting @kaggle: <a href="https://www.kaggle.com/prashant111/bagging-vs-boosting">https://www.kaggle.com/prashant111/bagging-vs-boosting</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23f9336d3cb0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/ensemble-learning-bagging-and-boosting-23f9336d3cb0">Ensemble learning: Bagging and Boosting</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to ICA: Independent Component Analysis]]></title>
            <link>https://medium.com/data-science/introduction-to-ica-independent-component-analysis-b2c3c4720cd9?source=rss-1c8d1cf684f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2c3c4720cd9</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[independent-component]]></category>
            <category><![CDATA[introduction]]></category>
            <category><![CDATA[ica]]></category>
            <dc:creator><![CDATA[Jonas Dieckmann]]></dc:creator>
            <pubDate>Tue, 14 Feb 2023 13:48:04 GMT</pubDate>
            <atom:updated>2023-03-21T14:46:15.325Z</atom:updated>
            <content:encoded><![CDATA[<p>Have you ever found yourself in a situation where you were trying to analyze a complex and highly correlated data set and felt overwhelmed by the amount of information? This is where Independent Component Analysis (ICA) comes in. ICA is a powerful technique in the field of data analysis that allows you to separate and identify the underlying independent sources in a multivariate data set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*yXuUHpd8gbKIHQ9YKWInLw.png" /><figcaption>Image by <a href="https://unsplash.com/">Unsplash</a></figcaption></figure><p>ICA is important because it provides a way to understand the hidden structure of a data set, and it can be used in a variety of applications, such as signal processing, brain imaging, finance, and many other fields. In addition, ICA can help extract the most relevant information from data, providing valuable insights that would otherwise be lost in a sea of correlations.</p><p>In this article, we will delve into #1 the <strong>fundamentals</strong> of ICA by discussing what cocktail parties might have to do with it, #2 the <strong>3-step-ICA-algorithm,</strong> and #3 how you can <strong>implement </strong>it in your data analysis projects. So, if you’re ready to unlock the full potential of your data, come along and join this journey!</p><h3>#1: Introduction and main idea</h3><p>Independent Component Analysis is one of the various unsupervised learning algorithms which means that we do not need to supervise the model before we can use it. The origin of this method comes from signal processing where we try to separate a multivariate signal into additive subcomponents. Let us jump into an explanation of the main idea:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ue-4dheZQNchwgUJKI3GyQ.png" /><figcaption>Image by author</figcaption></figure><p>Imagine some independent signals or variables. These signals can be represented as signal curves, with the first signal located at the top and the second signal at the bottom within the above image. As a result of the measurements, we did not receive a data set with the signals themselves, but a data set that includes measurements of these two signals which are unfortunately mixed into distinct linear combinations. The objective of ICA is to recover the original, unknown signals by separating the mixed data. The ultimate aim is to reconstruct the data such that each dimension is mutually independent.</p><p>To make this concept more tangible, the most well-known example of ICA, the “cocktail party problem,” will be utilized.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*voKND-AFJ_kWDdxcLqzZQA.png" /><figcaption>Image by <a href="https://unsplash.com/">Unsplash</a></figcaption></figure><h4>The cocktail party problem</h4><p>Imagine attending a cocktail party where multiple individuals are speaking simultaneously, making it difficult to follow a single conversation. It is noteworthy that humans possess the ability to separate individual voice streams in such situations. Technically, this becomes slightly challenging.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TwGH8SGNpzwkUxFsNnKbvA.png" /><figcaption>Cocktail party problem. Image by author.</figcaption></figure><p>Suppose we record the conversations of two groups in the party using two microphones. This results in two mixed signals, with the first measurement having a greater influence of the first group and a lesser influence of the second group, while the second measurement has a greater influence of the second group.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*XZyXZosbTFKtZ0smdtW18Q.png" /><figcaption>Image by author</figcaption></figure><p>The general framework for this can be represented in vector notation here in the grey box. The measurements in <strong>vector X</strong> are actually the signals from <strong>vector S</strong> multiplied with some <strong>mixing coefficients</strong>, represented in <strong>matrix A</strong>. Since we want to extract the full conversations (original signals), we need to solve this for vector S.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/911/1*ONy4a2Y9mJ3167IXDkPSGg.png" /><figcaption>Image by author</figcaption></figure><h4>ICA vs. PCA</h4><p>You have probably already guessed that ICA is in some way related to principal component analysis (PCA). This assumption is not so wrong. The ideas underlying both concepts are not that far apart, but they differ in the last stage, which we will see later.</p><p>Let’s summarise what PCA basically does: Suppose we have two variables that appear to be related. By maximizing the variances using the eigenvector and eigenvalues of these variables, we can convert them into principal components. In this particular example, PCA does a good job of identifying the principal direction of this relationship.</p><p>Let’s use the previous cocktail example as an example. In a very simple representation, we could imagine that the two measurements from microphones one and two have relationships that form something like a cross pattern. If we were to apply PCA in this case, we would get the wrong results, because PCA fails for data sets with more than one main direction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xTtZr8VeypzPaU7-bkzMLg.png" /><figcaption>Image by author</figcaption></figure><p>ICA, on the other hand, solves this problem by focusing on independent components instead of main components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1019/1*Zqi1kmqfq0T4TgM96tapQA.png" /><figcaption>Image by author</figcaption></figure><p>It is important to recall the established conceptual framework. The readings obtained from the microphones correspond to the original signals that have been multiplied by the mixing matrix A. By rearranging the equation with respect to vector S, the only necessary information to determine the original variables is matrix A. However, matrix A is unknown.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GeOh00z0Hz5O6ZyWMiQq1A.png" /><figcaption>Image by author</figcaption></figure><p>Hence, to gain a comprehensive understanding of matrix A and ultimately calculate the vector S, it is necessary to undertake inverse operations through a series of steps. These sequential inverse operations comprise the three stages of the ICA algorithm, which will now be analyzed in greater detail.</p><h3>#2: Separation process | the 3-step-ICA-algorithm</h3><p>Before proceeding to a practical demonstration in R, it is important to understand the three steps of the algorithm. The goal of the algorithm is to perform the multiplication of vector X with matrix A. Matrix A is comprised of three constituent parts, which are the result of multiplicative interactions between the different factors:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/987/1*vD76vLy6P7C4WlIjxowmGQ.png" /><figcaption>Image by author</figcaption></figure><p><strong>Step 1: Find the angle with maximal variance to rotate | estimate U^T </strong><br>The first component of the algorithm involves the use of the matrix U^T, which is based on the first angle Theta. The angle Theta can be derived from the primary direction of the data, as determined through Principal Component Analysis (PCA). This step rotates the graph to the position shown above.</p><p><strong>Step 2: Find the scaling of the principal components | estimate ∑^(-1)</strong><br>The second component involves stretching the figure, which is achieved through the Sigma^-1 step. This step employs the variances of sigma 1 and sigma 2 from the data, similar to the approach utilized in PCA.</p><p><strong>Step 3: Independence and kurtosis assumptions for rotation | estimate <em>V</em></strong><br>The final component, which distinguishes the current algorithm from PCA, involves the rotation of the signals around angle Phi. This step aims to rebuild the original dimensions of the signals by utilizing the independence and kurtosis assumptions for rotation.</p><p>In summary, the algorithm employs measurements and performs rotation around theta, stretching through the use of variances sigma 1 and 2, and finally, rotation around Phi. The mathematical background of these steps has been summarized in the following slide for reference.</p><p>As you can see, we can determine the <strong>inverse matrix A using only the two angles and the variances of the data</strong>, which is actually all we need to process the ICA algorithm. Take the measurements, rotate, and scale them. And finally, we rotate them again to get the final dimension.</p><h3>#3: Code examples with R using fastICA()</h3><p>I hope you have understood the basic idea of the ICA algorithm so far. It is not necessary to understand every single step mathematically, but it is helpful to understand the concept behind it. With this knowledge, I would like to work out a practical example with you to show the practical application of the ICA algorithm using a function called fastICA in R.</p><pre># install fastICA package in R<br>install.packages(&quot;fastICA&quot;)<br><br># load required libraries<br>library(MASS) # To use mvrnorm()<br>library(fastICA)</pre><p>We create two random data sets: signal 1 and signal 2 which could be imagined as voice signals from our two cocktail groups:</p><pre># random data for signal 1 <br>s1=as.numeric(0.7*sin((1:1000)/19+0.57*pi) + mvrnorm(n = 1000, mu = 0, Sigma = 0.004))<br>plot(s1, col=&quot;red&quot;, main = &quot;Signal 1&quot;, xlab = &quot;Time&quot;, ylab = &quot;Amplitude&quot;)<br><br># random data for signal 1 <br>s2=as.numeric(sin((1:1000)/33) + mvrnorm(n = 1000, mu = 0.03, Sigma = 0.005)) <br>plot(s2, col=&quot;blue&quot;, main = &quot;Signal 2&quot;,xlab = &quot;Time&quot;, ylab = &quot;Amplitude&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*FMwfVDXoO2V9T-X2mS_vRQ.png" /><figcaption>Screenshots from R-output: original signals. Image by author</figcaption></figure><p>The red curve stands for the first signal and the blue curve for the second. The shape does not matter in this case. What you should see is that both signals are different from each other. Let’s mix them now!</p><pre># measurements with mixed data x1 and x2<br>x1 &lt;- sine1-2*sine2<br>plot(x1, main = &quot;Linearly Mixed Signal 1&quot;, xlab = &quot;Time&quot;, ylab = &quot;Amplitude&quot;)<br><br>x2 &lt;- 1.73*sine1 +3.41*sine2<br>plot(x2, main = &quot;Linearly Mixed Signal 2&quot;, xlab = &quot;Time&quot;, ylab = &quot;Amplitude&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ODOFoNxmJlUz7vW6V3CfCw.png" /><figcaption>Screenshots from R-output: measurements. Image by author</figcaption></figure><p>As you can see above, we simulate two measurements using both signals. Therefore the signals within the measurements are not independent anymore. Both mixed signals can be imagined as the recordings of our two microphones in the cocktail example. We now forget about our two original signals and imagine, that these two measurements are the only information we have about this data.</p><p>Hence we want to separate them to finally get two independent signals:</p><pre># apply fastICA function to identify independent signals<br>measurements &lt;- t(rbind(x1,x2))<br><br>estimation &lt;- fastICA(measurements, 2, alg.typ = &quot;parallel&quot;, fun = &quot;logcosh&quot;, alpha = 1, method = &quot;C&quot;, row.norm = FALSE, maxit = 200, tol = 0.0001, verbose = TRUE)<br><br>plot(estimation$S[,1], col=&quot;red&quot;, main = &quot;Estimated signals&quot;, xlab = &quot;Time&quot;, ylab = &quot;Amplitude&quot;)<br>lines(estimation$S[,2], col=&quot;blue&quot;) <br>mtext(&quot;Signal 1 estimation in red, Signal 2 estimation in blue&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iSriZU7D4zikLtX0pGBqNA.png" /><figcaption>Screenshots from R-output: independent signals separated again. Image by author</figcaption></figure><p>The result of this algorithm is shown above. The red curve is the estimate of signal 1, while the blue curve estimates signal 2. And — no surprise — the algorithm has estimated almost the original signals, shown here on the right. You may have noticed that the red curve matches the expectation perfectly, while the blue curve appears to be inverted. This is because the algorithm cannot recover the exact amplitude of the source activity. But apart from that, the reconstruction has done a really good job here.</p><h3>Limitation and conclusion</h3><p><em>Let’s start with the bad news:</em> ICA can only separate linearly mixed sources and further, we cannot separate perfectly Gaussian-distributed sources because they would kill the third step of our algorithm. While we expect independent sources that are mixed within linear combinations, the ICA would find a space where even not-independent sources are maximally independent.</p><p><em>But now the good ones: </em>the ICA algorithm is a powerful method for different areas and is easily usable within open source packages for R; Mathlab and other systems. There are various examples, where ICA algorithms were used for applications: face recognition apps, predictions in the stock market, and many more. It is therefore an important and well-respected method in practical usage.</p><p><strong>In a nutshell: <br></strong>We introduced the Independent Component Analysis as an unsupervised learning algorithm. The main idea is to separate given measurements of linear combinations back into the original signals. This is called <em>reconstruction</em> and uses the three-step ICA algorithm. The most popular example to visualize the problem behind this method is again the cocktail party problem. But enough problems with cocktails for now.</p><blockquote>Time for real cocktails parties 🍹</blockquote><p><a href="https://medium.com/@jonas_dieckmann">Jonas Dieckmann - Medium</a></p><p>I hope you find it useful. Let me know your thoughts! And feel free to connect on LinkedIn at <a href="https://www.linkedin.com/in/jonas-dieckmann/">https://www.linkedin.com/in/jonas-dieckmann/</a> and/or to follow me here on medium.</p><h3>See also some of my other articles:</h3><ul><li><a href="https://towardsdatascience.com/ensemble-learning-bagging-and-boosting-23f9336d3cb0">Ensemble learning: Bagging and Boosting</a></li><li><a href="https://towardsdatascience.com/how-to-set-started-with-tensorflow-using-keras-api-and-google-colab-5421e5e4ef56">How to get started with TensorFlow using Keras API and Google Colab</a></li></ul><h3>References</h3><p>[1]: Bell, AJ; Sejnowski, TJ (1997). “The independent components of natural scenes are edge filters”. Vision Research. 37 (23): 3327– 3338. doi:10.1016/s0042–6989(97)00121–1. PMC 2882863. PMID 9425547. </p><p>[2]: Back, AD; Weigend, AS (1997). “A first application of independent component analysis to extracting structure from stock returns”. International Journal of Neural Systems. 8 (4): 473–484. doi:10.1142/s0129065797000458. PMID 9730022. </p><p>[3]: Barlett, MS (2001). Face image analysis by unsupervised learning. Boston: Kluwer International Series on Engineering and Computer Science. </p><p>[4]: Comon, Pierre (1994). “Independent component analysis, A new concept?” Signal Processing, Volume 36, Issue 3,Pages 287–314, ISSN 0165–1684, <a href="https://doi.org/10.1016/0165-1684(94)90029-9.">https://doi.org/10.1016/0165-1684(94)90029-9.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2c3c4720cd9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/introduction-to-ica-independent-component-analysis-b2c3c4720cd9">Introduction to ICA: Independent Component Analysis</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DASC-PM: a novel Process Model for Data Science Projects]]></title>
            <link>https://medium.com/data-science/dasc-pm-a-novel-process-model-for-data-science-projects-9f872f2534b1?source=rss-1c8d1cf684f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f872f2534b1</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[dasc-pm]]></category>
            <category><![CDATA[process-modelling]]></category>
            <category><![CDATA[crisp-dm]]></category>
            <category><![CDATA[project-management]]></category>
            <dc:creator><![CDATA[Jonas Dieckmann]]></dc:creator>
            <pubDate>Tue, 07 Feb 2023 07:29:44 GMT</pubDate>
            <atom:updated>2023-10-04T10:51:12.517Z</atom:updated>
            <content:encoded><![CDATA[<h3>DASC-PM: a Novel Process Model for Data Science Projects</h3><h4>Introduction of an alternative approach to the popular CRISP-DM</h4><p>With easy access to data, increasing computing power, and user-friendly analytics software, there has been a massive increase in the number of data science projects in various industries. While the early years felt like the Wild West, it is now much more common (and recommended) to follow specific frameworks for data-related projects.</p><p><strong>Process models </strong>provide a clear and structured approach to defining and organizing tasks, activities, and deliverables throughout the project lifecycle. By implementing a consistent process, teams can ensure that all project objectives are met and that the final deliverables are of high quality. In addition, process models help reduce the risk of delays, errors, and budget overruns, making them an essential part of the data science project management.</p><blockquote>Please note that the entire article is a quick introduction and a brief summary of the framework, which was produced by a <strong>larger group of authors</strong>, with me contributing to DASC-PM v1.1 as one of the co-authors:</blockquote><blockquote><strong><em>Schulz et al. (2022): </em></strong><em>“</em><a href="https://www.researchgate.net/publication/361428426_DASC-PM_v11_-_A_Process_Model_for_Data_Science_Projects"><em>DASC-PM v1.1 — A Process Model for Data Science Projects</em></a><em>”</em><strong><em> </em></strong><em>(2022), Publisher: NORDAKADEMIE gAG Hochschule der Wirtschaft, ISBN: 978–3–00–064898–4, DOI:</em><a href="http://dx.doi.org/10.25673/32872.2"><em>10.25673/32872.2</em></a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*kKwpscTt3ZhOh1w13BkERw.png" /><figcaption>Data Science Process Model DASC-PM (Schulz et al. 2022)</figcaption></figure><p>The “<strong>DA</strong>ta <strong>SC</strong>ience<strong> -</strong> <strong>P</strong>rocess <strong>M</strong>odel” (<a href="https://www.researchgate.net/publication/361428426_DASC-PM_v11_-_A_Process_Model_for_Data_Science_Projects">DASC-PM</a>) is a novel process model for data science projects that describes the key areas relevant to the project and the phases to be completed. It explains the typical tasks within the phases and depicts the project roles involved and the required competencies. The following article aims to introduce the main concepts and to work out the advantages compared to known concepts such as <a href="https://www.researchgate.net/publication/239585378_CRISP-DM_Towards_a_standard_process_model_for_data_mining">CRISP-DM</a>, <a href="https://docs.microsoft.com/en-us/azure/architecture/data-science-process/lifecycle">TDSP</a>, <a href="http://www2.cs.uregina.ca/~dbd/cs831/notes/kdd/1_kdd.html">KDD</a>, or <a href="https://www.datascience-pm.com/semma/">SEMMA</a>.</p><h3>The rise of a novel approach</h3><p>With the cross-industry standard process for data mining (CRISP-DM), there exists already a well-known “framework for carrying out data mining projects which is independent of both the industry sector and the technology used.” [4] Besides that, other relevant concepts such as TDSP, SEMMA, or KDD aim to provide comparable models that outperform each other on several details. However, it was of interest to take a step back to identify (meta-) requirements relevant to process models focusing on data science. The requirements were collected via a survey from April 2019 to February 2020, cover both scientific and practical aspects, and, thus, address the first research question [2]:</p><blockquote>Which theoretical and practical requirements are imposed upon data science process models?</blockquote><p>Data collection was conducted in a working group consisting of 22 experts, including 9 professors as well as 13 practitioners and scientists with relevant theoretical and practical experience in data science. Based on that, it was examined the extent to which the related process models fulfill the requirements collected previously. The below table offers an overview of the results of the investigation by the specific requirement and process model. Filled Harvey balls indicate that a requirement is addressed by the respective process model, half-filled ones that a requirement is at least mentioned, and empty ones that a requirement is neither mentioned nor addressed. [2]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/1*iKifIsTA6Hz9L35CTd7XtQ.png" /><figcaption>(Meta-) Requirements of a data science process model (Schulz et al. 2020)</figcaption></figure><p>Recognizing that none of the related, well-known process models could fulfill the 17 identified requirements placed upon process models for data science projects, <em>Schulz et al.</em> developed a novel data science process model called DASC-PM to address the second research question [2]:</p><blockquote>How can a data science process model that is aligned with relevant theoretical and practical requirements be conceptualized?</blockquote><h3>Brief introduction of the five DASC-PM phases</h3><p>In the following section, we will briefly capture the main ideas of the five core phases within the newly created process model: Project Order, Data Provision, Analysis, Deployment, and Application. [3]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*lPg0Ik_9-yFuIUU6K505lw.png" /><figcaption>Data Science Process Model DASC-PM (Schulz et al. 2022)</figcaption></figure><p><em>The corresponding areas and tasks will be visualized as the following:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/216/1*UoOS_S1ympWGyfe9bQHybg.png" /><figcaption>Legend of phase representation (Schulz et al. 2022)</figcaption></figure><h4>Phase 1: Project Order</h4><p>Problems existing within a domain trigger a use-case development. The promising use cases are subsequently configured to a data science project outline. All associated tasks are reflected in the project order phase. Through the early, relatively comprehensive consideration of the project, comprehensive abilities in almost all skill areas are also frequently required here. [3]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*ouS-hXzRNlUF0lHVpPORtw.png" /><figcaption>Phase 1: Project Order (Schulz et al. 2022)</figcaption></figure><h4>Phase 2: Data Provision</h4><p>Within the data provision phase, all activities that are allocable to the data key area are summarized, which is why the term used is broadly formulated. The phase contains data preparation (from recording to storage), data management, and exploratory data analysis. This phase results in a data source that is suited for further analysis. [3]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*DdYsablIXsWjAxbk_xhExw.png" /><figcaption>Phase 2: Data Provision (Schulz et al. 2022)</figcaption></figure><h4>Phase 3: Analysis</h4><p>In a data science project, either existing procedures can be used or a new procedure developed the decision in question is a separate challenge. The phrase, therefore, includes not only performing the analysis but also related activities. The artifact of the phase is an analysis result that has traversed a methodical and technical evaluation. [3]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/659/1*RFieEV4L7CNIg_fAnjXlIg.png" /><figcaption>Phase 3: Analysis (Schulz et al. 2022)</figcaption></figure><h4>Phase 4: Deployment</h4><p>In the deployment phase, an applicable form of the analysis results is created. Depending on the project, this can entail comprehensively considering technical, methodological, and professional tasks — or it can be handled pragmatically. The analysis artifact can include results as well as models or procedures and is provided to its target recipients in various forms. [3]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*hB2rXeT3Y12FaKFgo14MSQ.png" /><figcaption>Phase 4: Deployment (Schulz et al. 2022)</figcaption></figure><h4>Phase 5: Application</h4><p>Using artifacts after the project performance is not considered a primary part of a data science project. Monitoring is necessary, however, to check the models continuing suitability in the application and obtain findings from the application for ongoing and new developments (including developments for the purposes of iterative approaches). [3]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*H4-x7M-mphBXxovdlgd2Vg.png" /><figcaption>Phase 5: Application (Schulz et al. 2022)</figcaption></figure><h3>Overarching key areas</h3><p>Besides the five process steps, the model contains three overarching key areas which have to be taken into account in all phases of the project:</p><h4>Domain</h4><p>At many points in a data science process, broad background knowledge of the domain is needed. Examples are the identification of the analysis target or the correct understanding of data, its origin, quality, and connections. Other examples include the assessment and classification of analysis results in the application as well as subsequent practical use. The area “Domain” also encompasses rating strengths and weaknesses of existing solutions, conducting requirements analysis, supporting parametrization of models, and finally evaluating the success of the project. Legal, social, and ethical aspects of data science projects will also be discussed here. [1]</p><h4>Scientificity</h4><p>Just because data science projects are scientific in nature does not mean they claim to be complete, formalized, academic, or consistently research-orientated in general. Although this might certainly be the case for research projects, the aspect of their scientificity within a business context primarily refers to a solid methodology: a typically expected characteristic or minimum requirement of scientific work. [1]</p><p>The defined project order must be processed methodically in every project phase. Special mention must be made here of the project management and structured processing that is placed in the foreground by using a process mode. Details on the degree of scientificity required must be established while considering the project situation and domain specifics. [1]</p><h4>IT Infrastructure</h4><p>All the steps that a data science project traverses depend on the underlying IT infrastructure; the actual extent of IT support, however, should be individually assessed for each project. Even if the use of specific hardware and software is frequently determined within the organization, the limiting and empowering characteristics of the IT infrastructure (as well as the possibility of expanding the infrastructure, if applicable) must be considered in all project phases. [1]</p><h3>Summary</h3><p>The DASC-PM is the result of a scientific approach to collect, structure, and address (meta-) requirements for process models in the area of data science. Since all previous, well-known models did not meet all defined requirements, it was of interest to introduce a novel concept that allows researchers as well as practitioners in business &amp; industry to use the DASC-PM to structure data science projects in a phase-orientated way. However, the authors claim that the DASC-PM should not be considered a finished deliverable, but more as a framework that can be continuously improved through scientific and practical discourse.</p><p><a href="https://medium.com/@jonas_dieckmann">Jonas Dieckmann - Medium</a></p><p>I hope you find it interesting. Let me know your thoughts and feel free to connect on LinkedIn <a href="https://www.linkedin.com/in/jonas-dieckmann/">https://www.linkedin.com/in/jonas-dieckmann</a> and/or to follow me here on medium.</p><h3>See also some of my other articles:</h3><ul><li><a href="https://towardsdatascience.com/case-study-applying-a-data-science-process-model-to-a-real-world-scenario-93ae57b682bf">Case Study: Applying a Data Science Process Model to a Real-World Scenario</a></li><li><a href="https://towardsdatascience.com/ethics-in-ai-potential-root-causes-for-biased-algorithms-890091915aa3">Ethics in AI: Potential Root Causes for Biased Algorithms</a></li></ul><h3>References</h3><p><em>The whole article represents a brief summary and is based on the framework:</em></p><p>[1] <strong>Schulz et al. (2022):<em> </em></strong><em>“</em><a href="https://www.researchgate.net/publication/361428426_DASC-PM_v11_-_A_Process_Model_for_Data_Science_Projects">DASC-PM v1.1 — A Process Model for Data Science Projects</a>”<strong> </strong>(2022), Publisher: NORDAKADEMIE gAG Hochschule der Wirtschaft, ISBN: 978–3–00–064898–4, DOI:<a href="http://dx.doi.org/10.25673/32872.2">10.25673/32872.2</a></p><p><em>as well as the introduction:</em></p><p>[2] <strong>Schulz et al. (2020):</strong><em> </em>“Introducing DASC-PM: A Data Science Process Model” (2020). ACIS 2020 Proceedings. 45. <a href="https://aisel.aisnet.org/acis2020/45">https://aisel.aisnet.org/acis2020/45</a></p><p><em>An additional quotable source is provided in the following book:</em></p><p>[3] <strong>Kuehnel, S., Neuhaus, U., Kaufmann, J., Schulz, M., Alekozai, E.M. (2023)</strong>. “Using the Data Science Process Model Version 1.1 (DASC-PM v1.1) for Executing Data Science Projects: Procedures, Competencies, and Roles.” In: Barton, T., Müller, C. (eds) Apply Data Science. Springer Vieweg, Wiesbaden. <a href="https://doi.org/10.1007/978-3-658-38798-3_8">https://doi.org/10.1007/978-3-658-38798-3_8</a></p><p>Other references:</p><p>[4] <strong>Wirth, R., Hipp, J. (2000)</strong> “CRISP-DM: Towards a Standard Process Model for Data Mining”, Proc. 4th Int. Conference on Practical Applications of Knowledge Discovery and Data mining, pp. 29–39.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f872f2534b1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/dasc-pm-a-novel-process-model-for-data-science-projects-9f872f2534b1">DASC-PM: a novel Process Model for Data Science Projects</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ethics in AI: Potential Root Causes for Biased Algorithms]]></title>
            <link>https://medium.com/data-science/ethics-in-ai-potential-root-causes-for-biased-algorithms-890091915aa3?source=rss-1c8d1cf684f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/890091915aa3</guid>
            <category><![CDATA[data-for-change]]></category>
            <category><![CDATA[ai-ethics]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[bias]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Jonas Dieckmann]]></dc:creator>
            <pubDate>Fri, 27 Jan 2023 19:43:59 GMT</pubDate>
            <atom:updated>2023-03-21T14:40:58.804Z</atom:updated>
            <content:encoded><![CDATA[<h4>An alternative approach to understanding bias in data</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*06_gnx0LG7FtCrMwsHN61A.jpeg" /><figcaption>Image Credits: <a href="https://pixabay.com/vectors/lady-justice-allegory-justice-7313900/">Pixabay</a></figcaption></figure><p>A s the number of data science applications increases, so does the potential for abuse. It is easy to condemn developers or analytics teams for the results of their algorithms, but are they the main culprits? The following article tries to discuss the problem from a different angle and <strong>concludes that ethical abuse might be the real problem with data in our society</strong>.</p><blockquote>A better world won’t come about simply because we use data; data has its dark underside. ¹</blockquote><h3>Biased algorithms</h3><p>Today, discussions of data science are often associated with significant opportunities in business and industry to deliver more accurate predictive and classification solutions and improve people’s lives in the health and environmental sectors.² ³ Perhaps as important as the benefits are the data ethics challenges that should be considered when evaluating new solutions and approaches in data analytics. As a specific branch of ethics, data ethics addresses moral issues related to data, algorithms, and the respective practices to formulate and support morally good solutions.⁴ Overall, there seems to be a fine line between the use and misuse of data. It is common for companies to collect data not only to increase their profits but also to provide a tailored and targeted experience to their customers.⁵ However, when companies start to externally exploit their data for purposes other than those for which it was collected, ethical questions arise. In addition to these privacy-related issues, another challenge is data analysis bias, which can arise in several ways. Examples include the creation of survey questions by people with a particular intent or framing, the selective collection of data from groups with a particular background, or the underlying bias of those from whom the data is drawn.⁶</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*tvdtBxCrcV-5d44asf63kw.png" /><figcaption>Image Credits: <a href="https://pixabay.com/photos/play-figures-green-blue-play-wood-4541731/">Pixabay</a></figcaption></figure><h4>Popular examples of algorithmic discrimination</h4><p>The issues are not at all theoretical — they lead to real concerns and cause serious discussions as described in several case studies:</p><ul><li><strong>Racial discrimination in crime prediction</strong><br>A problematic and well-known example is the prediction of crime. A 2016 machine learning paper by Shanghai Jiao Tong University investigated whether crime can be detected in people based on facial feature analysis.⁷ The result was that members of the population who look “different” from the average are more likely to be suspected of committing crimes. ⁸ Another example of biased data science approaches is the risk assessment algorithm used in the United States to predict the likelihood of re-offending (repeated criminal behavior) by arrested individuals. It turned out that the underlying algorithm tended to predict the likelihood of recidivism of black defendants twice as high as that of white defendants, while white defendants tended to be incorrectly classified as low-risk when in fact they were recidivists.⁹</li><li><strong>Gender discrimination in recruitment</strong><br>Another area that is affected by biased algorithms is recruitment. In 2014, Amazon used machine learning-based software to evaluate and rank the CVs of potential candidates to find new top talent. In 2015, the company discovered that its new system was not evaluating applicants in a gender-neutral way, especially for software development jobs. Based on hiring over the last ten years, the system penalized all applications that included the word “women” in their CVs.¹⁰</li><li><strong>Stereotypes in the social media </strong><br>Gender and race bias can also be found in Facebook’s advertising. In their 2019 empirical study, Ali et al. discovered that there may be biased rules for delivering ads to specific users, based on the predicted relevance of those ads to different audiences. This relevance is often based on male/female stereotypes, classified and correlated through image analysis software that analyses digital advertising media.¹¹</li></ul><h4>Principle for big data ethics</h4><p>How can this be prevented? These cases are just a few of many instances where algorithms are used in a biased way in today’s world. Many experts discuss and evaluate ways to avoid bias in data analysis, which is expressed in one of the “5 Principles for Big Data Ethics” published by Herdiana: “Big Data should not institutionalize unfair biases such as racism or sexism.” ¹² While this principle is considered a general rule within data ethics, the examples given “such as racism or sexism” seem to be the most common occurrences. However, a more general description might be:</p><blockquote>“Big data should not institutionalize unfair biases that cause discrimination”.</blockquote><p>The Canadian Human Rights Commission (CHRC) describes discrimination as an act or decision that treats a person or group poorly for reasons called grounds. These grounds are protected by the Canadian Human Rights Act and include<em> race, national or ethnic origin, color, religion, age, sex, sexual orientation, gender identity or expression, marital status, family status, disability, genetic characteristics, and a conviction for which a pardon has been granted or registration suspended</em>. ¹³ In sociology, discrimination describes the belittling of others based on individual or group characteristics, practiced systemically and therefore constituting a gross violation of human rights.¹⁴ The European Convention on Human Rights (ECHR) also prohibits discrimination in all of the above areas in Article 14, adding to the CHRC list <em>language, political opinion,</em> and <em>property </em>as well.¹⁵</p><p>The European Economic and Social Committee (EESC) study entitled “The ethics of Big Data: Balancing economic benefits and ethical questions of Big Data in the EU policy context” describes algorithm bias as one of the critical ethical problems of Big Data. They define trust in algorithms as an ethical problem because most people think that machines are neutral by definition. In reality, this is not the case, and therefore the risk can be very high.¹⁶ But what are possible reasons for this lack of neutrality? It is hard to imagine that the best data scientists in the world have developed their algorithms with this goal in mind.</p><h3>Root causes analysis</h3><p>Algorithms are not biased. Nevertheless, there exist many popular examples of skewed outcomes from algorithms while deeper assessments of the causes and drivers of the current outcome are often superficial. Moreover, the possible alternatives and outcomes in a hypothetical world without algorithms are missing. Therefore, the question of biased algorithms should not focus exclusively on the neutrality of algorithms, but also highlight the (missing) causes. The following analysis, therefore, aims to further explore issues related to bias and how today’s ethical pressure on developers could be understood as a collective responsibility. Nevertheless, it is crucial to underline the importance of eliminating discrimination in both technical and non-technical world, as the two are directly linked and influence each other:</p><ol><li><strong>A biased society leads to biased data <br></strong> By design, <strong>discrimination takes place before the data is even collected, as datasets are historic collections that have been influenced by society itself</strong>. To think about the technical aspects of this issue, one needs to examine the possible causes of biased systems. In most papers and studies, the authors state that the algorithms themselves are biased and cause discrimination.¹⁷ ¹⁸ However, it is difficult to ascribe the very cause of discrimination to algorithms themselves, since algorithms perform decision rules based on a given input and the interactions of given parameters in an automated way. ¹⁹ The rationale behind machine learning algorithms is usually an optimization problem where rules are applied to parameters based on their past effect on the outcome.²⁰ The presence of bias in the interaction between a given variable (e.g. gender) and the target (e.g. suitability for a vacancy) is therefore not caused by the algorithm itself, but by the underlying data set. Looking at the Amazon recruiting example, the biased outcome (i.e., favoring male applications vs. female) can be attributed to the training dataset initially fed to the algorithm. Assuming that the majority of former employees in technical positions were male, the algorithm anticipated this would make a success factor. So discrimination does not begin with the algorithm or its developer as an individual, but rather with the very hiring behavior of the company, which has been perpetrated for decades.</li><li><strong>Hidden bias between variables<br></strong>Provocatively, <strong>one could argue that every dataset is biased to some extent since it originated in a likely biased environment</strong>. An algorithm can only factor in what is present in the (biased) data, and data usually only contains what is present in the (biased) non-technical world. A reasonable and common-sense approach would be to remove such discriminatory variables from the dataset before beginning to model. However, no guarantee removing a variable will remove the presence or influence of that variable on the rest of the dataset. Let us imagine a fictitious dataset containing the variables <em>gender</em> and <em>sport</em>. Both variables could have an interaction, such as a “correlation between men and football and women and riding”, although this blunt example may not fully represent our society, since there may be women who prefer football to riding and vice versa. However, due to an unbalanced (e.g., male-heavy) dataset, it is likely that the algorithm indirectly favors candidates with a football preference over others, as the training process is based on biased information. This example shows that removing critical variables can reduce the risk of discrimination, but is not sufficient to prevent it. Moreover, it is of great importance how balanced the data sets represent the different expressions of a variable. A sensitive (discriminatory) item may be less problematic if the expressions are evenly distributed. If a subset of entries is underrepresented, the algorithm might factor this in negatively. Therefore, it is difficult to establish whether a dataset is truly unbiased.²¹</li><li><strong>Technical decisions vs. human decisions </strong><br><strong>There is no guarantee that humans make less biased decisions than algorithms</strong>. In addition to the idea that any dataset can be biased and lead to biased results in an algorithmic process, another key indicator of unfair predictions/classifications is the lack of information, in the form of incomplete data.²² It is difficult to guarantee that a given dataset possesses full visibility to explain a target behavior. Common reasons for the lack of data may be limited access, active exclusion of information, or lack of knowledge about the marginal impact of additional information. Another risk for discrimination does not only result from the technical implementation but can also be rooted in the existing structures of our society. The advantage of Big Data in combination with machine learning algorithms is the possibility to apply human rules and decisions to a large amount of information. If algorithms are considered biased, one might wonder how humans would have processed the analysis of given data. Looking back to social media advertising, personalized advertising aims to make individual suggestions based on one’s preferences. This implies that algorithms support individualism. However, algorithms have the opposite effect, as they process automated and standardized decision rules based on general rather than individual patterns. Ultimately, it can be assumed that <strong>algorithms support collectivism </strong>rather than individualism.²³</li></ol><h3>Conclusion</h3><p>Our analysis has shown that the bias in algorithms is mainly a result of bias in today’s society. The latter supports the collection of biased datasets, which may contain (hidden) relationships, ultimately leading to biased results. Furthermore, we question whether human decision-making is less biased compared to algorithmic rules. Lastly, we have concluded that more complex algorithms are often too non-transparent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*V1RKzKRvx4DAqkS9ncvidQ.png" /><figcaption>Image Credits: Pixabay</figcaption></figure><p><strong>Mitigation strategy</strong><br>In discussions about the responsibility for biased results of machine learning algorithms, the developer or the data science team is often blamed for discriminatory behavior. On one hand, this argument is overly simplistic, and the apportionment of the blame needs to be further differentiated. On the other, any developer working with machine learning algorithms should be aware of the sensitivity of their task and their respective responsibilities. Since algorithms support collectivism, it is important to investigate the fairness of these collective algorithms that make decisions or suggestions for individuals. Literature suggests other techniques to reduce bias, such as a system design that randomizes a small sample of queries or the use of propensity weighting techniques.²⁴ For bias in human-based content, there are several ideas for quantifying discrimination and assessing the fairness of algorithms.²⁵</p><p><strong>Fairness as a shared responsibility: both in the technical and real world<br></strong>It is the responsibility of data teams to identify and minimize biased behavior in their data models. However, looking at the discriminatory factors examined in this article, it is a global responsibility to ensure that no one experiences such injustices, either through technical systems or human actions:</p><ol><li>Humans must treat humans equally and without discrimination</li><li>Algorithms must treat humans equally and without discrimination</li><li>We should not hide behind algorithms to justify biased and discriminatory human behavior</li></ol><p><strong>Outlook: now what?</strong><br>It seems important to evaluate decisions or predictions not only based on the outcome itself but also based on the ethical considerations involved in working out the solution. We also need to rethink systems that involve biased behavior. However, there is no evidence that the alternative (human-based decision-making) leads to more equality in general. Consequently, we should not only focus on the problems with bias <em>within</em> technology but rather broaden the discussion to include bias <em>outside</em> technology. Nevertheless, <em>precisely because of this</em>, <strong>unbiased systems should help improve our current social weaknesses</strong> and possibly lead to <strong>more equality in the world to come. </strong>People could use AI to more easily detect discrimination in the non-technological world. Because <strong>wherever we find bias in the data, there is also bias in society</strong>.</p><p><a href="https://medium.com/@jonas_dieckmann">Jonas Dieckmann - Medium</a></p><p>I hope you find it interesting. Let me know your thoughts and feel free to connect on LinkedIn <a href="https://www.linkedin.com/in/jonas-dieckmann/">https://www.linkedin.com/in/jonas-dieckmann</a> and/or to follow me here on medium.</p><h3>See also some of my other articles:</h3><ul><li><a href="https://towardsdatascience.com/how-to-set-started-with-tensorflow-using-keras-api-and-google-colab-5421e5e4ef56">How to get started with TensorFlow using Keras API and Google Colab</a></li><li><a href="https://towardsdatascience.com/case-study-applying-a-data-science-process-model-to-a-real-world-scenario-93ae57b682bf">Case Study: Applying a Data Science Process Model to a Real-World Scenario</a></li></ul><h3>References</h3><p>[1]: Loukides, M., Mason, H., &amp; Patil, D. (2018). Ethics and Data Science.</p><p>[2]: Floridi, L., &amp; Taddeo, M. (2016). What is data ethics? Philosophical Transactions Of The Royal Society A: Mathematical, Physical, And Engineering Sciences, 374(2083), 20160360. doi: 10.1098/rsta.2016.0360</p><p>[3]: Google (2021). AI FOR SOCIAL GOOD, Applying AI to some of the world’s biggest challenges Retrieved 10 March 2021, from <a href="https://ai.google/social-good">https://ai.google/social-good</a></p><p>[4] Mittelstadt, B., &amp; Floridi, L. (2015). The Ethics of Big Data: Current and Foreseeable Issues in Biomedical Contexts. Science And Engineering Ethics, 22(2), 303–341. DOI: 10.1007/s11948–015- 9652–2</p><p>[5] Hand, D. (2018). Aspects of Data Ethics in a Changing World: Where Are We Now? Big Data, 6(3), 176–190. DOI: 10.1089/big.2018.0083</p><p>[6] Broad, E., Smith, A., &amp; Wells, P. (2017). Helping organizations navigate ethical concerns in their data practices. Open Data Inst.</p><p>[7] Wu, Xiaolin &amp; Zhang, Xi. (2016). Automated Inference on Criminality using Face Images.</p><p>[8] Sullivan, B. (2016). A New Program Judges If You’re a Criminal From Your Facial Features. Retrieved 10 March 2021, from <a href="https://www.vice.com/en/article/d7ykmw/new-programdecides-criminality-from-facial-features">https://www.vice.com/en/article/d7ykmw/new-programdecides-criminality-from-facial-features</a></p><p>[9] Angwin, J., Larson, J., Mattu, S., &amp; Kirchner, L. (2016). Machine Bias. Retrieved 28 February 2021, from <a href="https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing">https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing</a></p><p>[10] Dastin, J. (2018). Amazon scraps secret AI recruiting tool that showed bias against women. Retrieved 28 February 2021, from <a href="https://www.reuters.com/article/us-amazon-com-jobsautomation-insight-idUSKCN1MK08G">https://www.reuters.com/article/us-amazon-com-jobsautomation-insight-idUSKCN1MK08G</a></p><p>[11] Ali, M., Sapiezynski, P., Bogen, M., Korolova, A., Mislove, A., &amp; Rieke, A. (2019). Discrimination through Optimization. Proceedings Of The ACM On Human-Computer Interaction, 3(CSCW), 1–30. DOI: 10.1145/3359301</p><p>[12] Herdiana (2013), “5 Principles for Big Data Ethics,” Journal of Chemical Information and Modeling. Retrieved 10 March 2021, from: <a href="https://aiforceos.com/2018/06/16/big-data-ethics">https://aiforceos.com/2018/06/16/big-data-ethics</a></p><p>[13] Canadian Human Rights Commission. What is Discrimination. Retrieved 28 February 2021, from <a href="https://www.chrc-ccdp.gc.ca/eng/content/what-discrimination">https://www.chrc-ccdp.gc.ca/eng/content/what-discrimination</a></p><p>[14] Hartfiel, G., &amp; Hillmann, K. (1994). Dictionary of Sociology. Stuttgart: Kröner</p><p>[15] Council of Europe (1950). European Convention for the Protection of Human Rights and Fundamental Freedoms. Retrieved 10 March 2021, from <a href="http://www.refworld.org/docid/3ae6b3b04">www.refworld.org/docid/3ae6b3b04</a></p><p>[16] European Economic and Social Committee. (2017). The ethics of Big Data.</p><p>[17] Eslami, M., Vaccaro, K., Karahalios, K. and Hamilton, K. (2017) ‘Be Careful; Things Can Be Worse than They Appear’: Understanding Biased Algorithms and Users’ Behavior Around Them in Rating Platforms. Proceedings of the International AAAI Conference on Web and Social Media, 11(1).</p><p>[18] Kim, P. (2017) Auditing Algorithms for Discrimination, 166 U. Pa. L. Rev. Retrieved 10 March 2021, from scholarship.law.upenn.edu/penn_law_review_online/vol166/iss1/10</p><p>[19] Mittelstadt, B., Allo, P., Taddeo, M., Wachter, S., &amp; Floridi, L. (2016). The ethics of algorithms: Mapping the debate. Big Data &amp; Society, 3(2), 205395171667967. DOI: 10.1177/2053951716679679</p><p>[20] Lehr, D., Ohm, P. (2017). Playing with the Data: What Legal Scholars Should Learn About Machine Learning, UCDL Review 51: 653, 671.</p><p>[21] Calders, T., &amp; Žliobaitė, I. (2013). Why Unbiased Computational Processes Can Lead to Discriminative Decision Procedures. Studies In Applied Philosophy, Epistemology, And Rational Ethics, 43–57. DOI: 10.1007/978–3–642–30487–3_3</p><p>[22] Rosen, R. (2019). Missing Data and Imputation. Retrieved 10 March 2021, from <a href="https://towardsdatascience.com/missing-data-and-imputation-89e9889268c8">https://towardsdatascience.com/missing-data-and-imputation-89e9889268c8</a></p><p>[23] Saake. I., Nassehi, A. (2004). Die Kulturalisierung der Ethik. Eine zeitdiagnostische Anwendung des Luhmannschen Kulturbegriffs. Frankfurt/M: 102, 135</p><p>[24] Ridgeway, G., Kovalchik, S., Griffin, B., &amp; Kabeto, M. (2015). Propensity Score Analysis with Survey Weighted Data. Journal Of Causal Inference, 3(2), 237–249. doi: 10.1515/jci-2014–0039</p><p>[25] Floridi, L. (2016). Faultless responsibility: on the nature and allocation of moral responsibility for distributed moral actions. Philosophical Transactions Of The Royal Society A: Mathematical, Physical, And Engineering Sciences, 374(2083), 20160112. doi: 10.1098/rsta.2016.0112</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=890091915aa3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/ethics-in-ai-potential-root-causes-for-biased-algorithms-890091915aa3">Ethics in AI: Potential Root Causes for Biased Algorithms</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>