<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Giannis Tolios on Medium]]></title>
        <description><![CDATA[Stories by Giannis Tolios on Medium]]></description>
        <link>https://medium.com/@giannistolios?source=rss-334ac6ae3ca3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*dMF50Zimwv8D6rlswVCwZg.jpeg</url>
            <title>Stories by Giannis Tolios on Medium</title>
            <link>https://medium.com/@giannistolios?source=rss-334ac6ae3ca3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 17:11:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@giannistolios/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Streamlining Data Science with MLJAR Studio]]></title>
            <link>https://giannistolios.medium.com/streamlining-data-science-with-mljar-studio-7406163e3387?source=rss-334ac6ae3ca3------2</link>
            <guid isPermaLink="false">https://medium.com/p/7406163e3387</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Giannis Tolios]]></dc:creator>
            <pubDate>Thu, 14 May 2026 14:08:22 GMT</pubDate>
            <atom:updated>2026-05-14T14:37:41.916Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Guide for Accelerated Machine Learning Workflows</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*O_M3Bp6wYWlYetHG" /></figure><p>Data science can be significantly challenging, especially for beginners in the field. Statistical analysis, data visualization and machine learning are complicated disciplines, requiring attention to detail and significant expertise. Data professionals should not become frustrated though, as modern tools can help them simplify complicated tasks, thus saving valuable time. For example, <a href="https://mljar.com/">MLJAR</a> is a <strong>leading data science company</strong> that has developed powerful software to assist data scientists. In this article, we are utilizing various MLJAR tools to create a streamlined data science workflow. More specifically, we are going to explore the functionality of <strong>AutoML </strong>and <strong>MLJAR Studio</strong> . Let’s begin working on the project!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*17nvCXmpvHnZM4Px" /></figure><p><a href="https://mljar.com/automl/">MLJAR AutoML</a> is an open source Python library aiming to automate machine learning experiments. AutoML provides a complete workflow to preprocess the data, train machine learning models and perform hyper-parameter tuning to find the best one. The library supports advanced automated machine learning capabilities, thus enabling optimized exploratory data analysis and ML model training. Furthermore, AutoML has been cited in more than 100 research papers, hence being established as a scientific tool. You can easily install MLJAR AutoML by running the following command on your terminal:</p><pre>pip install mljar-supervised</pre><p>If you prefer the conda package manager, enter this command instead :</p><pre>conda install -c conda-forge mljar-supervised</pre><p>AutoML can be easily used in any Python environment, but we are going to work with MLJAR Studio in this tutorial. Keep in mind that Studio includes every necessary library, so you don’t need to install AutoML separately!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XhYrpZgZApw80aZO" /></figure><p><a href="https://mljar.com/studio/">MLJAR Studio</a> is an amazing data science toolkit that accelerates your data science workflows by utilizing a powerful AI assistant. MLJAR provides free access to their customized AI model, specifically trained for data science tasks. Alternatively, you can also use OpenAI models or local AI with Ollama. Here are the main features of MLJAR Studio:</p><p><strong>AI Data Analyst:</strong> this is an AI-powered assistant utilized for notebook-based data analysis. You start by asking questions in natural language, then apply analysis on local datasets, and get Python code that you can inspect and execute!</p><p><strong>AI Code Assistant:</strong> this is an AI agent that helps you write code in data science projects. Whether you need help with data cleaning, visualization, or machine learning, the AI assistant provides optimized Python code for every task. The code blocks are easy to review, edit, and integrate right inside your Jupyter notebook.</p><p><strong>AutoLab Experiments:</strong> AutoLab Experiments let you run machine learning experiments in a structured workflow. You begin by defining a task, set the constraints, and let AI agents explore multiple solutions while keeping everything transparent and reproducible.</p><p>In the rest of this tutorial, we are going to explore practical use cases and workflows for each of the MLJAR Studio features. First of all, you need to download and install the software by visiting <a href="https://mljar.com/downloads">this link</a>. MLJAR is compatible with all major operating systems, including Microsoft Windows, Apple macOS and Linux. After installing the software, you have to install the Python environment and start using the <strong>free plan</strong>, or consider upgrading to pro / business. You are then ready to begin working on your data science projects!</p><h3>AI Data Analyst</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MiBwfJYXu4XgMmUs" /></figure><p>As mentioned previously, the AI Data Analyst helps you analyze the chosen dataset of your preference, by using natural language prompts. You can begin using this feature by clicking <strong>Create a New Analysis</strong> in the main menu, and then selecting <strong>AI Data Analyst</strong>. We then enter a prompt, asking the AI Data Analyst to load, analyze and visualize the <a href="https://archive.ics.uci.edu/dataset/53/iris">Iris dataset</a>. Iris is one of the most popular datasets used in statistics and machine learning, containing 3 classes that refer to Iris plant species, i.e. Setosa, Versicolour and Virginica.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XjN05Xb11lZ4azAYaIqDqw.gif" /></figure><p>As demonstrated in the video above, AI Data Analyst successfully loads the Iris dataset, then displays basic information about it and creates the requested figure. Every Jupyter notebook cell is explained in detail, so we comprehend the included Python code. Finally, the data analysis insights are presented, thus helping us understand our dataset, as well as the AI model output in general. By default, the Python code and detailed notebook cell explanations are hidden, hence letting users focus on the data insights exclusively, if they prefer to choose that option. Therefore, AI Data Analyst is <strong>an excellent tool for business users</strong> who require insights, without necessarily being familiar with the technical aspect of Python coding.</p><h3>AI Code Assistant</h3><p>After creating a project with the AI Data Analyst, we can simply click <strong>Open in AI-Assisted Notebook</strong> to enable this mode. Alternatively, we can open the main menu of MLJAR Studio and select AI-Assisted Notebook, to create a new Jupyter file (.ipynb). In contrast to AI Data Analyst which only supports text prompts, this mode is a fully functional Jupyter notebook environment. Therefore, you can add and execute cells with Python code, or include Markdown text cells. For additional information about using Jupyter notebooks, please read the <a href="https://jupyterlab.readthedocs.io/en/latest/user/index.html">JupyterLab documentation</a>. We are now going to test the <strong>AI Agent</strong> functionality, which is available in the MLJAR Studio sidebar (bottom left selection).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9ESFoZNC7fwlVZqq" /></figure><p>In the screenshot above, we see an example of the AI Agent being used. First of all, we entered the prompt asking to create a KDE plot for the petal_width variable, grouped by species. The AI agent returns a code cell that can be easily added to our notebook, by clicking <strong>Insert below active cell </strong>or<strong> Replace and Run</strong>. After executing the cell, the KDE plot we requested is shown. For additional information about the Python code cell, we can expand the AI agent <strong>thinking </strong>section, but this is optional.</p><h3>AutoLab Experiments</h3><p>AutoLab is a powerful feature helping us automate machine learning experiments, by using an AI agent. This lets you train and test multiple models on your chosen dataset, with minimal configuration. First of all, you need to click on <strong>AutoLab Experiments</strong> in the MLJAR Studio main menu. You are then presented with a configuration screen, where you need to specify the experiment settings, including dataset, scoring objective and various other details. Afterwards, you can automatically generate the <strong>AGENTS.md</strong> file, including all instructions and settings for the AI Agent, and finally run the experiment!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6GLqKVz_0A0Y-TAD" /></figure><p>When the experiment is completed, you can see all the results, including best run trial and associated metrics. Every trial generates a separate Jupyter notebook, so you can review them to understand the experiment in detail. This helps you easily identify the optimal machine learning model for a specific objective. You can obviously save and execute the notebook of your preference, thus having reproducible results and full control!</p><h3>Configuring the AI Provider</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/0*zFop8kkq6hoB3qAT" /></figure><p>As mentioned previously, MLJAR Studio supports various types of AI models and providers. The AI provider can be configured on the AI Agent settings. The available options are listed below:</p><ul><li><strong>MLJAR</strong>: This is the customized AI model provided by MLJAR. The free plan offers 50 prompts per month, but you can upgrade to pro / business and get significantly more. Click <a href="https://mljar.com/pricing/">here</a> to get additional information and plan comparisons.</li><li><strong>OpenAI</strong>: You can use OpenAI models by entering your API key and selecting the AI model of your preference.</li><li><strong>Ollama</strong>: This option lets you utilize local AI models running on your hardware. I suggest reading my <a href="https://medium.com/data-science-collective/a-brief-introduction-to-ollama-36f0ffd597c0">article about Ollama</a> for an extensive tutorial about installing and configuring the software.</li></ul><p>There is significant flexibility regarding AI models, as MLJAR provides numerous options to users. You can prioritize privacy by using Ollama, or choose an AI service provider for ease of use. This is also a preferable option, if your hardware is less powerful and doesn’t support running AI models!</p><p>Modern AI tools help data scientists save time and effort, without making any compromises regarding their work output. Furthermore, business users can also utilize AI to extract data insights and analysis, while not necessarily getting involved with Python code. MLJAR provides numerous tools offering value to diverse types of users in the data analytics space. I suggest downloading <a href="https://mljar.com/downloads/">MLJAR Studio</a> and start experimenting with the powerful features it provides. You can also check my introductory book about machine learning and time series forecasting, titled <a href="https://leanpub.com/pycaretbook/">Simplifying Machine Learning with PyCaret</a>. Finally, you can follow me on <a href="https://www.linkedin.com/in/giannis-tolios/">LinkedIn</a>, where I regularly post content about data science, statistics, data visualization and more!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7406163e3387" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Create AI Chatbots with Gradio]]></title>
            <link>https://medium.com/data-science-collective/create-ai-chatbots-with-gradio-04c281ba86af?source=rss-334ac6ae3ca3------2</link>
            <guid isPermaLink="false">https://medium.com/p/04c281ba86af</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[gradio]]></category>
            <category><![CDATA[ollama]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Giannis Tolios]]></dc:creator>
            <pubDate>Wed, 29 Apr 2026 13:32:04 GMT</pubDate>
            <atom:updated>2026-05-07T09:56:24.304Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to Develop a Web Interface for Large Language Models</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Z_z-t3GFNvdZvLh6" /><figcaption>Photo by <a href="https://unsplash.com/@andrewtneel?utm_source=medium&amp;utm_medium=referral">Andrew Neel</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Large language models are extensively used for tasks like text and code generation. Millions of people use online chatbots like ChatGPT, Claude and Gemini, accessed via their web interface. In my previous <a href="https://medium.com/data-science-collective/a-brief-introduction-to-ollama-36f0ffd597c0">article</a>, I explained how to set up Ollama and run language models locally on your computer. By following this approach, you avoid paying a monthly subscription to companies like OpenAI and Anthropic. Furthermore, you ensure data privacy and control over your LLM configuration. After experimenting with Ollama in the terminal, I suggesting creating a chat interface for your language model, a project that is easily accomplished with the Gradio Python library. This tutorial will utilize Gradio and Ollama to develop a simple user interface for large language models. Let’s begin working on the project!</p><h4>The Ollama Software</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*onhyed1uQbBDfDFfBHqEnA.png" /></figure><p><a href="https://ollama.com/">Ollama</a> was extensively covered in the previous <a href="https://medium.com/data-science-collective/a-brief-introduction-to-ollama-36f0ffd597c0">tutorial</a>, so please refer to that article for more information on software features and functionality. After installing Ollama on your computer, you need to download and run a model of your preference, by executing the following command. In this particular example we use Qwen 3.5 9B, the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts (MoE) models.</p><pre>ollama run qwen3.5</pre><p>After successfully downloading the model, you need to run Ollama in server mode, so it can be accessed via its application programming interface (API). This can be accomplished with the command below. To successfully serve the Qwen3.5 9B model, I suggest using an NVIDIA GPU with at least 8 GB of VRAM. Alternatively, you may consider installing a smaller language model like <a href="https://ollama.com/library/phi4-mini">Phi-4-mini</a>, and get optimal performance on less powerful systems.</p><pre>ollama serve</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dH-cfmg8WVVW5k1NndAapA.png" /></figure><p>As seen in the screenshot above, Ollama provides various information about server hardware configuration. This process should remain active in the background, so Gradio can seamlessly interface with the language model server. Therefore, you should make sure to keep the terminal window open. Furthermore, while the server keeps running, additional information will be displayed, such as time needed to complete every API request.</p><h4>The Gradio Library</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MeWSeYwIF3EKwSs0BhqdiQ.png" /></figure><p><a href="https://www.gradio.app/">Gradio</a> is an open-source web development library based on the Python programming language. Gradio can be used to easily develop machine learning applications, web dashboards and other data science projects. It is an extremely useful tool that can be used to develop prototypes, as well as fully fledged applications. The Gradio API provides numerous components that are suitable for machine learning applications, one of them being gr.ChatInterface(). This a high-level abstraction allowing you to develop a chatbot, by only writing a few lines of Python code. Furthermore, this Gradio component also supports multimodal chatbots, as well as extensive customization.</p><h4>The Chatbot Application</h4><p>We are now going to develop our chatbot application, based on Ollama and Gradio! I suggest creating an environment for this project by using a package manager like conda, uv or poetry, to keep track of all necessary libraries. This is only an optional suggestion, but generally considered a best practice in software development with Python!</p><pre>pip install gradio==6.5.0 ollama==0.6.1</pre><p>First of all, we need to install the necessary libraries on our computer, by executing this command on the terminal. Apart from Gradio, we also install the Ollama Python library to interface with the local server. We also specify the library versions, therefore ensuring maximum compatibility with the source code. Regardless, this project may also be compatible with other versions of both Ollama and Gradio, so feel free to experiment!</p><pre>import gradio as gr<br>import ollama<br><br># Safely extracts plain text from Gradio&#39;s message content format.<br>def extract_text(content):<br>    if isinstance(content, str):<br>        return content<br>    if isinstance(content, list):<br>        return &quot;&quot;.join(item.get(&quot;text&quot;, &quot;&quot;) for item in content if isinstance(item, dict))<br>    return str(content)</pre><p>We begin working on the Python source code, by importing the Gradio and Ollama libraries. Afterwards, we define extract_text a function that extracts plain text from Gradio messages by returning unmodified strings, or joining text values from dictionary items in lists. This function will be used to properly handle text input in the chatbot interface, as seen below.</p><pre># Chat with streaming responses.<br>def create_stream(message, history, temperature, system_prompt):<br>    messages = []<br><br>    # insert a system prompt if available<br>    if system_prompt:<br>        messages.append({<br>            &quot;role&quot;: &quot;system&quot;,<br>            &quot;content&quot;: system_prompt<br>        })<br><br>    for msg in history:<br>        # msg can be a dict or a ChatMessage object depending on Gradio version/configuration<br>        role = msg.get(&quot;role&quot;) if isinstance(msg, dict) else msg.role<br>        content = msg.get(&quot;content&quot;) if isinstance(msg, dict) else msg.content<br>        messages.append({<br>            &quot;role&quot;: role, <br>            &quot;content&quot;: extract_text(content)<br>        })<br>            <br>    messages.append({<br>        &#39;role&#39;: &#39;user&#39;, <br>        &#39;content&#39;: extract_text(message)<br>    })<br>    <br>    # Stream response from Ollama<br>    response_text = &quot;&quot;<br>    for chunk in ollama.chat(<br>        model=&#39;qwen3.5:4b&#39;,<br>        messages=messages,<br>        options={&#39;temperature&#39;: temperature},<br>        stream=True, think=False<br>    ):<br>        response_text += chunk[&#39;message&#39;][&#39;content&#39;]<br>        yield response_text</pre><p>We now create another function named create_stream that handles much of the chatbot application functionality. First of all, this function checks whether a system prompt has been entered by the user. This is a basic instruction that will be processed by the LLM before every other message. Afterwards, all messages of the current conversation are added in the context, so they are considered in every LLM reply. The ollama.chat component is used to access the Ollama model of our preference, as well as configure the necessary settings. In this case, we disable ‘thinking’ for faster performance, and enable streaming to get real-time incremental replies.</p><pre># Create Gradio chat UI component<br>demo = gr.ChatInterface(<br>    fn=create_stream,<br>    editable=True, save_history=True, stop_btn=False,<br>    title=&quot;🤖 Ollama Chatbot&quot;,<br>    description=&quot;Enter your prompt and watch the response in real-time!&quot;,<br>    additional_inputs=[<br>        gr.Slider(<br>            minimum=0.0, <br>            maximum=1.0, <br>            value=0.7, <br>            step=0.1, <br>            label=&quot;Temperature&quot;<br>        ),<br>        gr.Textbox(label=&quot;System prompt&quot;, placeholder=&quot;Enter a system prompt:&quot;)<br>    ]<br>)<br><br>if __name__ == &quot;__main__&quot;:<br>    demo.queue().launch() </pre><p>Τhe Gradio ChatInterface component is used to create our main user interface. First of all, we specify the chat interface around will wrap around the chat_stream function. Afterwards, we enable the boolean parameter save_history, so chat history is saved to the browser’s local storage, and previous conversations are displayed in a side panel. We also incude some additional inputs for the language model, i.e. temperature and system prompt. Temperature is a fundamental hyperparameter of language models, with values ranging from 0 to 1. Simply put, the temperature value controls the level of randomness and creativity in the LLM output.</p><p>For example, lower temperature values result in predictable factual replies, while higher values will lead to creative and occasionally incoherent answers. In practice, the temperature value of 0.7 is often used, thus balancing between creativity and factual answers. System prompt is the default instruction provided to the model before every user prompt. You can use system prompts to configure a specific language tone or any other preference, e.g. “always talk in rhyme”. Feel free to experiment with various system prompts and temperature values, to see what effect they have on the final result!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0fZynhXS_w38TjOnCV0emA.gif" /></figure><p>To start the chatbot web application, simply execute python app.py and visit the link displayed in your terminal. Before doing that, you need to ensure the Ollama server is running in another terminal window, with the following command: ollama serve. Alternatively, you can run the Windows Ollama application in the background, so it interfaces with the Python code. Make sure that Ollama runs either as a Windows application or the terminal exclusively, otherwise you are going to get an error. In the video above, you can see a brief demonstration of the app running. First, we enter a prompt asking our chatbot to describe the Eiffel tower. Afterwards, we enter a system prompt instructing the AI model to reply like a pirate, thus getting a hilarious result!</p><h3>Conclusion</h3><p>In this article, we developed a simple chatbot application with Gradio &amp; Ollama, and explored the advantages of executing LLM locally. If you want to learn more about large language models, I suggest starting with <a href="https://jalammar.github.io/illustrated-transformer/">The Illustrated Transformer</a>, an excellent article by Jay Alammar (also available as a book). In case you are comfortable with scientific papers, I encourage studying the landmark paper titled <a href="https://arxiv.org/abs/1706.03762">Attention is All You Need</a>, where the Transformer architecture was first introduced. You can also check my introductory book about machine learning and time series forecasting, titled <a href="https://leanpub.com/pycaretbook/">Simplifying Machine Learning with PyCaret</a> (<strong>now freely available</strong>). Finally, you can follow me on <a href="https://www.linkedin.com/in/giannis-tolios/">LinkedIn</a>, where I regularly post content about data science, statistics, data visualization and more!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=04c281ba86af" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/create-ai-chatbots-with-gradio-04c281ba86af">Create AI Chatbots with Gradio</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Brief Introduction to Ollama]]></title>
            <link>https://medium.com/data-science-collective/a-brief-introduction-to-ollama-36f0ffd597c0?source=rss-334ac6ae3ca3------2</link>
            <guid isPermaLink="false">https://medium.com/p/36f0ffd597c0</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[ollama]]></category>
            <category><![CDATA[chatgpt]]></category>
            <dc:creator><![CDATA[Giannis Tolios]]></dc:creator>
            <pubDate>Tue, 01 Jul 2025 11:35:43 GMT</pubDate>
            <atom:updated>2026-04-02T18:29:31.314Z</atom:updated>
            <content:encoded><![CDATA[<h4>Run Large Language Models on your PC</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Z2v7ss92SxMd6xK1" /><figcaption>Photo by <a href="https://unsplash.com/@cgower?utm_source=medium&amp;utm_medium=referral">Christopher Gower</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Generative AI is one of the hottest topics, with models like GPT-5.4, Claude and Gemini disrupting the industry and gaining extensive media coverage. Large language models (LLM) are typically accessed via a web page, or an application programming interface (API). Most casual users will prefer this type of experience, letting them enter their prompts in a chat interface. Regardless, there are numerous benefits in running LLM locally, without relying on external providers like OpenAI. Here are the most important advantages:</p><ul><li>No need for subscription or API credits, which can be costly</li><li>Ability to configure and customize the language model</li><li>Significantly increased privacy and security (important for sensitive data)</li><li>Independence from third parties like OpenAI or Anthropic</li></ul><p>Furthermore, practical experience with LLM is a signifcantly desirable skill in the modern data science job market. Therefore, experimenting with language models on your computer is an easy way to get acquainted with the technology and enhance your personal skill set. In the rest of this article, I am going to provide a brief tutorial about running language models on standard computer hardware, by utilizing the Ollama application. Let’s begin!</p><h3>The Ollama Software</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*j3t82JaiaudwlCKT.png" /></figure><p>Ollama is a user-friendly application that lets you download and run various LLM, including Llama, Qwen, DeepSeek-R1, Mistral, Phi and many more. Ollama has a command line interface, allowing for great customization and flexibility, that will benefit advanced users and professionals. Regardless, the basic functionality of Ollama is easily accessible, making it suitable for beginners as well! Ollama is available for all major operating systems, such as Windows, Linux and macOS. You can visit the official <a href="https://ollama.com/">Ollama website</a> and get more information, as well as extensive <a href="https://github.com/ollama/ollama/tree/main/docs">documentation</a> about the software features and functionality.</p><h3>Installing Ollama Locally</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*VlZfzJXOKmGJoo_Q.png" /></figure><p>I am going to provide a brief tutorial for the Windows version of Ollama, but the process is mostly similar for all operating systems. We begin by downloading the Ollama installation file from the official website, and executing it. This process may require some time to complete, while the CUDA library is being installed on your computer. Upon completion, the ollama command will be available on Windows terminal. By default, the Ollama server executes on Windows startup, but this can be disabled if preferable.</p><h3>The Qwen 3.5 Language Model</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*4uY0hnh_fuG1etNy.png" /></figure><p><a href="https://ollama.com/library/qwen3.5">Qwen 3.5</a> is a family of large language models developed by Alibaba Cloud. The models are open source and may be utilized for research of commercial purposes under the Apache 2.0 license. Qwen 3.5 makes a significant leap forward, as the 9B version manages to outperform larger models like GPT-OSS-120B, while running on a consumer PC. The models integrate breakthroughs in multimodal learning, architectural efficiency and reinforcement learning scale. Furthermore, the models have global linguistic coverage by supporting 201 languages and dialects, thus enabling worldwide deployment. Qwen 3.5 models are open source and can be easily accessed via Hugging Face, Ollama and other free platforms.</p><p>More specifically, we are going to focus on Qwen 3.5 9B, a relatively small model with 9 billion parameters that can be executed on modern computers. Personally, I am using a Dell laptop equipped with the NVIDIA RTX 3060 GPU (6GB VRAM). In case you have trouble running the model, I suggest trying the 2B/4B version instead. Furthermore, Qwen 3.5 9B shows remarkable performance on benchmarks for coding, logical reasoning, multilingualism, instruction following and others. The Qwen 3.5 9B model can be easily executed by entering the following command on the Windows terminal.</p><pre>ollama run qwen3.5:9b</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*5MUT0zl3kns9en2c.png" /></figure><p>As seen in the screenshot above, the model will first be downloaded, in case it isn’t available on your computer. After the download is completed, Ollama executes the model, hence letting you interact by entering prompts in the terminal. Alternatively, If you want to downlaod a model without executing it, try the ollama pull command. In the following example, I asked a question about the Mariana Trench, and the Qwen 3.5 model responded with an elaborate reply. Before asking the question, I disabled “thinking” to get quicker responses. I encourage you to try entering your own prompts, thus letting you evaluate the model’s abilities!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*-jqI0YWUj9nPsWsW.png" /></figure><h3>Other Models and Features</h3><p>As mentioned previously, Ollama supports a wide variety of models, including DeepSeek-R1, Llama, Mistral and various others. Most of the Ollama models come in various sizes, with the purpose of supporting various hardware configurations and use cases. Apart from small models like Qwen 3.5 9B, you can even download the full version of DeepSeek-R1 that has 671 billion paremeters! Notably, DeepSeek-R1 and similar models require enterprise-grade hardware to execute, so average users should prefer smaller ones. You can find more information about the Ollama model library, by visiting <a href="https://ollama.com/models">this link</a>. I suggest experimenting with different models to find the most suitable for your specific needs!</p><p>Furthermore, Ollama is supported by a vibrant community that develops various additional tools and software, hence extending its functionality. For example, <a href="https://github.com/open-webui/open-webui">Open WebUI</a> and <a href="https://github.com/ivanfioravanti/chatbot-ollama">Chatbot Ollama</a> enable you to create a chat user interface that is compatible with Ollama models. There are numerous other interesting applications and utilities available, such as codding assistants, text summarizers, browser extensions, Discord bots, and much more. You can visit the Ollama <a href="https://github.com/ollama/ollama/tree/main?tab=readme-ov-file#community-integrations">Github page</a> for an extensive list of community projects, and consider supporting them if possible!</p><h3>Conclusion</h3><p>In this article, we explored the main functionality of Ollama, as well as the advantages of executing LLM locally on your computer. If you want to learn more about large language models, I suggest starting with <a href="https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/">this excellent introduction</a> by Stephen Wolfram (also available as a book), which is suitable for educated laypeople with no background in machine learning. In case you are comfortable with scientific papers, I encourage studying the landmark paper titled <a href="https://arxiv.org/abs/1706.03762">Attention is All You Need</a>, where the Transformer architecture was first introduced. You can also check my introductory book about machine learning and time series forecasting, titled <a href="https://leanpub.com/pycaretbook/">Simplifying Machine Learning with PyCaret</a>. Finally, you can follow me on <a href="https://www.linkedin.com/in/giannis-tolios/">LinkedIn</a>, where I regularly post content about data science, statistics, data visualization and more!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36f0ffd597c0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/a-brief-introduction-to-ollama-36f0ffd597c0">A Brief Introduction to Ollama</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring Dimensionality Reduction with Python]]></title>
            <link>https://giannistolios.medium.com/exploring-dimensionality-reduction-with-python-a0d769a8783f?source=rss-334ac6ae3ca3------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0d769a8783f</guid>
            <category><![CDATA[dimensionality-reduction]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Giannis Tolios]]></dc:creator>
            <pubDate>Wed, 24 Jan 2024 12:01:38 GMT</pubDate>
            <atom:updated>2024-01-25T21:52:07.926Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Comparison of PCA, t-SNE and UMAP on the Palmer Dataset</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jhZOWMn92-SJqXUZ" /><figcaption>Photo by <a href="https://unsplash.com/@ricaros?utm_source=medium&amp;utm_medium=referral">Danial Igdery</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Modern datasets typically have numerous columns, hence making it difficult to process and visualize them. Additionally, machine learning models perform poorly on such datasets, due to the curse of dimensionality, a phenomenon that emerges in high-dimensional feature spaces [1]. Dimensionality reduction is a machine learning technique that aims to decrease the dataset columns, while retaining most of the valuable information. Furthermore, dimensionality reduction can improve the accuracy of machine learning models, while reducing the computational resources and time required to train them. There are several techniques for dimensionality reduction, including Principal Component Analysis (PCA), t-Distributed Stochastic Neighbor Embedding (t-SNE), and Uniform Manifold Approximation and Projection (UMAP). In this article, we are going to compare the various techniques, by utilizing the scikit-learn Python library, as well as the Palmer Penguins dataset. Let’s begin!</p><h3>The Palmer Penguins Dataset</h3><p>The Palmer Penguins dataset is widely used to teach various statistical concepts, while being freely available for everyone. Palmer Penguins is provided as an alternative to the established Iris dataset, which is considered obsolete and inadequate by some researchers [2]. The dataset classes are Chinstrap, Gentoo and Adelie, which are penguin species living in the Palmer Archipelago of Antarctica. Furthermore, the features include numeric and categorical variables, such as bill length, body mass, sex and island. Additionally, the data was collected by Dr. Kristen Gorman in the Palmer Station, which is part of the the United States Long-Term Ecological Research (US LTER) network. Real-world data are usually more complicated than Palmer Penguins, but working with toy datasets allows beginners to become acquainted with statistics and machine learning, and get a grasp of the associated techniques. In the rest of this article, we are going to explore the Palmer Penguins dataset, and apply various dimensionality reduction techniques on it.</p><pre>import pandas as pd<br>import matplotlib.pyplot as plt<br>import matplotlib as mpl<br>import seaborn as sns<br>from sklearn.preprocessing import StandardScaler<br>from sklearn.decomposition import PCA<br>from sklearn.manifold import TSNE<br>import umap<br><br>mpl.rcParams[&#39;figure.dpi&#39;] = 300<br>plt.style.use(&#39;seaborn-v0_8-muted&#39;)</pre><p>We begin by importing all the necessary Python libraries for our project, such as pandas, Matplotlib and Seaborn. Furthermore, we also import the PCA and t-SNE classes of scikit-learn, as well as the umap-learn library. The latter will be used in the dimensionality reduction examples of the following article sections. Finally, we set the figure DPI to 300 for high resolution plots, and change the Matplotlib style sheet, but those settings are strictly optional.</p><pre>df = sns.load_dataset(&#39;penguins&#39;).dropna(thresh = 6)<br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Re3nDS1kgeGvbEywshj6Jw.png" /><figcaption>Image by Author</figcaption></figure><p>As mentioned previously, this article will focus on the Palmer Penguins dataset, so we create a pandas dataframe based on it. This dataset is included with the Seaborn library, so we load it on a pandas dataframe, by using the load_dataset() function. Afterwards, we use the pandas head() function to display the first rows, thus getting a better understanding of the data. As we can see, there are four numeric features comprised of the physical characteristics of each penguin, such as bill length and bill depth. In addition, there are some categorical features as well, including species, sex and island.</p><pre>sns.pairplot(df, hue = &#39;species&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qXculPXXL3OKvxJf67oNWg.png" /><figcaption>Image by Author</figcaption></figure><p>Next, we are going to visualize the dataset, with the purpose of applying exploratory data analysis and extracting some useful insights. This will be accomplished by utilizing pairplot(), a Seaborn function that creates scatterplot matrices, as well as KDE plots. Furthermore, we also used the hue parameter to highlight the differences between each penguin species. The plot above contains a wealth of information, so we are only focusing on the most important aspects. First of all, there are numerous linear relationships between variable pairs, such as body mass and flipper length. Notably, there are some occurrences of Simpson&#39;s paradox[3], where two variables appear to have negative correlation, which is inversed when taking into account a third categorical variable. For example, bill depth and flipper length have a Pearson correlation coefficient of 𝑟 = −0.58, thus suggesting an inverse linear relationship between them. Regardless, we can easily verify that the variables are linearly related for each individual species, by visually inspecting the plot. Simpson&#39;s paradox highlights the fact that analysts should be diligent to avoid mistakes.</p><h3>Principal Component Analysis</h3><p>Principal Component Analysis (PCA) is a widely used technique for dimensionality reduction in machine learning and data analysis. PCA transforms the original high dimensional data into a lower dimensional space, while preserving most of the variance. This is accomplished by applying a linear orthogonal transformation to a different system of coordinates, where the first principal component explain most of the data variance. PCA helps us effectively reduce dimensionality, resulting in a dataset that is easier to visualize and interpret. In this section, we are going to apply PCA on the Penguins dataset, by utilizing the scikit-learn library.</p><pre>cols_num = [&#39;bill_length_mm&#39;, &#39;bill_depth_mm&#39;,<br>            &#39;flipper_length_mm&#39;, &#39;body_mass_g&#39;]<br>cols_cat = [&#39;species&#39;, &#39;island&#39;, &#39;sex&#39;]<br><br>scaler = StandardScaler()<br><br>X = df[cols_num]<br>X = scaler.fit_transform(X)<br><br>y = df[&#39;species&#39;].astype(&#39;category&#39;).cat.codes</pre><p>After loading the dataset to a pandas dataframe, we also create the X and y variables, i.e. the features and class label. In addition, we converted the class to the category data type, which includes both text and numeric representations of categorical variables. Furthermore, we use the StandardScaler() class, that helps us standardize the features by removing the mean and scaling to unit variance. It is necessary to apply scaling, because PCA assumes the data is normally distributed, while also being sensitive to the variance of each variable.</p><pre>pca = PCA(n_components = 2)<br>components = pca.fit_transform(X)<br>explained_var = pca.explained_variance_ratio_<br><br>print(f&#39;&#39;&#39;The explained variance of PC1 is {explained_var[0]:.2%}<br>The explained variance of PC2 is {explained_var[1]:.2%}<br>The total explained variance is {explained_var.sum():.2%}&#39;&#39;&#39;)</pre><pre>The explained variance of PC1 is 68.84%<br>The explained variance of PC2 is 19.31%<br>The total explained variance is 88.16%</pre><p>We create a PCA model with two principal components, by utilizing the PCA() class of the scikit-learn library. Afterwards, we display the explained_variance_ratio_ list, to check the explained variance of each principal component. As we can see, the two components explain almost 90% of the total variance, resulting in an accurate representation of the original dataset.</p><pre>df_ = df.join(pd.DataFrame(components, index = df.index,<br>                           columns = [&#39;pc_1&#39;, &#39;pc_2&#39;]))<br><br>fig, ax = plt.subplots(figsize = (8,5))<br>ax.grid(False)<br>ax.set_frame_on(False)<br><br>sns.scatterplot(data=df_, x=&#39;pc_1&#39;, y=&#39;pc_2&#39;, hue=&#39;species&#39;, ax =ax)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wv57kibT9QR33bUNa8u6aw.png" /><figcaption>Image by Author</figcaption></figure><p>We add the principal components to the original pandas dataframe, by using the join() function. After doing that, we plot them by using the scatterplot() Seaborn function. Evidently, the Gentoo class is clearly separated from the rest, while there is some overlap between Adelie and Chinstrap, indicating their similarities. Adding more principal components will help us retain the entire dataset variance, and possibly create a classifier machine learning model based on the data.</p><h3>t-Distributed Stochastic Neighbor Embedding</h3><p>t-Distributed Stochastic Neighbor Embedding (t-SNE) is a non-linear machine learning algorithm that is commonly used for dimensionality reduction and data visualization[5]. This algorithm is particularly useful for embedding high-dimensional data into fewer dimensions, with the most common option being a 2D space. t-SNE is an iterative algorithm that starts by computing the similarity between instances and then maps these similarities to a low-dimensional space, while preserving their structure. There are two hyperparameters that significantly affect t-SNE: the number of iterations and perplexity, which is the number of neighboring points that must be considered. When being compared to PCA, t-SNE is better at preserving relationships between data points and visualizing complex patterns, while PCA is more suitable for retaining the dataset variance.</p><pre>tsne = TSNE(n_components = 2, perplexity = 50)<br>components = tsne.fit_transform(X)<br><br>df_ = df.join(pd.DataFrame(components, index = df.index,<br>                           columns = [&#39;pc_1&#39;, &#39;pc_2&#39;]))<br><br>fig, ax = plt.subplots(figsize = (8,5))<br>ax.grid(False)<br>ax.set_frame_on(False)<br><br>sns.scatterplot(data=df_, x=&#39;pc_1&#39;, y=&#39;pc_2&#39;, hue=&#39;species&#39;, ax =ax)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z-5ZDuFG9K3Ucz2aLWt9ag.png" /><figcaption>Image by Author</figcaption></figure><p>We create a t-SNE model with two components, by utilizing the scikit-learn TSNE() class. After doing so, we plot the result with the scatterplot() Seaborn function. As we can see, the Palmer penguin species are separated in clearly defined clusters, with Gentoo being less similar to Adelie and Chinstrap. As mentioned previously, the perplexity and n_iter parameters can affect the t-SNE algorithm drastically, so it may be useful to experiment with different values and check the results.</p><h3>Uniform Manifold Approximation and Projection</h3><p>Uniform Manifold Approximation and Projection (UMAP) is a relatively new dimensionality reduction technique that has gained popularity in recent years, and is based in Riemannian geometry and algebraic topology [6]. UMAP is a non-linear method that seeks to preserve the data structure in a lower-dimensional space, making it similar to t-SNE. However, according to its creators, UMAP is designed to be more efficient and scalable, making it suitable for larger datasets. UMAP constructs a high-dimensional graph representation of the data and then projects it onto a low-dimensional space, by applying a series of optimization steps. This results in a visual representation of the data that is both accurate and interpretable. In comparison to PCA and t-SNE, UMAP offers a good balance of accuracy, efficiency, and scalability, making it a popular choice for dimensionality reduction in machine learning and data analysis.</p><pre>umap_model = umap.UMAP(n_components = 2, n_neighbors = 10,<br>                       random_state = 42)<br>components = umap_model.fit_transform(X)<br><br>df_ = df.join(pd.DataFrame(components, index = df.index,<br>                           columns = [&#39;pc_1&#39;, &#39;pc_2&#39;]))<br><br>fig, ax = plt.subplots(figsize = (8,5))<br>ax.grid(False)<br>ax.set_frame_on(False)<br><br>sns.scatterplot(data=df_, x=&#39;pc_1&#39;, y=&#39;pc_2&#39;, hue=&#39;species&#39;, ax =ax)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*POKlyGt1CZ7DpU7TtYQFzw.png" /><figcaption>Image by Author</figcaption></figure><p>The UMAP model is not included in scikit-learn, so we have to install the <a href="https://umap-learn.readthedocs.io/en/latest/">umap-learn library</a> separately, by using pip or conda. The UMAP() class helps us create a UMAP model that is fitted on our dataset, and is configured with various hyperparameters. In this particular case, we set n_neighbors to 15, and specified the random_state parameter to ensure reproducibility. By examining the scatter plot, we can see that UMAP captured the local structure successfully. Similarly to t-SNE, it can be useful to experiment with different hyperparameter values and compare the results.</p><h3>Conclusion</h3><p>In this article, we explored various dimensionality reduction techniques, by utilizing standard Python libraries, as well as the Palmer Penguins dataset. We have observed that t-SNE and UMAP have separated the Penguin species more successfully, hence making them suitable for data visualization. In general, those techniques are effective at capturing local structure, whereas PCA is better at retaining the global dataset structure. Considering this is an introductory article, I encourage you to look into newer dimensionality reduction techniques, such as PaCMAP or TriMAP [7]. I also suggest that you follow me on <a href="https://www.linkedin.com/in/giannis-tolios-0020b067/">LinkedIn</a>, where I regularly post data science content. You can also visit my <a href="https://giannis.io/">personal website</a> or check my book, titled <a href="https://leanpub.com/pycaretbook/">Simplifying Machine Learning with PyCaret</a>.</p><h3>References</h3><p>[1] Verleysen, Michel, and Damien François. “The curse of dimensionality in data mining and time series prediction.” <em>International work-conference on artificial neural networks</em>. Springer Berlin Heidelberg, 2005.</p><p>[2] Horst, Allison M., Alison Presmanes Hill, and Kristen B. Gorman. “Palmer Archipelago Penguins Data in the palmerpenguins R Package-An Alternative to Anderson’s Irises.” <em>R Journal</em> 14.1 (2022).</p><p>[3] Norton, H. James, and George Divine. “Simpson’s paradox… and how to avoid it.” <em>Significance</em> 12.4 (2015): 40–43.</p><p>[4] Shlens, Jonathon. “A tutorial on principal component analysis.” <em>arXiv preprint arXiv:1404.1100</em> (2014).</p><p>[5] Van der Maaten, Laurens, and Geoffrey Hinton. “Visualizing data using t-SNE.” <em>Journal of machine learning research</em> 9.11 (2008).</p><p>[6] McInnes, Leland, John Healy, and James Melville. “Umap: Uniform manifold approximation and projection for dimension reduction.” <em>arXiv preprint arXiv:1802.03426</em> (2018).</p><p>[7] Wang, Yingfan, et al. “Understanding how dimension reduction tools work: an empirical approach to deciphering t-SNE, UMAP, TriMAP, and PaCMAP for data visualization.” <em>The Journal of Machine Learning Research</em> 22.1 (2021): 9129–9201.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0d769a8783f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Statistical Hypothesis Testing with Python]]></title>
            <link>https://medium.com/data-science/statistical-hypothesis-testing-with-python-6a2f38c12486?source=rss-334ac6ae3ca3------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a2f38c12486</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[hypothesis-testing]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pingouin]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Giannis Tolios]]></dc:creator>
            <pubDate>Thu, 25 Aug 2022 14:22:18 GMT</pubDate>
            <atom:updated>2022-11-12T17:35:32.256Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using Pingouin to Examine a Case Study of ANOVA</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eeEsNJyP-fuvQWlb" /><figcaption>Photo by <a href="https://unsplash.com/@homajob?utm_source=medium&amp;utm_medium=referral">Scott Graham</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Hypothesis testing is an inferential statistics method that lets us determine population characteristics by analyzing a sample dataset. The mathematical tools necessary for hypothesis testing were formalized in the early 20th century by statisticians Ronald Fisher, Jerzy Neyman and Egon Pearson¹. Their influential work established concepts like the null hypothesis and p-values, with those tools becoming a fundamental part of modern scientific research. It should be noted that Fisher and Neyman-Pearson had engaged in academic rivalry, but eventually a combination of their differing approaches became established as the modern form of hypothesis testing². Apart from academic research, hypothesis testing is particularly useful to data scientists, as it lets them conduct A/B tests and other experiments. In this article, we are going to examine a case study of hypothesis testing on the seeds dataset, by using the Pingouin Python library.</p><h3>The Basic Steps of Hypothesis Testing</h3><p>The first step in hypothesis testing is coming up with the research hypothesis, a statement that can be tested statistically and involves the comparison of variables, e.g. drug X can lower blood pressure more than a placebo. After doing that, we specify the null hypothesis <strong>H₀ </strong>which states the effect is not present in the population. In contrast, the alternative hypothesis <strong>H₁ </strong>states the effect is actually present in the population. The next step is data collection, which can be accomplished with experiments, surveys, interviews, and other methods depending on the type of research. For example, A/B tests collect user feedback from different website versions to evaluate their performance. You may also use datasets that have been created for other purposes, a method known as secondary data analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yhE_afNPwE8mmTyzD0J4Jw.png" /><figcaption>Overview of Statistical Tests — <a href="https://philipppro.github.io/Statistical_tests_overview/">Image by Philipp Probst</a> (MIT License)</figcaption></figure><p>Afterwards, we need to decide which test is most suitable for our hypothesis. There are numerous available, including t-test, analysis of variance (ANOVA), Chi-squared, Kruskal-Wallis and many more. Choosing the appropriate test depends on a number of factors, including the type of variables and their distribution. Parametric tests like ANOVA are based on various assumptions, so we need to assess whether our dataset satisfies them. The table above provides an overview of all basic hypothesis tests, and can be a valuable tool when trying to find the most suitable. Keep in mind there are more hypothesis tests available, but this table covers the fundamental cases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nuOatoMGJuxDgUPCxVz3Og.png" /><figcaption>Type I and Type II Errors — Image by Author</figcaption></figure><p>Afterwards, we need to specify the significance level α (alpha), which is a threshold for rejecting the null hypothesis, typically set to 0.05. Therefore, a hypothesis test resulting to a p-value &gt; 0.05 means the null hypothesis can’t be rejected. In contrast, a p-value ≤ 0.05 allows us to reject the null hypothesis and accept the alternative hypothesis. More specifically, p-value is the probability of an observed effect occurring with the null hypothesis being true. Furthermore, the significance level α is equal to the probability of committing a type I error, i.e. rejecting the null hypothesis when it is true (false positive). In addition, β (beta) is the probability of committing a type II error, i.e. failing to reject the null hypothesis when it is false (false negative). Another important concept is statistical power, which is the probability of the null hypothesis being correctly rejected, and is defined as 1-β. After completing the previous steps, we execute the hypothesis test and state our conclusion, either rejecting the null hypothesis or not.</p><h3>The Pingouin Library</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0sbSyg0VzVsSc808fcZoyQ.png" /><figcaption>Pingouin logo — image by <a href="https://pingouin-stats.org/">https://pingouin-stats.org/</a></figcaption></figure><p><a href="https://pingouin-stats.org/index.html">Pingouin</a> is an open source Python library that supports a wide variety of hypothesis tests and statistical models³. The library includes numerous tests like ANOVA, t-test, Chi-squared, Kruskal-Wallis, Mann-Whitney, Wilcoxon signed-rank and others, hence covering a wide variety of cases. Furthermore, Pingouin lets you calculate the correlation coefficient between two variables, as well as create linear and logistic regression models. Pingouin is user-friendly yet powerful, as it returns extensive results for all tests, making it a great addition to the scientific Python ecosystem. In the rest of this article, we are going to use Pingouin to run hypothesis tests and interpret the provided results. Feel free to check the official <a href="https://pingouin-stats.org/api.html">library documentation</a> for extensive details about its functionality, and consider <a href="https://pingouin-stats.org/contributing.html">making a contribution</a> if you want.</p><h3>The Seeds Dataset</h3><p>The case study of this article is based on <a href="https://archive.ics.uci.edu/ml/datasets/seeds">the seeds dataset</a>, which is freely provided by the UCI Machine Learning Repository. This dataset contains information about samples of 3 wheat varieties, i.e. Kama, Rosa and Canadian⁴. Furthermore, the dataset includes various geometric properties for each wheat kernel, including area, perimeter, compactness, kernel length, kernel width and more. This dataset is widely used for machine learning tasks, such as classification and clustering, but we are utilizing it for hypothesis testing. More specifically, our goal is to assess the geometric differences between wheat varieties.</p><h3>A Case Study of ANOVA</h3><p>We are now going to examine a practical example of hypothesis testing, by utilizing the Pingouin library and the seeds dataset. Our research hypothesis is that compactness values are related to wheat variety, so we state the null and alternative hypothesis:</p><blockquote><strong><em>H₀</em></strong><em>: All wheat varieties have the same mean compactness.</em></blockquote><blockquote><strong><em>H₁</em></strong><em>: The wheat varieties have different mean compactness.</em></blockquote><p>After stating our hypotheses, we move on to the coding section that is based on Python 3.9 and Anaconda. In case you are interested, the full code of this article is available as a Jupyter notebook, so I encourage you to clone the associated <a href="https://github.com/derevirn/hypothesis-test">Github repository</a>.</p><pre>import pandas as pd<br>import pingouin as pg<br>import matplotlib.pyplot as plt<br>import matplotlib as mpl<br>import seaborn as sns<br>mpl.rcParams[&#39;figure.dpi&#39;] = 300<br>plt.style.use(&#39;seaborn-whitegrid&#39;)<br><br>df = pd.read_csv(&#39;data/seeds.csv&#39;)<br><br>dv = &#39;compactness&#39;<br>iv = &#39;variety&#39;<br>df.groupby(iv).mean()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nSfO2DDXMXXfcn6PfF_20w.png" /><figcaption>Image by Author</figcaption></figure><p>We begin by importing the necessary Python libraries and loading the seeds dataset to a pandas dataframe. Afterwards, we use the groupby() function to group dataset rows by wheat variety, and calculate the mean value for each column. As we can see, the mean values of most variables are significantly different for each variety. Compactness seems to be an exception, with all wheat varieties having similar mean values, so we are going to examine this variable in detail.</p><pre>df.boxplot(column = dv, by = iv, figsize = (8, 6), grid = False)<br><br>plt.box(False)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E44h3_oCMTjXPz8ZUYwTBw.png" /><figcaption>Image by Author</figcaption></figure><p>We use the boxplot() pandas function to create box plots for the compactness variable. Evidently, the Kama and Rosa varieties have similar quartiles, with median values that are nearly identical. In contrast, the Canadian variety appears to slightly differ from the rest, but we need to verify this with a hypothesis test. We want to compare the mean compactness value of all wheat varieties, i.e. there is a numeric dependent variable and an independent variable with three categories. Hence, the most suitable test for this case is one-way ANOVA.</p><pre>fig, ax = plt.subplots(figsize = (8, 6))<br>ax.grid(False)<br>ax.set_frame_on(False)<br><br>sns.kdeplot(data = df, x = dv, hue = iv,<br>            fill = False, ax = ax)<br>plt.show()<br><br>pg.normality(df, dv = dv, group = iv, method = &#39;shapiro&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ExtEM_Fzv7BrzLE30HhD_g.png" /><figcaption>Image by Author</figcaption></figure><p>As a parametric test, ANOVA is based on various assumptions about the dataset, one of them being that all group samples are normally distributed⁵. We can visually evaluate this, by using the kdeplot() Seaborn function to create a KDE plot for each wheat variety. Furthermore, we use the Pingouin normality() function to run a Shapiro-Wilk normality test⁶, which confirms that all samples are normally distributed. Keep in mind that Shapiro-Wilk is not particularly accurate on large samples, so tests like Jarque-Bera or Omnibus are preferable in those cases. Furthermore, research has indicated that ANOVA can be robust to violations of this assumption⁷, so slight deviations from the normal distribution are not a serious concern. Still, you should always evaluate whether your dataset satisfies test assumptions, and consider using a non-parametric test otherwise.</p><pre>fig, axes = plt.subplots(2, 2, figsize = (10, 8))<br>axes[1,1].set_axis_off()<br><br>categories = df[iv].unique()<br>for ax, cat in zip(axes.flatten(), categories):<br>    mask = df[iv] == cat<br>    sample = df.loc[mask, dv]<br>    pg.qqplot(sample, ax = ax)<br>    ax.set_title(f&quot;Q-Q Plot for category {cat}&quot;)<br>    ax.grid(False)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lCg8pqbXsk3Hh5RhOmAIMg.png" /><figcaption>Image by Author</figcaption></figure><p>Apart from tests like Shapiro-Wilk, creating a Q-Q plot is another way to evaluate sample normality. This is a scatter plot that lets you easily compare the quantiles of the normal and sample distribution. If the sample distribution is normal, all points will be near the line y = x. We can easily create a Q-Q plot for various theoretical distributions, by using the Pingouin qqplot() function. Furthermore, a best-fit line is also included in the plot, based on a linear regression model. Evidently, all sample quantiles are nearly identical to the normal distribution, further confirming the Shapiro-Wilk test and the KDE plot visual assessment.</p><pre>pg.homoscedasticity(df, dv = dv, group = iv, method = &#39;levene&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5LPbf2hm6BT_NsJxRZr3KA.png" /></figure><p>The ANOVA test is also based on the assumption that all samples have equal variance, a property known as homoscedasticity. The homoscedasticity() Pingouin function lets us easily evaluate this by using the Levene test, a typical approach in assessing the equality of variances⁸. According to the Levene test results, group samples don’t satisfy the assumption of homoscedasticity, i.e. they have unequal variances. We can overcome this problem by using the Welch ANOVA test, which is more robust to violations of this assumption, in comparison to the classic ANOVA⁹.</p><pre>df_anova = pg.welch_anova(df, dv = dv, between = iv)<br>df_anova</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IrfZdi3u8tjk2jVaW6fovA.png" /><figcaption>Image by Author</figcaption></figure><p>After executing the Welch ANOVA test, we examine the resulting dataframe to evaluate the results. First of all, the F-value indicates that variation between sample means is high, as compared to variation within the samples. The partial Eta-squared value represents effect size, thus helping us calculate statistical power. Furthermore, the p-value is almost equal to zero, making it particularly lower than the significance level (α = 0.05). Hence, we can reject the null hypothesis and accept the alternative hypothesis, i.e. the wheat varieties have different mean compactness values.</p><pre>pg.pairwise_gameshowell(df, dv = dv, between = iv)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yvxGJ3UUPav9vo5tYYzkmw.png" /><figcaption>Image by Author</figcaption></figure><p>After rejecting the null hypothesis of ANOVA, it is advisable to execute a post-hoc test to determine which group differences are statistically significant. We opted for the Games-Howell test, because it is robust to heterogeneity of variances, hence making it complementary to Welch ANOVA¹⁰. Evidently, the difference between Canadian and the other varieties is statistically significant. In contrast, the mean compactness values of Kama and Rosa varieties don’t have a significant difference.</p><h3>Conclusion</h3><p>In this article, I introduced the fundamental concepts of statistical hypothesis testing, by using the Pingouin library and seeds dataset. Hopefully, I helped you understand those concepts, as hypothesis testing is a challenging topic that leads to numerous misconceptions. Feel free to read <a href="https://openintro-ims.netlify.app/index.html">Introduction to Modern Statistics</a>, an excellent book that delves deeper into the topic, while being friendly to beginners. I also encourage you to share your thoughts in the comments, or follow me on <a href="https://www.linkedin.com/in/giannis-tolios-0020b067/">LinkedIn</a>, where I regularly post content about data science. You can also visit my <a href="https://giannis.io/">personal website</a> or check my latest book, titled <a href="https://leanpub.com/pycaretbook/">Simplifying Machine Learning with PyCaret</a>.</p><h3>References</h3><p>[1] Biau, David Jean, Brigitte M. Jolles, and Raphaël Porcher. “P value and the theory of hypothesis testing: an explanation for new researchers.” Clinical Orthopaedics and Related Research® 468.3 (2010): 885–892.</p><p>[2] Lenhard, Johannes. “Models and statistical inference: The controversy between Fisher and Neyman–Pearson.” The British journal for the philosophy of science (2020).</p><p>[3] Vallat, Raphael. “Pingouin: statistics in Python.” J. Open Source Softw. 3.31 (2018): 1026.</p><p>[4] Charytanowicz, Małgorzata, et al. “Complete gradient clustering algorithm for features analysis of x-ray images.” Information technologies in biomedicine (2010): 15–24.</p><p>[5] Scheffe, Henry. The analysis of variance. Vol. 72. John Wiley &amp; Sons, 1999.</p><p>[6] Shapiro, Samuel Sanford, and Martin B. Wilk. “An analysis of variance test for normality (complete samples).” Biometrika 52.3/4 (1965): 591–611.</p><p>[7] Schmider, Emanuel, et al. “Is it really robust? Reinvestigating the robustness of ANOVA against violations of the normal distribution assumption.” Methodology: European Journal of Research Methods for the Behavioral and Social Sciences 6.4 (2010): 147.</p><p>[8] Levene, Howard. “Robust tests for equality of variances.” Contributions to probability and statistics. Essays in honor of Harold Hotelling (1961): 279–292.</p><p>[9] Liu, Hangcheng. “Comparing Welch ANOVA, a Kruskal-Wallis test, and traditional ANOVA in case of heterogeneity of variance.” (2015).</p><p>[10] Games, Paul A., and John F. Howell. “Pairwise multiple comparison procedures with unequal n’s and/or variances: a Monte Carlo study.” Journal of Educational Statistics 1.2 (1976): 113–125.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a2f38c12486" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/statistical-hypothesis-testing-with-python-6a2f38c12486">Statistical Hypothesis Testing with Python</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Guide for Aspiring Data Scientists]]></title>
            <link>https://giannistolios.medium.com/a-guide-for-aspiring-data-scientists-4efceb70f1ef?source=rss-334ac6ae3ca3------2</link>
            <guid isPermaLink="false">https://medium.com/p/4efceb70f1ef</guid>
            <category><![CDATA[beginners-guide]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Giannis Tolios]]></dc:creator>
            <pubDate>Fri, 29 Apr 2022 14:29:25 GMT</pubDate>
            <atom:updated>2022-04-29T21:24:24.459Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to Successfully Enter the Data Science Field</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*d6yKzQx0STnBQd0a" /><figcaption>Photo by <a href="https://unsplash.com/@mjessier?utm_source=medium&amp;utm_medium=referral">Myriam Jessier</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I decided to publish this article for a number of reasons. First of all, I’ve been working as a freelance data scientist for some years, hence acquiring valuable experience that could prove useful for those wishing to enter the field. Second, numerous people have contacted me in the past year, asking for guidance and resources about data science. Unfortunately, I don’t have time to give personal advice to everyone, so I hope this will be a helpful guide for beginners. In this article, I included a number of steps you can follow to become proficient in data science, by acquiring theoretical knowledge, as well as practical skills. Many of the books and courses that are mentioned here are freely available, as I believe that education should be accessible to all.</p><p>Data science combines software engineering, statistics and machine learning, with the goal of extracting business insights from raw data. It can be a truly rewarding field of work, offering intellectual stimulation and significant monetary rewards, while being at the forefront of technology. Regardless, this field is also full of challenges and difficulties, that can be discouraging to beginners. Having a thoughtful plan can give you a significant advantage as an aspiring data scientist, so I am providing the following detailed steps, based on my personal experience. Furthermore, I sincerely hope this guide will be helpful to newcomers who are interested in data science, and wish to start their journey in the field!</p><h3>Make Sure Data Science Suits you</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DDL-m-Iic2S9tkgb" /><figcaption>Photo by <a href="https://unsplash.com/@iam_muzan?utm_source=medium&amp;utm_medium=referral">Yosep Surahman</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Machine learning has advanced impressively in the past years, leading to massive media exposure. You can constantly read news about the latest achievements of machine learning in scientific research, medicine, business etc. This created significant hype around the technology, so countless people became interested in the field. For example, more than 4 million people have enrolled in the famous <a href="https://www.coursera.org/learn/machine-learning">Machine Learning course by Andrew Ng on Coursera</a>, indicating the huge interest for this subject. Subsequently, a huge amount of competition emerged in the job market, especially for junior positions.</p><p>Given those facts, you should understand that commitment and hard work are necessary to succeed as a data scientist. Before committing though, make sure that data science is a good fit for you. This type of work requires quantitative thinking, analytical skills, and constant learning. Data scientists need to enhance their skills by becoming acquainted with new software tools and libraries, acquire domain knowledge, and stay up to speed with the latest research. Not everyone has the same talents and aptitude, so I suggest that you take an introductory data science course to see if you actually enjoy it, before deciding to pursue this career.</p><h3>Learn Data Science the Right Way</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qSRaaWeEH1V6WsFg" /><figcaption>Photo by <a href="https://unsplash.com/@windows?utm_source=medium&amp;utm_medium=referral">Windows</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Due to the growing popularity of data science and machine learning, there are numerous online courses, books and degrees available for those interested. Having a relevant degree is not absolutely necessary, but my postgraduate studies in computer science have certainly helped me. Any kind of quantitative degree will benefit you, as familiarity with mathematics and computer programming is essential. In case you need to freshen your math knowledge, I suggest the excellent <a href="https://mml-book.github.io/">Mathematics for Machine Learning book</a>, which is freely available. Furthermore, <a href="https://openintro-ims.netlify.app/">Introduction to Modern Statistics</a> is a free textbook that will teach you the fundamentals of statistics, a necessary skill for all data scientists.</p><p>One of the most popular machine learning books is <a href="https://www.statlearning.com/">An Introduction to Statistical Learning</a> by Gareth James et al. It’s a great introduction to all the theoretical concepts, including regression, classification, support vector machines, clustering and decision trees. The book authors have recently published the second edition, adding chapters about deep learning, survival analysis and other topics. They’ve also released an <a href="https://web.stanford.edu/~hastie/ISLRv2_website.pdf">ebook version</a> of the book as a free download, so make sure to check it out. Furthermore, <a href="https://www.packtpub.com/product/machine-learning-with-pytorch-and-scikit-learn/9781801819312">Machine Learning with PyTorch and Scikit-Learn</a> by Sebastian Raschka is another amazing book, covering all the fundamental topics in machine learning and deep learning. Finally, <a href="https://leanpub.com/pycaretbook/">Simplifying Machine Learning with PyCaret</a> is a book that I published last year, offering a beginner-friendly introduction to machine learning, based on the PyCaret library.</p><p>Online courses provided by <a href="https://www.coursera.org/browse/data-science">Coursera</a>, <a href="https://www.edx.org/course/subject/data-science">edX</a> and <a href="https://www.dataquest.io/">Dataquest</a> can also be great starting points for beginners, but you shouldn’t focus on course certifications, as your experience is what really matters. That being said, Stanford University provides free access to the <a href="http://cs229.stanford.edu/syllabus-summer2020.html">CS229: Machine Learning course</a>. The syllabus covers linear algebra, statistics, supervised learning, unsupervised learning, deep learning and other topics. You can watch the lectures on YouTube and download the class notes if you want. It is truly fascinating that a top-ranking university provides that kind of educational content, free for everyone to enjoy!</p><h3>Create a Strong Portfolio</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Hgkqu16GB5Bsv-_Y" /><figcaption>Photo by <a href="https://unsplash.com/@clarktibbs?utm_source=medium&amp;utm_medium=referral">Clark Tibbs</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The best way to gain experience before finding your first data science job, is to work on personal projects. Start by finding a topic that interests you, then download a dataset from <a href="https://www.kaggle.com/">Kaggle</a> or <a href="https://datasetsearch.research.google.com/">Google Dataset Search</a>, and work on your project! It is even better to create your own dataset by scraping a website, as this is necessary in some real-world projects. All that may sound daunting to a beginner, but there’s no other way to become skillful and valuable to employers. Working through the examples of online courses and books is simply not enough, as you need to stand out from the rest.</p><p>After working on your personal projects, sharing them with others is the next logical step. <a href="https://github.com/">Github</a> is a free service that lets you create repositories for your projects, that can either be public or private. Creating a strong portfolio of data science projects on Github is the best way to promote yourself to employers, as it showcases your practical skills and accomplishments. Here’s a list with suggested data science projects:</p><p>• A data analysis and visualization project.</p><p>• A project focusing on statistical testing (t-test, ANOVA &amp; chi-square).</p><p>• A project focusing on basic ML tasks, such as classification and regression.</p><p>• A time series analysis and forecasting project.</p><p>• A natural language processing project.</p><p>You should keep in mind it’s not necessary to create all those projects to have a strong portfolio. Furthermore, you should avoid using toy datasets like Iris/Titanic Passengers because they have been analyzed by millions of people and are considered trivial. Instead of that, try finding new datasets in a domain that interests you. This will motivate you to focus on the project and get a better result! In addition, make sure to document each project and write an executive summary or blog post with your insights. Communication is a key skill for data scientists, so don’t underestimate it.</p><h3>Apply for Jobs</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iRwfwVQjia-EQtgC" /><figcaption>Photo by <a href="https://unsplash.com/@eprouzet?utm_source=medium&amp;utm_medium=referral">Eric Prouzet</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>After acquiring the fundamental data science skills and creating a portfolio with your personal projects, the next step is applying for a job. Employees may feel insecure when looking for their first data science job, due to lack of experience and other factors. Most people will struggle with such feelings at some point in life, so you shouldn’t be discouraged by them. Instead, you can deal with insecurity in a productive way, by becoming motivated to improve your skills and knowledge.</p><p><a href="https://www.linkedin.com/">LinkedIn</a> is a great website to look for data science job listings, as most companies use it nowadays. Furthermore, <a href="https://www.glassdoor.com/">Glassdoor</a> is another great resource, offering numerous listings related to data science, machine learning and other associated fields. Finally, you may want to pursue a career as a freelancer, by using platforms like <a href="https://www.upwork.com/">Upwork</a>, <a href="https://fiverr.com/">Fiverr</a> or <a href="https://www.freelancer.com/">Freelancer</a>. Personally, I am quite familiar with the advantages and disadvantages of freelancing, so I will consider writing an article focusing on the topic.</p><h3>Connect with the Right People</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2RiGNye00MgQQelE" /><figcaption>Photo by <a href="https://unsplash.com/@brookecagle?utm_source=medium&amp;utm_medium=referral">Brooke Cagle</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Nobody can achieve truly great things by themselves, so connecting with other like-minded professionals should be a priority to everyone. <a href="https://www.linkedin.com/">LinkedIn</a> is obviously a great way to accomplish that, either by contacting people on your recommendation feed, or joining some groups. Messaging strangers may feel awkward at times, but this shouldn’t discourage you from communicating with people who are willing to collaborate. You should simply be respectful and avoid becoming burdensome in case they are busy, or simply not interested.</p><p>Another great way of connecting with people are hackathons. In case you’re not familiar with the concept, hackathons are events where teams of participants attempt to develop a software prototype in a few days, typically a weekend. Afterwards, the hackathon judges decide which team delivered the best result, and award the members with a monetary prize, or the opportunity to start a company based on their idea. Hackathons are an excellent way to meet people who share your passion, and even join a new startup company if you are lucky.</p><h3>Conclusion</h3><p>I hope this article helps beginners avoid common mistakes, and evolve as professionals in the field of data science. It is obviously not a definitive guide, but rather my personal advice to newcomers, so I welcome any constructive criticism and differing opinions. Do you agree with my tips for aspiring data scientists? Would you add any other helpful advice? I encourage you to share your thoughts in the comments, or follow me on <a href="https://www.linkedin.com/in/giannis-tolios-0020b067/">LinkedIn</a> where I regularly post content about data science, climate change and other topics!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4efceb70f1ef" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Forecasting Atmospheric CO2 with Python]]></title>
            <link>https://medium.com/data-science/forecasting-atmospheric-co2-concentration-with-python-c4a99e4cf142?source=rss-334ac6ae3ca3------2</link>
            <guid isPermaLink="false">https://medium.com/p/c4a99e4cf142</guid>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[climate-change]]></category>
            <category><![CDATA[data-for-change]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Giannis Tolios]]></dc:creator>
            <pubDate>Tue, 22 Mar 2022 15:20:41 GMT</pubDate>
            <atom:updated>2026-06-11T12:48:48.110Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to Create Time Series Forecasting Models with Darts</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gSOMLehEPA1NSmbR" /><figcaption>Photo by <a href="https://unsplash.com/@andreasfelske?utm_source=medium&amp;utm_medium=referral">Andreas Felske</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Climate change is undoubtedly one of the greatest challenges that humanity is facing, with experts considering it an existential threat for our species. In 2021, unprecedented heatwaves and great wildfires were recorded worldwide, while floods brought destruction in Europe and Asia. According to the Intergovernmental Panel for Climate Change (IPCC) Sixth Assessment Report, the increasing frequency of extreme weather events is linked to climate change, and drastic measures need to be imposed globally to deal with this issue¹. Otherwise, millions of people will be affected, with their quality of life being significantly decreased in the following years.</p><p>Greenhouse gases (GHG) such as carbon dioxide (CO2) and methane (CH4) trap heat in the atmosphere, hence keeping our planet warm and friendly to biological species. Regardless, human activities such as burning fossil fuels, lead to enormous quantities of GHG being emitted, thus excessively increasing the mean global temperature of Earth². Therefore, transitioning towards a sustainable global economy is imperative, so we can mitigate climate change and secure the prosperity of our species. In this article, we are going to apply time series forecasting on atmospheric CO2 concentration data, hence getting the opportunity to explore the intersection of machine learning and climate change.</p><h3>The Darts Library</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ChznHDcIXrvVhpOKe66nnw.png" /><figcaption>Darts Logo — Image by Unit8</figcaption></figure><p>The <a href="https://unit8co.github.io/darts/">darts library</a> developers aim to simplify time series analysis and forecasting with Python. Darts supports a variety of forecasting approaches, ranging from classical statistical models like ARIMA and exponential smoothing, as well as novel methods based on machine learning and deep learning. Furthermore, darts includes various functions that let us understand the statistical properties of time series, as well as evaluate the accuracy of forecasting models. If you want to learn more about darts, you can read <a href="https://medium.com/unit8-machine-learning-publication/darts-time-series-made-easy-in-python-5ac2947a8878">this detailed introduction</a> to the library, or refer to the <a href="https://unit8co.github.io/darts/generated_api/darts.html">official API documentation</a>. The darts library will let us create various forecasting models based on the Mauna Loa CO2 dataset. Furthermore, we are going to compare the accuracy of those models and use the best one to forecast the atmospheric CO2 concentration values for 2022.</p><h3>The Mauna Loa CO2 Dataset</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vfPaYL7eJ2vSj0gfDm1aSA.jpeg" /><figcaption>The Mauna Loa Observatory — Photo by NOAA</figcaption></figure><p>The Mauna Lua volcano is home to the eponymous <a href="https://gml.noaa.gov/obop/mlo/">Mauna Loa Observatory (MLO)</a>, a research facility that has been monitoring the atmosphere since 1950, with its remote location providing ideal conditions to record climate data. In 1958, Charles David Keeling established the CO2 monitoring program of the MLO, and started recording scientific evidence for the rapidly increasing CO2 concentration of the atmosphere³. The Mauna Loa CO2 dataset was downloaded from the Scripps Institution of Oceanography, and includes monthly atmospheric CO2 concentration values in parts per million (ppm), from 1958 to 2021⁴. Furthermore, the dataset also contains seasonally adjusted and smoothed versions of the data, but our analysis will exclusively focus on the standard time series.</p><h3>Time Series Analysis</h3><p>In this section, we are going to extract some insights about the dataset and its statistical properties. This will be accomplished by using various types of plots and other time series analysis techniques.</p><pre>import pandas as pd<br>import matplotlib.pyplot as plt<br>import matplotlib as mpl<br>from statsmodels.tsa.seasonal import seasonal_decompose<br>from statsmodels.graphics.tsaplots import plot_acf, plot_pacf<br>from darts import TimeSeries<br>from darts.models import *<br>from darts.metrics import *<br>from darts.dataprocessing.transformers import Scaler<br>import logging<br><br>mpl.rcParams[&#39;figure.dpi&#39;] = 300<br>logging.disable(logging.CRITICAL)</pre><p>We begin by importing the Python libraries that are necessary for our project, including pandas, Matplotlib, as well as various functions from the statsmodels and darts libraries. After doing that, we set the Matplotlib figure DPI to 300, so we get high-resolution plots for this article, but it is optional.</p><pre>df = pd.read_csv(&#39;data/monthly_in_situ_co2_mlo.csv&#39;,<br>                  comment = &#39;&quot;&#39;, header = [0,1,2], na_values = &#39;-99.99&#39;)<br><br>cols = [&#39; &#39;.join(col).replace(&#39; &#39;, &#39;&#39;) for col in df.columns.values]<br>df.set_axis(cols, axis = 1, inplace = True)<br><br># Converting Excel date format to datetime<br># and setting as dataframe index<br>df[&#39;datetime&#39;] = pd.to_datetime(df[&#39;DateExcel&#39;], origin = &#39;1899-12-30&#39;, unit = &#39;D&#39;)<br><br>df.set_index(&#39;datetime&#39;, inplace = True)<br><br>df = df[[&#39;CO2filled[ppm]&#39;]]<br>df.rename(columns = {&#39;CO2filled[ppm]&#39;: &#39;CO2&#39;}, inplace = True)<br>df.dropna(inplace = True)<br>df = df.resample(&#39;M&#39;).sum()</pre><p>After importing the libraries, we continue by loading the dataset to a pandas dataframe. We then apply some data preprocessing techniques to it, such as removing unnecessary columns, setting a monthly datetime index and dropping the null values.</p><pre>df.plot(figsize=(8,5))<br>plt.title(&#39;Monthly CO2 Concentration (ppm)&#39;)<br><br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*veQOZXrASHG4Mh-hrSMSUQ.png" /><figcaption>Image by Author</figcaption></figure><p>After cleaning the dataset, we use the plot() pandas function to create a simple line plot of the time series. Evidently, there’s a clear upward trend, highlighting the fact that atmospheric CO2 concentration has been rapidly increasing in the past decades. Furthermore, there’s also seasonality due to the planet’s natural carbon cycle, as plants capture and release CO2 in different seasons of the year⁵. Specifically, when plants start growing in Spring, they remove CO2 from the atmosphere by photosynthesizing. In contrast, when deciduous trees lose their leaves in Autumn, CO2 is released due to respiration.</p><pre>mpl.rcParams[&#39;figure.figsize&#39;] = (8, 6)<br><br>result = seasonal_decompose(df)<br>result.plot()<br><br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KVFJQG1GWxxFTOPsqeU3mA.png" /><figcaption>Image by Author</figcaption></figure><p>We use the seasonal_decompose() statsmodels function to apply decomposition on the time series, i.e. extract the trend, seasonal and residuals components. After doing that, we plot those components and get a better understanding of the trend and seasonality that was mentioned before. In this particular case, we could already identify the components by visually inspecting the line plot, but seasonal decomposition is significantly helpful in more complex cases.</p><pre>fig, ax = plt.subplots(figsize = (8,5))<br><br>plot_acf(df, ax = ax)<br><br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YaWA4mFD_H8IvNG3_X6Cvg.png" /><figcaption>Image by Author</figcaption></figure><p>We use the plot_acf() statsmodels function to plot the autocorrelation function (ACF) of the time series, i.e. the linear relationship between its lagged values. Because of the time series trend, we can see that autocorrelation is high for small lags and starts to gradually decrease after lag 5. The ACF should also highlight the time series seasonal component, but it is indiscernible in this case.</p><h3>Time Series Forecasting</h3><p>In this section, we are going to train various forecasting models on the CO2 dataset and compare their performance. After doing that, we’ll pick the most accurate model and create a forecast for 2022 based on it.</p><pre>series = TimeSeries.from_dataframe(df)<br><br>start = pd.Timestamp(&#39;123115&#39;)<br>df_metrics = pd.DataFrame()<br><br>def plot_backtest(series, forecast, model_name):<br>    idx = -144<br>    series[idx:].plot(label=&#39;Actual Values&#39;)<br>    forecast[idx:].plot(label= &#39;Forecast&#39;)<br>    plt.title(model_name)<br>    plt.show()<br>    <br>def print_metrics(series, forecast, model_name):<br>    mae_ = mae(series, forecast)<br>    rmse_ = rmse(series, forecast)<br>    mape_ = mape(series, forecast)<br>    smape_ = smape(series, forecast)<br>    r2_score_ = r2_score(series, forecast)<br>    <br>    dict_ = {&#39;MAE&#39;: mae_, &#39;RMSE&#39;: rmse_,<br>             &#39;MAPE&#39;: mape_, &#39;SMAPE&#39;: smape_, <br>             &#39;R2&#39;: r2_score_}<br>    <br>    df = pd.DataFrame(dict_, index = [model_name])<br>    <br>    return(df.round(decimals = 2))  </pre><p>We begin by loading the pandas dataframe to a TimeSeries object, as it is required by the Darts library. After doing that, we create the plot_backtest() and print_metrics() utility functions that will let us plot the forecasts and display various model metrics, including MAE, RMSE, MAPE, SMAPE and R².</p><h4>Creating a Naive Forecasting Model</h4><p>Setting a baseline accuracy is standard practice, so we are going to do that by creating a naive model. This will help us evaluate the performance of more complex models, that should theoretically have higher accuracy compared to the baseline. In this case, we are going to create a naive seasonal model that always predicts the value of K steps ago, with K being equal to the seasonal period.</p><pre>model = NaiveSeasonal(K = 12)<br>model_name = &#39;Naive Seasonal&#39;<br><br>plt.figure(figsize = (8, 5))<br><br>forecast = model.historical_forecasts(series, start=start, forecast_horizon=12, verbose=True)<br>plot_backtest(series, forecast, model_name)<br>df_naive = print_metrics(series, forecast, model_name)<br>df_metrics = df_metrics.append(df_naive)<br><br>plt.show()<br>df_naive</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RDmZlJZvqXBXoIlz5DRWjQ.png" /><figcaption>Image by Author</figcaption></figure><p>To test the naive seasonal model, we use the historical_forecasts() function with a forecast horizon of 12 months. This function consecutively trains the model on an expanding window, and keeps the last value of each forecast by default. This technique is known as backtesting or time series cross-validation, and it is more sophisticated compared to the typical train/test method. After generating the forecast, we display the plot and metrics, by using the utility functions. As we can see, the naive seasonal model performed reasonably well, with a SMAPE value of 0.59%.</p><h4>Creating an Exponential Smoothing Forecasting Model</h4><p>Now that we have set a baseline accuracy, we are going to create a forecasting model based on Holt-Winter’s exponential smoothing, a classical approach that has been successfully used since the 1960s⁶.</p><pre>model = ExponentialSmoothing(seasonal_periods = 12)<br>model_name = &#39;Exponential Smoothing&#39;<br><br>plt.figure(figsize = (8, 5))<br><br>forecast = model.historical_forecasts(series, start=start, forecast_horizon=12, verbose=True)<br>plot_backtest(series, forecast, model_name)<br>df_exp = print_metrics(series, forecast, model_name)<br>df_metrics = df_metrics.append(df_exp)<br><br>plt.show()<br>df_exp</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4u8vdPjjIGT-ljVeKcUrYw.png" /><figcaption>Image by Author</figcaption></figure><p>As previously, we use the historical_forecasts() function, as well as the utility functions to evaluate model performance. Evidently, the exponential smoothing model significantly outperforms the naive model, with a SMAPE value of 0.1%. We can also visually confirm this, by observing that the forecasted values are almost identical to the plotted dataset values.</p><h4>Creating a Linear Regression Forecasting Model</h4><p>In the past years, machine learning models have been widely used in time series forecasting, as an alternative to classical approaches. By simply adding the lagged values as features to the dataset, we can transform time series forecasting into a regression task⁷. Therefore, we are able to use any <a href="https://scikit-learn.org/stable/">scikit-learn</a> regression model or other libraries that have a compatible API, including XGBoost and LightGBM. In this case, we are going to create a linear regression model based on the scikit-learn library.</p><pre>model = LinearRegressionModel(lags = 12)<br>model_name = &#39;Linear Regression&#39;<br><br>plt.figure(figsize = (8, 5))<br><br>forecast = model.historical_forecasts(series, start=start, forecast_horizon=12, verbose=True)<br>plot_backtest(series, forecast, model_name)<br>df_lr = print_metrics(series, forecast, model_name)<br>df_metrics = df_metrics.append(df_lr)<br><br>plt.show()<br>df_lr</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L7FzWYPQzsmlPH8l_yYw-Q.png" /><figcaption>Image by Author</figcaption></figure><p>By using historical_forecasts() and the utility functions, we test the linear regression model and subsequently display the results. As we can see, the model has excellent performance, with a SMAPE value of 0.11%.</p><h4>Creating a Temporal Convolutional Network Forecasting Model</h4><p>In recent years, deep learning has become popular in time series forecasting, with recurrent neural networks being the standard choice in research, as well as practical applications. Regardless, the temporal convolutional network is an alternative architecture that offers promising results⁸, so we are going to test its performance.</p><pre>model = TCNModel(<br>    input_chunk_length=24,<br>    output_chunk_length=12,<br>    n_epochs=100,<br>    dropout=0.1,<br>    dilation_base=3,<br>    weight_norm=True,<br>    kernel_size=5,<br>    num_filters=3,<br>    random_state=0,<br>)<br><br>model_name = &#39;TCN&#39;<br><br>plt.figure(figsize = (8, 5))<br><br>scaler = Scaler()<br>scaled_series = scaler.fit_transform(series)<br>forecast = model.historical_forecasts(scaled_series, start=start,<br>                                      forecast_horizon=12, verbose=True)<br>plot_backtest(series, scaler.inverse_transform(forecast), model_name)<br>df_dl = print_metrics(series, scaler.inverse_transform(forecast), model_name)<br>df_metrics = df_metrics.append(df_dl)<br><br>plt.show()<br>df_dl</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*od5DT390R4NydoGEcCbNlQ.png" /><figcaption>Image by Author</figcaption></figure><p>As previously, we use historical_forecasts() and the utility functions to test the temporal convolutional network model and display the results. Furthermore, we also normalize the time series, by using the Scaler() class. Evidently, the TCN model has significantly better performance compared to the baseline, with a SMAPE value of 0.11%.</p><h4>Creating a Forecast</h4><p>As expected, all models outperformed the naive seasonal model, by offering excellent accuracy. Regardless, we can see that exponential smoothing provided the best performance, hence our forecast for the atmospheric CO2 concentration of 2022 will be based on it.</p><pre>model = ExponentialSmoothing(seasonal_periods = 12)<br>model_name = &#39;Exponential Smoothing&#39;<br><br>model.fit(series)<br>forecast = model.predict(12)<br><br>plot_backtest(series, forecast, model_name)<br>print(forecast.pd_dataframe())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SX-IyHqTAN4Mec2Jzl9SyQ.png" /><figcaption>Image by Author</figcaption></figure><p>We use the fit() function to fit the exponential smoothing model on the entire dataset, and afterwards display the results. As we can visually observe, the time series components have been successfully identified by the model. Furthermore, we can compare our result to the <a href="https://www.metoffice.gov.uk/research/climate/seasonal-to-decadal/long-range/forecasts/co2-forecast-for-2022">official CO2 forecast</a> of the United Kingdom’s Meteorological Office (Met Office) for 2022. As expected, the two forecasts have nearly identical values, hence validating the accuracy of the exponential smoothing model. It may seem surprising that exponential smoothing provided the best results, instead of the machine learning and deep learning models. Regardless, we should keep in mind that newer models aren’t necessarily better in every case, compared to the classical methods. Furthermore, machine learning models can be optimized by applying hyperparameter tuning, a technique that is time-consuming and complicated, but can offer significant improvements.</p><h3>Conclusion</h3><p>In this article, we have explored one of the ways that data science can be utilized to tackle climate change, a thriving and diverse field of research that constantly grows⁹, so I encourage you to learn more about it. Sadly, we also discovered that atmospheric CO2 concentration is expected to increase this year, highlighting the fact that emissions must be reduced, to ensure the prosperity of our species. Finally, the code and data of this article are available at this <a href="https://github.com/derevirn/co2-forecasting">Github repository</a>, so feel free to clone it. I also encourage you to share your thoughts in the comments, or follow me on <a href="https://www.linkedin.com/in/giannis-tolios-0020b067/">LinkedIn</a> where I regularly post content about data science, climate change and other topics. You can also visit my <a href="https://giannis.io/">personal website</a> or check my latest book, titled <a href="https://leanpub.com/pycaretbook/">Simplifying Machine Learning with PyCaret</a>.</p><h3>References</h3><p>[1] Zhai, P., et al. “IPCC 2021: Climate Change 2021: The Physical Science Basis. Contribution of Working Group I to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change.” (2021).</p><p>[2] Houghton, John. “Global warming.” Reports on progress in physics 68.6 (2005): 1343.</p><p>[3] Harris, Daniel C. “Charles David Keeling and the story of atmospheric CO2 measurements.” Analytical chemistry 82.19 (2010): 7865–7870.</p><p>[4] Keeling, Charles D., et al. “Exchanges of atmospheric CO2 and 13CO2 with the terrestrial biosphere and oceans from 1978 to 2000. I. Global aspects.” (2001).</p><p>[5] Keeling, Charles D. “The concentration and isotopic abundances of atmospheric carbon dioxide in rural areas.” Geochimica et cosmochimica acta 13.4 (1958): 322–334.</p><p>[6] Winters, Peter R. “Forecasting sales by exponentially weighted moving averages.” Management science 6.3 (1960): 324–342.</p><p>[7] Dietterich, Thomas G. “Machine learning for sequential data: A review.” Joint IAPR international workshops on statistical techniques in pattern recognition (SPR) and structural and syntactic pattern recognition (SSPR). Springer, Berlin, Heidelberg, 2002.</p><p>[8] Bai, Shaojie, J. Zico Kolter, and Vladlen Koltun. “An empirical evaluation of generic convolutional and recurrent networks for sequence modeling.” arXiv preprint arXiv:1803.01271 (2018).</p><p>[9] Rolnick, David, et al. “Tackling climate change with machine learning.” ACM Computing Surveys (CSUR) 55.2 (2022): 1–96.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4a99e4cf142" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/forecasting-atmospheric-co2-concentration-with-python-c4a99e4cf142">Forecasting Atmospheric CO2 with Python</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Amazon SageMaker Studio Lab: A Great Alternative to Google Colab]]></title>
            <link>https://medium.com/data-science/amazon-sagemaker-studio-lab-a-great-alternative-to-google-colab-7194de6ef69a?source=rss-334ac6ae3ca3------2</link>
            <guid isPermaLink="false">https://medium.com/p/7194de6ef69a</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[sagemaker]]></category>
            <category><![CDATA[amazon]]></category>
            <dc:creator><![CDATA[Giannis Tolios]]></dc:creator>
            <pubDate>Mon, 06 Dec 2021 18:07:33 GMT</pubDate>
            <atom:updated>2021-12-08T05:20:18.299Z</atom:updated>
            <content:encoded><![CDATA[<h4>Introducing the New Free Machine Learning Platform of AWS</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uQIxhRYkemS5niH1" /><figcaption>Photo by <a href="https://unsplash.com/@homajob?utm_source=medium&amp;utm_medium=referral">Scott Graham</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Getting started with machine learning can be a daunting experience, especially if you are not technically oriented and experienced with computers. <a href="https://www.anaconda.com/products/individual">Anaconda</a> is an amazing tool, but it requires some expertise to set up and use correctly. Furthermore, having a powerful computer is necessary to train machine learning models on big datasets, making it especially true if you are interested in deep learning. Fortunately, there are various alternatives that let beginners experiment with machine learning, by easily executing their code on a cloud service. <a href="https://colab.research.google.com/">Google Colab</a> has become an industry standard in the past years, as it is a user-friendly service that can be easily accessed, simply by creating a Google account. Amazon has recently introduced <a href="https://studiolab.sagemaker.aws/">SageMaker Studio Lab</a>, an alternative service that offers useful features and functionality. In this article, I am going to present its main features, as well as provide a short tutorial to help you get familiar with the service. Let’s get started!</p><h3>Introducing the Service</h3><p>SageMaker Studio Lab provides free access to AWS runtimes that are optimized for machine learning and deep learning tasks. In case you are interested in technical specifications, the CPU runtime is based on the <a href="https://aws.amazon.com/ec2/instance-types/t3/">T3.xlarge</a> instance, while the GPU runtime runs on <a href="https://aws.amazon.com/ec2/instance-types/g4/">G4dn.xlarge</a>. Each runtime comes with JupyterLab, a web-based interface that lets you create and execute Jupyter notebooks, the de facto standard of data science. Furthermore, the service provides 15 GB of persistent storage, letting you store datasets and experiment results.</p><p>Having access to AWS runtimes can be immensely useful. For example, people that have trouble setting up Anaconda and JupyterLab on their own computers, could benefit significantly from it. Furthermore, even experienced users don’t always have access to powerful hardware, so utilizing a cloud service is a great option in that case. Finally, GPU prices have increased in the past years, due to hardware shortages and various other reasons. Therefore, utilizing the GPU runtime of SageMaker Studio Lab will be invaluable for those who can’t afford buying a GPU card!</p><h3>Comparison with Google Colab</h3><p>Google Colab is the main alternative to SakeMaker Studio Lab, so I’m going to briefly compare them. Google Colab is an established service, used by millions of data scientists and machine learning engineers, so most professionals in the field are already familiar with it. In my opinion, Google Colab is streamlined and easy to use, while also having advanced collaboration features, an area where SageMaker Studio Lab is somewhat lacking at the moment.</p><p>Regardless, it should be noted that Google Colab doesn’t provide persistent storage, with data being lost every time the instance is restarted. Furthermore, Google Colab usually assigns the Tesla K80 GPU to free accounts, whereas <a href="https://colab.research.google.com/signup">faster GPUs are reserved for subscribers of the premium Colab Pro service</a>. On the other hand, <a href="https://studiolab.sagemaker.aws/faq">the GPU runtime of SageMaker Studio Lab utilizes the Tesla T4 model</a>, a GPU that is significantly better compared to Tesla K80, thus making the service compelling for those interested in deep learning. Regardless, both services are great choices for executing Jupyter notebooks on the cloud.</p><h3>Getting Started with SageMaker Studio Lab</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RR4GhsX-6AAge65qA_mo6g.png" /><figcaption>Image by Author</figcaption></figure><p>First of all, you need to visit <a href="https://studiolab.sagemaker.aws/requestAccount">this link</a> and complete the form to request a free SageMaker Studio Lab account. According to Amazon, requests are typically approved in 1 to 5 business days, but I got access to my account a few hours after submitting the form, so you may get lucky. After your account has been approved, you can log in to the service with your credentials.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SXy23N2bsReT07AmXRYsKA.png" /><figcaption>Image by Author</figcaption></figure><p>After logging in to SageMaker Studio Lab, you simply have to select your preferred instance type, that can either be based on CPU or GPU. In this case, I have decided to start the CPU runtime, as seen on the screenshot. After clicking the “Start runtime” button, the session starts and remains active for the following 12 hours. If you choose the GPU runtime, the maximum time is limited to 4 hours. The runtime will be restarted when the time limit is reached, but your files will be saved to the persistent storage. We can now click the “Open Project” button to start the JupyterLab interface.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xU8YoeSj7lP69WFKeYZPWQ.png" /><figcaption>Image by Author</figcaption></figure><p>We are now using the JupyterLab environment that provides advanced features, such as executing Python code, writing Markdown text, running Linux commands on the terminal, as well as various others. Furthermore, you can install any Python package you want by using thepip and conda package managers. In case you are not familiar with JupyterLab, I suggest that you read <a href="https://jupyterlab.readthedocs.io/en/stable/">the official documentation</a> to get familiar with it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VKn_6XNUv7gWqG7vd5TYlQ.png" /><figcaption>Image by Author</figcaption></figure><p>If you want to get acquainted with SageMaker Studio Lab, you can clone the studio-lab-examples repository and experiment with the provided notebooks. As seen on the screenshot, this can be easily accomplished by starting a terminal and running the following command. You can obviously clone any other Github repository that might be useful as well.</p><pre>git clone <a href="https://github.com/aws/studio-lab-examples.git">https://github.com/aws/studio-lab-examples.git</a></pre><h3>Executing a Jupyter Notebook</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yi6_MovesZKFGeM31p12vg.png" /><figcaption>Image by Author</figcaption></figure><p>After cloning the studio-lab-examples repository, we are now going to execute one of the notebooks it contains. I have chosen the EDA_weather_climate.ipynb notebook, but the process is similar for the rest. First of all, we need to open the NOAA_Exploratory_Analysis directory and right click on the env_eda.yml file. After doing that, we select the ‘Build Conda Environment’ option to create a Conda environment with all the necessary packages for this notebook. The commands will be executed on a new terminal and the process may need a few minutes to be completed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wSHYwDaSDOBHLUQsfL0sRw.png" /></figure><p>After the environment is created, we can activate it by selecting the eda kernel. Keep in mind that creating a new environment for every notebook isn’t necessary, but it is considered best practice so we can avoid package dependency conflicts and other issues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j8QSZVw5m2rwX-Y6kAP-1A.png" /><figcaption>Image by Author</figcaption></figure><p>After activating the eda Conda environment, we simply need to select “Run All Cells” to execute all notebook cells and render the resulting graphs, as seen in the screenshot above.</p><h3>Conclusion</h3><p>As it is evident, Amazon SageMaker Studio Lab is a compelling service that provides easy access to a JupyterLab environment, letting anybody work on machine learning projects. Its advanced features and functionality make it a great alternative to Google Colab, thus introducing competition that may incentivize Google to improve its service, perhaps by offering persistent storage to Colab users. Therefore, I encourage you to experiment with SageMaker Studio Lab and see if it suits your needs! Feel free to share your thoughts in the comments, or follow me on <a href="https://www.linkedin.com/in/giannis-tolios-0020b067/">LinkedIn</a> where I regularly post content about data science and other topics. You can also visit my <a href="https://giannis.io/">personal website</a> or check my latest book, titled <a href="https://leanpub.com/pycaretbook/">Simplifying Machine Learning with PyCaret</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7194de6ef69a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/amazon-sagemaker-studio-lab-a-great-alternative-to-google-colab-7194de6ef69a">Amazon SageMaker Studio Lab: A Great Alternative to Google Colab</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Clustering Made Easy with PyCaret]]></title>
            <link>https://medium.com/data-science/clustering-made-easy-with-pycaret-656316c0b080?source=rss-334ac6ae3ca3------2</link>
            <guid isPermaLink="false">https://medium.com/p/656316c0b080</guid>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[pycaret]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Giannis Tolios]]></dc:creator>
            <pubDate>Fri, 15 Oct 2021 12:32:11 GMT</pubDate>
            <atom:updated>2021-10-15T13:41:36.428Z</atom:updated>
            <content:encoded><![CDATA[<h4>Low-code Machine Learning with a Powerful Python Library</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*k7m_lgr--sBapUz9" /><figcaption>Photo by <a href="https://unsplash.com/@luckaslh?utm_source=medium&amp;utm_medium=referral">Lucas Hobbs</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote>The content of this article was originally published in my latest book, Simplifying Machine Learning with PyCaret. You can <a href="https://leanpub.com/pycaretbook/">click here</a> to learn more about it.</blockquote><p>One of the fundamental tasks in unsupervised machine learning is clustering. The goal in this task is to categorize instances of a given dataset in different clusters, based on their common characteristics. Clustering has many practical applications in various fields, including market research, social network analysis, bioinformatics, medicine and others. In this article, we are going to examine a clustering case study by using <a href="https://pycaret.org/">PyCaret</a>, a Python library that supports all basic machine learning tasks, such as regression, classification, clustering and anomaly detection. PyCaret simplifies the machine learning workflow by following a low-code approach, thus making it a great choice for beginners, as well as experts that want to quickly prototype ML models.</p><h3>Software Requirements</h3><p>The code in this article should work on all major operating systems, i.e. Microsoft Windows, Linux and Apple macOS. You will need to have Python 3 installed on your computer, as well as <a href="https://jupyterlab.readthedocs.io/">JupyterLab</a>. I suggest that you use <a href="https://www.anaconda.com/">Anaconda</a>, a machine learning and data science toolkit that includes numerous helpful libraries and software packages. Anaconda can be freely downloaded at this <a href="https://www.anaconda.com/products/individual">link</a>. Alternatively, you can use a cloud service like <a href="https://colab.research.google.com/">Google Colab</a> to run Python code without worrying about installing anything on your machine. You can either create a new Jupyter notebook and enter the code, or download it from <a href="https://github.com/derevirn/pycaret-book/tree/main/04_Clustering">this Github repository</a>.</p><h3>Installing PyCaret</h3><p>The PyCaret library can be installed locally by executing the following command on your Anaconda terminal. You can also execute the same command on Google Colab or a similar service, to install the library on a remote server.</p><p>pip install pycaret[full]==2.3.4</p><p>After executing this command, PyCaret will be installed and you’ll be able to run all code examples of the article. It is recommended to install the optional dependencies as well, by including the [full] specifier. Furthermore, installing the correct package version ensures maximum compatibility, as I used PyCaret ver. 2.3.4 while working on this article. Finally, <a href="https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html#managing-envs">creating a conda environment</a> for PyCaret is considered to be best practice, as it will help you avoid conflicts with other packages and make sure you always have the correct dependencies installed.</p><h3>K-Means Clustering</h3><p>K-Means clustering¹ is one of the most popular and simplest clustering methods, making it easy to understand and implement in code. It is defined in the following formula.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8x056xyQFZjcujF_MTQYQw.png" /></figure><p><em>K</em> is the number of all clusters, while <em>C</em> represents each individual cluster. Our goal is to minimize <em>W</em>, which is the measure of within-cluster variation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FflGcxgGmxBUWcb_XwAmaQ.png" /></figure><p>There are various ways to define within-cluster variation, but the most common one is squared euclidean distance, as seen in the above equation. This results in the following form of K-Means clustering, with <em>W </em>being replaced by the euclidean distance formula.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O8hpohcdKZ3A2MICxFF3rA.png" /></figure><h3>Clustering with PyCaret</h3><p>K-Means is a widely used method, but there are numerous others available, such as Affinity Propagation², Spectral Clustering³, Agglomerative Clustering⁴, Mean Shift Clustering⁵ and Density-Based Spatial Clustering (DBSCAN)⁶. We are now going to see how the PyCaret clustering module can help us easily train a model and evaluate its performance.</p><pre>import numpy as np<br>import pandas as pd<br>import matplotlib.pyplot as plt<br>import matplotlib as mpl<br>import seaborn as sns<br>from pycaret.clustering import *<br>from sklearn.datasets import make_blobs<br>mpl.rcParams[&#39;figure.dpi&#39;] = 300</pre><p>We begin by importing some standard Python libraries, including NumPy, pandas, Matplotlib and Seaborn. We also import the PyCaret clustering functions, as well as the make_blobs() scikit-learn function that can be used to generate datasets. Finally, we set the Matplotlib figure DPI to 300, so we get high-resolution plots. Having this setting enabled isn’t necessary, so you can remove the last line if you want.</p><h3>Generating a Synthetic Dataset</h3><pre>cols = [&#39;column1&#39;, &#39;column2&#39;, &#39;column3&#39;,<br>        &#39;column4&#39;, &#39;column5&#39;]</pre><pre>arr = make_blobs(n_samples = 1000, n_features = 5, random_state =20,<br>                 centers = 3, cluster_std = 1)</pre><pre>data = pd.DataFrame(data = arr[0], columns = cols)<br>data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j4h3aMUpFifpz6zQIwwo3g.png" /><figcaption>Image by author</figcaption></figure><pre>data.info()</pre><pre>&lt;class &#39;pandas.core.frame.DataFrame&#39;&gt;<br>RangeIndex: 1000 entries, 0 to 999<br>Data columns (total 5 columns):<br> #   Column   Non-Null Count  Dtype  <br>---  ------   --------------  -----  <br> 0   column1  1000 non-null   float64<br> 1   column2  1000 non-null   float64<br> 2   column3  1000 non-null   float64<br> 3   column4  1000 non-null   float64<br> 4   column5  1000 non-null   float64<br>dtypes: float64(5)<br>memory usage: 39.2 KB</pre><p>Instead of loading a real-world dataset, we are going to generate a synthetic one by using the make_blobs() scikit-learn function. This function generates datasets that are suitable for clustering models, and has various parameters that can be modified according to our needs. In this case, we created a dataset with 1000 instances, 5 features and 3 distinct clusters. Using a synthetic dataset to test our clustering model has various benefits, the main being that we already know the actual number of clusters, so we can evaluate model performance easily. Real-world data are typically more complicated, i.e. they don’t always have clearly separated clusters, but working with a simple dataset lets you become acquainted with the tools and workflow.</p><h3>Exploratory Data Analysis</h3><pre>data.hist(bins = 30, figsize = (12,10), grid = False)</pre><pre>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MGJ72lZxVq6JJ4CukvSveg.png" /><figcaption>Image by author</figcaption></figure><p>The hist() pandas function lets us easily visualize the distribution of each variable. We can see that all variable distributions are either bimodal or multimodal, i.e. they have two or more peaks. This typically happens when the dataset contains multiple groups with different characteristics. In this case, the dataset was specifically created to contain 3 distinct clusters, so it is reasonable for the variables to have multimodal distributions.</p><pre>plt.figure(figsize=(10, 8))<br>sns.heatmap(data.corr().round(decimals=2), annot=True)</pre><pre>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xX__o9N6oIw57ip0j-hm5Q.png" /><figcaption>Image by author</figcaption></figure><p>We used the corr() pandas function, as well as the heatmap() Seaborn function to create a heatmap that visualizes the correlation values of all variable pairs. We can see that column2 and column3 have a strong linear relationship, with a correlation value of 0.93. The same is true for column3 and column4, having a correlation value of 0.75. On the other hand, column1 is inversely correlated with all the other columns, but especially column5, with a value of -0.85.</p><pre>plot_kws = {&#39;scatter_kws&#39;: {&#39;s&#39;: 2}, &#39;line_kws&#39;: {&#39;color&#39;: &#39;red&#39;}}<br>sns.pairplot(data, kind=&#39;reg&#39;, vars=data.columns[:-1], plot_kws=plot_kws)</pre><pre>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N-OabBhWhNC-R5P4h-h-jQ.png" /><figcaption>Image by author</figcaption></figure><p>We used the pairplot() Seaborn function to create a scatter plot matrix for the synthetic dataset, with the diagonal having the histogram of each variable. The dataset clusters are visible in the scatter plots, indicating that they are clearly separated from each other. As it was previously observed in the correlation heatmap, we can see that some of the variable pairs have strong linear relationships, while others have inverse linear relationships. This is highlighted by the regression lines that have been included in each scatter plot, by setting the kind parameter of the pairplot() function to reg.</p><h3>Initializing the PyCaret Environment</h3><pre>cluster = setup(data, session_id = 7652)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0e1xOiGfw5vZLYK5b572yQ.png" /><figcaption>Image by author</figcaption></figure><p>After completing the Exploratory Data Analysis (EDA), we are now going to use the setup() function to initialize the PyCaret environment. By doing this, a pipeline that prepares the data for model training and deployment will be created. In this case, the default settings are acceptable, so we aren’t going to modify any of the parameters. Regardless, this powerful function has numerous data preprocessing abilities, so you can refer to the documentation page of the <a href="https://pycaret.readthedocs.io/en/latest/api/clustering.html">PyCaret Clustering module</a> to read more details about it.</p><h3>Creating a Model</h3><pre>model = create_model(&#39;kmeans&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YPfxA_XBUczKv3HKgpVnfw.png" /><figcaption>Image by author</figcaption></figure><p>The create_model() function lets us easily create and evaluate the clustering model of our preference, such as the K-Means algorithm. This function creates 4 clusters by default, so we could simply set the num_clusters parameter to 3, as this is the correct number. Instead of doing that, we are going to follow an approach that generalizes for real-world datasets, where the cluster number is typically unknown. After executing the function, a number of performance metrics are printed, including Silhouette⁷, Calinski-Harabasz⁸ and Davies-Bouldin⁹. We are going to focus on the Silhouette Coefficient, which is defined in the following equation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I1lMDXBcZ0Wrzg5YMVJG9w.png" /></figure><ul><li><em>s(i)</em> is the Silhouette Coefficient of the dataset instance <em>i</em>.</li><li><em>a(i)</em> is the mean intra-cluster distance of <em>i</em>.</li><li><em>b(i)</em> is the mean nearest-cluster distance of <em>i</em>.</li></ul><p>The resulting metric value is the mean Silhouette Coefficient of all instances, having a range between -1 and 1. Negative values indicate that an instance has been assigned to the wrong cluster, while values near 0 indicate that clusters are overlapping. On the other hand, positive values close to 1 indicate correct assignment. In our example, the value is 0.5822, suggesting that model performance can be improved by finding the optimal number of clusters for the dataset. Next, we are going to see how we can accomplish that by using the elbow method.</p><pre>plot_model(model, &#39;elbow&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N6OHN0yMiHM8-eTg-23jYw.png" /><figcaption>Image by author</figcaption></figure><p>The plot_model() function lets us create various useful graphs for our model. In this case, we created an elbow plot that will help us find the optimal number of clusters for the K-Means model. The elbow method trains the clustering model for a range of <em>K</em> values, and visualizes the distortion score for each one of them¹⁰. The point of inflection on the curve –known as the elbow– is an indication of the optimal value for <em>K</em>. As expected, the plot has an elbow at <em>K</em> = 3, highlighted by the dashed vertical line.</p><pre>model = create_model(&#39;kmeans&#39;, num_clusters = 3)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mdpIzfoQQgt5ReenaFhvbw.png" /><figcaption>Image by author</figcaption></figure><p>After using the elbow method to find the optimal number of clusters, we train the K-Means model again. As we can see, the mean Silhouette Coefficient increased to 0.7972, indicating improved model performance and better cluster assignment for each dataset instance.</p><h3>Plotting the Model</h3><pre>plot_model(model, &#39;cluster&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t-QwyDLJV010bxWmvSWzrA.png" /><figcaption>Image by author</figcaption></figure><p>As seen earlier, plot_model() is a useful function that can be used to plot various kinds of graphs for our clustering model. In this case, we created a 2D Principal Component Analysis (PCA) plot for the K-Means model. PCA can be used to project data to a lower-dimensional space while preserving most of the variance¹¹, a technique known as dimensionality reduction. After applying PCA to the synthetic dataset, the original 5 features have been reduced to 2 principal components. Furthermore, we can see that the clusters are clearly separated, and all of the dataset instances have been assigned to the correct cluster.</p><h3>Saving and Assigning the Model</h3><pre>save_model(model, &#39;clustering_model&#39;)</pre><pre>results = assign_model(model)<br>results.head(10)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*guU5qLo43UTEbK0qV5SOew.png" /><figcaption>Image by author</figcaption></figure><p>The save_model() function lets us save the clustering model to the local disk for future use or deployment as an application. The model is stored as a pickle file that can be loaded using the complementary load_model() function. Furthermore, the assign_model() function returns the synthetic dataset with an additional column for the cluster labels that were assigned to the dataset instances.</p><h3>Conclusion</h3><p>Hopefully, the case study I provided in this article will help you get a grasp of the PyCaret clustering module. I also encourage readers to experiment with other datasets and practice the aforementioned techniques by themselves. In case you want to learn more about PyCaret, you can check <a href="https://leanpub.com/pycaretbook/">Simplifying Machine Learning with Pycaret</a>, the book I recently published about the libary. Feel free to share your thoughts in the comments, or follow me on <a href="https://www.linkedin.com/in/giannis-tolios-0020b067/">LinkedIn</a> where I regularly post content about data science and other topics.</p><h3>References</h3><p>[1]: Steinley, Douglas. “K‐means clustering: a half‐century synthesis.” British Journal of Mathematical and Statistical Psychology 59.1 (2006): 1–34.</p><p>[2]: Dueck, Delbert. Affinity propagation: clustering data by passing messages. Toronto: University of Toronto, 2009.</p><p>[3]: Von Luxburg, Ulrike. “A tutorial on spectral clustering.” Statistics and computing 17.4 (2007): 395–416.</p><p>[4]: Ackermann, Marcel R., et al. “Analysis of agglomerative clustering.” Algorithmica 69.1 (2014): 184–215.</p><p>[5]: Derpanis, Konstantinos G. “Mean shift clustering.” Lecture Notes (2005): 32.</p><p>[6]: Khan, Kamran, et al. “DBSCAN: Past, present and future.” The fifth international conference on the applications of digital information and web technologies (ICADIWT 2014). IEEE, 2014.</p><p>[7]: Rousseeuw, Peter J. “Silhouettes: a graphical aid to the interpretation and validation of cluster analysis.” Journal of computational and applied mathematics 20 (1987): 53–65.</p><p>[8]: Caliński, Tadeusz, and Jerzy Harabasz. “A dendrite method for cluster analysis.” Communications in Statistics-theory and Methods 3.1 (1974): 1–27.</p><p>[9]: Davies, David L., and Donald W. Bouldin. “A cluster separation measure.” IEEE transactions on pattern analysis and machine intelligence 2 (1979): 224–227.</p><p>[10]: Yuan, Chunhui, and Haitao Yang. “Research on K-value selection method of K-means clustering algorithm.” J 2.2 (2019): 226–235.</p><p>[11]: Abdi, Hervé, and Lynne J. Williams. “Principal component analysis.” Wiley interdisciplinary reviews: computational statistics 2.4 (2010): 433–459.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=656316c0b080" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/clustering-made-easy-with-pycaret-656316c0b080">Clustering Made Easy with PyCaret</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Created a Fake News Detector with Python]]></title>
            <link>https://medium.com/data-science/how-i-created-a-fake-news-detector-with-python-65b1234123c4?source=rss-334ac6ae3ca3------2</link>
            <guid isPermaLink="false">https://medium.com/p/65b1234123c4</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[fake-news]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[streamlit]]></category>
            <category><![CDATA[spacy]]></category>
            <dc:creator><![CDATA[Giannis Tolios]]></dc:creator>
            <pubDate>Wed, 14 Apr 2021 12:42:27 GMT</pubDate>
            <atom:updated>2021-10-08T22:03:16.808Z</atom:updated>
            <content:encoded><![CDATA[<h4>Developing a fake news detection app with spaCy and Streamlit</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8FeBJ2G3Dtciejh6poq9UQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@markuswinkler?utm_source=medium&amp;utm_medium=referral">Markus Winkler</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The proliferation of fake news is a significant challenge for modern democratic societies. Inaccurate information can affect the health and well-being of people, especially during the challenging times of the COVID-19 pandemic. Furthermore, disinformation erodes public trust in democratic institutions, by preventing citizens from making rational decisions based on verifiable facts. A disturbing study has shown that fake news reach more people and spread faster than actual facts, especially on social media. MIT researchers have discovered that fake news are 70% more likely to be shared on platforms like Twitter and Facebook¹.</p><p>Fake news campaigns are a form of modern information warfare, used by states and other entities to undermine the power and legitimacy of their opponents. According to EU authorities, <a href="https://www.politico.eu/article/russia-china-disinformation-coronavirus-covid19-facebook-google/">european countries have been targeted by chinese and russian disinformation campaigns</a>, spreading falsehoods about numerous topics, including the COVID-19 pandemic. <a href="https://eeas.europa.eu/headquarters/headquarters-homepage/2116/-questions-and-answers-about-the-east-stratcom-task-force_en">The East StratCom Task Force</a> has been set up to deal with that problem, by monitoring and debunking fake news about EU member states.</p><p>Fact-checkers are individuals that verify the factual correctness of published news. Those professionals debunk fake news by identifying their false claims. Research has shown that traditional fact-checking can be augmented by machine learning and natural language processing (NLP) algorithms². In this article, I am going to explain how I developed a web application that detects fake news written in my native language (Greek), by using the Python programming language.</p><h3>The Greek Fake News Dataset</h3><p>The success of every machine learning project depends on having a proper and reliable dataset. There are numerous publicly available fake news datasets, such as LIAR³ and FakeNewsNet⁴, but unfortunately most of them are comprised of english articles exclusively. As I couldn’t find any datasets including articles in Greek, I decided to create my own. <a href="https://github.com/derevirn/gfn-detector/tree/main/data">The Greek Fake News (GFN) dataset</a> is comprised of real and fake news written in the greek language, and can be used to train text classification models, as well as other NLP tasks.</p><p>The dataset was created based on the following methodology. First of all, real news items were collected from a number of reputable greek newspapers and websites. I added news from a variety of topics, mostly focusing on politics, the economy, the COVID-19 pandemic and world news. To identify fake news articles, I consulted <a href="https://www.ellinikahoaxes.gr/">Ellinika Hoaxes</a>, a greek fact-checking website that has been certified by the <a href="https://www.poynter.org/ifcn/">International Fact-Checking Network</a> (IFCN). A sample of news items verified to be false were also added to the dataset. After that process was completed, the resulting dataset was used to train the text classification model of the Greek Fake News Detector application.</p><h3>The spaCy Python Library</h3><p>There are numerous advanced Python libraries that can be used for natural language processing tasks. One of the most popular is <a href="https://spacy.io/">spaCy</a> , a NLP library that comes with pre-trained models, as well as support for tokenization and training for more than 60 languages. spaCy includes components for named entity recognition (NER), part-of-speech tagging, sentence segmentation, text classification, lemmatization, morphological analysis, and others. Furthermore, spaCy is robust and production-ready software that can be used in real-world products. This library was used to create the text classification model of the Greek Fake News Detector application.</p><h3>The Streamlit Framework</h3><p><a href="https://www.streamlit.io/">Streamlit</a> is a Python framework that lets you build web apps for data science projects very quickly. You can easily create a user interface with various widgets, in a few lines of code. Furthermore, Streamlit is a great tool for deploying machine learning models to the web, and adding great visualizations of your data. Streamlit also has a powerful caching mechanism, that optimizes the performance of your app. Furthermore, <a href="https://streamlit.io/sharing">Streamlit Sharing</a> is a service provided freely by the library creators, that lets you easily deploy and share your app with others. A detailed introduction to Streamlit is available <a href="https://towardsdatascience.com/streamlit-101-an-in-depth-introduction-fc8aad9492f2">here</a>.</p><h3>Developing the Web Application</h3><p>I decided to develop the <a href="https://share.streamlit.io/derevirn/gfn-detector/main/app.py">Greek Fake News Detector</a> for a number of a reasons. First of all, I wanted to acquaint myself with the spaCy library and NLP in general, thus enhancing my skill set and evolving as a professional. Second, I wanted to showcase the potential of using machine learning to deal with the issue of fake news, in a way that is accesible to non-experts. The best way to achieve that was by developing a simple prototype in the form of a web application. Streamlit is an ideal tool for this purpose, so I decided to utilize it. I will now explain the source code functionality, starting with the text classification model training. I originally used a Jupyter notebook, but for the purposes of this article, the code was converted to the following Python file, named gfn_train.py.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d61d1e4a53742960ad4d609392d975e3/href">https://medium.com/media/d61d1e4a53742960ad4d609392d975e3/href</a></iframe><p>First of all, we import the necessary Python libraries and define two helper functions. The load_data() function shuffles the data, assigns a category to each news article, and splits the dataset into train and test subsets. The evaluate() function calculates various metrics such as precision, recall and F-score, that can help us evaluate the text classifier performance. After defining the helper functions, we load the spaCy pre-trained model. I’ve chosen the el_core_news_md model, as we’re working with articles written in Greek.</p><p>After doing that, we load the GFN dataset to a pandas dataframe, and clean it by removing some unwanted characters. Afterwards, we add the textcat component to our pre-trained model. This component will be trained with the GFN dataset, to create the text classification model. We then disable the other components, as we only need to train textcat . Afterwards, we use the load_data() and update() functions to load the dataset and train the model respectively. The evaluate() function that we defined earlier, is used to print the training metrics and performance. After the training is complete, the model is saved by using the to_disk() function. We are now going to examine app.py, the main file of the Streamlit web application.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fc66f253ada46d4cddbe7da96a5bef28/href">https://medium.com/media/fc66f253ada46d4cddbe7da96a5bef28/href</a></iframe><p>We begin by importing Streamlit, spaCy, and other necessary libraries. After doing that, we define the get_nlp_model() function that loads the spaCy text classification model we trained earlier. This function is marked with the @st.cache decorator, that lets Streamlit store the model in a local cache, thus improving performance. Afterwards, we define the generate_output() function that prints the classification result, using the markdown() function and some common HTML tags. After that, the article text is printed, along with a word cloud for visualization purposes.</p><p>We then create the application layout, using various Streamlit widgets. First of all, we set the page title and description. Second, we create a radio button widget that is used for input type selection. By doing that, users can select between entering the article URL or text. In case the user selects article URL as input type, the text is scraped using the get_page_text() function. Otherwise, the user can paste the article in a multi-line text input. In both cases, a button widget is used to call the generate_output() function, thus classifying the article and printing the result. Finally, we can execute the streamlit run app.py command to run the application locally, or use the free Streamlit Sharing service to deploy it.</p><h3>Conclusion</h3><p>I hope that after reading this article, you’ll be more knowledgeable about the potential of using NLP and machine learning to deal with the serious problem of fake news. Furthermore, I encourage you to experiment and create your own fake news detection application, as modifying the code to train the model on a different dataset is simple. In case you want to clone the Github repository of this project, it is available <a href="https://github.com/derevirn/gfn-detector">here</a>. Feel free to share your thoughts in the comments, or follow me on <a href="https://www.linkedin.com/in/giannis-tolios-0020b067/">LinkedIn</a> where I regularly post content about data science and other topics. You can also visit my <a href="https://giannis.io/">personal website</a> or check my latest book, titled <a href="https://leanpub.com/pycaretbook/">Simplifying Machine Learning with PyCaret</a>.</p><h3>References</h3><p>[1] Vosoughi, Soroush, Deb Roy, and Sinan Aral. “The spread of true and false news online.” <em>Science</em> 359.6380 (2018): 1146–1151.</p><p>[2] Oshikawa, Ray, Jing Qian, and William Yang Wang. “A survey on natural language processing for fake news detection.” <em>arXiv:1811.00770</em> (2018).</p><p>[3] Wang, William Yang. “” liar, liar pants on fire”: A new benchmark dataset for fake news detection.” <em>arXiv:1705.00648</em> (2017).</p><p>[4] Shu, Kai, et al. “Fakenewsnet: A data repository with news content, social context, and spatiotemporal information for studying fake news on social media.” <em>Big Data</em> 8.3 (2020): 171–188.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=65b1234123c4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-i-created-a-fake-news-detector-with-python-65b1234123c4">How I Created a Fake News Detector with Python</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>