<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Judeleonard on Medium]]></title>
        <description><![CDATA[Stories by Judeleonard on Medium]]></description>
        <link>https://medium.com/@judeleonard86?source=rss-bb6e58077747------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*kMEr_UWB6svHe-ctamtcoQ.jpeg</url>
            <title>Stories by Judeleonard on Medium</title>
            <link>https://medium.com/@judeleonard86?source=rss-bb6e58077747------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 09:02:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@judeleonard86/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Dynamic Duo of   Modern Data Stack — Harnessing the Power of DBT and Airflow to Streamline…]]></title>
            <link>https://blog.devgenius.io/the-dynamic-duo-of-modern-data-stack-harnessing-the-power-of-dbt-and-airflow-to-streamline-d4e51e6f6625?source=rss-bb6e58077747------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4e51e6f6625</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Judeleonard]]></dc:creator>
            <pubDate>Mon, 28 Oct 2024 14:34:32 GMT</pubDate>
            <atom:updated>2024-11-04T13:20:01.852Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Dynamic Duo of Modern Data Stack — Harnessing the Power of DBT and Airflow to Streamline Data Workflows.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2gU8wXUCg76JvxwdR17Gcg.png" /></figure><p>It’s been a while, hasn’t it? For the longest time, I’ve been so deeply immersed in the world of building, coding, and problem-solving that I let the art of sharing my progress slip through the cracks. I often got caught up in the thrill of integrating new technologies to develop fit-for-purpose datasets, refining processes, and tackling challenges. Somewhere along the way, I forgot to pause and write about these experiences.</p><p>This article marks a fresh start for me — a conscious effort to reconnect with writing and share the exciting projects I’ve been working on, especially my journey so far in the world of data engineering that has transformed my data workflows. Writing this piece is part of my journey to reignite that spark for storytelling. I hope it inspires me to keep sharing more insights and lessons moving forward.</p><p>If you find this article helpful or learn something new, give this article a clap! Your feedback will go a long way in motivating me to keep at it. Let’s dive into how dbt and Airflow became a game changer in my work.</p><p>As a data engineer with a passion for optimizing workflows and managing large-scale data systems, one of my key projects has been integrating dbt and Airflow into a seamless data pipeline. This journey began as I sought more robust ways to transform, manage, and orchestrate data models. While dbt was my go-to for managing my data models and transformations, Airflow provided the orchestration I needed to handle end-to-end data pipelines. Before this time everything was handled via Airflow to orchestrate a bunch of Python scripts that call other SQL files to perform certain database operations as tasks. However, as the organization&#39;s data grew, there were lots of data requests from the marketing team down to the Business Intelligence team, as we needed to provide them with the requested data to optimize their marketing campaigns to reach their target audience. Then also there is the Business Intelligence team that requests data to track tons of KPIs.</p><p>Things started getting messy and overwhelming as my team and I needed to quickly come up with a new working solution to completely migrate all our data pipelines while ensuring the different teams still got all the data they needed to do their job. We had to bootstrap a new solution that incorporated dbt from the previous working branch whilst still ensuring there was no downtime in data delivery across different teams.</p><h4>Why dbt for Data Transformations?</h4><p>When I first explored tools for managing data transformations,<a href="https://docs.getdbt.com/docs/introduction"> dbt (Data Build Tool) </a>stood out. It simplifies SQL-based transformations and is designed to support a modular approach to building data models. With dbt, I could create reusable SQL transformations and easily manage dependencies between tables. Just like we already know, dbt handles the “ T ” in all your ELT workflows (Extract, Load, and Transform) which means that you can still have your “ E ” and “ L ” logic in any language of your choice making it even more dynamic to fit into any data workflow. The fun part, it is Python-friendly. I have seen many people assume dbt to be a complete ELT tool, thereby using the dbt-seed command to load staging tables. This might actually solve the problem but not following best practices and for someone who has benchmarked this process, I can say it&#39;s not scalable especially when you are trying to load large tables from your transactional database. The expectation is to harness the dbt seed command to load static data i.e. data that doesn&#39;t change frequently. You can find more details about dbt seed best practices from this dbt <a href="https://docs.getdbt.com/docs/build/seeds#:~:text=Seeds%20should%20not%20be%20used,or%20user%20IDs%20of%20employees.">docs</a> page.</p><p>Here are some of the reasons I chose dbt:</p><ul><li><strong>Documentation and Community Support</strong>: If there is one thing I admire about dbt, it&#39;s their well-curated documentation to help you easily kickstart working with the tool as well as their community support forum. This means if you get stuck along the way or need a little bit of clarity around the choices you are making you can easily ping them and be sure to get a response to point you in the right direction. You can also do a similarity search of your query against a list of past conversations to help them keep things simple and avoid redundancy. This is me giving their documentation a 5-star</li><li><strong>Modular Development</strong>: dbt encourages breaking down complex transformations into smaller, manageable parts, promoting cleaner and more maintainable code.</li><li><strong>Version Control</strong>: With dbt’s integration with git, it became easier to track changes in our transformations, which improved collaboration amongst my team and ensured transparency.</li><li><strong>Testing</strong>: dbt’s in-built testing framework and its support for custom tests allowed me to ensure the integrity of our data models. Ensuring all downstream data are well tested for further analysis.</li></ul><h4><strong>Introducing Airflow (Astronomer) for Orchestration</strong></h4><p>While dbt was handling transformations efficiently, Airflow brought the orchestration layer into the equation. It allowed us to schedule, monitor, and execute dbt tasks as part of a larger workflow. It also helped us cut costs, the fact is, there is no way to truly scale your dbt workflow without using the dbt cloud of course which doesn’t come free, hence the need for Airflow. Airflow’s DAG structure was particularly appealing for managing dependencies between tasks, ensuring that all our transformations ran in the correct sequence.</p><p>P.S. I would also like to point out that working on this solution was more of an iterative process as some steps did not work initially and you will have to continue scanning through the documentation and perhaps engaging the communities to discover what’s possible and following best practices. I will do my best to explain some of the approaches we had to move away from and what worked for us.</p><h4><strong>Airflow and dbt Together: Key Implementation Insights</strong></h4><p>Most of the time, revamping an existing solution requires a complete redesign of your entire workflow. Here is the breakdown of our integration:</p><ol><li><strong>Airflow DAGs for dbt Models</strong><br>This is where we had to make sketches of DAGs in Airflow to reflect the hierarchy of dependencies in our dbt models. Each dbt model run was encapsulated as a task within the DAG, ensuring that transformations were executed in the correct order and upstream tasks that contain artifacts, datasets, and variables needed by dbt were executed before executing the transformation logic. However, to make things simpler and more manageable for us we resorted to having models per Dag kinda design rather than having especially large models appended as tasks which obviously would make our Dags look busy, cluttered, and unmanageable, to say the least. In this way, we set these models apart from any sort of long-chain dependencies and can easily reuse or pull any upstream artifact required by each model, hence conserving even more resources.</li><li><strong>Project Structure and Configuration</strong><br>This is where requirement gathering is crucial after which you decide how to organize your data models, configurations, variables, etc. If you are already familiar with dbt you will agree with me that one of the crucial steps is configuring your profiles.yml which connects to your target system (data warehouse) in turn this config file is not expected to expose your database secrets. Remember airflow and dbt run on entirely two different environments so we needed to find a way to provide airflow with these credentials at Dag runtime.</li></ol><pre># sample dbt_project.yml config<br><br>profile: &#39;project_name&#39;<br><br># These configurations specify where dbt should look for different types of files.<br>model-paths: [&quot;models&quot;]<br>test-paths: [&quot;tests&quot;]<br>macro-paths: [&quot;macros&quot;]<br>snapshot-paths: [&quot;snapshots&quot;]<br><br>target-path: &quot;target&quot;<br>clean-targets:<br>  - &quot;target&quot;<br>  - &quot;dbt_packages&quot;<br><br>models:<br>  project_name:<br>    +materialized: view<br>  <br>    model_A:<br>      +materialized: table<br>      +schema: &#39;analytics&#39;<br>    src:<br>      +materialized: ephemeral<br>      +schema: &#39;staging&#39;</pre><p>Here are the different ways we approached this problem:</p><p><strong>a. Using the dbt BashOperator</strong>. This is expected to pull the credentials from the airflow metastore and export them as environment variables at execution runtime upon running the dbt command following the naming convention variables we have set in the profiles.yml file. Here we supply the credentials as metadata within the BashOperator during execution. This particular approach was inspired by a conversation from <a href="https://stackoverflow.com/questions/70989049/managing-dbt-profile-file-in-mwaa">StackOverflow.</a></p><pre># sample profiles.yml file<br>project_name:<br>  target: dev<br>  outputs:<br>    dev:<br>      type: postgres<br>      host: &quot;{{ env_var(&#39;DB_HOST&#39;) }}&quot; <br>      user: &quot;{{ env_var(&#39;DB_USER&#39;) }}&quot;<br>      password: &quot;{{ env_var(&#39;DB_PASSWORD&#39;) }}&quot; <br>      port: &quot;{{ env_var(&#39;DB_PORT&#39;, &#39;5432&#39;) }}&quot;<br>      dbname: &quot;{{ env_var(&#39;DB_NAME&#39;) }}&quot;<br>      schema: public<br>      threads: 4<br>      connect_timeout: 20</pre><p>Then within our Dag, we call the BashOperator task:</p><pre>from airflow.operators.bash_operator import BashOperator<br>from airflow.models import Variable<br><br># load the profile credentials <br># assuming you named the json file profile_credentials<br>profile_credentials = Variable.get(&quot;profile_credentials&quot;, deserialize_json=True)<br><br>with DAG(<br>    dag_id=&#39;test profiles config&#39;,<br>    start_date=days_ago(1),<br>    schedule_interval=None,<br>    catchup=False,<br>) as dag:<br>    <br>    run_dbt = BashOperator(<br>        task_id=&#39;run_dbt&#39;,<br>        bash_command=&quot;&quot;&quot;<br>        export DB_HOST={{ profile_credentials[&#39;DB_HOST&#39;] }}<br>        export DB_USER={{ profile_credentials[&#39;DB_USER&#39;] }}<br>        export DB_PASSWORD={{ profile_credentials[&#39;DB_PASSWORD&#39;] }}<br>        export DB_PORT={{ profile_credentials[&#39;DB_PORT&#39;] }}<br>        export DB_NAME={{ profile_credentials[&#39;DB_NAME&#39;] }}<br>        export DB_SCHEMA={{ profile_credentials[&#39;DB_SCHEMA&#39;] }}<br>        dbt run --project-dir /path/to/your/dbt/project --profiles-dir /path/to/your/dbt/profiles<br>        &quot;&quot;&quot;,<br>        env=profile_credentials,<br>    )</pre><p><strong>Key Takeaway Utilizing the dbt BashOperator Approach</strong></p><p>After so much back and forths trying to get this approach working and discovering airflow does not consider the dbt project dir and profiles-dir supplied. We also tried to provide the path to dbt executable file in a virtual environment but that still didn’t work. Digging deeper here are my few takes:</p><ul><li><strong>Debugging headache: </strong>This was a major setback for us trying to utilize the dbt within the bash operator as there was limited error logging within Airflow as it keeps returning generic errors like “Airflow Exceptions” that don&#39;t point to the right issue making RCA even more difficult.</li><li><strong>Lacking structure: </strong>dbt Bashoperator<strong> </strong>runs individual shell commands without inherent support for task grouping or model-level dependency management, which I find suitable for complex dbt projects. This limitation makes organizing and managing multiple dbt models challenging, as you must create separate tasks and manually define dependencies, leading to cluttered and harder-to-read DAGs.</li></ul><p><strong>b</strong>. <strong>Introducing Astronomer-cosmos: </strong>In our workflow, one major challenge was creating an efficient way to orchestrate and manage dbt commands within a larger data pipeline. That’s where <strong>Astronomer-Cosmos</strong> came in as a solution with its seamless compatibility and ease of integration. Astronomer-Cosmos allows Airflow to natively run dbt commands, eliminating the need for complex workarounds and providing a highly reliable and flexible orchestration layer.</p><p>Through Astronomer-Cosmos, we could manage each step of the data transformation process, set dependencies, monitor, and trigger dbt jobs at exactly the right points in the pipeline. This compatibility enabled the workflow to be streamlined and automated, ultimately enhancing productivity and reliability across all data processes. Whether it was loading, testing, or modeling data, Astronomer-Cosmos provided an orchestration backbone allowing us to focus on optimizing our models without getting bogged down by manual orchestration.</p><p><strong>Key Takeaway Utilizing the Astronomer-Cosmos Approach</strong></p><p>First, we needed to ensure Airflow could access our profile secrets within Airflow metastore. Cosmos provided convenient ways to parse all dbt configurations and access them within a DbtTaskGroup Operator. In this case, we assume Postgres is the target data warehouse.</p><pre># define cosmos configuration to be used across all dags<br># config.py<br>from cosmos.config import ProfileConfig, ProjectConfig<br>from cosmos import ExecutionConfig<br>from cosmos.profiles import PostgresUserPasswordProfileMapping<br><br><br>project_root = &quot;/usr/local/airflow&quot;<br><br>DBT_PROFILE_CONFIG = ProfileConfig(<br>    profile_name=&#39;project_name&#39;,<br>    target_name=&#39;dev&#39;,<br>    profile_mapping=PostgresUserPasswordProfileMapping(<br>        conn_id=&quot;dbt_connection_id&quot;,<br>        profile_args={&quot;schema&quot;: &quot;public&quot;,<br>                      &quot;threads&quot;: 4<br>        <br>        },<br>    ),<br>)<br><br>DBT_PROJECT_CONFIG = ProjectConfig(<br>    dbt_project_path=f&quot;{project_root}/include/dbt/&quot;,<br>)<br><br>DBT_EXECUTABLE_PATH = ExecutionConfig(<br>    dbt_executable_path=f&quot;{project_root}/dbt_venv/bin/dbt&quot;<br>)</pre><p>You can also adapt the above profile configuration to any flavor of your choice that connects to other cloud warehouse Technologies, such as BigQuery and Snowflake. See this <a href="https://astronomer.github.io/astronomer-cosmos/profiles/GoogleCloudServiceAccountDict.html">documentation</a>.</p><p>Then we can call in the DbtTaskGroup operator within the dag as</p><pre>from config import DBT_PROJECT_CONFIG, DBT_PROFILE_CONFIG, DBT_EXECUTABLE_PATH<br>from cosmos.airflow.task_group import DbtTaskGroup<br>from cosmos.constants import LoadMode<br>from cosmos.config import RenderConfig<br>from airflow.models import DAG<br><br><br>with DAG(<br>    dag_id=&#39;test profiles config&#39;,<br>    start_date=days_ago(1),<br>    schedule_interval=None,<br>    catchup=False,<br>) as dag:<br><br>    model_A = DbtTaskGroup(<br>        group_id=&#39;transform_data&#39;,<br>        project_config=DBT_PROJECT_CONFIG,<br>        profile_config=DBT_PROFILE_CONFIG,<br>        execution_config=DBT_EXECUTABLE_PATH,<br>        render_config=RenderConfig(<br>            load_method=LoadMode.DBT_LS,<br>            select=[&#39;path:models/model_name&#39;], # folder containing your transformation logic<br>            emit_datasets=False<br>        ),<br>        operator_args={<br>                &quot;install_deps&quot;: True<br>            }<br>    )</pre><ul><li><strong>Enhanced error handling and logging: </strong>with DbtTaskGroup you benefit from native logging and error handling specific to dbt as each task within the DbtTaskGroup can log details of model executions, errors, and configurations in a way that makes tracing issues easier for better RCA.</li><li><strong>Efficient environment variable and profile management</strong>: Cosmos supports direct configuration with Airflow’s environment variables and profiles, making it easy to dynamically set up dbt profiles, targets, and environment configurations. This approach is more streamlined than setting environment variables manually within a <strong>BashOperator</strong> and makes the DAGs cleaner and more secure since configurations are managed centrally.</li></ul><p>3. <strong>Testing and Data Quality with dbt</strong></p><p>Testing was one of the most important aspects of our integration. Using <strong>dbt_expectations</strong> to develop generic tests and custom tests with reusable macros and assert methods to validate data quality, adding a critical layer of reliability, especially across all our data product models. We had our tests integrated into 3 different layers which are the source stage, the intermediate stage, and the final stage. We used these stages to form a test chain which implies that the intermediate stage expects all tests within the source stage to pass before executing, and the same for the final stage downstream.</p><p>4. <strong>Setting up CI/CD workflow to Manage automatic deployment</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MoXULSbW_zqx7buMMjTY4A.png" /><figcaption>Our typical workflow setup to manage ci/cd Ops.</figcaption></figure><p>In our typical CI/CD setup, we prioritized automating every stage of the deployment pipeline, ensuring our team could focus more on building models while the integration and delivery ran seamlessly. Our approach leaned on GitHub for tracking changes and managing pull requests and <a href="https://forge.laravel.com/docs/introduction.html">Laravel Forge</a> to handle production deployments which we configured for maximum automation and precision.</p><p><strong>Our GitHub workflow for Collaborative Development</strong></p><p>Our workflow begins with tracking schema and transformation updates directly in GitHub, where each change is proposed through a pull request (PR). Once a PR is opened, collaborators can suggest changes and improvements, keeping us aligned on best practices. To make collaboration even more efficient, we set up automated triggers to run tests on a staging server as soon as any created pull request is merged into our staging branch.</p><p>We configured this staging server to replicate the production environment as closely as possible, mocking the same configurations, data structures, and resource allocations to ensure consistency. Every pull request, whether it’s a dbt model update or a new Airflow DAG, goes through this staging environment. Here, our test suite validates everything from schema alignment to data integrity checks. If the tests pass successfully, the PR is then automatically merged into the production branch in GitHub, where it’s ready for deployment</p><p><strong>Utilizing Laravel Forge for production deployment automation</strong></p><p>After a successful merge into the production branch, our focus shifts to Laravel Forge for deployment. We configured our production branch in Laravel Forge to detect any changes pushed to GitHub, so every new merge triggers an automatic deployment. Forge monitors the production branch, running our deployment scripts meticulously with each update, ensuring our dbt models and Airflow tasks are consistently up to date.</p><p>Forge allows us to centralize production deployment on a single platform, making it easy to manage configurations, monitor server health, and control release rollouts. Since Forge handles deployment, our focus stays on creating optimized deployment scripts. This setup also means we don’t have to worry about manually deploying each time, letting us prioritize development, testing, and the actual data work while knowing that everything will be seamlessly updated in production.</p><p><em>By configuring GitHub, staging tests, and Laravel Forge in harmony, we created a CI/CD pipeline that continuously integrates and delivers code without manual intervention. This robust setup keeps our dbt models and Airflow tasks reliable, tested, and production-ready while allowing our team to innovate quickly without risking stability. Even when incidents arise, our robust alert system sends an error message to the data team with a full traceback error pointing exactly where the error occurred for easy fixes.</em></p><p><strong>Final remark</strong></p><p>As we continue to optimize our setup, we’re focusing on finding the best methods for dynamically detecting schema changes within our data models. Automating this process would provide more streamlined updates, helping us keep our transformations aligned with any upstream changes without interrupting workflows thereby creating a full refresh across all our data models.</p><p>For now, though, our schema is relatively stable, with changes occurring only occasionally. When schema updates are anticipated, they’re flagged by our software engineers ahead of time, often in stand-up meetings. This proactive communication provides the data team with clear documentation on what’s expected to change. Once the details are outlined, a proposed pull request (PR) is created, allowing us to carefully review and understand the adjustments before they’re integrated.</p><p>While fully automating schema detection is our long-term goal, our current approach ensures nothing slips through the cracks. By keeping everyone on the same page and making thoughtful updates, we’re laying the groundwork for a data pipeline that’s as resilient as it is.</p><p>Let me know your thoughts in the comments. 😃</p><p>Let’s connect on <a href="https://www.linkedin.com/in/jude-ndu-78ab38175">LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4e51e6f6625" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/the-dynamic-duo-of-modern-data-stack-harnessing-the-power-of-dbt-and-airflow-to-streamline-d4e51e6f6625">The Dynamic Duo of   Modern Data Stack — Harnessing the Power of DBT and Airflow to Streamline…</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Analysis on UK 2016 Road Safety Data]]></title>
            <link>https://medium.com/@judeleonard86/an-analysis-on-uk-2016-road-safety-data-d2aaca2bd6d8?source=rss-bb6e58077747------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2aaca2bd6d8</guid>
            <category><![CDATA[road-safety]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Judeleonard]]></dc:creator>
            <pubDate>Sat, 31 Oct 2020 16:55:18 GMT</pubDate>
            <atom:updated>2020-10-31T16:55:18.934Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>A data science open source group project provided by <a href="https://hamoye.com/app/courses/concepts">Hamoye</a> to be done by the interns (including myself)</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*K5F4R_gl_f2OfbU0BKILmA.jpeg" /><figcaption>Image from <a href="http://iamroadsmart.com/">http://iamroadsmart.com/</a></figcaption></figure><h4>Introduction</h4><p>According to the World Health Organization, worldwide, an estimated 1.2 million people are killed in road crashes each year and as many as 50 million are injured. Projections indicate that these figures will increase by about 65% over the next 20 years unless there is new commitment to prevention [1]. There has been a rising concern about road death rates, and government agencies keep working seriously to aviod chances of road accidents.</p><p>However, for the interest of this project, the details of the UK road safety data contains already occured accidents, hence we chose to analyze the data and bring it to future use on how road accidents in UK can be curbed or managed.</p><p>The data for this project was found on <a href="https://www.kaggle.com/bluehorseshoe/uk-2016-road-safety-data/data">Kaggle</a>. It comprises five data sets in a folder, which included the casualties data, details of vehicles involved in the accidents, the vehicles data, and records of accidents.</p><h4>Problem Statement</h4><p>When accidents occur, the level of severity differs. On an extreme, it could be fatal, while on the very minimal, it could result to having some damages. Either way there is always a question of how badly an accident was. This heightened concern tilted the course of this project to analyze the data, and seek for patterns in the records, which could be used to predict the severity of a road accident.</p><h4>Data Exploration</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/487/1*mnCG8xD-FLSTileyL8fZTg.png" /></figure><p>The data contained over 80% of urban road accidents. This can be attributed to the very high population of road users in urban areas.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*dCHBXyVrHZvipQJZQjE_YQ.png" /></figure><p>The plot shows that when accidents occur, there are higher chances that the impact will be very severe on the driver.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/1*R7rrtndtmohPqSzfv3n2ww.png" /></figure><p>Road accidents in 2019 reached the peak in November, a winter season. The darker, shorter days, coupled with increased chance of ice and snow increases risk for all road users. In addition, clear and sunny winter days often result in strong glare, which can be blinding for road users driving into the sun [2].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*jg-WoYyOgQIvXKLMqUDKkA.jpeg" /></figure><p>The plot revealed there is great variability in the accident severity classes. Indicating there were more reported cases of slight accident with 82.86%, and serious and fatal with 15.90% and 1.24% respectively. This facilitated the feature selection by studying the different factors that attribute to each category of accident severity.</p><h4>Data Preprocessing</h4><p>After exploring the data, the next step was to preprocess it before using it to develop a predictive model. The data preprocessing steps carried out were:</p><blockquote>Data cleaning</blockquote><p>After running different exploratory data analysis, the irrelevant columns including features with high percentage of missing values were dropped.</p><blockquote>Feature engineering and transformation</blockquote><p>Feature engineering was carried out on the date column by converting it to a datetime format in order to extract the years and months columns separately, this will help facilitate an in depth understanding of variations in accident occurrences across the different time stamp and as well enhance accurate model predictions with respect to time.</p><p>The cleaned data sets were merged and was standardized using the MinMax scaler. This will help bring all features to a common scale thereby controlling the model bias tradeoffs.</p><h4>Model Implementation</h4><p>The different models implemented were:</p><blockquote>1. Random forest classifier</blockquote><blockquote>2. Lightgbm classifier</blockquote><blockquote>3. Xgboost classifier</blockquote><blockquote>4. ExtraTrees classifier</blockquote><h4>Metric Result Evaluation</h4><p>The metrics for evaluation were: precision score, recall score, accuracy score and F1 score.</p><blockquote>Observations :</blockquote><blockquote>Following the different models fitted, the Random Forest and ExtraTress classifiers had the best F1_scores of 0.8475 and 0.8467 respectively.</blockquote><blockquote>The feature importance for the random forest classifier is given below.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*CerEkuCE_X7t1bFnIta__w.jpeg" /></figure><h4>Conclusion</h4><p>The model is reliable enough to predict severity in an accident occurrence. This approach can be used as a directive to provide strategies on how to prevent or reduce severe incidents.</p><p>You can access the project on <a href="https://github.com/judeleonard/stage-f-01-road-safety">GitHub</a></p><p><em>Special thanks to all contributors of this work, both in the previous and my current team (as at the time of this writing).</em></p><h4>References</h4><ol><li>Peden M, Scurfield R, Sleet D, Mohan D, Hyder AA, Jarawan E, et al. World report on road traffic injury prevention. Geneva: World Health Organization (WHO). 2004. [cited 2019 November 4]. Available from URL: <a href="https://www.who.int/violence_injury_prevention/publications/road_traffic/world_report/en/">https://www.who.int/violence_injury_prevention/publications/road_traffic/world_report/en/</a></li><li><a href="https://www.insurethebox.com/author/insurethebox/">insurethebox</a>, January 12, 2017. Available from URL: <a href="https://www.insurethebox.com/wp-content/uploads/2017/01/ITB17.01-Winter-Driving-tips-final.pdf">https://www.insurethebox.com/wp-content/uploads/2017/01/ITB17.01-Winter-Driving-tips-final.pdf</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2aaca2bd6d8" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>