<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Elias Dogbatsey on Medium]]></title>
        <description><![CDATA[Stories by Elias Dogbatsey on Medium]]></description>
        <link>https://medium.com/@dogbatseyelias73?source=rss-c03308032e99------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*BlcV1rIQr3ytZ7humb8XxA.jpeg</url>
            <title>Stories by Elias Dogbatsey on Medium</title>
            <link>https://medium.com/@dogbatseyelias73?source=rss-c03308032e99------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 16:53:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dogbatseyelias73/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Recommendation System Analysis & Modeling]]></title>
            <link>https://medium.com/@dogbatseyelias73/recommendation-system-analysis-modeling-c89635c73b3b?source=rss-c03308032e99------2</link>
            <guid isPermaLink="false">https://medium.com/p/c89635c73b3b</guid>
            <category><![CDATA[hybrid-system]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <dc:creator><![CDATA[Elias Dogbatsey]]></dc:creator>
            <pubDate>Tue, 15 Apr 2025 09:29:00 GMT</pubDate>
            <atom:updated>2025-04-15T09:29:00.843Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*da4Z40M7sZXqkIDsHu_0rQ.jpeg" /></figure><h3>Introduction:</h3><p>Recommendation systems are the driving force behind personalized experiences in e-commerce, media streaming, and social platforms. From Netflix suggesting your next binge-watch to Amazon recommending products you might like, these systems enhance user engagement and satisfaction.</p><p>As part of my Recommendation System Analysis &amp; Modeling project, I am developing a system that delivers personalized recommendations based on user behavior. This journal covers the first two weeks of my progress, including project setup, data exploration, preprocessing, and feature engineering.</p><h3>Week 1: Understanding the Dataset &amp; Setting Up the Project</h3><h4><strong>Project Scope</strong></h4><p>The goal was to build a recommendation system using historical user behavior to provide personalized recommendations. The system had to:<br>1. Predict relevant products for users based on past interactions.<br>2. handle large-scale e-commerce data efficiently.<br>3. Implement both CF and CBF approaches to improve accuracy.</p><p><strong>Exploring the Dataset</strong></p><p>The dataset consisted of three files:<br>1. <strong>events.csv: </strong>Contains user interactions (view, add to cart, purchase).<br>2. <strong>item_properties.csv:</strong> Stores item details (category, availability, price).<br>3. <strong>Category_tree.csv</strong>: represents the item category hierarchy.</p><p>Key Observations:</p><ul><li>Over 2.7M user interactions with 1.4M unique visitors.</li><li>20M+ item property records, updated weekly.</li><li>Hierarchical product categories are useful for content-based filtering.</li></ul><h4>Project Setup</h4><p>Created a GitHub repository and set up the development environment with:<br>Python (Pandas, NumPy, Scikit-Learn, Surprise) for analysis and modeling.<br>Jupyter Notebook for interactive data exploration.<br>Google Colab for large dataset processing.</p><p><strong>To guide the analysis, seven key questions were formulated:</strong><br>1. What are the most popular items among users?<br>2. How does user behavior impact recommendations?<br>3. What factors influence user engagement?<br>4. How can we balance recommendation accuracy &amp; diversity?<br>5. How do different recommendation models compare in performance?<br>6. What is the impact of seasonality on recommendations?<br>7. How can we optimize recommendation efficiency for real-time performance?</p><h3>Week 2: Data Cleaning &amp; Feature Engineering</h3><p>The item_properties.csv dataset contained timestamps stored in Unix epoch format (milliseconds). To make them readable, I converted them to datetime format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fj4bE_m3q1bmOI-01jvBvw.png" /></figure><p>The item_properties.csv file contains a property column with different types of attributes stored in a single column. Some important attributes include:</p><ul><li><strong>categoryid</strong>: Represents the category of the item.</li><li><strong>Availability: </strong>Indicates whether the item was in stock (1 = Available, 0 = Not Available, and ‘Unknown’).</li></ul><ol><li>To extract categoryid, I filtered only the rows where property == &#39;categoryid&#39; and stored the corresponding values. And converted the values to numeric</li></ol><pre>item_properties[&#39;category_id&#39;] = item_properties.apply(lambda x: x[&#39;value&#39;] if x[&#39;property&#39;] == &#39;categoryid&#39; else None, axis=1)<br><br>item_properties[&#39;category_id&#39;] = pd.to_numeric(item_properties[&#39;category_id&#39;], errors=&#39;coerce&#39;)</pre><p>2. To extract ‘Availability’, I checked where property == &#39;available&#39; and mapped values accordingly.</p><pre>item_properties[&#39;Availability&#39;] = &#39;Unknown&#39;  # Default value<br>item_properties.loc[(item_properties[&#39;property&#39;] == &#39;available&#39;) &amp; (item_properties[&#39;value&#39;] == &#39;1&#39;), &#39;Availability&#39;] = &#39;Available&#39;<br>item_properties.loc[(item_properties[&#39;property&#39;] == &#39;available&#39;) &amp; (item_properties[&#39;value&#39;] == &#39;0&#39;), &#39;Availability&#39;] = &#39;Not Available&#39;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KkGBVJb40u7JAZwmQZz5QA.png" /><figcaption>Output after doing extractions</figcaption></figure><h3>Week 3: Merging and further Data Cleaning</h3><h4>Merging large datasets without crashing</h4><p>Google Colab kept crashing while merging 20M+ item properties with 2.7M events.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*zS7HzPV2s7u0rPTv8i_ZvA.png" /></figure><p>Switched to GPU as a runtime type but still faced the same issue, so I came up with two solutions to address this:</p><ol><li>Reduce ‘item_properties’ dataset size before merging</li></ol><pre>item_properties_filtered = item_properties[item_properties[&#39;property&#39;].isin([&#39;categoryid&#39;, &#39;available&#39;])]<br>item_properties_pivot = item_properties_filtered.pivot(index=&#39;item_id&#39;, columns=&#39;property&#39;, values=&#39;value&#39;).reset_index()</pre><p>2. Use batch processing for merging</p><pre>batch_size = 500000  <br>merged_batches = []  <br><br>for batch in np.array_split(events_cleaned, len(events_cleaned) // batch_size + 1):  <br>    batch_merged = pd.merge(batch, item_properties_pivot, on=&#39;item_id&#39;, how=&#39;left&#39;)  <br>    merged_batches.append(batch_merged)<br><br>final_data = pd.concat(merged_batches, ignore_index=True)</pre><p>This approach prevented memory overload and sped up processing.</p><h4>Further data cleaning</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ApMEQA6vfNDgczVustwhJA.png" /></figure><h4>Detecting and removing abnormal Users (Possibly bots)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/1*EFUDpvgttMZJ5YNr66dFjw.png" /></figure><h4>Building the Baseline Recommendation Model</h4><p>With the merged dataset ready, I built a baseline recommendation system using popularity-based recommendations.</p><ol><li><strong>Define Interaction Scores for Events</strong><br>To rank items effectively, I assigned scores to different interaction types:<br>view: 1 point<br>add to cart: 2 points<br>purchase: 5 points</li></ol><pre>event_type_mapping = {&#39;view&#39;: 1, &#39;addtocart&#39;: 3, &#39;transaction&#39;: 5}<br>events_merged[&#39;event_score&#39;] = events_merged[&#39;event_type&#39;].map(event_type_mapping)</pre><p>2. <strong>Identify Most Popular Items by Interaction Score<br></strong>Using interaction scores, I identified top trending items for new users.</p><pre># Compute item popularity based on total interaction score<br>popular_items = events_merged.groupby(&#39;item_id&#39;)[&#39;event_score&#39;].sum().reset_index()<br><br># Sort items by popularity<br>popular_items = popular_items.sort_values(by=&#39;event_score&#39;, ascending=False)<br><br>print(popular_items.head(10))  # Display top trending products</pre><h3>Week 4: Building and Implementing the Hybrid Recommendation System</h3><p>With the merged dataset ready, I built a baseline recommendation system using popularity-based recommendations.</p><p>To rank items effectively, I assigned scores to different interaction types:</p><p>view = 1</p><p>add to cart = 3</p><p>purchase = 5</p><pre>event_type_mapping = {&#39;view&#39;: 1, &#39;addtocart&#39;: 3, &#39;transaction&#39;: 5}<br>events_merged[&#39;event_score&#39;] = events_merged[&#39;event_type&#39;].map(event_type_mapping)</pre><h4>Identifying Most Popular Items by Interaction Score</h4><p>Using interactions scores, I identified top trending items for news.</p><pre># Computing item popularity based on total interaction score<br>popular_items = events_merged.groupby(&#39;item_id&#39;)[&#39;event_score&#39;].sum().reset_index()<br><br># Sorting items by popularity<br>popular_items = popular_items.sort_values(by=&#39;event_score&#39;, ascending=False)<br><br>print(popular_items.head(10))</pre><p>This recommends trending products for new users with no history. Also, it is efficient as a fallback model when CF or CBF cannot provide results.</p><p>With a cleaned and optimized dataset, the next step was to build the hybrid recommendation system by:</p><p><strong>Collaborative Filtering (CF):</strong> Used Matrix Factorization (SVD) to predict user preferences.<br> <strong>Content-Based Filtering (CBF):</strong> Computed cosine similarity between item categories.<br> <strong>Hybrid Approach:</strong> Combined CF &amp; CBF for a more accurate model.</p><pre>def hybrid_recommend(user_id, top_n=5):<br>    # CF-based recommendations<br>    cf_preds = model.predict(user_id, item_id) for item_id in events_merged[&#39;item_id&#39;]<br>    cf_top = sorted(cf_preds, key=lambda x: x.est, reverse=True)[:top_n]<br><br>    # CBF-based recommendations<br>    user_items = events_merged[events_merged[&#39;user_id&#39;] == user_id][&#39;item_id&#39;].unique()<br>    cb_scores = similarity_matrix[user_items].mean(axis=0)<br>    cb_top = events_merged.iloc[cb_scores.argsort()[::-1][:top_n]]<br><br>    # Combine CF &amp; CBF results<br>    hybrid_results = pd.concat([cf_top, cb_top]).drop_duplicates().head(top_n)<br>    return hybrid_results</pre><pre>from surprise import SVD, Dataset, Reader<br>from surprise.model_selection import train_test_split<br><br>reader = Reader(rating_scale=(1, 5))<br>data = Dataset.load_from_df(final_data[[&#39;user_id&#39;, &#39;item_id&#39;, &#39;event_score&#39;]], reader)<br><br># Training CF model<br>trainset, testset = train_test_split(data, test_size=0.2)<br>model = SVD()<br>model.fit(trainset)<br><br># Making predictions<br>predictions = model.test(testset)</pre><h4><strong>Content-Based Filtering (Item Similarity)</strong></h4><p>Used TF-IDF &amp; Cosine Similarity on category_id and popularity score.</p><pre>from sklearn.feature_extraction.text import TfidfVectorizer<br>from sklearn.metrics.pairwise import cosine_similarity<br><br># Preparing category &amp; popularity-based features<br>final_data[&#39;combined_features&#39;] = final_data[&#39;category_id&#39;].astype(str) + &#39; &#39; + final_data[&#39;item_popularity&#39;].astype(str)<br><br># Computing TF-IDF similarity<br>tfidf = TfidfVectorizer()<br>tfidf_matrix = tfidf.fit_transform(final_data[&#39;combined_features&#39;])<br>similarity_matrix = cosine_similarity(tfidf_matrix)</pre><h4>Hybrid Recommendation System</h4><p>The hybrid approach combines CF &amp; CBF:<br>CF predicts what users like based on similar users.<br>CBF recommends items with similar properties.<br>Blended scores improve accuracy &amp; diversity.</p><pre>def hybrid_recommend(user_id, top_n=5):<br>    # CF-based recommendations<br>    cf_preds = model.predict(user_id, item_id) for item_id in final_data[&#39;item_id&#39;]<br>    cf_top = sorted(cf_preds, key=lambda x: x.est, reverse=True)[:top_n]<br><br>    # CBF-based recommendations<br>    user_items = final_data[final_data[&#39;user_id&#39;] == user_id][&#39;item_id&#39;].unique()<br>    cb_scores = similarity_matrix[user_items].mean(axis=0)<br>    cb_top = final_data.iloc[cb_scores.argsort()[::-1][:top_n]]<br><br>    # Combine CF &amp; CBF results<br>    hybrid_results = pd.concat([cf_top, cb_top]).drop_duplicates().head(top_n)<br>    return hybrid_results</pre><h3>Evaluation Metrics &amp; Results</h3><p>The models were evaluated using:</p><ol><li><strong>Root Mean Squared Error (RMSE):</strong> Measures prediction accuracy.</li><li><strong>Precision &amp; Recall: </strong>Evaluates recommendation relevance.</li><li><strong>Diversity Score: </strong>Ensures recommendations are varied.</li></ol><h3>Results Summary</h3><ul><li><strong>CF (SVD) Model:</strong> RMSE = 0.85, Precision = 78%</li><li><strong>CBF Model:</strong> Precision = 74%, Recall = 68%</li><li><strong>Hybrid Model:</strong> Precision = 82<strong>%</strong>, Recall = 75<strong>%</strong></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c89635c73b3b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Environmental Impact of Food Production]]></title>
            <link>https://medium.com/@dogbatseyelias73/environmental-impact-of-food-production-5fc361b0fadb?source=rss-c03308032e99------2</link>
            <guid isPermaLink="false">https://medium.com/p/5fc361b0fadb</guid>
            <dc:creator><![CDATA[Elias Dogbatsey]]></dc:creator>
            <pubDate>Wed, 09 Apr 2025 12:59:53 GMT</pubDate>
            <atom:updated>2025-04-09T12:59:53.693Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SIyfFUUvKfSv75hOP35fww.jpeg" /></figure><h4>Introduction</h4><p>Food production is one of the biggest contributors to climate change, deforestation, and water scarcity. Understanding its environmental impact is essential for making informed choices about food sustainability. In this project, I used data science techniques to analyze the carbon footprint, land use, and water consumption of different food products.</p><p>This article documents my four-week journey, from setting up the project to uncovering insights and visualizing results.</p><h3>Week 1: Project Planning &amp; Setup</h3><h4>Defining the Problem</h4><p>The goal of this project was to analyze the environmental footprint of food production using data science. I sought to answer key business questions, such as:</p><ol><li>Which food products have the highest and lowest carbon footprints</li><li>How does land use vary across food products?</li><li>Which food categories contribute most to eutrophication?</li><li>What is the relationship between water use and greenhouse gas emissions?</li><li>How do plant-based and animal-based foods compare in terms of environmental impact?</li><li>Are there trade-offs between environmental metrics (e.g., high land use vs. low water use)?</li><li>What recommendations can be made for sustainable food choices?</li></ol><h4>Setting Up the GitHub Repository</h4><p>I created a GitHub repository to store all project-related files, including:</p><ul><li>A README.md file outlining objectives, methodology, and tools used.</li><li>Python scripts for data cleaning, analysis, and visualization.</li><li>Data visualizations illustrating key insights.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MCWyY7usPKIcxgp1V5YCQQ.png" /></figure><p><strong>Tools Selected:</strong><br> <strong>Python (pandas, seaborn, matplotlib, plotly):</strong> For data analysis &amp; visualization.<br> <strong>Jupyter Notebook:</strong> For interactive coding.<br> <strong>GitHub:</strong> For version control and documentation.<br> <strong>Power BI:</strong> For interactive dashboard.</p><h3>Week 2: Data Exploration &amp; Cleaning</h3><h4>Understanding the Dataset</h4><p>The dataset contained 43 food products and their 23 environmental impact metrics, including:</p><ul><li>Greenhouse gas emissions (kg CO₂ per kg product)</li><li>Land use (m² per kg product)</li><li>Water usage (liters per kg product)</li><li>Eutrophication impacts (nutrient pollution per kg product)</li></ul><h4>Data Cleaning Challenges and Solutions</h4><ol><li><strong>Missing Values</strong></li></ol><ul><li>Some columns had more than 35% missing data (e.g., eutrophication per 100g protein).</li><li><strong>Solution</strong>: Dropped columns with a lot of missing values and used median imputation for the rest.</li></ul><pre>threshold = len(df) * 0.65<br>df_cleaned = df.loc[:, df.isnull().mean() &lt; 0.35].copy()<br>print(f&quot;Dropped columns: {set(df.columns) - set(df_cleaned.columns)}&quot;)</pre><p><strong>2. Inconsistent Groupings</strong></p><ul><li>Some foods (e.g., “Pig Meat”) were incorrectly labeled as “Plant-based.”</li><li><strong>Solution</strong>: Manually corrected misclassifications.</li></ul><p><strong>3. Negative Values</strong></p><ul><li>Some foods (e.g., “Nuts”) had negative Land use change values (indicating carbon sequestration).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*0xl5AfG9SbM0MJ4QGHaTOQ.png" /></figure><ul><li><strong>Solution</strong>: Verified with domain knowledge since I already have a background in Natural Resources, and retained them.</li></ul><p>Renamed the ‘Packging’ column to ‘Packaging’</p><pre>df_cleaned.rename(columns={&#39;Packging&#39;: &#39;Packaging&#39;}, inplace=True)</pre><h4>Key Findings from EDA</h4><ul><li><strong>Highest Emissions</strong>: Beef herd (59.6 kg CO₂eq/kg), Lamb and Mutton (24.5), and Cheese (21.2).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sX_Pti4bNaMVqS7tBAjLEw.png" /></figure><ul><li><strong>Lowest Emissions</strong>: Peas (0.8), potatoes (0.3), and bananas (0.8).</li><li><strong>Water Hogs</strong>: Nuts (4,133 L/kg), beef (1,451 L/kg).</li><li><strong>Land Hogs</strong>: Lamb (369 m²/kg), beef (326 m²/kg).</li></ul><h3>Week 3: Modeling</h3><h4>Clustering Foods by Environmental Impact</h4><p>Used K-Means clustering to group foods into Low, Moderate, and High environmental impact groups based on their emissions, water use, land use, and eutrophication potential.</p><p>The data was scaled to ensure equal weighting.</p><pre>import pandas as pd<br>from sklearn.preprocessing import StandardScaler<br>from sklearn.cluster import KMeans<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br><br>features = [<br>    &#39;Total_emissions&#39;,<br>    &#39;Freshwater withdrawals per kilogram (liters per kilogram)&#39;,<br>    &#39;Land use per kilogram (m² per kilogram)&#39;,<br>    &#39;Eutrophying emissions per kilogram (gPO₄eq per kilogram)&#39;<br>]<br>X = data[features]<br>scaler = StandardScaler()<br>X_scaled = scaler.fit_transform(X)</pre><p>The elbow method was used to determine the optimal clusters, 3.</p><pre>wcss = []<br>for i in range(1, 11):<br>    kmeans = KMeans(n_clusters=i, random_state=42)<br>    kmeans.fit(X_scaled)<br>    wcss.append(kmeans.inertia_)<br><br>plt.plot(range(1, 11), wcss, marker=&#39;o&#39;)<br>plt.title(&#39;Elbow Method&#39;)<br>plt.xlabel(&#39;Number of Clusters&#39;)<br>plt.ylabel(&#39;WCSS (Within-Cluster Sum of Squares)&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*fEMBf5JBWnXOloV2iqJrkw.png" /><figcaption>The elbow occurs at 3 clusters (low/moderate/high impact).</figcaption></figure><p>Applied K-Means clustering to group foods based on their environmental profiles.</p><pre>kmeans = KMeans(n_clusters=3, random_state=42)<br>data[&#39;Impact_Cluster&#39;] = kmeans.fit_predict(X_scaled)<br><br>cluster_labels = {<br>    0: &#39;Low-impact&#39;,<br>    1: &#39;Moderate-impact&#39;,<br>    2: &#39;High-impact&#39;<br>}<br>data[&#39;Impact_Label&#39;] = data[&#39;Impact_Cluster&#39;].map(cluster_labels)</pre><p>Results from the three food clusters</p><ol><li>Low-impact foods.</li></ol><p>Examples: Peas, potatoes, bananas, apples, wheat, and rye. They have low impact on the environment because they have low water and land requirements and also a minimal carbon footprint. These foods must be prioritized because they are eco-friendly.</p><p>2. Moderate-impact foods.</p><p>Examples: Nuts, rice, dark chocolate, soybean oil. Even though they have low emissions, they are extremely water-intensive. Also, dark chocolate uses a lot of land in terms of production due to cocoa farming.</p><p>3. High-impact foods.</p><p>Examples: Beef, lamb, cheese, pork, farmed. This can be confirmed in the EDA report, where these foods are part of the top 10 emmitors. People must reduce the consumption of these foods.</p><p>In all, this clustering analysis makes it clear that small dietary shifts can have massive environmental benefits. By choosing low-impact foods, we can reduce our carbon footprint, conserve water, and protect ecosystems.</p><h4>Predicting total emissions based on other environmental factors using a random forest regressor</h4><pre>from sklearn.ensemble import RandomForestRegressor<br>from sklearn.model_selection import train_test_split<br>from sklearn.metrics import r2_score, mean_squared_error<br>from sklearn.preprocessing import OneHotEncoder<br>from sklearn.compose import ColumnTransformer<br><br>data[&#39;is_animal&#39;] = data[&#39;Food_Group&#39;].apply(lambda x: 1 if x == &#39;Animal&#39; else 0)<br><br>features = [<br>    &#39;is_animal&#39;,  <br>    &#39;Land use per kilogram (m² per kilogram)&#39;,<br>    &#39;Freshwater withdrawals per kilogram (liters per kilogram)&#39;,<br>    &#39;Eutrophying emissions per kilogram (gPO₄eq per kilogram)&#39;<br>]<br>X = data[features]<br>y = data[&#39;Total_emissions&#39;]<br><br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)<br><br>rf = RandomForestRegressor(n_estimators=100, random_state=42)<br>rf.fit(X_train, y_train)<br><br>y_pred = rf.predict(X_test)<br>print(f&quot;R²: {r2_score(y_test, y_pred):.2f}&quot;) <br>print(f&quot;RMSE: {np.sqrt(mean_squared_error(y_test, y_pred)):.2f}&quot;)<br><br>importance = pd.DataFrame({<br>    &#39;Feature&#39;: features,<br>    &#39;Importance&#39;: rf.feature_importances_<br>}).sort_values(&#39;Importance&#39;, ascending=False)<br>print(&quot;\nFeature Importance:&quot;)<br>print(importance)</pre><h4>Results:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/1*o_jdifzydwE81vGfrUH1Cg.png" /></figure><p>R² Score: 0.60</p><p>RMSE: 4.97</p><h3>Week 4: Documentation and Finalization</h3><h4>Power BI Dashboard</h4><p>Created an interactive dashboard with:<br>✔ Filters (Plant/Animal, Emissions/Water/Land).<br>✔ Top Emitters bar chart.<br>✔ Lifecycle Impact treemap.<br>✔ Water vs. Land Trade-offs scatter plot.</p><h4>Project Repository</h4><p>All codes, datasets, and the Power BI dashboard are available on GitHub.</p><h3>Conclusion</h3><p>This project revealed clear disparities in food sustainability, with animal products being the worst offenders. By making data-driven choices, consumers and policymakers can significantly reduce environmental impact.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5fc361b0fadb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transborder Freight Data Analysis]]></title>
            <link>https://medium.com/@dogbatseyelias73/transborder-freight-data-analysis-35cdc125f3c9?source=rss-c03308032e99------2</link>
            <guid isPermaLink="false">https://medium.com/p/35cdc125f3c9</guid>
            <dc:creator><![CDATA[Elias Dogbatsey]]></dc:creator>
            <pubDate>Fri, 14 Feb 2025 17:56:56 GMT</pubDate>
            <atom:updated>2025-03-27T11:04:52.543Z</atom:updated>
            <content:encoded><![CDATA[<h3>About Project:</h3><p>I worked on the Transborder Freight Data Analysis Project to analyze the transportation data provided by the Bureau of Transportation Statistics (BTS) to uncover insights on freight transportation&#39;s efficiency, safety, and environmental impacts across various modes (road, rail, air, and water). Also, the project employed the CRISP-D framework, ensuring a structured, analytical approach to derive meaningful insights from the data.</p><h3>Week 1: Project Setup and Understanding the Data</h3><p>The first week focused on laying the foundation for the project. This includes:</p><p>Setting up a GitHub repository to store all project files, including scripts, documentation, and progress updates, shown by the daily commits.</p><p>I identified and documented my preferred tools for the analysis, including Python (in Google Colabs), Power BI, and Excel.</p><p>Understanding the dataset was crucial before proceeding, so I studied the BTS datasets to know the best consolidation method to use, how to handle missing values and other data preprocessing methods that may be needed for the analysis.</p><p>To guide my analysis, I formulated key analytical questions such as:</p><ol><li>Which transportation mode is most commonly used for cross-border freight movement?</li><li>How does freight transportation volume vary across different months?</li><li>Which transportation mode has the lowest cost per ton-mile?</li><li>Where do the most frequent delays occur in freight transportation?</li><li>What are the busiest freight routes based on shipment weight?</li><li>Which country contributes the most to cross-border freight movement?</li><li>How does the trade type impact freight transportation volume?</li></ol><h3><strong>Week 2: Data Cleaning and Preprocessing:</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1aht1VMacr5UOoA_nWe3UQ.png" /><figcaption>Data consolidation in Google Colab</figcaption></figure><p>The BTS data was available in multiple files for each month yearly, requiring integration. I merged different data from each to get a unified dataset for that year, then merged all yearly datasets (2020 — 2024) to get a unified dataset for analysis. I used the <strong>‘d.concat()’</strong> method to merge all datasets, as shown below.</p><pre># Concatenated dataset for all years<br>final_dataset = pd.concat([m1, m2, m3, m4, m5])<br>final_dataset.head(5)<br><br># Where m1, m2, m3, m4, and m5 represent the dataframes for the years.</pre><p>During the data cleaning and preprocessing phase, I used contextual filling to handle missing values and inconsistencies in a way that maintains the integrity of the data. Instead of dropping rows with missing values, I filled in missing data using relevant context. For example, if a particular freight shipment was missing the mode of transportation, I inferred it based on similar shipments from the same country or with the same trade type. Instead of leaving missing categorical values <strong>(like DF) </strong>blank, I assigned the most frequent value in that category. By employing contextual filling, the dataset remained complete and useful for analysis, and no unnecessary data was lost due to missing values.</p><p>I used Power Query in Power BI to do some of the conversions. First, the data type is changed to <strong>‘object,’ and </strong>then the mapping codes in the dataset are added to the descriptive labels in the data dictionary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ls7gOOoMHyofc8g2eXXCfA.png" /></figure><p>The picture below shows a summary of missing values in the data frame.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g7Yyl4CTFZGiX3gb2AQtJg.png" /></figure><p>I conducted an initial assessment of the dataset, identifying key patterns. Certain Canadian provinces and U.S. states rely more on rail transport, while others prefer trucking routes.</p><p>I identified and analyzed missing values, and I found missing values in key columns:</p><ol><li><strong>U.S. State</strong> <strong>(14.04%) </strong>- I filled in the missing values using mode (most frequent value) per Mexican State and Canadian Province.</li><li><strong>Port/District Name (62.93%)</strong> -Filled missing values using mode for the respective U.S. State</li><li><strong>Mexican State</strong> <strong>(71.77%)</strong> -Replaced “XX” with “OT” (Unknown).</li><li><strong>Canadian Province</strong> <strong>(47.86%)</strong>-</li><li><strong>Commodity Classification (23.02%)</strong> — Filled using mode within <strong>U.S. State, Mexican State, and Canadian Province </strong>columns.</li></ol><p>I then implemented data handling strategies using mode imputation for categorical columns (Commodity Classification, U.S. State, and Port/District Name). I fixed inconsistent category labels ( “Packaging Method” had an unexpected ‘1’ value, “Mexican State” had an expected “XX” value). Also, I ensured uniform representation of trade types, transportation modes, and location names.</p><p>After cleaning and contextual filling, I did a final data validation process to ensure the dataset was in the right structure for analysis.</p><h3>3. Exploratory Data Analysis (EDA) and visualization</h3><p>Having cleaned the data, I used Matplotlib and Seaborn to create visualizations of freight trends, congestion points, and other key insights about freight transportation.</p><p><strong>Analytical Question 1: </strong>Which transportation mode is most commonly used for cross-border freight movement?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/1*q9DJ3qT64Qmhl5QdproLhA.png" /></figure><p>A <strong>bar chart </strong>showing the frequency of each mode of transportation revealed:</p><ul><li>🚛 Trucking is the dominant mode, accounting for over 60% of freight movement.</li><li>🚆 Rail transport is the second most used, preferred for heavy shipments.</li><li>✈ Air freight is used less frequently, mainly for high-value, time-sensitive goods.</li></ul><p><strong>Key Takeaway:</strong> Road and rail transport handle the majority of cross-border freight movement due to their cost-effectiveness and accessibility.</p><p><strong>Analytical Question 2:</strong> How does freight transportation volume vary across different months?</p><p>To identify seasonal trends, I analyzed monthly variations in shipment weight.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*PfGhvZbMn_ykIRxnYQWwpw.png" /></figure><p>Peak trade activity occurs in Q2 and A4, aligning with major supply chain cycles and holiday-driven demand. However, Trade volumes dipped in Q1, which could be due to reduced production. And Imports show higher variation, while Export remains relatively stable.</p><p><strong>Key takeaway: </strong>Businesses should optimize freight operations around seasonal demand fluctuations.</p><p><strong>Analytical Question 3: </strong>Which transportation mode has the lowest cost per ton-mile?</p><p>I compared freight charges relative to shipment weight to determine the most cost-effective transportation method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/1*fxGqiyKirX3t5Jn9pdQBZQ.png" /></figure><p>Rail, Vessels, and Foreign Trade Zones (FTZs) have mid-range costs, offering flexibility but higher fuel expenses.</p><p>The pipeline is the most expensive, with a significantly higher cost per ton-mile.</p><p><strong>Key takeaway: </strong>For major cost-effective transportation, Mail (U.S. Postal Service) is the preferred mode, while trucks remain dominant for shorter routes.</p><p><strong>Analytical Question 4: </strong>Where do the most frequent delays occur in freight transportation?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*G2g_GtU7wK6VvzwCI_V-dg.png" /></figure><p>Certain ports experience higher delays due to congestion or high shipment volume.</p><p><strong>Analytical Question 5: </strong>What are the busiest freight routes based on shipment weight?</p><p>To understand trade flow, I analyzed the top 10 freight routes by total weight shipped.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U1eaObn3ohDaNHYKzoKY3Q.png" /></figure><p>The U.S-Canada border dominates freight movement, particularly Illinois-Canada trade routes. Since Illinois borders the Great Lakes, on which international commerce comes from Canada, water transport is more active on this route.</p><p><strong>Analytical Question 6: </strong>Which country contributes the most to cross-border freight movement?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*0gRFWGvyKBSUvkRmO-tONw.png" /></figure><p>The bar chart illustrates the total number of shipments between the U.S. and its two major trade partners. The objective of this analytical question is to understand which country contributes more to cross-border freight movement.</p><p>Canada has approximately 4 million shipments, which is significantly higher than Mexico’s 2.6 million shipments. This suggests a stronger trade relationship between the U.S. and Canada compared to Mexico. Low shipment frequency for U.S.-Mexico trade could be a result of strict border control and customs regulations.</p><p><strong>Analytical Question 7: </strong>How does the trade type impact freight transportation volume?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*aLVtudABpjyuj5sDScco8A.png" /></figure><p>This stacked bar chart visualizes the impact of trade type (Exports vs. Imports) on freight transportation volume across different modes of transportation. The goal is to analyze which transportation modes are dominant for exports and imports.</p><p><strong>Insights: </strong>Truck shipments are the most frequently used transportation mode for both exports and imports. The number of truck shipments for exports is higher than for imports, suggesting a stronger U.S. outbound logistic network.</p><p>FTZ Shipments appear only in small numbers, as these zones serve specific trade regulations. While, Mail (U.S. Postal Service) is an insignificant contributor, indicating that most freight trade involves bulk or large shipments rather than postal services.</p><h3>Week 4: Project Journal Update</h3><p>Here, I finalized the PowerPoint Presentation for the project by structuring slides to present key findings, visualizations, and recommendations, and also incorporated insights from EDA<strong> </strong>and Power BI dashboards.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ifqg_yiBjO-KQN_Q_sfSNA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=35cdc125f3c9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>