<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by DropletOfData on Medium]]></title>
        <description><![CDATA[Stories by DropletOfData on Medium]]></description>
        <link>https://medium.com/@DropletOfData?source=rss-d2bd81983783------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*QnE6isQ_zRaEO-ZV09JbrA.png</url>
            <title>Stories by DropletOfData on Medium</title>
            <link>https://medium.com/@DropletOfData?source=rss-d2bd81983783------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 01:30:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@DropletOfData/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From Correlation to Causation: Modeling Appliance Energy Consumption with Causal ML]]></title>
            <link>https://medium.com/@DropletOfData/from-correlation-to-causation-modeling-appliance-energy-consumption-with-causal-ml-7e361747d156?source=rss-d2bd81983783------2</link>
            <guid isPermaLink="false">https://medium.com/p/7e361747d156</guid>
            <category><![CDATA[explainable-ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[causal-inference]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[energy-analytics]]></category>
            <dc:creator><![CDATA[DropletOfData]]></dc:creator>
            <pubDate>Tue, 23 Dec 2025 15:29:44 GMT</pubDate>
            <atom:updated>2025-12-23T15:29:44.661Z</atom:updated>
            <content:encoded><![CDATA[<p>Most energy forecasting systems optimize for accuracy and stop there. That’s a problem.</p><p>High-performing black-box models can tell us <em>what</em> will happen, but not <em>why</em> — which makes them fragile, hard to trust, and nearly impossible to act on. This study tackles that gap head-on by integrating <strong>causal inference</strong>, <strong>interpretable ML</strong>, and <strong>time-series modeling</strong> to answer a more useful question:</p><blockquote>Which environmental factors actually <strong>cause</strong> changes in appliance energy consumption — and how can we exploit that causality to build better models?</blockquote><p>This post summarizes a full dissertation that systematically moves from correlation-heavy baselines to causally grounded, deployable forecasting systems .</p><h3>1. Problem Setup and Dataset</h3><p>The analysis uses the <strong>Appliances Energy Prediction dataset</strong> from the UCI ML Repository:</p><ul><li>19,735 observations</li><li>10-minute resolution</li><li>~4.5 months of data</li><li>Indoor temperature &amp; humidity (multiple rooms)</li><li>Outdoor weather variables</li><li>Target: <strong>appliance energy consumption (Wh)</strong></li></ul><p>This dataset is deceptively rich: it mixes <strong>highly correlated indoor sensors</strong>, <strong>exogenous weather signals</strong>, and <strong>human behavior proxies (lights)</strong> — a perfect storm for spurious correlations.</p><h3>2. Exploratory Analysis: Patterns Are Not Causes</h3><h3>Temporal structure</h3><p>Energy usage shows:</p><ul><li>Strong <strong>daily seasonality</strong></li><li>Consistent <strong>evening peaks</strong></li><li>Lower overnight consumption</li></ul><p>Stationarity checks (ADF statistic ≈ −21.6, p ≪ 0.001) confirm the series is suitable for <strong>Granger causality, VAR, and SARIMAX</strong> modeling.</p><h3>Correlation structure</h3><p>A Pearson heatmap reveals:</p><ul><li><strong>Lights</strong> strongly correlated with appliance usage</li><li>Indoor temperature &amp; humidity highly correlated <em>with each other</em></li><li>Weak direct correlation between many environmental variables and the target</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q3tjyoxegRFe1jOyq_THoQ.png" /></figure><p><strong>Key takeaway:</strong> correlation alone is insufficient. Many features look important but may simply be co-moving.</p><h3>3. Baseline Models: Accuracy Without Insight</h3><p>Three standard models establish a reference point:</p><p>Model, R², MSE</p><p>Ridge Regression, <strong>0.086, 7573.76</strong></p><p>LSTM, 0.056, 7824.56</p><p>XGBoost, -0.443, 11958.13</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iZnl04UiLSxrr2vxkh-JnA.png" /></figure><p>Surprisingly, <strong>Ridge outperforms both XGBoost and LSTM</strong>. The reason is structural:</p><ul><li>High multicollinearity</li><li>Noisy features</li><li>Limited true signal</li></ul><p>This already hints that <strong>feature quality matters more than model complexity</strong>.</p><h3>4. Moving Toward Causality</h3><h3>4.1 Granger causality (temporal predictiveness)</h3><p>Granger tests identify which variables improve forecasts when their <em>past values</em> are included.</p><p>Significant (p &lt; 0.05):</p><ul><li>Indoor humidity (RH₁, RH₂, RH₃, …)</li><li>Indoor temperature (T₂, T₃, T₆, T₈)</li><li>Lights</li><li>Outdoor temperature &amp; pressure</li></ul><p>Not significant:</p><ul><li>Visibility</li><li>Random variables (rv1, rv2) → good negative controls</li></ul><p>This filters noise, but <strong>Granger ≠ true causation</strong>.</p><h3>4.2 Feature selection convergence (RFE, RFECV, Random Forest)</h3><p>Multiple methods converge on a <strong>stable core feature set</strong>:</p><ul><li>Indoor temperature &amp; humidity dominate</li><li>Outdoor weather is secondary</li><li>Time features (hour, month) matter structurally</li></ul><p>Agreement across methods is critical — it reduces the chance of model-specific artifacts.</p><h3>5. Formal Causal Inference with DoWhy</h3><p>To move beyond predictive heuristics, the study applies <strong>Pearl’s causal framework</strong> via <strong>DoWhy</strong>:</p><ul><li>Treatments: environmental variables</li><li>Outcome: appliance energy consumption</li><li>Identification: backdoor adjustment</li><li>Validation: placebo tests + random confounders</li></ul><h3>Estimated causal effects (examples):</h3><ul><li><strong>RH₁ → +0.59</strong> (strong positive causal effect)</li><li><strong>RH₂ → −0.53</strong></li><li><strong>T₃ → +0.50</strong></li><li><strong>T_out → −0.53</strong></li><li><strong>Lights → +0.15</strong></li></ul><p>Random variables collapse to ~0, validating the setup.</p><p><strong>Interpretation:</strong><br>Indoor climate variables <em>cause</em> energy usage changes, not just correlate with them. Outdoor temperature reduces usage indirectly (less heating demand), while lighting acts as a behavioral proxy.</p><h3>6. Interpreting Models with SHAP</h3><p>Causal validity alone isn’t enough — the model must <em>use</em> those variables.</p><p>SHAP analysis confirms:</p><ul><li>Causally identified features dominate prediction explanations</li><li>Nonlinear interactions (e.g., humidity × temperature) matter</li><li>Effects vary by regime, not linearly</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hNNr5ha49-y569sjPxN5oA.png" /></figure><p>This alignment between <strong>causal theory</strong> and <strong>model behavior</strong> is rare — and essential.</p><h3>7. Time-Series Modeling with Causal Features</h3><h3>Vector Autoregression (VAR)</h3><p>Three VAR systems are compared:</p><ul><li>Indoor-only</li><li>Indoor + outdoor</li><li>Outdoor-only</li></ul><p>Best model: <strong>indoor variables only</strong></p><ul><li>Lowest BIC</li><li>Clean residuals</li><li>Strong autoregressive structure</li></ul><p>This confirms that <strong>short-term energy dynamics are driven primarily by indoor conditions</strong>.</p><h3>LSTM and SARIMAX: causal features win</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*suBL7_J4qir8x61InNIpvQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*O3UwFjbmANNNXZF519iRUA.png" /></figure><p>That’s:</p><ul><li><strong>~8× improvement for LSTM</strong></li><li><strong>~3.5× improvement for SARIMAX</strong></li></ul><p>Causal filtering:</p><ul><li>Reduces noise</li><li>Improves generalization</li><li>Simplifies models</li><li>Increases trust</li></ul><h3>8. Why This Matters</h3><p>This work demonstrates something practical and non-obvious:</p><blockquote><strong><em>Causality is not just about interpretability — it materially improves performance.</em></strong></blockquote><p>By removing non-causal features:</p><ul><li>Deep models train faster and generalize better</li><li>Classical models become competitive again</li><li>Predictions become actionable</li></ul><p>This directly enables:</p><ul><li>Counterfactual simulation (“what if we lower humidity?”)</li><li>Energy-aware control systems</li><li>Trustworthy deployment in smart buildings</li></ul><h3>References (selected)</h3><ul><li><a href="https://archive.illc.uva.nl/cil/uploaded_files/inlineitem/Pearl_2009_Causality.pdf">Pearl, <em>Causality</em> (2009) — theoretical foundation</a></li><li><a href="https://www.pywhy.org/dowhy/v0.10.1/user_guide/refuting_causal_estimates/index.html">DoWhy documentation — causal estimation &amp; refutation</a></li><li><a href="https://www.researchgate.net/publication/317062430_A_Unified_Approach_to_Interpreting_Model_Predictions">Lundberg &amp; Lee (2017) — SHAP explainability</a></li><li><a href="https://archive.ics.uci.edu/ml/datasets/Appliances+energy+prediction">UCI Appliances Energy Dataset — empirical backbone</a></li></ul><p>For complete documentation of the project and code snippets, checkout <a href="https://github.com/Komal-Soni/A-Study-on-Causal-Models/tree/main">https://github.com/Komal-Soni/A-Study-on-Causal-Models/tree/main</a></p><h3>Final word</h3><p>Most ML systems stop at prediction.<br>This study shows that <strong>causal structure is the difference between a model that scores well — and one you can actually deploy</strong>.</p><p>If you care about robustness, intervention, and real-world impact, causality isn’t optional anymore.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e361747d156" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine Learning, what, why, and where?]]></title>
            <link>https://medium.com/@DropletOfData/machine-learning-what-why-and-where-7efa02adfcee?source=rss-d2bd81983783------2</link>
            <guid isPermaLink="false">https://medium.com/p/7efa02adfcee</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[DropletOfData]]></dc:creator>
            <pubDate>Fri, 01 Dec 2023 20:41:23 GMT</pubDate>
            <atom:updated>2023-12-01T20:41:23.721Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EeRcQpHukeNsZMIIlJ_93A.jpeg" /></figure><p>One can imagine the enormous amount of data available today. This data hides many insights and values. It can be used in ways that go far beyond your imagination. Here enters the concept of machine learning.</p><h4><strong>What is Machine Learning?</strong></h4><p>Machine Learning is the science of programming computers so that they can learn from the data with or without supervision or based on rewards and penalties.</p><p>In general terms, Machine Learning is a field of study that grants computers the ability to learn without explicit programming.</p><p>In engineering terms, Machine Learning involves algorithms that enhance their performance (P) at a specific task (T) through experience (E). A well-defined learning task is given by &lt;P, T, E&gt;.</p><p><strong>How do we define Learning Tasks?</strong></p><p>Let&#39;s say we are working on a problem where we need to recognize handwritten words.</p><p>Our goal is to improve on-task T, with respect to performance metric P, based on experience E. Hence, for this scenario,<br>T: Recognize handwritten words<br>P: Percentage of words correctly classified<br>E: Database of human-labelled images of handwritten words</p><h4>Why do we need Machine Learning?</h4><p>We use machine learning in the following scenarios:</p><ol><li>When we do not have human expertise on the subject:</li></ol><p>In situations where there is a lack of human expertise, such as exploring unfamiliar or distant environments like Mars, ML algorithms can be employed to analyze data, make decisions, and navigate autonomously. This is especially true in space exploration, where real-time decision-making is crucial and human intervention may not be feasible.</p><p>2. When we as humans can not explain our expertise</p><p>In cases where humans possess expertise but cannot explicitly articulate the rules or patterns they use, ML can be employed to learn and replicate these patterns. Biometric systems, such as fingerprint or facial recognition, often involve complex patterns that are difficult for humans to explain explicitly. ML algorithms excel at learning and recognizing such intricate patterns.</p><p>3. When the models must be customized</p><p>Personalized medicine aims to tailor medical treatment to the individual characteristics of each patient. ML is used to analyze vast amounts of patient data, including genetics, lifestyle, and medical history, to identify personalized treatment plans. The customization aspect requires sophisticated algorithms to recognize patterns and correlations specific to each patient, improving the effectiveness of medical interventions.</p><p>4. When Learning Isn’t Always Useful:</p><p>In certain tasks, learning may not be necessary, and traditional rule-based systems or algorithms can suffice. Payroll calculation is a good example where the rules are well-defined, and there is no need for a system to continuously adapt or learn. Traditional algorithms can efficiently perform such calculations without the need for ML, saving computational resources and simplifying the process.</p><h4>Where do we use Machine Learning?</h4><p>Machine Learning has found applications across various industries and domains due to its ability to analyze large amounts of data, identify patterns, and make predictions or decisions. The following are common areas where machine learning is widely employed.</p><ol><li>Healthcare:</li></ol><ul><li>Disease Diagnosis: ML is employed for analyzing medical images (like X-rays and MRIs) and identifying patterns associated with diseases.</li><li>Drug Discovery: ML algorithms help in identifying potential drug candidates and predicting their effectiveness.</li><li>Personalized Medicine: ML is used to tailor treatment plans based on individual patient data.</li></ul><p>2. Finance:</p><ul><li>Fraud Detection: ML algorithms can identify patterns indicative of fraudulent activities in financial transactions.</li><li>Credit Scoring: ML models assess credit risk by analyzing various factors to determine creditworthiness.</li><li>Algorithmic Trading: ML is used for developing trading strategies and predicting stock market trends.</li></ul><p>3. E-commerce and Retail:</p><ul><li>Recommendation Systems: ML algorithms analyze user behavior to provide personalized product recommendations.</li><li>Inventory Management: ML helps optimize inventory levels based on historical data and demand forecasting.</li><li>Price Optimization: ML models analyze market trends and competitor pricing to optimize product pricing.</li></ul><p>4. Marketing and Advertising:</p><ul><li>Customer Segmentation: ML is used to identify and categorize customer segments for targeted marketing campaigns.</li><li>Ad Targeting: ML algorithms analyze user behavior to target ads more effectively.</li><li>Predictive Analytics: ML models predict customer behavior and trends for better marketing strategies.</li></ul><p>5. Technology and Social Media:</p><ul><li>Natural Language Processing (NLP): ML is used for language translation, sentiment analysis, and chatbots.</li><li>Image and Speech Recognition: ML algorithms enable the recognition of images and speech, enhancing user interfaces and accessibility.</li></ul><p>6. Manufacturing and Industry:</p><ul><li>Predictive Maintenance: ML models predict equipment failures to optimize maintenance schedules.</li><li>Quality Control: ML is used to identify defects in manufacturing processes.</li><li>Supply Chain Optimization: ML helps optimize supply chain logistics and predict demand.</li></ul><p>7. Transportation:</p><ul><li>Autonomous Vehicles: ML is crucial for enabling vehicles to perceive and navigate their surroundings.</li><li>Traffic Prediction: ML models analyze historical data to predict traffic patterns and optimize routes.</li></ul><p>8. Education:</p><ul><li>Adaptive Learning Platforms: ML is used to personalize educational content based on individual student performance.</li><li>Predictive Analytics: ML models help identify students at risk of falling behind and recommend interventions.</li></ul><p>These are just a few examples, and the applications of machine learning continue to expand as technology advances and new challenges arise across various industries. The key is to identify problems where data-driven insights, predictions, or automation can provide significant value.</p><p>In conclusion, machine learning stands as a transformative force, revolutionizing our approach to real-world challenges across diverse sectors. By harnessing the power of data-driven insights, predictions, and automation, machine learning not only enhances efficiency and precision but also unlocks unprecedented solutions to complex issues. Its adaptability and capacity to learn from experience empower industries to innovate, making strides toward a future where technological advancements continue to address and overcome the ever-evolving challenges of our world.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7efa02adfcee" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>