<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Bowen Jin on Medium]]></title>
        <description><![CDATA[Stories by Bowen Jin on Medium]]></description>
        <link>https://medium.com/@jinbowen517?source=rss-eedb0617b186------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*5knOcNlZEVDrd9bD</url>
            <title>Stories by Bowen Jin on Medium</title>
            <link>https://medium.com/@jinbowen517?source=rss-eedb0617b186------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 02:31:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jinbowen517/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Causal Inference]]></title>
            <link>https://medium.com/aiskunks/causal-inference-8d753e0b4f3c?source=rss-eedb0617b186------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d753e0b4f3c</guid>
            <dc:creator><![CDATA[Bowen Jin]]></dc:creator>
            <pubDate>Fri, 28 Apr 2023 21:00:49 GMT</pubDate>
            <atom:updated>2023-04-28T21:00:49.797Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Authors</em></p><p><em>Peichen</em> Han, <em>Bowen</em> Jin</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G-Op_uSZcEpPpxsDqVqafg.png" /></figure><h3>What is Causal Inference</h3><p>Causal inference, a powerful tool in statistics and data science, involves pinpointing the precise impact of a particular phenomenon within a larger system. It’s an invaluable process for revealing the causal relationships between variables and uncovering the underlying reasons behind a phenomenon. Causal inference is widely used across fields such as biomedicine, economics, and social sciences, where it can be utilized to determine the impact of smoking on cancer, detect gender discrimination during recruitment processes, and estimate the quantitative causal effect of an event. This process allows individuals to make better-informed decisions and interventions by exploring how education levels affect future income or analyzing the response of an outcome variable in the face of changing causes.</p><h3>Why Causal Inference</h3><h3>Causation and Correlation</h3><p>Correlation is a very common concept in statistics and data science. We have become accustomed to all kinds of data analysis through correlation. However, correlation has its limits. The graph below shows the relationship between margarine consumption and the divorce rate in Maine</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8gV8_vP_AxapSNn8xsD8Bg.png" /></figure><p>The graph clearly shows a high correlation between these two variables. However, drawing a causal relationship between margarine consumption and divorce rates seems illogical. It’s essential to understand that correlation does not necessarily mean causation. Correlation is typically symmetrical, while causation is asymmetrical. While correlation can exist without causation, causality often leads to correlation at a statistical level. In practical terms, it’s easier to comprehend and apply concepts that have causality rather than those that merely have correlation. With the advancement of data science, it’s only natural to consider causality and related principles.</p><h3>Simpson&#39;s paradox</h3><p>The table below shows success rates and number of successful and failed cases of two treatments for kidney stones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/379/1*A23fCBkySXo5xLxRZDh49w.png" /></figure><p>Based on the information presented, which option would you select? If you only focus on large or small stones, you will discover that Treatment A is the superior choice. However, if you take into account both large and small stones, it appears that Treatment B is the more effective option. Although this may seem like an insurmountable paradox, doctors may not face this issue in reality. This is due to the fact that doctors are aware of the causal structure that underpins the data. The causal relationship between treatment, stone size, and failure rate may resemble the following in actuality:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/527/1*wrtepTExQqUCfbmdRzTEfg.png" /></figure><p>The causal relationship depicted in the figure above may be interpreted as follows, based on the actual scene. Patients who have larger stone sizes are inherently more likely to experience treatment failure. In addition, medical practitioners tend to use treatment B for small stones and treatment A for large stones. Consequently, treatment A is more frequently associated with failure. This means that stone size is a common factor influencing both the choice of treatment and the success rate. To avoid any confusion arising from such confounding factors, it is essential to focus on the relationship between treatment and outcomes when considering stones of the same size. Doing so can enable us to determine which treatment is superior, as evidenced by its higher success rate when applied to either large or small stones. By considering causality in this way, we can better address practical problems and enhance our performance.</p><h3>Examples</h3><h3>DoWhy</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*A-KbGkJHXJEwuG1ePw8S5A.png" /></figure><p>DoWhy is a Python library released by Microsoft for causal inference based on graphic model and potential outcome model. It provides a principled method to transform a given problem into a causal graph, and provides a unified interface for many commonly used causal inference methods, and combines two main causal inference frameworks. You can also use it to check correctness of assumptions and robustness of estimates automatically.</p><h3>Simulate Dataset</h3><p>In this article, for better learning, we first use dowhy library to simulate a causal dataset for causal inference.</p><p>Import library</p><pre>import numpy as np<br>import pandas as pd<br>import matplotlib.pyplot as plt<br><br>import dowhy<br>import dowhy.datasets<br>from dowhy import CausalModel</pre><p>Generate simulated data, in this case we simulate a linear relationship between different variables</p><pre>data = dowhy.datasets.linear_dataset(beta=10, #real causal effects<br>        num_common_causes=5, #confounders<br>        num_instruments=2, #instrumental variable<br>        num_effect_modifiers=1, #Effect modifier variable<br>        num_samples=10000, #number of samples<br>        treatment_is_binary=True,<br>        num_discrete_common_causes=1)<br>df = data[&quot;df&quot;] <br>print(data[&quot;dot_graph&quot;])</pre><pre>digraph {v0-&gt;y;W0-&gt; v0; W1-&gt; v0; W2-&gt; v0; W3-&gt; v0; W4-&gt; v0;Z0-&gt; v0; Z1-&gt; v0;W0-&gt; y; W1-&gt; y; W2-&gt; y; W3-&gt; y; W4-&gt; y;X0-&gt; y;}</pre><pre>print(df.head())</pre><pre>X0   Z0        Z1        W0        W1        W2        W3 W4    v0  \<br>0  1.047721  1.0  0.946104 -0.956627 -0.584224  0.660085  0.608764  3  True   <br>1  2.538196  1.0  0.317835 -1.204281 -0.826717  1.662989  1.070910  0  True   <br>2 -0.374068  0.0  0.672964 -0.850264 -0.255517  1.665433 -0.699174  2  True   <br>3 -1.150634  1.0  0.375710 -0.909048 -0.437055 -0.127924  1.596841  3  True   <br>4  1.368289  1.0  0.990431 -0.566798 -0.166837  1.614959  1.530499  3  True   <br><br>           y  <br>0  23.689074  <br>1  24.733789  <br>2  17.815255  <br>3  16.351174  <br>4  36.342800</pre><h3>Model</h3><p>DoWhy creates a causal graphical model for each question to keep causal assumptions explicit. The causal graph does not need to be complete, you can provide only partial graphs to represent prior knowledge of certain variables, and DoWhy supports automatically treating the remaining variables as potential confounding factors. Currently, DoWhy supports causal assumptions in these forms:<br>1、Graph: Provide a cause-and-effect graph in the form of gml or dot, which can be in file or string format.<br>2、Named variable sets: directly provide the type of variables, including cofounders, instrumental variables, effect modifiers, front-door variables, etc.<br>we will use 1 in this article.</p><pre>model=CausalModel(<br>        data = df,<br>        treatment=data[&quot;treatment_name&quot;],<br>        outcome=data[&quot;outcome_name&quot;],<br>        graph=data[&quot;gml_graph&quot;]<br>        )<br>model.view_model()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*EVdgJRXP9APfyozRlCtg3g.png" /></figure><h3>Identify</h3><p>Based on the constructed causal graph, DoWhy identifies causal effects in all possible ways. Specifically, graph-based criteria and do-integration are used to find expressions that identify causal effects. Supported identification criteria are:<br>1、Back-door criterion<br>2、Front-door criterion<br>3、Instrumental Variables<br>4、Mediation-Direct and indirect effect identification<br>In this article, we will focus on back-door criterion.</p><pre>identified_estimand = model.identify_effect()<br>print(identified_estimand)</pre><pre>Estimand type: EstimandType.NONPARAMETRIC_ATE<br><br>### Estimand : 1<br>Estimand name: backdoor<br>Estimand expression:<br>  d                       <br>─────(E[y|W2,W0,W3,W4,W1])<br>d[v₀]                     <br>Estimand assumption 1, Unconfoundedness: If U→{v0} and U→y then P(y|v0,W2,W0,W3,W4,W1,U) = P(y|v0,W2,W0,W3,W4,W1)<br><br>### Estimand : 2<br>Estimand name: iv<br>Estimand expression:<br> ⎡                              -1⎤<br> ⎢    d        ⎛    d          ⎞  ⎥<br>E⎢─────────(y)⋅⎜─────────([v₀])⎟  ⎥<br> ⎣d[Z₁  Z₀]    ⎝d[Z₁  Z₀]      ⎠  ⎦<br>Estimand assumption 1, As-if-random: If U→→y then ¬(U →→{Z1,Z0})<br>Estimand assumption 2, Exclusion: If we remove {Z1,Z0}→{v0}, then ¬({Z1,Z0}→y)<br><br>### Estimand : 3<br>Estimand name: frontdoor<br>No such variable(s) found!</pre><h3>back-door criterion</h3><p>Given a pair of ordered variables (X, Y) in a directed acyclic graph (DAG), if the variable set Z (can be empty) satisfies:</p><p>There are no descendant nodes of X in Z.<br>Z blocks every path between X and Y that points to X.<br>If Z satisfies the above two points, it is said that Z satisfies the backdoor criterion about (X, Y).</p><p>In the above example of kidney stones, what we want is the causal relationship of treatment ---&gt; failure rate. At this time, size satisfies the back-door criterion, so we can obtain correct causal effects by blocking non-causal association treatment &lt;--- size ---&gt; failure rate.</p><h3>Estimate</h3><p>DoWhy supports all estimation methods based on the identification criteria described above, and additionally provides nonparametric confidence space and permutation tests to test the statistical significance of the resulting estimates.<br>In this article, we will mainly use method based on back-door criterion.</p><pre>estimate = model.estimate_effect(identified_estimand,<br>        method_name=&quot;backdoor.propensity_score_stratification&quot;)<br>print(estimate)</pre><pre>*** Causal Estimate ***<br><br>## Identified estimand<br>Estimand type: EstimandType.NONPARAMETRIC_ATE<br><br>### Estimand : 1<br>Estimand name: backdoor<br>Estimand expression:<br>  d                       <br>─────(E[y|W2,W0,W3,W4,W1])<br>d[v₀]                     <br>Estimand assumption 1, Unconfoundedness: If U→{v0} and U→y then P(y|v0,W2,W0,W3,W4,W1,U) = P(y|v0,W2,W0,W3,W4,W1)<br><br>## Realized estimand<br>b: y~v0+W2+W0+W3+W4+W1<br>Target units: ate<br><br>## Estimate<br>Mean value: 15.994236372775601</pre><h3>Refuse</h3><p>DoWhy supports a variety of refutation methods to verify the correctness of the estimate, the specific list is as follows:</p><p>1、Add random confounder: Estimate whether the causal effect will change after adding a random variable as a confounder (expected result: no)<br>2、Placebo intervention: Does the causal effect change after replacing the real intervention variable with an independent random variable (expected result: zero causal effect)<br>3、Dummy outcome: Does the causal effect change after replacing the real outcome variable with an independent random variable (expected result: zeroing out the causal effect)<br>4、Simulation Outcome: Does the causal effect change when the dataset is replaced by a dataset generated based on a simulation that approximates the data-generating process for a given dataset (expected result: matches the effect parameters of the data-generating process)<br>5、Add unobserved confounders: Sensitivity to causal effects after adding an additional confounder related to intervention and outcome (expected result: not overly sensitive)<br>6、Data subset validation: Does the causal effect change when a given dataset is replaced by a random subset (expected result: no)<br>7、Bootstrap Validation: Does the causal effect change when a given dataset is replaced by a bootstrap sample of the same dataset (expected result: no)</p><p>In this article, we will use 1 and 6.</p><pre># add random confounder<br>res_random=model.refute_estimate(identified_estimand, estimate, method_name=&quot;random_common_cause&quot;)<br>print(res_random)</pre><pre>Refute: Add a random common cause<br>Estimated effect:15.994236372775601<br>New effect:15.994236372775594<br>p value:2.0</pre><pre># data subset validation<br>res_subset=model.refute_estimate(identified_estimand, estimate,<br>        method_name=&quot;data_subset_refuter&quot;, subset_fraction=0.9)<br>print(res_subset)</pre><pre>Refute: Use a subset of data<br>Estimated effect:15.994236372775601<br>New effect:16.013462963117345<br>p value:0.92</pre><h3>Another Example With Real Data</h3><h3>Dataset</h3><p>The dataset we used describes graduation rates at a four-year college.<br>Import the dataset first.</p><pre>graduate = pd.read_csv(&#39;../content/sample_data/graduation_rate.csv&#39;)<br>graduate</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/1*cAWO5LT_ifmqfL7wqeP-Bw.png" /></figure><p>Name of columns are too long, rename them.</p><pre>graduate.rename(columns={&#39;ACT composite score&#39;:&#39;act&#39;, &#39;SAT total score&#39;:&#39;sat&#39;, &#39;parental level of education&#39;:&#39;pa_edu&#39;, &#39;parental income&#39;:&#39;pa_in&#39;, &#39;high school gpa&#39;:&#39;h_gpa&#39;, &#39;college gpa&#39;:&#39;c_gpa&#39;, &#39;years to graduate&#39;:&#39;years&#39;}, inplace=True)<br>graduate</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/1*cAWO5LT_ifmqfL7wqeP-Bw.png" /></figure><p>Investigating the correlation between high school GPA and college GPA is the objective of our study using this particular dataset. We have neglected to take into account the time spent on graduation as it is not relevant to our research. Our hypothesis is that students who have parents with higher education or income levels tend to have access to better resources, resulting in higher GPAs. Furthermore, we have included an indicator, U, to account for any other unobserved variables that may impact the results.</p><pre>causal_graph = &quot;&quot;&quot;digraph {<br>  act;<br>  sat;<br>  h_gpa;<br>  c_gpa;<br>  pa_edu;<br>  pa_in;<br>  U[label=&quot;Unobserved Confounders&quot;];<br>  act -&gt; h_gpa;<br>  sat -&gt; h_gpa;<br>  h_gpa -&gt; c_gpa;<br>  pa_edu -&gt; h_gpa; pa_edu -&gt; c_gpa;<br>  pa_in -&gt; h_gpa; pa_in -&gt; c_gpa;<br>  U -&gt; h_gpa; U -&gt; c_gpa;<br>}&quot;&quot;&quot;</pre><p>Bulid model based on causal graph above.</p><pre>model= dowhy.CausalModel(<br>        data = graduate,<br>        graph=causal_graph.replace(&quot;\n&quot;, &quot; &quot;),<br>        treatment=&#39;h_gpa&#39;,<br>        outcome=&#39;c_gpa&#39;)<br>model.view_model()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*dpg3ImWQrx5tSwEVA3-kcQ.png" /></figure><p>Identify</p><pre>identified_estimand = model.identify_effect()<br>print(identified_estimand)</pre><pre>### Estimand : 1<br>Estimand name: backdoor<br>No such variable(s) found!</pre><pre>### Estimand : 2<br>Estimand name: iv<br>Estimand expression:<br> ⎡                                         -1⎤<br> ⎢     d             ⎛     d              ⎞  ⎥<br>E⎢───────────(c_gpa)⋅⎜───────────([h_gpa])⎟  ⎥<br> ⎣d[act  sat]        ⎝d[act  sat]         ⎠  ⎦<br>Estimand assumption 1, As-if-random: If U→→c_gpa then ¬(U →→{act,sat})<br>Estimand assumption 2, Exclusion: If we remove {act,sat}→{h_gpa}, then ¬({act,sat}→c_gpa)</pre><pre>### Estimand : 3<br>Estimand name: frontdoor<br>No such variable(s) found!</pre><p>Estimate(based on instrumental variable equation)</p><pre>estimate = model.estimate_effect(identified_estimand,<br>                  method_name=&quot;iv.instrumental_variable&quot;,test_significance=True)<br>print(estimate)</pre><pre><br>## Identified estimand<br>Estimand type: EstimandType.NONPARAMETRIC_ATE</pre><pre>### Estimand : 1<br>Estimand name: iv<br>Estimand expression:<br> ⎡                                         -1⎤<br> ⎢     d             ⎛     d              ⎞  ⎥<br>E⎢───────────(c_gpa)⋅⎜───────────([h_gpa])⎟  ⎥<br> ⎣d[act  sat]        ⎝d[act  sat]         ⎠  ⎦<br>Estimand assumption 1, As-if-random: If U→→c_gpa then ¬(U →→{act,sat})<br>Estimand assumption 2, Exclusion: If we remove {act,sat}→{h_gpa}, then ¬({act,sat}→c_gpa)</pre><pre>## Realized estimand<br>Realized estimand: Wald Estimator<br>Realized estimand type: EstimandType.NONPARAMETRIC_ATE<br>Estimand expression:<br> ⎡   d           ⎤  -1⎡   d           ⎤<br>E⎢────────(c_gpa)⎥⋅E  ⎢────────(h_gpa)⎥<br> ⎣dact,sat       ⎦    ⎣dact,sat       ⎦<br>Estimand assumption 1, As-if-random: If U→→c_gpa then ¬(U →→{act,sat})<br>Estimand assumption 2, Exclusion: If we remove {act,sat}→{h_gpa}, then ¬({act,sat}→c_gpa)<br>Estimand assumption 3, treatment_effect_homogeneity: Each unit&#39;s treatment [&#39;h_gpa&#39;] is affected in the same way by common causes of [&#39;h_gpa&#39;] and c_gpa<br>Estimand assumption 4, outcome_effect_homogeneity: Each unit&#39;s outcome c_gpa is affected in the same way by common causes of [&#39;h_gpa&#39;] and c_gpa</pre><pre>Target units: ate</pre><pre>## Estimate<br>Mean value: 0.9084360711049295</pre><p>In the end, we check the robustness of the estimate using a Placebo refutation test.</p><pre>ref = model.refute_estimate(identified_estimand, estimate, method_name=&quot;placebo_treatment_refuter&quot;, placebo_type=&quot;permute&quot;)<br>print(ref)</pre><pre>Refute: Use a Placebo Treatment<br>Estimated effect:0.9084360711049295<br>New effect:0.9058590125918934<br>p value:0.0</pre><p>Very close, The results are consistent with common sense.</p><h3>Reference:</h3><p>[1]: Graduation Rate <a href="https://www.kaggle.com/datasets/rkiattisak/graduation-rate">https://www.kaggle.com/datasets/rkiattisak/graduation-rate</a><br>[2]: Simpson’s paradox <a href="https://en.wikipedia.org/wiki/Simpson%27s_paradox">https://en.wikipedia.org/wiki/Simpson%27s_paradox</a><br>[3]: Causaulity <a href="https://zhuanlan.zhihu.com/p/269625734">https://zhuanlan.zhihu.com/p/269625734</a><br>[4]: dowhy <a href="https://github.com/py-why/dowhy">https://github.com/py-why/dowhy</a><br>[5]: Causal Effects <a href="https://docs.google.com/presentation/d/1yKx_z5aXk6tEQZsSbKI5LfRR4Sa6XDHYyMqg9Zgztts/edit#slide=id.p">https://docs.google.com/presentation/d/1yKx_z5aXk6tEQZsSbKI5LfRR4Sa6XDHYyMqg9Zgztts/edit#slide=id.p</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d753e0b4f3c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/aiskunks/causal-inference-8d753e0b4f3c">Causal Inference</a> was originally published in <a href="https://medium.com/aiskunks">AI Skunks</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Crash Course in Forecasting Participation: Time Series Analysis]]></title>
            <link>https://medium.com/aiskunks/crash-course-in-forecasting-participation-time-series-analysis-e088f456ffff?source=rss-eedb0617b186------2</link>
            <guid isPermaLink="false">https://medium.com/p/e088f456ffff</guid>
            <dc:creator><![CDATA[Bowen Jin]]></dc:creator>
            <pubDate>Mon, 10 Apr 2023 12:51:28 GMT</pubDate>
            <atom:updated>2023-04-10T12:51:28.216Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Authors</em></p><p><em>Peichen</em> Han, <em>Bowen</em> Jin</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N5gUEqDzYg0eU7Ds072e1Q.png" /></figure><h3>Data set and our target</h3><p>The provided dataset contains information about water bodies in the Tuscany region of Italy. There are four different waterbodies included in the dataset: aquifers, rivers, lakes, and rainfall. For each waterbody, there are various features included, such as water levels, temperature, volume, and rainfall, among others. The data covers a period of several years, depending on the waterbody, and the frequency of the data varies from daily to monthly measurements. The dataset is intended for use in predicting future water levels or other features for each waterbody.</p><p>We will be using the Lake Bilancino dataset for our time series forecasting. Specifically, we will be forecasting the ‘Lake_Level’ feature, which represents the depth of the water level in the lake.</p><h4>Target</h4><p>We hope that readers can learn how to draw time series graphs and what is ARIMA model-based time series forecasting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PvGlg6iOBFonB_z8j4LlaA.png" /></figure><h3>Access Dataset</h3><pre>Lake_input = pd.read_csv(&#39;../content/sample_data/Lake_Bilancino.csv&#39;)<br><br>Lake = Lake_input.copy()<br><br>print(Lake.info())</pre><pre>&lt;class &#39;pandas.core.frame.DataFrame&#39;&gt;<br>RangeIndex: 6603 entries, 0 to 6602<br>Data columns (total 9 columns):<br> #   Column                Non-Null Count  Dtype  <br>---  ------                --------------  -----  <br> 0   Date                  6603 non-null   object <br> 1   Rainfall_S_Piero      6026 non-null   float64<br> 2   Rainfall_Mangona      6026 non-null   float64<br> 3   Rainfall_S_Agata      6026 non-null   float64<br> 4   Rainfall_Cavallina    6026 non-null   float64<br> 5   Rainfall_Le_Croci     6026 non-null   float64<br> 6   Temperature_Le_Croci  6025 non-null   float64<br> 7   Lake_Level            6603 non-null   float64<br> 8   Flow_Rate             6582 non-null   float64<br>dtypes: float64(8), object(1)<br>memory usage: 464.4+ KB<br>None</pre><h3>Time series</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mnmrsEokOHOpfKuWmqiItg.png" /></figure><p>For each subplot, the title indicates the corresponding feature name and the y-label indicates the unit of measurement. The plots show the overall trend and seasonality of each feature over time, as well as any irregular patterns or outliers.</p><p>The visualization can provide insights into the characteristics of each feature, such as the periodicity and seasonality of the ‘Lake_Level’ feature, the sharp decrease in ‘Flow_Rate_Mangona’ around 2011, and the increasing trend in ‘Rainfall_S_Piero’ and ‘Rainfall_Mangona’ after 2017. This information can be useful in understanding the behavior of the dataset and selecting appropriate models for time series forecasting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RYWiRUaiLL1tNeK-B5shZw.jpeg" /><figcaption>Lake Bilancino</figcaption></figure><h3>ARIMA Model Analysis</h3><p>The basic idea of the ARIMA model is to regard the data sequence formed by the forecast object over time as a random sequence and use a certain mathematical model to approximate this sequence. This model, once identified, can predict future values from the past and present values of the time series. Modern statistical methods and econometric models have been able to help companies predict the future to some extent.</p><p>The basic steps of model building are as follows:</p><ol><li>Obtain the time series data of the observed system.</li><li>Plot the data and determine whether it is a stationary time series. For non-stationary time series, perform a differencing operation to convert it into a stationary time series. If it is already stationary, proceed to the next step.</li><li>Obtain the autocorrelation coefficient ACF and partial autocorrelation coefficient PACF for the stationary time series. Analyze the ACF and PACF plots to determine the best values for the autoregressive parameter p and the moving average parameter q.</li><li>Using the values of d, p, and q, construct the ARIMA model. Check the model by examining its residual plots to ensure it is consistent with the characteristics of the observed data. If not, go back to step 3 and adjust the values of p and q accordingly.</li></ol><p>But here we use ADF instead of ACF for stationary check.</p><h3>Stationary check</h3><h4>What is stationary</h4><p>When we talk about time series, there are two types of stationarity: Strict-sense stationary and wide-sense stationarity.<br>Strict-sense stationary time series requires that every two subsequences with any given length in the time series satisfy the same joint distribution. Consequently, all statistical properties of the sequence do not change when shifted in time. Obviously, It&#39;s a very strict condition in practice. So we still have weak wide-sense stationarity.<br>Wide-sense stationarity(WSS) only requires that 1st moment (i.e. the mean) and autocovariance do not vary with respect to time and that the 2nd moment is finite for all times. Any strictly stationary process which has a finite mean and a covariance is also WSS.</p><p>Obviously, as long as the series is stationary, we can use past data to predict the future. For this, we need a way to detect whether the series is stationary. This way is augmented Dickey-Fuller test.</p><h4>ADF</h4><p>In statistics, an augmented Dickey–Fuller test (ADF) tests the null hypothesis that a unit root is present in a time series sample. The alternative hypothesis is different depending on which version of the test is used, but is usually stationarity or trend-stationarity. It is an augmented version of the Dickey–Fuller test for a larger and more complicated set of time series models.</p><p>The augmented Dickey–Fuller (ADF) statistic, used in the test, is a negative number. The more negative it is, the stronger the rejection of the hypothesis that there is a unit root at some level of confidence.</p><p>In python, you can use existing tools for ADF.</p><pre>from statsmodels.tsa.stattools import adfuller.</pre><p>Take the temp time series in this dataset as an example.</p><pre>Lake = Lake.rename(columns={&#39;Temperature_Le_Croci&#39;:&#39;tem&#39;,&#39;Date&#39;:&#39;date&#39;})<br>tem_adf = adfuller(Lake[&#39;tem&#39;].values)<br>tem_adf</pre><pre><br>tem_adf = adfuller(Lake[&#39;tem&#39;].values)<br>tem_adf<br>(-4.607163579535117,<br> 0.00012529899651088334,<br> 16,<br> 6008,<br> {&#39;1%&#39;: -3.4314388974936767,<br>  &#39;5%&#39;: -2.8620211927157615,<br>  &#39;10%&#39;: -2.5670261364086677},<br> 24542.31575729264)</pre><p>In tem_adf, the parameters are: {<br>Test Statistic Value,<br>p-value,<br>Lags Used,<br>Number of Observations Used,<br>{Critical Value(1%),<br>Critical Value(5%),<br>Critical Value(10%)},<br>icbest }</p><p>If the p-value is equal to or below the predetermined significance level of 0.05, this indicates that the data is considered stationary. In regards to the temperature time series we are analyzing, the p-value is observed to be 0.000125, which is less than 0.05, highlighting that it is already stationary. However, we will still discuss the methods for making a time series stationary.</p><h4>How to make time-series stationary</h4><p>In general, for this problem, we have two approaches:</p><ol><li>Make a transformation: for example, a logarithm or a square root to stabilize a non-constant variance;</li></ol><p>2. Differentiate: subtract the current value from the previous one.</p><h4>Differentiate</h4><pre>tem_diff = Lake[&#39;tem&#39;].diff(1)<br>tem_diff_adf = adfuller(tem_diff[1:].values)<br>tem_diff_adf<br>(-26.28195776196397,<br> 0.0,<br> 15,<br> 6008,<br> {&#39;1%&#39;: -3.4314388974936767,<br>  &#39;5%&#39;: -2.8620211927157615,<br>  &#39;10%&#39;: -2.5670261364086677},<br> 24558.0834117153)</pre><p>Our p-value &lt; 0.05, so we do not need to Differentiate again.</p><h4>Choose value of p and q</h4><p>For the classic baseline model, we will use ARIMA. ACF and PACF graphs: After the time series has been stationary by differentiation, the next step in fitting the ARIMA model is to determine whether the terms AR or MA are needed to correct any autocorrelation that remains in the differentiated series. By looking at the graphs of the autocorrelation function (ACF) and partial autocorrelation (PACF) of the difference series, we will be able to pre-determine the number of required AR and/or MA terms.</p><p>Autocorrelation function (ACF): P = Periods with a delay, for example: (if P = 3, then we will use the three previous periods of our time series in the autoregressive part of the calculation) P helps to adjust the line that is set to predict the series. P corresponds to the parameter MA</p><p>Partial Autocorrelation Function (PACF): D = In the ARIMA model, we transform a time series into a stationary one (a series without trend or seasonality) using differentiation. D refers to the number of different transformations required by the time series to achieve stationarity. D corresponds to the AR parameter. Autocorrelation graphs help determine seasonality.</p><p>Based on the graph below, autocorrelation continues to decrease as the delay increases, confirming the absence of a linear relationship between observations separated by large “delays”.</p><p>For the AR process, it is expected that the ACF schedule will gradually decrease, and at the same time, the PACF should have a sharp drop after p significant delays. To determine the MA process, we expect the opposite from the ACF and PACF graphs, which means that: ACF should show a sharp drop after a certain qth number of shifts, while PACF should show a geometric or gradual downward trend.</p><pre>f, ax = plt.subplots(nrows=2, ncols=1, figsize=(16, 8))<br># caculate acf and pacf<br>plot_acf(data_diff,lags=30, ax=ax[0])<br>plot_pacf(data_diff,lags=30, ax=ax[1])<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IVACsGyTwJSYDKdRo7Sypg.png" /></figure><p>When choosing the values of p and q, in addition to observing ACF and PACF, we can also use AIC and BIC.</p><p>The Akaike information criterion (AIC) is an estimator of prediction error and thereby relative quality of statistical models for a given set of data. Given a collection of models for the data, AIC estimates the quality of each model, relative to each of the other models. Thus, AIC provides a means for model selection.</p><p>The Bayesian information criterion (BIC) or Schwarz information criterion (also SIC, SBC, SBIC) is a criterion for model selection among a finite set of models; models with lower BIC are generally preferred. It is based, in part, on the likelihood function and it is closely related to the Akaike information criterion (AIC).</p><p>In this article, we use AIC to choose p and q. Through the following code, we can get the combination of p and q with the smallest AIC. However, in order to reduce calculation, here we set maximum of p is 6 and maximum of q is 4.</p><pre>sm.tsa.arma_order_select_ic(data,max_ar=6,max_ma=4,ic=&#39;aic&#39;)[&#39;aic_min_order&#39;]<br>(2, 3)</pre><p>In summary, our ARIMA model’s parameters p and q were determined by utilizing the Akaike information criterion (AIC) — a reliable method for assessing a model’s quality compared to others. By leveraging the arma_order_select_ic function in the stats models library, we found that our dataset performed best when p=2 and q=3. This allowed us to confidently select the most suitable parameters for our ARIMA model. Since this dataset is originally stationary then d = 0.</p><h3>Generate model</h3><pre>model = ARIMA(data, order=(2,3,0)).fit()<br>model.summary() </pre><pre>SARIMAX Results<br>Dep. Variable: tem No. Observations: 6025<br>Model: ARIMA(2, 3, 0) Log Likelihood -15619.888<br>Date: Sat, 08 Apr 2023 AIC 31245.777<br>Time: 04:44:11 BIC 31265.886<br>Sample: 01-02-2004 HQIC 31252.759<br>- 06-30-2020  <br>Covariance Type: opg  <br>coef std err z P&gt;|z| [0.025 0.975]<br>ar.L1 -0.9616 0.010 -95.594 0.000 -0.981 -0.942<br>ar.L2 -0.4725 0.010 -45.075 0.000 -0.493 -0.452<br>sigma2 10.4812 0.155 67.677 0.000 10.178 10.785<br>Ljung-Box (L1) (Q): 226.08 Jarque-Bera (JB): 291.34<br>Prob(Q): 0.00 Prob(JB): 0.00<br>Heteroskedasticity (H): 1.08 Skew: 0.09<br>Prob(H) (two-sided): 0.08 Kurtosis: 4.06</pre><h3>Predict with model</h3><pre>predictions_ARIMA = pd.Series(model.fittedvalues, copy=True)<br>predictions_ARIMA</pre><pre>date<br>2004-01-02     0.000000<br>2004-01-03    13.000018<br>2004-01-04     0.291693<br>2004-01-05    -0.850156<br>2004-01-06     0.007128<br>                ...    <br>2020-06-26    21.889392<br>2020-06-27    23.011570<br>2020-06-28    24.564236<br>2020-06-29    19.835548<br>2020-06-30    24.023303<br>Length: 6025, dtype: float64</pre><h3>Result Visualization</h3><pre>plt.figure(figsize=(40, 20))<br>plt.plot(data,label=&quot;data&quot;)<br>plt.xlabel(&#39;date&#39;,fontsize=12,verticalalignment=&#39;top&#39;)<br>plt.ylabel(&#39;tem&#39;,fontsize=14,horizontalalignment=&#39;center&#39;)<br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZrIl9gymXRDuVxIMwNx5Iw.png" /></figure><pre>plt.figure(figsize=(40, 20))<br>plt.plot(predictions_ARIMA,label=&quot;forecast&quot;)<br>plt.xlabel(&#39;date&#39;,fontsize=12,verticalalignment=&#39;top&#39;)<br>plt.ylabel(&#39;tem&#39;,fontsize=14,horizontalalignment=&#39;center&#39;)<br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*InwahwllI6IlXdf_4HEZyg.png" /></figure><pre>plt.figure(figsize=(40, 20))<br>plt.plot(predictions_ARIMA,label=&quot;forecast&quot;)<br>plt.plot(data,label=&quot;data&quot;)<br>plt.xlabel(&#39;date&#39;,fontsize=12,verticalalignment=&#39;top&#39;)<br>plt.ylabel(&#39;tem&#39;,fontsize=14,horizontalalignment=&#39;center&#39;)<br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UMRwxI7sKE_kY4RyU8HQgw.png" /></figure><h3>Advantages and Disadvantages of ARIMA, and result analysis</h3><p>In relation to the dataset of Lake Bilancino, an ARIMA model was utilized to predict the lake level as time progressed. The outcomes demonstrated that the model had the ability to generally depict the trend of the initial data, but fell short in accurately representing the data. This can be attributed to the uncomplicated nature of the ARIMA model, which possesses limitations when it comes to fitting in situations involving nonlinear relationships within data. In spite of this, this model still provided valuable comprehensive knowledge and observations about the trends and formations present at the lake level as time progressed. In summary, the Lake Bilancino dataset serves as an effective case study for examining various time-series forecasting techniques and their practical applications in real-world circumstances.</p><h3>Reference:</h3><p>[1] Data Source: ACEA Water Prediction Competition. (n.d.). Retrieved from <a href="https://www.kaggle.com/c/acea-water-prediction/data">https://www.kaggle.com/c/acea-water-prediction/data</a></p><p>[2] Tutorials: Gupta, P. (n.d.). ARIMA Model — Time Series Forecasting (Python). Machine Learning Plus. Retrieved from <a href="https://www.machinelearningplus.com/time-series/arima-model-time-series-forecasting-python/">https://www.machinelearningplus.com/time-series/arima-model-time-series-forecasting-python/</a></p><p>[3] Research Papers: Mannocchi, F., Carleo, E., &amp; Capparelli, G. (2020). Water Table Level Prediction Using Machine Learning Techniques: An Application to a Shallow Aquifer in Central Italy. Water, 12(7), 1965. <a href="https://doi.org/10.3390/w12071965">https://doi.org/10.3390/w12071965</a></p><p>[4] Ziadat, F. M., Al-Ansari, N., Abdellatif, M., &amp; Senan, A. (2015). Groundwater Level Prediction Using ARIMA Modeling Technique: A Case Study in Basrah Province. Environmental Earth Sciences, 74(8), 6879–6895. <a href="https://doi.org/10.1007/s12665-015-4725-5">https://doi.org/10.1007/s12665-015-4725-5</a></p><p>[5] Blog Posts: Mansoor, M. (2020, July 23). Time Series Forecasting using ARIMA models. Towards Data Science. Retrieved from <a href="https://towardsdatascience.com/time-series-forecasting-using-arima-models-6f33eb80a0d6">https://towardsdatascience.com/time-series-forecasting-using-arima-models-6f33eb80a0d6</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e088f456ffff" width="1" height="1" alt=""><hr><p><a href="https://medium.com/aiskunks/crash-course-in-forecasting-participation-time-series-analysis-e088f456ffff">Crash Course in Forecasting Participation: Time Series Analysis</a> was originally published in <a href="https://medium.com/aiskunks">AI Skunks</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Crash Crouse in Data : Data Humanism]]></title>
            <link>https://medium.com/aiskunks/crash-crouse-in-data-data-humanism-72e06a4f36a0?source=rss-eedb0617b186------2</link>
            <guid isPermaLink="false">https://medium.com/p/72e06a4f36a0</guid>
            <dc:creator><![CDATA[Bowen Jin]]></dc:creator>
            <pubDate>Mon, 27 Mar 2023 16:24:09 GMT</pubDate>
            <atom:updated>2023-04-13T16:28:31.187Z</atom:updated>
            <content:encoded><![CDATA[<h3>Crash Course in Data: Data Humanism</h3><p>Connecting people and data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*1eLppubkYV-DEd9L" /></figure><p>Data humanism is a philosophy that centers around data processing and visualization, prioritizing human needs and values. Rather than relying on traditional infographics, this approach emphasizes a more individualized and nuanced perspective on data visualization, highlighting the importance of context and personal experience. A prominent figure in this field is Giorgia Lupi, author of the data humanism manifesto, which provides a framework for researchers and designers to approach data in a more ethical and human-centered manner.</p><h3>Humanism And Dataism</h3><p>Humanism originated from an ideological tendency, which tends to care about human personality, emphasizes the maintenance of human dignity, advocates a tolerant secular culture, opposes violence and discrimination, pursues freedom, equality and self-worth, and eventually develops into a philosophy and a theoretical system of worldview. From the above description, we can also know that Humanism has an important feature. It holds a view that each person is valuable in himself or herself.</p><p>For a long time, humanism was considered advanced and universal. But With the development of computer science and biology, another theory was born — dataism. Dataism believes that the universe is made up of data streams, and the value of any phenomenon or entity lies in its contribution to data processing. It is the only brand-new value created by humans after the creation of humanism. For dataism, the highest value is information streams. Therefore, freedom of information is the highest good. Dataism equates human experience with data patterns, undermines the human authority and source of meaning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*geKKS2mPuBJ5Ippm" /></figure><h3>Data Humanism</h3><p>Humanism and Dataism can be said to have many opposites, especially in recognizing personal value. But recently, a new idea was born in the exploration of data-related technologies and means. That’s what I’m going to talk about next, Data Humanism. Data humanism believes data is very important, but humans should not be abandoned. We should find a way to make data and humans closely connected. And this method is data visualization. People can better understand your purpose by transforming data into various visual images and feel the connection between this content and their life. The result of data visualization is a combination of art and data science. It is scientific and also takes into account the author’s self-expression. It can be said to be a good combination of humanism and data. And also, Data humanism is a philosophy and approach to data analysis and decision-making that emphasizes the importance of understanding the human context and impact of data. It stresses the need to consider the ethical and societal implications of data usage and ensure that data is used in a way that benefits individuals and society. Data humanism also emphasizes the importance of transparency and accountability in data usage and encourages the active participation of individuals and communities in the data decision-making process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*ELuIZm3sysMbFq2Z" /></figure><h3>Data Humanism and Data Visualization</h3><p>Most of the previous explanations of data humanism in this article stay in terms of history and central ideas. These are abstract interpretations, but for data science, data humanism is a concrete way of working. In fact, in my mind, data humanism is actually a kind of advanced data visualization which adds more humanistic considerations and artistic pursuits than traditional data visualization. So obviously, in order to understand data humanism, we need to understand data visualization first. Next, I will use some pictures and two examples to help you understand data visualization.</p><h3>Data Visualization and examples</h3><p>Consider the following matrix, and you need to find all the numbers 9 from this matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/165/0*oMSzgs6bPLsW-cxk.png" /></figure><p>This is not a difficult task, but it still takes a lot of time. So what if the matrix becomes like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/126/0*lWB3IInh7bBg4AdU.png" /></figure><p>I marked all the nines with green squares so that you can complete the task in a second.</p><p>This is the power of vision, in this case, the matrix is equivalent to the raw data we got, and the green square marks are an easy way to visualize the key points. Of course, in general, we will not use this simple method but use various charts for data visualization.</p><h3>Basic Chart</h3><p>First, let’s take a look at the first data set we will use — — <strong>Covid Deaths and Cases Worldwide</strong></p><pre>import pandas as pd   <br>import numpy as np<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br>import regex as re</pre><pre>df=pd.read_csv(&#39;../content/sample_data/covid_worldwide.csv&#39;, index_col=0)<br>df.describe()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/0*BHraEeA5c2tX4H0q.png" /></figure><p>The data in this dataset is string, this is not conducive to our analysis, so we convert all data to float type.</p><pre>df[&#39;Total Deaths&#39;] = df[&#39;Total Deaths&#39;].astype(str).str.replace(&#39;,&#39;, &#39;&#39;).astype(float)<br>df[&#39;Total Cases&#39;] = df[&#39;Total Cases&#39;].astype(str).str.replace(&#39;,&#39;, &#39;&#39;).astype(float)<br>df[&#39;Total Recovered&#39;] = df[&#39;Total Recovered&#39;].astype(str).str.replace(&#39;,&#39;, &#39;&#39;).astype(float)<br>df[&#39;Active Cases&#39;] = df[&#39;Active Cases&#39;].astype(str).str.replace(&#39;,&#39;, &#39;&#39;).astype(float)<br>df[&#39;Total Test&#39;] = df[&#39;Total Test&#39;].astype(str).str.replace(&#39;,&#39;, &#39;&#39;).astype(float)<br>df[&#39;Population&#39;] = df[&#39;Population&#39;].astype(str).str.replace(&#39;,&#39;, &#39;&#39;).astype(float)<br>df.head(10)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/0*wfJ6L8GF4IcFxKY0.png" /></figure><p>Here, if we want to explore the number of deaths in the top ten death countries, we can use the <strong>histogram</strong>. Of course, before drawing, we also need to sort the required data.</p><pre>topDeath = df.sort_values(by=&quot;Total Deaths&quot;,ascending=False)<br>topDeath = topDeath.head(10)<br>topDeath</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/0*OFEHeEokA6S5ggFC.png" /></figure><pre>plt.bar([x for x in range(10)], topDeath[&#39;Total Deaths&#39;], label=&quot;Deaths&quot;, width=0.4)<br>plt.xticks([x for x in range(10)], topDeath[&#39;Country&#39;], rotation=330)<br>plt.xlabel(&quot;Country&quot;)<br>plt.ylabel(&quot;Numbers&quot;)<br>plt.title(&quot;Top 10 Deaths Country&quot;)<br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/391/0*HmyYO_RuEHXZo6A-.png" /></figure><p>This is the simplest histogram, we can intuitively see the number of deaths in each country.</p><p>But sometimes, we want to know more. For example, we want to explore the relationship between the number of tests and the number of cases in the countries with the top ten highest numbers of deaths.</p><p>In this case, we can use a <strong>two-column histogram.</strong></p><pre>totalWidth=0.8 <br>labelNums=2 <br>barWidth=totalWidth/labelNums <br>seriesNums=10 </pre><pre>plt.bar([x for x in range(seriesNums)], topDeath[&#39;Total Test&#39;], label=&quot;Test&quot;, width=barWidth)<br>plt.bar([x+barWidth for x in range(seriesNums)], topDeath[&#39;Total Cases&#39;], label=&quot;Cases&quot;, width=barWidth)</pre><pre>plt.xticks([x+barWidth/2*(labelNums-1) for x in range(seriesNums)], topDeath[&#39;Country&#39;], rotation=330)<br>plt.xlabel(&quot;Country&quot;)<br>plt.ylabel(&quot;Numbers&quot;)<br>plt.title(&quot;Tests and Cases of Top 10 Deaths Country&quot;)<br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/385/0*im5-cn-yrxL6Rm9R.png" /></figure><p>In the figure above, we compare the relationship between the number of tests and the number of cases in the countries with the top ten highest number of deaths.</p><p>In the similar way, we can also draw a histogram with three or more columns.</p><p>We also have another way to represent the same situation, which is to use a <strong>stacked-histogram</strong></p><pre>plt.bar([x for x in range(10)], topDeath[&#39;Total Test&#39;], label=&quot;Test&quot;, width=0.4)<br>plt.bar([x for x in range(10)], topDeath[&#39;Total Cases&#39;], label=&quot;Cases&quot;, width=0.4, bottom=topDeath[&#39;Total Test&#39;])</pre><pre>plt.xticks([x for x in range(10)], topDeath[&#39;Country&#39;], rotation=330)<br>plt.xlabel(&quot;Country&quot;)<br>plt.ylabel(&quot;Numbers&quot;)<br>plt.title(&quot;Tests and Cases of Top 10 Deaths Country&quot;)<br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*7gZXYAkcSdddZ4VF.png" /></figure><p>After looking at so many histograms, we can find that histograms have their limitations. The histogram does not have a clear overall concept. Although we can compare different parts of data, but if you want to know the proportion of each different part, you cannot draw conclusions intuitively.</p><p>And in this case you can use <strong>pie chart</strong>，let’s take the sum of the deaths of the top ten deaths countries as 1 to see what the proportion of each country is.</p><pre>plt.title(&#39;Deaths proportion of top ten deaths country&#39;)<br>plt.pie(topDeath[&#39;Total Deaths&#39;],autopct=&#39;%1.1f%%&#39;, labels=topDeath[&#39;Country&#39;])<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/316/0*r6tMnJZFp6XXqMAt.png" /></figure><p>In this example, we explore several basic ways of data visualization, using various methods, that can be used to compare the relationship between one or more aspects of different objects.</p><p>Next, let’s move on to another example. We’re going to illustrate how to use a graph to show the changing trend of a certain property of an object.</p><p>Dataset 2 — <strong>Rainfall in India</strong></p><pre>df2=pd.read_csv(&#39;/content/sample_data/district wise rainfall normal.csv&#39;, index_col=0)<br>df2</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kqknhZDIzaqNEAxm.png" /></figure><p>In this data set, we know the monthly rainfall for each region of India, and these data have a clear time order.</p><p>If we want to know the rainfall in the NICOBAR area during the year, we can use a <strong>scatterplot</strong></p><pre>nico=df2[[&#39;DISTRICT&#39;, &#39;JAN&#39;, &#39;FEB&#39;, &#39;MAR&#39;, &#39;APR&#39;, &#39;MAY&#39;, &#39;JUN&#39;, &#39;JUL&#39; ,&#39;AUG&#39;, &#39;SEP&#39;, &#39;OCT&#39;, &#39;NOV&#39;, &#39;DEC&#39;]].head(1)<br>plt.scatter([&#39;JAN&#39;, &#39;FEB&#39;, &#39;MAR&#39;, &#39;APR&#39;, &#39;MAY&#39;, &#39;JUN&#39;, &#39;JUL&#39; ,&#39;AUG&#39;, &#39;SEP&#39;, &#39;OCT&#39;, &#39;NOV&#39;, &#39;DEC&#39;], nico[[&#39;JAN&#39;, &#39;FEB&#39;, &#39;MAR&#39;, &#39;APR&#39;, &#39;MAY&#39;, &#39;JUN&#39;, &#39;JUL&#39; ,&#39;AUG&#39;, &#39;SEP&#39;, &#39;OCT&#39;, &#39;NOV&#39;, &#39;DEC&#39;]])<br>plt.xlabel(&#39;month&#39;)<br>plt.ylabel(&#39;rainfall&#39;)<br>plt.title(&#39;Monthly Rainfall in NICOBAR&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/0*Vgb24bwph6Qmsn_P.png" /></figure><p>In the picture above, we can indeed vaguely find out the trend of rainfall changes, but it is not really intuitive. Scatter plots are more used in classification. In order to clearly know the trend of rainfall over time, we should use <strong>line chart</strong>.</p><pre>plt.plot([&#39;JAN&#39;, &#39;FEB&#39;, &#39;MAR&#39;, &#39;APR&#39;, &#39;MAY&#39;, &#39;JUN&#39;, &#39;JUL&#39; ,&#39;AUG&#39;, &#39;SEP&#39;, &#39;OCT&#39;, &#39;NOV&#39;, &#39;DEC&#39;], [nico[x] for x in [&#39;JAN&#39;, &#39;FEB&#39;, &#39;MAR&#39;, &#39;APR&#39;, &#39;MAY&#39;, &#39;JUN&#39;, &#39;JUL&#39; ,&#39;AUG&#39;, &#39;SEP&#39;, &#39;OCT&#39;, &#39;NOV&#39;, &#39;DEC&#39;]],alpha=0.8, label=&#39;NICOBAR&#39;)<br>plt.xlabel(&#39;month&#39;)<br>plt.ylabel(&#39;rainfall&#39;)<br>plt.title(&#39;Monthly Rainfall in NICOBAR&#39;)<br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/410/0*RljfwEnh4QGJoHcr.png" /></figure><p>In this way, we can clearly see the trend of rainfall over time. Of course, we can also add data from more regions to compare the trends of them.</p><pre>sa=df2[[&#39;DISTRICT&#39;, &#39;JAN&#39;, &#39;FEB&#39;, &#39;MAR&#39;, &#39;APR&#39;, &#39;MAY&#39;, &#39;JUN&#39;, &#39;JUL&#39; ,&#39;AUG&#39;, &#39;SEP&#39;, &#39;OCT&#39;, &#39;NOV&#39;, &#39;DEC&#39;]][1:2]<br>plt.plot([&#39;JAN&#39;, &#39;FEB&#39;, &#39;MAR&#39;, &#39;APR&#39;, &#39;MAY&#39;, &#39;JUN&#39;, &#39;JUL&#39; ,&#39;AUG&#39;, &#39;SEP&#39;, &#39;OCT&#39;, &#39;NOV&#39;, &#39;DEC&#39;], [nico[x] for x in [&#39;JAN&#39;, &#39;FEB&#39;, &#39;MAR&#39;, &#39;APR&#39;, &#39;MAY&#39;, &#39;JUN&#39;, &#39;JUL&#39; ,&#39;AUG&#39;, &#39;SEP&#39;, &#39;OCT&#39;, &#39;NOV&#39;, &#39;DEC&#39;]],alpha=0.8, label=&#39;NICOBAR&#39;)<br>plt.plot([&#39;JAN&#39;, &#39;FEB&#39;, &#39;MAR&#39;, &#39;APR&#39;, &#39;MAY&#39;, &#39;JUN&#39;, &#39;JUL&#39; ,&#39;AUG&#39;, &#39;SEP&#39;, &#39;OCT&#39;, &#39;NOV&#39;, &#39;DEC&#39;], [sa[x] for x in [&#39;JAN&#39;, &#39;FEB&#39;, &#39;MAR&#39;, &#39;APR&#39;, &#39;MAY&#39;, &#39;JUN&#39;, &#39;JUL&#39; ,&#39;AUG&#39;, &#39;SEP&#39;, &#39;OCT&#39;, &#39;NOV&#39;, &#39;DEC&#39;]],alpha=0.8, label=&#39;SOUTH ANDAMAN&#39;)<br>plt.xlabel(&#39;month&#39;)<br>plt.ylabel(&#39;rainfall&#39;)<br>plt.title(&#39;Monthly Rainfall in NICOBAR and SOUTH ANDAMAN&#39;)<br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/0*aAiYT3-Xd2fTFZ4X.png" /></figure><p>Apparently the rainy season in SOUTH ANDAMAN is more violent</p><p>Through the above two examples, we have learned a lot of basic charts for data visualization. After years of development, in fact, the current data visualization have more intuitive and diverse forms.</p><h3>Advanced Data Visualizaion</h3><p>First of all, of course, through continuous development, we have more types of charts, such as mekko chart, radar chart, circular rose diagram and so on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/0*XdwyDDYfwTBoSQeq.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/0*fmvws0_gNohUB4Hu.png" /></figure><h3>BI(Business Intelligence) System</h3><p>In addition to the increase in the types of these charts, the biggest progress in the data visualization is the birth of Business Intelligence system.</p><p>It refers to an operation interaction system of specific advanced analysis technology, which can allow business personnel to participate more in data processing and chart production by accessing enterprise data, thereby releasing the development pressure of technical personnel and increasing the participation of other personnel Accuracy and control over data, making analysis work more agile and efficient.</p><p>Compared with various previous data visualization methods, the biggest change in BI lies in the enhancement of interactivity. The data is made into a UI interface. Users can dynamically and interactively display high-dimensional data through HTML5, JS and other technologies. And even if users have no technical background, they can also select and drill down on the data they are interested in.</p><h3>In the End, Data Humanism</h3><p>In the latest data visualization attempts, data has become a variety of interactive, dynamic and concise elements.</p><p>data humanism wants to go a step further, to add some elements of humanism — artistry.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xcFUD6qfCYjVSyB_.png" /></figure><p>Each week, people represented their activities in a unique and creative way. They decided to represent each week with a bouquet of flowers, one for each activity. This person assigned a different activity to each flower, including fitness, family time, food, and more. To track their progress, this person came up with a clever system: the more they engaged in an activity, the more petals they earned for the corresponding flower. Their goal was to win up to 5 petals for each flower by completing a week’s worth of goals. This method allowed the person to visually see their progress and identify areas for improvement. At the end of each week, they had a beautiful bouquet of flowers representing their activities and accomplishments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/0*AkjZNvX-mOW1LDcr.png" /></figure><ul><li>It is a vision that wants their bouquet to evolve as they grow as individuals. They envisioned that the composition of the flowers would change as they built stronger habits and discovered new interests.</li><li>During the second week of the experiment, the first digital family gathering took place. They found this activity to be very fulfilling and decided to make it a weekly habit. To represent this new activity, this person added a new flower, purple salvia, to their bouquet and began to track its development.</li><li>This person is excited about the possibility of their bouquet growing and developing in this way and is looking forward to seeing the changes in their personal development.</li></ul><h3>Conclusion</h3><p>We must acknowledge that while data humanism brings about an optimistic approach to integrating ethical and human-centered principles into data-driven technologies, we must scrutinize its potential limitations and flaws. The difficulty in executing data humanism principles lies in the intricacy of data, the competing interests of stakeholders, and rapid technological advances. Those who are benefiting from the present data-driven processes may resist the implementation of these principles, creating obstacles to widespread adoption. To tackle these challenges, continued research, dialogue, and interdisciplinary collaboration are imperative to find a delicate balance between the idealistic vision of data humanism and the reality of our dependence on data. It is vital that data humanism does not become an overused buzzword but indeed becomes a practical framework to develop a fair and ethical digital society.</p><h3>Reference:</h3><p>[1] Qlik-oss. (n.d.). Sn-mekko-chart — 梅科图. GitHub. <a href="https://github.com/qlik-oss/sn-mekko-chart">https://github.com/qlik-oss/sn-mekko-chart</a></p><p>[2] Zhihu. (n.d.). 什么是梅克图？. Retrieved February 23, 2023, from <a href="https://www.zhihu.com/question/52240981">https://www.zhihu.com/question/52240981</a></p><p>[3] Kaggle. (n.d.). Rainfall in India. Retrieved February 23, 2023, from <a href="https://www.kaggle.com/datasets/rajanand/rainfall-in-india?select=district+wise+rainfall+normal.csv">https://www.kaggle.com/datasets/rajanand/rainfall-in-india?select=district+wise+rainfall+normal.csv</a></p><p>[4] Kaggle. (n.d.). Covid deaths and cases worldwide. Retrieved February 23, 2023, from <a href="https://www.kaggle.com/code/finnheaslop/beginner-covid-deaths-and-cases-work-in-prog/notebook">https://www.kaggle.com/code/finnheaslop/beginner-covid-deaths-and-cases-work-in-prog/notebook</a></p><p>[5] Zhihu. (n.d.). 饼图与其他图形. Retrieved February 23, 2023, from <a href="https://zhuanlan.zhihu.com/p/345262150">https://zhuanlan.zhihu.com/p/345262150</a></p><p>[6] Zhihu. (n.d.). 什么是数据可视化？. Retrieved February 23, 2023, from <a href="https://zhuanlan.zhihu.com/p/162338503">https://zhuanlan.zhihu.com/p/162338503</a></p><p>[7] Khowala, D. (2018, June 7). Data Humanism: Visualizing data to connect people with numbers. Devika Khowala. <a href="https://devikakhowala.com/data-humanism">https://devikakhowala.com/data-humanism</a></p><p>[8] Zhihu. (n.d.). 数据可视化. Retrieved February 23, 2023, from <a href="https://zhuanlan.zhihu.com/p/439354353">https://zhuanlan.zhihu.com/p/439354353</a></p><p>[9] Winkler, L. (2019, July 15). Data Visualization for Humans: How I Turned My Data Into Watercolour Art. Towards Data Science. <a href="https://towardsdatascience.com/data-visualization-for-humans-how-i-turned-my-data-into-watercolour-art-651d6acb16a3">https://towardsdatascience.com/data-visualization-for-humans-how-i-turned-my-data-into-watercolour-art-651d6acb16a3</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72e06a4f36a0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/aiskunks/crash-crouse-in-data-data-humanism-72e06a4f36a0">Crash Crouse in Data : Data Humanism</a> was originally published in <a href="https://medium.com/aiskunks">AI Skunks</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>