<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Simon Greaves on Medium]]></title>
        <description><![CDATA[Stories by Simon Greaves on Medium]]></description>
        <link>https://medium.com/@simongreaves?source=rss-6f9a3d06845e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*0L4pHSfIrzdVLvnLGlDX1A.png</url>
            <title>Stories by Simon Greaves on Medium</title>
            <link>https://medium.com/@simongreaves?source=rss-6f9a3d06845e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 08:23:09 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@simongreaves/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Forecasting the Evolution of the Active Blood Donor Base In Canada Using a Markov Process]]></title>
            <link>https://medium.com/@simongreaves/forecasting-the-evolution-of-the-active-blood-donor-base-in-canada-using-a-markov-process-96c86611934b?source=rss-6f9a3d06845e------2</link>
            <guid isPermaLink="false">https://medium.com/p/96c86611934b</guid>
            <category><![CDATA[markov-models]]></category>
            <category><![CDATA[forecasting-models]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Simon Greaves]]></dc:creator>
            <pubDate>Wed, 02 Jul 2025 15:21:07 GMT</pubDate>
            <atom:updated>2025-07-02T15:21:07.087Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/515/1*2XHZeacYNN4FYXECMjTOKg.png" /><figcaption>Image: Simplified example graph of donor transition process</figcaption></figure><h3>Introduction</h3><p>Managing a sustainable blood supply requires an accurate understanding of donor retention and attrition. In this project, we developed a Markov-based forecasting model to predict the evolution of the active donor base over the next five years. By analyzing donor transitions between different states — new, continuing, reinstated, and lapsed — we estimated future donor availability, providing insights to support operational planning and donor recruitment strategies.</p><h3>Understanding Donor Transitions</h3><p>Donors are categorized into four mutually exclusive states:</p><ul><li><strong>NEW</strong>: First-time donors in the current year.</li><li><strong>CONTINUING</strong>: Donors who also donated in the previous year.</li><li><strong>REINSTATED</strong>: Previously lapsed donors who return to donate.</li><li><strong>LAPSED</strong>: Donors who did not donate in the current year but had donated in the past.</li></ul><p>At any given time, donors transition between these states based on their donation patterns, making this a natural Markov process. Each year, new donors enter the system, some continuing donors lapse, and some lapsed donors reinstate, changing the composition of the active donor base.</p><h3>Constructing the Transition Probability Matrix</h3><p>To model these transitions, we:</p><p><strong>1. Collected Historical Data</strong>: Donation appointment records from multiple years were retrieved, including de-identified donor numbers and donation dates.</p><p><strong>2. Defined Transition Criteria</strong>:</p><ul><li>A donor donating for the first time in the dataset’s time frame is classified as new.</li><li>A donor who donated in both the previous and current year is continuing.</li><li>A donor who skipped at least one year and then donated again is reinstated.</li><li>A donor who donated in a prior year but not in the current year is lapsed.</li></ul><p><strong>3. Computed Transition Probabilities</strong>:</p><ul><li>For each donor, historical donation behavior was analyzed to determine the probability of transitioning between states.</li><li>A transition probability matrix (TPM) was constructed, where each row represents the current state and each column represents the probability of moving to a different state in the next year.</li><li>Sub-states were introduced based on donation frequency over one year, with 7 being the maximum donations possible over a year (e.g., continuing_1, continuing_2, …, continuing_7) to capture variability in donor behavior. The maximum number of donations is due to the effects of blood donation on iron levels.</li><li>Lapsed donors were further classified by duration in months(e.g., lapsed_12, lapsed_24) to improve transition estimates. The transition probability matrix takes the form:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/192/1*Xj0i3Zb5h2Yv1YOwp3eXHg.png" /></figure><p>where <em>state i</em> is a state in the previous year and <em>state j</em> is a state in the current year, with the probability of moving from state i to state j is given by the formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*qEgkSVkOwIZukeyHfuu1wQ.png" /></figure><p>Each element represents the probability of a donor transitioning from state to state in the next year, based on empirical data. The probability of moving from one state to another is calculated as the ratio of the number of donors who transitioned to the target state in the following year to the total number of donors in the current state. Each state also consists of 7 sub-states where the number of donations over the year represents the sub-state.</p><h3>Forecasting the Active Donor Base</h3><p>Using the transition matrix, we estimated the number of active donors each year:</p><p><strong>1. Matrix Multiplication for State Evolution</strong>:</p><ul><li>The donor base in year <em>Dt+1 </em>was obtained by multiplying the donor distribution vector in year <em>t</em> by the transition matrix <em>P, </em>then adding the new donors in year <em>t+1</em>:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/151/1*oHM2rfcyYstgFnng44danQ.png" /></figure><p><strong>2. Iterative Projections</strong>:</p><ul><li>This process was repeated iteratively over a five-year horizon to track how donor numbers evolve.</li></ul><p><strong>3. Estimating Total Donations</strong>:</p><ul><li>The total expected donations were derived by multiplying each donor segment by the number of donations per year:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/307/1*8vzWR_x8GjKrWMPvAcizxw.png" /></figure><p>where <em>sub-state_jd</em> represents the number of donors in <em>statej </em>estimated to donate <em>d</em> times.</p><h3>Broader Context</h3><p>While this article focuses on transition matrix calculation and donor base forecasting, the project also included:</p><ul><li>Comparing forecasted donations to expected demand.</li><li>Defining an optimization model to determine new donor acquisition targets.</li><li>Creating a weighted allocation model to identify the optimal FSAs in which to seek out the required new donors</li></ul><p>This approach currently supports strategic decision-making in our planning, helping to balance supply and demand efficiently.</p><h3>Conclusion</h3><p>While this article simplifies the reality of this project, it illustrates how well it reduced the complexity of various inter-connected processes into one single, computable Markov Process. This Markov-based forecasting approach provides a structured method for forecasting the active donor base, allowing for more informed planning in donor recruitment and retention strategies. Markov models are a very useful alternative to common time-series analysis methods when forecasting KPIs that are inter-connected and where individuals states or stages are clearly defined and identifiable.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96c86611934b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[On A/B Testing — T-Test]]></title>
            <link>https://medium.com/@simongreaves/on-a-b-testing-t-test-4d3816f77adf?source=rss-6f9a3d06845e------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d3816f77adf</guid>
            <category><![CDATA[ab-testing]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[t-test]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Simon Greaves]]></dc:creator>
            <pubDate>Mon, 16 Jun 2025 18:00:23 GMT</pubDate>
            <atom:updated>2025-06-16T18:00:23.894Z</atom:updated>
            <content:encoded><![CDATA[<h3>On A/B Testing — T-test</h3><p>(And when it’s not enough)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*FAgaaIYwpTKlXgUMOwz86w.png" /></figure><p><em>A/B testing is </em>how you decide which version of a product, process, webpage, or ad performs better. At its core, A/B testing is about comparing two or more variants and determining which one performs best — with statistical confidence. In this series, we’ll explore all the major types of tests used in A/B testing, beginning with one of the simplest and most common: the <strong>T-test</strong>.</p><h3>What Is a T-test?</h3><p>A T-test is a statistical method used to determine whether there’s a significant difference between the <strong>means</strong> of two groups. It’s the go-to method when:</p><ul><li>You have <strong>two variants</strong> (e.g., Version A vs. Version B).</li><li>Your <strong>metric of interest is continuous</strong> (like time on site, revenue, click-through rate, etc.).</li><li>Your <strong>data is (approximately) normally distributed</strong>.</li><li>You don’t know the population standard deviation.</li></ul><h3>A Simple Example: Comparing conversion rates</h3><p>Let’s say we’re testing two processes to handle blood donations.</p><ul><li><strong>Version A</strong> processed 100 donations.</li><li><strong>Version B</strong> processed 120 donations.</li></ul><p>You might be tempted to say B is better — 120 &gt; 100 — but to know if the difference is <em>statistically significant</em> you could use a t-test. This could be for comparing two similar locations using two different versions of a process, or the same location over two separate time periods.</p><h3>Types of T-tests</h3><ol><li><strong>Independent Samples T-test</strong>: Compares means between <em>two separate groups</em>.</li><li><strong>Paired Samples T-test</strong>: Compares means from <em>the same group at two time points</em> (before vs. after).</li></ol><h3>Example 1: Do two groups have different Ferritin levels?</h3><p>A blood operator is concerned about iron depletion in high-frequency donors. To investigate, they compare <strong>ferritin levels</strong> (a marker of iron stores) between donors of regular vs high-frequency.</p><p>You can compare the ferritin levels to see if there’s a statistically significant difference between the two.</p><h3>Python example (Independent T-test)</h3><pre>import scipy.stats as stats<br>import numpy as np<br><br># Simulated ferritin levels<br>group_base = np.array([140, 145, 132, 155, 120, 135, 144, 145, 129, 135])<br>group_high = np.array([125, 135, 130, 122, 117, 138, 136, 111, 137, 136])<br><br># Perform independent T-test<br>t_stat, p_value = stats.ttest_ind(group_base, group_high)<br><br>print(f&quot;Group Base average ferritin level: {np.mean(group_base)}&quot;)<br>print(f&quot;Group High average ferritin level: {np.mean(group_high)}&quot;)<br><br>print(f&quot;T-statistic: {t_stat:.2f}&quot;)<br>print(f&quot;P-value: {p_value:.4f}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/410/1*GX_b07EwDjJLYxn4fYNc7A.png" /><figcaption>Output — Example 1</figcaption></figure><p>Interpretation: If the p-value is less than 0.05, the difference is considered statistically significant.</p><h3>Example 2: Did wait times drop after a new scheduling system?</h3><p>A blood clinic introduces a new donor scheduling system. You want to test whether average wait times decreased.</p><p>Compare average wait times (in minutes) for 10 donors <em>before</em> and <em>after</em> the system went live.</p><h3>Python example (Paired T-test)</h3><pre># Wait times in minutes before and after system launch<br>before = np.array([18, 19, 21, 17, 16, 19, 12, 11, 17, 18])<br>after = np.array([10, 14, 12, 11, 19, 13, 15, 12, 14, 10])<br><br># Perform paired T-test<br>t_stat, p_value = stats.ttest_rel(before, after)<br><br>print(f&quot;Average wait time before launch: {np.mean(before)}&quot;)<br>print(f&quot;Average wait time after launch: {np.mean(after)}&quot;)<br><br>print(f&quot;T-statistic: {t_stat:.2f}&quot;)<br>print(f&quot;P-value: {p_value:.4f}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/379/1*sbQrSYeu1SZLQ0F38lIbKw.png" /><figcaption>Output — Example 2</figcaption></figure><p>Interpretation is same as before.</p><h3>Best practices</h3><ul><li><strong>Data Visualization -</strong> Boxplots or histograms can help spot outliers and non-normal distributions. This is important as averages are highly affected by outliers and may skew your results.</li><li><strong>Check assumptions -</strong> Use a normality test (e.g., Shapiro-Wilk) if unsure.</li></ul><h3>In reality</h3><p>In reality things are almost always not as simple as comparing two groups.</p><h4>When a T-test is not enough</h4><ul><li>Comparing <strong>3+ groups -&gt; ANOVA</strong></li><li>Binary outcome -&gt; Chi-square/Logistic Regression</li><li>Non-normal or unequal variance -&gt; Mann Whitney U or Welch’s T-test</li><li>Multiple outcomes per subject -&gt; MANOVA or Mixed-Effect Models</li><li>Need to control for other covariates -&gt; Regression</li><li>Measure change over a time series -&gt; ARIMA</li></ul><h3>To conclude</h3><p>So that’s it for the T-test— simple and effective when applied correctly, just make sure that it’s enough for what you need and that your data meets all the test’s assumptions before you use it.</p><p>I didn’t include R examples, but if there’s an interest, send me a note.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d3816f77adf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Grouped Time Series Forecast Confidence Intervals With Monte Carlo Simulation]]></title>
            <link>https://medium.com/@simongreaves/grouped-time-series-forecast-confidence-intervals-with-monte-carlo-simulation-31cb9313ac9f?source=rss-6f9a3d06845e------2</link>
            <guid isPermaLink="false">https://medium.com/p/31cb9313ac9f</guid>
            <category><![CDATA[monte-carlo-simulation]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[confidence-interval]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[forecasting]]></category>
            <dc:creator><![CDATA[Simon Greaves]]></dc:creator>
            <pubDate>Wed, 15 Jan 2025 15:44:46 GMT</pubDate>
            <atom:updated>2025-01-15T15:44:46.542Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*YprsXbVYHUn_l2R8asquhw.png" /></figure><h3>Introduction</h3><p>Time series forecasting is crucial for predicting future values based on historical data. While there are many resources for forecasting single time series, real-world scenarios often involve grouped time series. For example, forecasting blood collections for a blood donor organization requires predictions for individual clinics and a grouped total.</p><p>Summing individual forecasts gives a grouped prediction, but accurate confidence intervals are harder to calculate due to the complex interaction of uncertainties across forecasts.</p><p>This article demonstrates a solution using Monte Carlo simulation to sample forecast distributions, simulate grouped totals, and derive confidence intervals. This method better captures uncertainty, supporting more informed decisions. You can view the R code.</p><p>We’ll illustrate this process through three steps:</p><ol><li>Simulating three distinct time series.</li><li>Forecasting with Facebook’s Prophet model</li><li>Applying Monte Carlo simulation to aggregate results.</li></ol><p>Note that this method will be just as applicable if other statistics or methods of grouping time series are needed, such as averages, percentiles, etc.</p><h3>Why Traditional Methods Fall Short</h3><p>Traditional methods for aggregating time series forecasts often focus on summing point forecasts, which works well for predictions but falls short when estimating uncertainty. Confidence intervals cannot simply be added together because the variability of individual forecasts won’t align or cancel out as expected.</p><p>Additionally, these methods do not account for the correlations between individual time series, which can result in underestimating or overestimating the confidence range for the grouped total. This limitation makes traditional approaches less reliable for grouped time series forecasting, especially when precision is critical.</p><p>Monte Carlo simulation addresses these issues by generating a distribution for the aggregate through repeated sampling, capturing the full scope of variability and interaction between forecasts.</p><h3>Overview of the Solution</h3><p>This solution uses Monte Carlo simulation to overcome the limitations of traditional methods. It involves the following steps:</p><ol><li><strong>Simulating time series data</strong>: Generate sample time series with distinct trends, seasonality, randomness, and weekday effects to represent real-world scenarios.</li><li><strong>Forecasting individual series</strong>: Use Facebook’s Prophet model to forecast each series, capturing trends and seasonality while providing confidence intervals.</li><li><strong>Simulating grouped forecasts</strong>: Apply Monte Carlo simulation by sampling from the forecast distributions of each series and aggregating the samples repeatedly to form a distribution for the total.</li></ol><p>This process not only generates a grouped forecast but also provides accurate confidence intervals that reflect the uncertainty inherent in grouped forecasts. The next sections will demonstrate each step in detail.</p><h3>Step-by-Step Process</h3><h4>Simulating Time Series Data</h4><p>We will not go into detail for the creation of the simulated time series data, as this could be an article on its own. But I’ll state the general steps taken.</p><p>We generate three distinct time series with varying behaviors:</p><ul><li><strong>Trend</strong>: A steady increase or decrease over time.</li><li><strong>Seasonality</strong>: Periodic patterns such as yearly fluctuations.</li><li><strong>Randomness</strong>: Noise or variability to represent unpredictable factors.</li><li><strong>Weekday effects:</strong> Simulating the impact of different days of the week.</li></ul><p>We will split this time series data into ‘train’ and ‘test’ sets, with train data consisting of 2020 to 2023 data, and test data consisting of all 2024 data.</p><pre>pkgs &lt;- c(&quot;dplyr&quot;, &quot;forecast&quot;, &quot;ggplot2&quot;, &quot;tidyquant&quot;, &quot;lubridate&quot;)<br>inst &lt;- lapply(pkgs, library, character.only = TRUE)<br><br>simulate_time_series &lt;- function(n, id, start_date, min_value, max_value, growth_rate, weekday_effects, yearly_seasonal_comp, weekly_seasonal_comp) {<br>  # Daily data<br>  freq &lt;- 365<br>  trend &lt;- min_value + max_value / (1 + exp(-growth_rate * (seq_len(n))))<br>  # Yearly seasonality<br>  yearly_seasonal &lt;- yearly_seasonal_comp * sin(2 * pi * seq_len(n) / freq)<br>  # Weekly seasonality<br>  weekly_seasonal &lt;- weekly_seasonal_comp * sin(2 * pi * seq_len(n) / 7)<br>  seasonal &lt;- yearly_seasonal + weekly_seasonal<br>  # Random noise<br>  noise &lt;- rnorm(n, mean = 0, sd = 2)  # Reduced noise for daily data<br>  # Combine components<br>  y &lt;- trend + seasonal + noise<br>  # Create date vector starting at start_date<br>  time &lt;- seq(start_date, by = &quot;day&quot;, length.out = n)<br>  # Set up weekday effects<br>  effects &lt;- data.frame(weekday = c(&quot;Sun&quot;, &quot;Mon&quot;, &quot;Tue&quot;, &quot;Wed&quot;, &quot;Thu&quot;, &quot;Fri&quot;, &quot;Sat&quot;),<br>                        effect = weekday_effects)<br>  data &lt;- data.frame(date=time, value=y, id=id, weekday = wday(time, label = TRUE))<br>  data &lt;- add_weekday_effect(df=data, effects) |&gt;<br>    select(date, id, value)<br>  return(data)<br>}<br><br>add_weekday_effect &lt;- function(df, weekday_effects) {<br>  df &lt;- df |&gt;<br>    left_join(weekday_effects, by = &quot;weekday&quot;) |&gt;<br>    mutate(value = value * runif(n(), effect - 0.025, effect + 0.025))<br>  return(df)<br>}<br><br># Set seed<br>set.seed(123)<br># Set up time series parameters<br>start_date &lt;- as.Date(&quot;2020-01-01&quot;)<br>train_test_threshold &lt;- as.Date(&quot;2024-01-01&quot;)<br>n &lt;- 365 * 5<br>ts_1 &lt;- simulate_time_series(n=n, id=&quot;1&quot;, start_date=start_date, min_value=100, max_value=200, growth_rate=0.0025, weekday_effects = c(1, 1, 1.08, 1, 1, 1, 1), yearly_seasonal_comp=5, weekly_seasonal_comp=5)<br>ts_2 &lt;- simulate_time_series(n=n, id=&quot;2&quot;, start_date=start_date, min_value=200, max_value=100, growth_rate=0.005, weekday_effects = c(1, 1, 1, 1, 0.95, 1, 1), yearly_seasonal_comp=2, weekly_seasonal_comp=7)<br>ts_3 &lt;- simulate_time_series(n=n, id=&quot;3&quot;, start_date=start_date, min_value=200, max_value=100, growth_rate=-0.005, weekday_effects = c(0.93, 1, 1, 1, 1, 1, 1), yearly_seasonal_comp=7, weekly_seasonal_comp=3)<br><br># Merge all data into one dataframe<br>df &lt;- rbind(ts_1, ts_2, ts_3)<br><br># Plot the values<br>df |&gt;<br>  ggplot(aes(x = date, y = value, group = id, color = id)) +<br>  geom_point() +<br>  labs(title = &quot;Simulated Time Series&quot;, x = &quot;&quot;, y = &quot;Value&quot;,<br>       subtitle = &quot;Three time series with different characteristics&quot;) +<br>  scale_y_continuous() +<br>  theme_tq()<br><br># Aggregate into one dataframe<br>df_agg &lt;- df |&gt;<br>  group_by(date) |&gt;<br>  summarize(value = sum(value)) |&gt;<br>  mutate(train = date &lt; train_test_threshold)<br><br># Plot the grouped dataframe<br>df_agg |&gt;<br>  ggplot(aes(x = date, y = value, group = train, color = train)) +<br>  geom_point() +<br>  labs(title = &quot;Grouped Time Series&quot;, x = &quot;&quot;, y = &quot;Value&quot;,<br>       subtitle = &quot;Sum of the three time series&quot;) +<br>  scale_y_continuous() +<br>  theme_tq()</pre><p>This approach to data simulation offers several advantages:</p><ul><li><strong>Reproducibility</strong>: The use of a seed ensures that the same “random” data can be generated consistently.</li><li><strong>Ground truth</strong>: Since you know the underlying components of the data, you can better evaluate your models’ ability to capture these elements.</li><li><strong>Varying patterns</strong>: This data will help us show how this methodology can account for very specific and varying patterns.</li><li><strong>Experimentation</strong>: You can create different scenarios and see how your model performs under each of them.</li></ul><p>You can see the simulated values for the time series (1, 2, and 3) in the image below. Note that you can see patterns of weekdays on which the different time series have radically different behaviors (Series 1 has an upward trend with a positive Tuesday effect, Series 2 an upward trend with a negative Thursday effect, and Series 3 a downward trend with a negative Sunday effect).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*-oviEDrwSx1N-aaLqdsloQ.png" /><figcaption>Fig. 1: Simulated Time Series from 2020 to 2025</figcaption></figure><p>Below we can see the grouped actual values, split into train and test data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*3-THXsRypKyxCUe25yGzTg.png" /><figcaption>Fig. 2: Grouped data from 2020 to 2025 (grouped by test-train)</figcaption></figure><h4>Forecasting with Prophet</h4><p>To forecast each time series, we employ Facebook’s Prophet model, a powerful tool designed for automatic forecasting of univariate time series data. Prophet is particularly adept at handling data with strong seasonal effects.</p><pre>pkgs &lt;- c(&quot;Rcpp&quot;, &quot;prophet&quot;)<br>inst &lt;- lapply(pkgs, library, character.only = TRUE)<br><br># Individual clinic forecasting<br>forecast_series &lt;- function(dat, series_id, forecast_horizon = 365) {<br>  # Add regressors<br>  start_date &lt;- as.Date(min(dat$ds))<br>  dat &lt;- dat |&gt;<br>    mutate(weekday = wday(ds, label = TRUE),<br>           time_regressor = as.numeric(as.Date(ds) - start_date),<br>           time_regressor_log = log(time_regressor + 1))<br><br>  # Prepare the data for Prophet<br>  m &lt;- prophet(daily.seasonality = FALSE,<br>               weekly.seasonality = TRUE,<br>               yearly.seasonality = TRUE,<br>               growth = &#39;linear&#39;,<br>               changepoint_range = 0.95,<br>               seasonality.mode=&quot;multiplicative&quot;) |&gt;<br>    add_regressor(&#39;weekday&#39;) |&gt;<br>    add_regressor(&#39;time_regressor&#39;) |&gt;<br>    add_regressor(&#39;time_regressor_log&#39;) |&gt;<br>    fit.prophet(dat |&gt; select(ds, y, weekday, time_regressor, time_regressor_log))<br>  <br>  # Create future dataframe<br>  future &lt;- make_future_dataframe(m, periods = forecast_horizon) |&gt;<br>    mutate(weekday = wday(ds, label = TRUE),<br>           time_regressor = as.numeric(as.Date(ds) - start_date),<br>           time_regressor_log = log(time_regressor + 1))<br>  <br>  # Make forecast<br>  prophet_forecast &lt;- predict(m, future)<br>  <br>  # Select only the columns we need and rename to match input data<br>  forecast_df &lt;- prophet_forecast |&gt;<br>    select(ds, yhat, yhat_lower, yhat_upper) |&gt;<br>    rename(y = yhat) |&gt;<br>    mutate(id = series_id,<br>           weekday = wday(ds, label = TRUE),<br>           train = FALSE,<br>           ds = as.Date(ds))<br>  <br>  # Reorder columns to match input data<br>  forecast_df &lt;- forecast_df |&gt;<br>    select(id, ds, y, train, weekday, yhat_lower, yhat_upper)<br>  <br>  return(forecast_df)<br>}<br><br>plot_series_forecast &lt;- function(series_data, forecasts, series_id, start_date) {<br>  # Combine actual data and forecasts<br>  combined_data &lt;- bind_rows(<br>    series_data |&gt;<br>      filter(id == series_id &amp; ds &gt;= start_date) |&gt;<br>      mutate(type = &quot;Actual&quot;),<br>    forecasts |&gt;<br>      filter(id == series_id &amp; ds &gt;= start_date) |&gt;<br>      mutate(type = &quot;Forecast&quot;)) |&gt;<br>    mutate(weekday = wday(ds, label = TRUE))<br>  <br>  ggplot(combined_data, aes(x = ds, y = y)) +<br>    geom_point(data = combined_data |&gt; filter(type == &quot;Actual&quot;),<br>               aes(color = weekday), alpha = 0.6) +<br>    geom_line(data = combined_data |&gt; filter(type == &quot;Forecast&quot;),<br>              color = &quot;blue&quot;, size = 1, alpha = 0.5) +<br>    geom_ribbon(data = combined_data |&gt; filter(type == &quot;Forecast&quot;),<br>                aes(ymin = yhat_lower, ymax = yhat_upper),<br>                fill = &quot;cyan&quot;, alpha = 0.35) +<br>    labs(title = paste(&quot;Forecast for series&quot;, series_id),<br>         x = &quot;Date&quot;, y = &quot;Value&quot;, color = &quot;Weekday&quot;) +<br>    theme_minimal() +<br>    theme(legend.position = &quot;bottom&quot;)<br>}<br><br># Prepare data for Prophet<br>df_prophet &lt;- df |&gt;<br>  mutate(train = date &lt; train_test_threshold) |&gt;<br>  rename(ds = date,<br>         y = value)<br><br># Apply forecast function to each series<br>series_forecasts &lt;- NULL<br>series_ids &lt;- unique(df_prophet$id)<br>for (series_id in series_ids) {<br>  series_data &lt;- df_prophet |&gt;<br>    filter(train &amp; id == series_id)<br>  series_forecasts &lt;- series_forecasts |&gt;<br>    bind_rows(forecast_series(series_data, series_id))<br>}</pre><p>The <em>forecast_series</em> function is defined to handle individual time series forecasting. This function prepares the data, fits a Prophet model with specific parameters, and generates forecasts. It includes regressors for weekday, time, and log-time to capture various patterns in the data.</p><p>The model is applied to each unique series in our dataset. Prophet automatically detects trends and seasonality from the time-series data.</p><p>The output includes point forecasts along with upper and lower bounds, providing a measure of forecast uncertainty. These confidence intervals are crucial for the subsequent Monte Carlo simulation, as they form the basis for sampling and aggregating forecasts.</p><p>By leveraging Prophet’s capabilities, we obtain forecasts that capture the unique characteristics of each time series, setting a solid foundation for our aggregation process.</p><p>If we plot the entire time window, we’ll get a cluttered image. Instead, plotting from 2023 onwards allows us to observe the forecast closely. The model predicts weekday effects accurately, reflecting higher or lower values based on patterns in the original series. As an example, TS-3 shows a significant drop-off on Sundays, which Prophet has picked up on, forecasting much lower values on Sundays than other weekdays.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*gyYLsgNYx3Dq6YHQNEUJ1w.png" /><figcaption>Fig. 3: Time series 3 from 2023 to 2025</figcaption></figure><h4>Monte Carlo Simulation</h4><p>Once the forecasts are ready, Monte Carlo simulation is applied:</p><ul><li><strong>Sampling forecast distributions</strong>: For each time point, random samples are drawn from the forecast distributions of each time series.</li><li><strong>Aggregating samples</strong>: The sampled values are summed across the time series to simulate the grouped total.</li><li><strong>Repeating the process</strong>: This is done thousands of times to build a robust distribution for the grouped total.</li></ul><pre>pkgs &lt;- c(&quot;purrr&quot;)<br>inst &lt;- lapply(pkgs, library, character.only = TRUE)<br><br># Function to generate one simulation<br>simulate_forecast &lt;- function() {<br>  series_forecasts |&gt;<br>    group_by(id, ds) |&gt;<br>    mutate(simulated_y = rnorm(n(), y, (yhat_upper - yhat_lower) / 3.92)) |&gt;<br>    group_by(ds) |&gt;<br>    summarise(total_y = sum(simulated_y))<br>}<br><br># Set the number for the Monte Carlo<br>n_simulations = 1000<br><br># Run simulations<br>simulations &lt;- map(1:n_simulations, ~ simulate_forecast()) |&gt;<br>  bind_rows(.id = &quot;simulation&quot;)<br><br># Calculate aggregate forecast and confidence intervals<br>aggregate_forecast &lt;- simulations |&gt;<br>  group_by(ds) |&gt;<br>  summarise(<br>    y = mean(total_y),<br>    yhat_lower = quantile(total_y, 0.025),<br>    yhat_upper = quantile(total_y, 0.975)) |&gt;<br>  mutate(type = &quot;Forecast&quot;,<br>         train = ds &lt; train_test_threshold)<br><br># Calculate actual aggregate totals<br>actual_totals &lt;- df_prophet |&gt;<br>  group_by(ds, train) |&gt;<br>  summarise(y = sum(y)) |&gt;<br>  mutate(type = &quot;Actual&quot;,<br>         train = ds &lt; train_test_threshold)<br><br># Combine actual totals and aggregate forecast<br>combined_data &lt;- bind_rows(<br>  actual_totals,<br>  aggregate_forecast) |&gt;<br>  mutate(<br>    yhat_lower = ifelse(is.na(yhat_lower), 0, yhat_lower),<br>    yhat_upper = ifelse(is.na(yhat_upper), 0, yhat_upper)) |&gt;<br>  group_by(ds, train, type) |&gt;<br>  summarize(y = sum(y),<br>            yhat_lower = sum(yhat_lower),<br>            yhat_upper = sum(yhat_upper)) |&gt;<br>  mutate(weekday = wday(ds, label = TRUE))<br><br><br># Plot data: Keep only the last year of test data to reduce clutter<br>plot_data &lt;- combined_data |&gt; filter(ds &gt;= plot_start_date)<br># Plot daily<br>ggplot(plot_data, aes(x = ds, y = y)) +<br>  geom_point(data = plot_data |&gt; filter(type == &quot;Actual&quot;),<br>             aes(color = weekday), alpha = 0.6) +<br>  geom_ribbon(data = plot_data |&gt; filter(type == &quot;Forecast&quot;),<br>              aes(ymin = yhat_lower, ymax = yhat_upper),<br>              fill = &quot;cyan&quot;, alpha = 0.8) +<br>  geom_line(data = plot_data |&gt; filter(type == &quot;Forecast&quot;),<br>            color = &quot;blue&quot;, size = 1, alpha = 0.5) +<br>  labs(title = &quot;Aggregate Forecast vs Actuals (daily)&quot;,<br>       x = &quot;Date&quot;, y = &quot;Value&quot;) +<br>  theme_minimal()<br><br># Aggregate simulation data weekly<br>combined_simulations_weekly &lt;- simulations |&gt;<br>  mutate(wk = floor_date(ds, &quot;week&quot;, 1)) |&gt;<br>  group_by(simulation, wk) |&gt;<br>  summarise(<br>    days = n(),<br>    y = sum(total_y)) |&gt;<br>  filter(days == 7)<br><br># Plot weekly aggregate forecast with CIs<br>plot_data_wk &lt;- combined_data_weekly |&gt; filter(wk &gt;= plot_start_date)<br>ggplot(plot_data_wk, aes(x = wk, y = y)) +<br>  geom_point(data = plot_data_wk |&gt; filter(type == &quot;Actual&quot;),<br>             alpha = 0.6) +<br>  geom_line(data = plot_data_wk |&gt; filter(type == &quot;Forecast&quot;),<br>            color = &quot;blue&quot;, size = 1, alpha = 0.5) +<br>  geom_ribbon(data = plot_data_wk |&gt; filter(type == &quot;Forecast&quot;),<br>              aes(ymin = yhat_lower, ymax = yhat_upper),<br>              fill = &quot;cyan&quot;, alpha = 0.4) +<br>  labs(title = &quot;Aggregate Forecast vs Actuals (weekly)&quot;,<br>       x = &quot;Date&quot;, y = &quot;Value&quot;) +<br>  theme_minimal()</pre><h4>Plotting Results</h4><p>Finally, we plot the simulated distribution. The daily plot may look crowded, but the forecast still predicts higher and lower values based on the expected weekday effect (TS-1 Tuesday effect and TS-3 Sunday effect are clearly visible in blue and purple). This behavior originates from the three original time series and is faithfully represented in the grouped forecast. Let’s look at it starting in year 2023:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fHdrFxdCpbgvTZAoGn0BXg.png" /><figcaption>Fig. 4: Grouped forecast for time series 1, 2, and 3 from 2023 to 2025</figcaption></figure><p>However, the confidence intervals are hard to see in daily plots. Aggregating weekly data provides a clearer view, showcasing reliable results and better highlighting the confidence intervals’ performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*KKbeXCImX11Yd3EIqTZo8Q.png" /><figcaption>Fig. 5: Grouped forecast aggregated weekly vs weekly actuals from 2023 to 2025</figcaption></figure><p>The image above demonstrates how well the grouped forecast follows the actual values and we can see how the 95% confidence interval captures the range of likely outcomes, properly reflecting the variability of the grouped time series. This method offers clear insights into the grouped forecast and its uncertainty, enabling better decision-making.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*NEFJq8VIr48jcRBvQ2gN0A.png" /><figcaption>Fig. 6: Grouped forecast vs grouped actuals for 2020 to 2025</figcaption></figure><h3>Conclusion</h3><p>Creating a grouped forecast requires careful consideration of the correlations and uncertainties of the individual time series. Monte Carlo simulation offers a practical approach to aggregate forecasts while preserving uncertainty, providing accurate confidence intervals.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=31cb9313ac9f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simulating Auto-Correlated Data in Python: Code conversion]]></title>
            <link>https://medium.com/@simongreaves/simulating-auto-correlated-data-in-python-code-conversion-58bb4fd99b9c?source=rss-6f9a3d06845e------2</link>
            <guid isPermaLink="false">https://medium.com/p/58bb4fd99b9c</guid>
            <category><![CDATA[r-to-python]]></category>
            <category><![CDATA[autocorrelation]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-simulation]]></category>
            <category><![CDATA[longitudinal-data]]></category>
            <dc:creator><![CDATA[Simon Greaves]]></dc:creator>
            <pubDate>Wed, 23 Oct 2024 03:37:45 GMT</pubDate>
            <atom:updated>2024-10-29T16:20:29.384Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/403/0*PQ2_WbRIVjh4MCcw" /></figure><p>In the previous article, we explored a detailed R script designed to simulate measurement data across a large number of individuals. This script utilized R’s dplyr and other powerful tools to efficiently generate and manipulate synthetic data, offering insights into complex data structures and statistical modeling.</p><p>In this article, we will implement the same logic using Python. By leveraging pandas and numpy, we will recreate the data generation process while maintaining the integrity of the original R workflow. This is a Python exercise for me, in aims to provide a comparison between the R and Python approaches in order to gain a deeper understanding of how to adapt data simulation techniques across different programming environments.</p><p>The code can be found here: <a href="https://github.com/segreaves/Python-RepeatedMeasuresDataSimulation.git">GitHub repo</a></p><h4>Import Libraries</h4><p>We start by importing essential Python libraries for data manipulation and visualization:</p><pre>import numpy as np<br>import pandas as pd<br>import seaborn as sns<br>import matplotlib.pyplot as plt</pre><p>Set the random seed:</p><pre>np.random.seed(42)</pre><h4>Define Parameters</h4><p>Next, we define the parameters for our simulation:</p><ul><li><strong>Number of instances (</strong>n_instances<strong>)</strong>: Represents the total number of individuals in the study.</li><li><strong>Lambda parameter (</strong>lambda_param<strong>)</strong>: Used to determine the number of measurements for each individual, following an exponential distribution. This value cannot be named simply ‘lambda’ like in the R version of this code, this is because in Python lambda is a key word, reserved for the creation of anonymous functions.</li><li><strong>Gender Probability (</strong>prob_male<strong>)</strong>: The likelihood of an individual being male.</li><li><strong>Measurement Baseline and Offsets</strong>: Parameters such as the baseline value of measurements (baseline), gender-specific offsets, and the slopes indicating how measurements change over time.</li></ul><h4>Simulate Individual Profiles</h4><p>Using these parameters, we set up a pandas DataFrame to represent individual profiles:</p><pre>wide = pd.DataFrame({<br>    &#39;id&#39;: np.arange(1, n_instances + 1),<br>    &#39;total_measurements&#39;: np.ceil(np.random.exponential(1/lambda_param, n_instances)).astype(int),<br>    &#39;gender&#39;: np.random.binomial(1, prob_male, n_instances),<br>    &#39;id_offset&#39;: np.random.normal(0, 1, n_instances)<br>})</pre><ul><li>id: A unique identifier for each individual.</li><li>total_measurements: Number of measurements each individual will have, derived using an exponential distribution.</li><li>gender: A binary indicator (0 for female, 1 for male) assigned based on the specified probability.</li><li>id_offset: A random offset value for each individual&#39;s measurements to introduce some variation.</li></ul><h4>Expand to a “Long” Format</h4><p>Next, we reshape the data into a “long” format, where each measurement is represented by a row:</p><pre>long = wide.loc[wide.index.repeat(wide[&#39;total_measurements&#39;])].reset_index(drop=True)</pre><p>In this step:</p><ul><li>loc and repeat: Create multiple rows for each individual based on their total number of measurements. It’s optional to reset the index.</li></ul><p>We then add a simulated time component to represent days between measurements and simulate measurement success:</p><pre>long[&#39;days&#39;] = np.random.uniform(0, max_days, len(long))<br>long[&#39;day&#39;] = long.groupby(&#39;id&#39;)[&#39;days&#39;].cumsum()<br>long[&#39;n_appointment&#39;] = long.groupby(&#39;id&#39;).cumcount() + 1<br>long[&#39;successful_measurement&#39;] = np.random.binomial(1, measurement_prob, len(long))<br>long[&#39;n_measurements&#39;] = long.groupby(&#39;id&#39;)[&#39;successful_measurement&#39;].cumsum()<br>long[&#39;n_successful_measurements&#39;] = long.groupby(&#39;id&#39;)[&#39;successful_measurement&#39;].transform(&#39;sum&#39;)</pre><ul><li>days: Randomly generated intervals (in days) between measurements.</li><li>day: Cumulative sum of days, giving the total time elapsed for each measurement.</li><li>n_appointment: Tracks the appointment number (or measurement count) for each individual.</li><li>successful_measurement: Indicates if a given measurement was successful (1) or not (0).</li><li>n_measurements: Cumulative count of attempts for each individual.</li><li>n_successful_measurements: Total successful measurements for each individual.</li></ul><h4>Calculate Measurement Values</h4><p>Finally, we compute the measurement values based on gender, baseline values, slopes, and individual offsets. We need to ensures that only successful measurements receive a value using np.where(long[‘successful_measurement’] == 1, if_true, if_false), then we calculate the if_true value using the baseline plus id_offset value for every individual and add the male_baseline_offset and male_slope_offset to males only:</p><pre>long[&#39;value&#39;] = np.where(<br>    long[&#39;successful_measurement&#39;] == 1,<br>    baseline + long[&#39;id_offset&#39;] + long[&#39;gender&#39;] * male_baseline_offset + <br>    slope * long[&#39;day&#39;] + long[&#39;gender&#39;] * male_slope_offset * long[&#39;day&#39;] + <br>    np.random.normal(0, 1, len(long)),<br>    np.nan<br>)</pre><ul><li>np.random.normal: Adds random noise to simulate variability.</li><li>Note that if the measurement is not successful (long[‘successful_measurement’] == 1) then valuewill be np.nan.</li></ul><h4>Visualization</h4><p>The final step is to view the data and assess the results.</p><pre>fig = sns.lmplot(x=&#39;day&#39;,<br>                 y=&#39;value&#39;,<br>                 hue=&#39;sex&#39;,<br>                 palette={&#39;M&#39;: &#39;#4a90e2&#39;, &#39;F&#39;: &#39;#e57373&#39;},<br>                 scatter_kws={&#39;alpha&#39;: 0.2},<br>                 data=long)<br><br># Add title<br>plt.title(&#39;Regression of values by gender&#39;)<br># Set x and y labels<br>fig.set_axis_labels(&#39;Days&#39;, &#39;Value&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*4lRl_d41FWtADmoViposFA.png" /></figure><h4>Conclusion</h4><p>This example demonstrates how Python can be used to simulate complex datasets. Such simulated data is valuable for testing and validating analytical models and approaches before applying them to real-world data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58bb4fd99b9c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simulating Auto-Correlated Data in R: A Step-by-Step Guide]]></title>
            <link>https://medium.com/@simongreaves/simulating-auto-correlated-data-in-r-a-step-by-step-guide-1c09f48a8edb?source=rss-6f9a3d06845e------2</link>
            <guid isPermaLink="false">https://medium.com/p/1c09f48a8edb</guid>
            <category><![CDATA[r]]></category>
            <category><![CDATA[data-simulation]]></category>
            <category><![CDATA[longitudinal-data]]></category>
            <category><![CDATA[data-manipulation]]></category>
            <category><![CDATA[dplyr]]></category>
            <dc:creator><![CDATA[Simon Greaves]]></dc:creator>
            <pubDate>Fri, 20 Sep 2024 20:44:03 GMT</pubDate>
            <atom:updated>2024-09-22T18:04:10.372Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*SPnVD-_cklYRKUKIpbyOGg.png" /></figure><p>Simulating data is a valuable technique for model exploration/validation or for when real-world data simply isn’t available. Data scientists frequently encounter longitudinal datasets, where each individual’s repeated measurements over time are stored in separate rows, connected by a common identifier. But creating this type of a data set can get increasingly complicated when you want the data to maintain or respect certain effects and relationships. This article touches on some data manipulation tricks for creating such a data set.</p><p>The code can be found here:</p><p><a href="https://github.com/segreaves/R-RepeatedMeasuresDataSimulation.git">https://github.com/segreaves/R-RepeatedMeasuresDataSimulation.git</a></p><p>Here’s a quick overview of the process. We’ll create a data set of 10,000 individuals (IDs). Each one will have one or more measurement attempts (with decreasing probability). Additionally, we will model a probability of measurement failure, in which case the measurement value will be NA. We will also create a time variable that simulates time between measurement attempts. Finally we will make our measurement value decrease over time at different rates for males and females. Here’s the step-by-step process:</p><ul><li>Create individual IDs and their individual-level variables</li><li>For each individual, create a set of measures with the characteristics that we want to make sure our model picks up</li><li>Convert this data set into a “long” format, where each measurement will be in a separate row</li></ul><h3>Set up the environment</h3><pre>pkgs &lt;- c(&#39;dplyr&#39;, &#39;ggplot2&#39;)<br>inst &lt;- lapply(pkgs, library, character.only = TRUE)</pre><p>We’ll use the Dplyr package for data manipulation and Ggplot to visualize our results.</p><h3>Define the simulation parameters</h3><pre>set.seed(42)<br><br># Number of individual id instances<br>n_instances &lt;- 10000<br># Exp. parameter<br>lambda &lt;- 0.75<br># Model gender<br>prob_male &lt;- 0.5<br># Set up measurement baseline and slope varying by gender<br>baseline &lt;- 100<br>male_baseline_offset &lt;- 10<br>slope &lt;- -0.25<br>male_slope_offset &lt;- -0.1<br># Simulate a 10% change of no-shows<br>measurement_prob &lt;- 0.9<br># Simulate a maximum of 7 days between measurement appointments<br>max_days &lt;- 7</pre><ul><li><strong>Number of individuals (</strong>n_instances<strong>)</strong>: We&#39;ll simulate data for 10,000 individuals.</li><li><strong>Exponential parameter Lambda (</strong>lambda<strong>)</strong>: This parameter will allow us to have some control over the total number of measurements that each individual will have. Though any method for generating a random integer here would do.</li><li><strong>Probability for being assigned male (</strong>prob_male<strong>)</strong>: We&#39;ll assign gender to individuals with a 50% probability of being male.</li><li><strong>Baseline and slope (</strong>baseline<strong>,</strong> male_baseline_offset<strong>,</strong> slope<strong>, </strong>male_slope_offset<strong>)</strong>: These parameters allow us to vary the starting value and rate of change for measurements by gender. This would be a fictitious measurement at day 0 along with the rate of decrease each day for males and females.</li><li><strong>Probability of successful measurement (</strong>measurement_prob<strong>)</strong>: We simulate a 90% chance that an individual will have a measurement.</li><li><strong>Maximum days between measurements (</strong>max_days<strong>)</strong>: We&#39;ll simulate up to 7 days between appointments.</li></ul><h3>Create the Data</h3><p>Now, let’s create a data frame to store the individual profiles. The id_offset variable can be used to create offsets for each id, this allows us to create autocorrelated individuals.:</p><pre># Set up individual profiles<br>wide &lt;- data.frame(id = 1:n_instances,<br>                   total_measurements = ceiling(rexp(n_instances, lambda)),<br>                   gender = rbinom(n_instances,<br>                                   size = 1,<br>                                   prob = prob_male),<br>                   id_offset = rnorm(mean = 0, sd = 1, n = n_instances))<br>head(wide, 4)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/1*otNW5w_HGaGfQ6QQUQsnSw.png" /><figcaption>Fig. 1 First 4individuals (wide format)</figcaption></figure><p>Next, we’ll create the long data frame that expands the wide format to include each measurement for each individual. We will create a number of variables:</p><ul><li><strong>Days between measurements (</strong>days<strong>)</strong>: Number of days since previous appointment (or start of study).</li><li><strong>Day (</strong>day<strong>)</strong>: The day of measurement since the start of the study (day 0).</li><li><strong>Sex of individual (</strong>sex<strong>)</strong>: Sex or gender of individual. This variable has the same value for all rows within a single ID.</li><li><strong>Appointment number (</strong>n_appointment<strong>)</strong>: Same as row number by ID.</li><li><strong>Measurement not NA (</strong>successful_measurement<strong>)</strong>: Does the measurement have a value or is it NA?</li><li><strong>Number of measurements so far (</strong>n_measurements<strong>)</strong>: Number of successful measurements up to this point.</li><li><strong>Total successful measurements (</strong>n_successful_measurements<strong>)</strong>: The total successful measurements over the whole period.</li><li><strong>Fictitious value (</strong>value<strong>)</strong>: The fictitious value we are modelling. Thanks to the gender offset, and id offset, these values are correlated along, gender and individual (ID) lines.</li></ul><pre># Re-format wide to long on number of measurements<br>long &lt;- wide[rep(seq_len(nrow(wide)), wide$total_measurements), ] %&gt;%<br>  mutate(<br>    # Days between measurements<br>    days = runif(n = n(), min = 0, max = max_days)) %&gt;%<br>  group_by(id) %&gt;%<br>  mutate(<br>    # Gender<br>    sex = c(&#39;F&#39;, &#39;M&#39;)[gender + 1],<br>    # Day of measurement<br>    day = cumsum(days),<br>    # Number of appointment<br>    n_appointment = row_number(),<br>    # Was measurement successful or is it NA?<br>    successful_measurement = rbinom(n = n(), size = 1, prob = measurement_prob),<br>    # Successful measurements up to this point<br>    n_measurements = cumsum(successful_measurement),<br>    # Total successful measurements<br>    n_succesfull_measurements = sum(successful_measurement),<br>    # Measurement value<br>    value = if_else(successful_measurement == 1,<br>                 baseline + id_offset + gender * male_baseline_offset +<br>                   slope * day + gender * male_slope_offset * day +<br>                   rnorm(n = n(), mean = 0, sd = 1),<br>                 NA)<br>  )<br><br>head(data.frame(long), 8)</pre><p>Some of these variables aren’t needed, but it is worth showing how easy it is to create descriptive data with Dplyr. Note that while the wide contains one row per ID, the long dataset contains multiple rows per ID.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*Aqiln_PMF0qRNMFTcFc3XQ.png" /><figcaption>Fig. 2 First 7 individuals (long format)</figcaption></figure><p>The generic value term is the measure that gets repeated over time. To test how well a model picks up on differences in slopes, baseline differences, number of classes (individuals) or data points (measurements), you simply need to play with the simulation parameters. Additionally, the noise component on the value definition is something that warrants attention.</p><h3>Visualization</h3><p>The final step is to visualize the data and ensure it is what you intended. This step is easy with the Ggplot package.</p><pre># plot the data<br>long %&gt;%<br>  subset(successful_measurement == TRUE) %&gt;%<br>  ggplot(aes(x = day, y = value, color = sex)) +<br>  geom_point(alpha = 0.2) +  # Add transparency<br>  geom_smooth(method = &quot;lm&quot;, se = FALSE) +  # Regression lines<br>  labs(<br>    title = &quot;Regression of Values by Gender&quot;,<br>    x = &quot;Days&quot;,<br>    y = &quot;Value&quot;,<br>    color = &quot;Gender&quot;<br>  ) +<br>  theme(plot.title = element_text(hjust = 0.5))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/1*sRDqIzbCmbSOMIzIh4umzA.png" /><figcaption>Fig. 1 Scatterplot of value by time of measurement</figcaption></figure><h3>Conclusion</h3><p>This example demonstrates how R can be used to simulate complex datasets. Such simulated data is valuable for testing and validating analytical models and approaches before applying them to real-world data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c09f48a8edb" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>