Statistical Analysis with Python — Part 6: Time Series Analysis

The ultimate article series on the practical use of time series analysis.

Time, as we all know is a complex and fascinating concept. Scientists, philosophers and poets have been pondering about it’s nature since eons. One of the most profound quotes I’ve read about time comes from Einstein: “The distinction between past, present, and future is only a stubbornly persistent illusion”. There’s much to unpack from this quote, but let’s focus on a few key insights that will help us better understand time series analysis.

1. Past Influences the Present and the Future — Einstein proposed that the time that has passed is connected to the time that is currently passing and the time yet to pass. I know, I used the word ‘pass’ quite a bit, but the truth is, time will pass regardless. This principle lies at the heart of autocorrelation analysis, where past data points influence future ones — much like in stock predictions.
2. Illusion of Change — Time as we perceive it is an illusion — it appears to move forward, but underlying patterns often repeat. Some things change, some do not. How do we know which does and which does not? The answer is stationary time series analysis. In stationary time series, statistical properties (mean, variance) remain constant over time, implying an underlying structure that doesn’t change despite the passage of time.
3. The Relativity of Time in Forecasting: How Models Perceive Time Differently — Much like humans, some models want to live through each moment and some just want it to pass. Can someone please just skip this part?. Forecasting models (ARIMA, LSTMs, etc.) perceive time differently. Some treat it as linear, while others capture complex dependencies, much like relativity’s view of time being observer-dependent.
4. Causal Vs. Non Causal Models — Causal models assume a strict past-to-future relationship, whereas some modern models (like deep learning) capture patterns without explicitly differentiating past from future, reinforcing Einstein’s view that time separation is not absolute.

Now, let’s move on to the technical and definitive part of the article which you are here for.

What is Time Series Analysis?

Time series analysis is a statistical method used to examine data collected over time. Its goal is to uncover patterns, trends, and seasonal variations, helping to make predictions and derive meaningful insights. This technique is commonly applied in fields like finance, economics, sales forecasting, and weather prediction.

It answers: What is the past trying to tell us?

Types of Time Series Analysis

1. Trend Analysis: The Path of the Horizon — Trend analysis focuses on identifying the long-term direction of a time series. It reveals whether the data is moving upward, downward, or staying stable over an extended period. Trends can be linear or non-linear. Imagine tracking a company’s stock price over the past decade. Trend analysis shows that despite market fluctuations, the stock has been steadily rising — indicating growth, much like the steady climb of a mountain trail.
2. Seasonal Analysis: The Rhythm of Nature — Seasonal analysis identifies regular, predictable patterns or cycles within data that repeat at fixed intervals. These cycles can be driven by time-based factors such as months, quarters, or seasons. Think of how retail sales surge every December due to the holiday shopping season. Seasonal analysis captures this recurring pattern, helping businesses plan for the surge, just like the predictable blooming of flowers each spring.
3. Cyclic Analysis: The Ocean’s Waves — Cyclic analysis detects long-term fluctuations that do not follow fixed intervals, often influenced by external factors like the economy or market cycles. These cycles are typically irregular and can span unpredictable periods. Imagine tracking the economic growth and recession cycles of a country. Cyclic analysis identifies these broader trends — like the ebb and flow of ocean waves — giving businesses the insights they need to adapt strategies through economic highs and lows.
4. Stationarity Analysis: The Grounded Terrain — Stationarity analysis checks if a time series’ statistical properties (mean, variance) remain constant over time. A stationary series is stable, while a non-stationary one may require transformations (like differencing). Picture temperature data over many years. If the average temperature rises steadily or fluctuates wildly, the data isn’t stationary. Stationarity analysis helps determine whether the underlying landscape is stable or changing — like testing whether the ground beneath your feet is shifting or solid.
5. Autocorrelation Analysis: The Echo of the Past — Autocorrelation measures the correlation between a time series and its past values, showing how past data points influence future ones. A strong autocorrelation means past values have significant predictive power over the future. Imagine you’re predicting tomorrow’s temperature based on today’s weather. Autocorrelation reveals how today’s temperature influences the next day’s forecast — like an echo that reverberates through time.
6. Decomposition: The Data Alchemist — Decomposition breaks down a time series into its core components — trend, seasonality, and residuals (random noise). This helps separate the predictable patterns from the unpredictable noise. Think of analyzing monthly electricity consumption. Decomposition shows that most consumption is driven by a trend (increasing usage) and seasonality (higher use in winter), while the residuals capture outliers like a sudden drop due to a power outage, just like a skilled alchemist separating gold from impurities.
7. Forecasting: The Crystal Ball of Time — Forecasting uses historical data to predict future values, applying models like ARIMA, exponential smoothing, or machine learning. It transforms past patterns into future predictions. Imagine forecasting next week’s weather by analyzing patterns in temperature, humidity, and wind from the past few years. Forecasting is like looking into a crystal ball — grounded in data — revealing what’s to come by understanding what has happened before.
8. Anomaly Detection: The Detective’s Lens — Anomaly detection identifies data points that deviate from expected patterns, often signalling errors, unusual events, or critical insights like fraud or system failures. Picture bank transactions — an anomaly might occur when a customer’s account shows a sudden withdrawal that doesn’t match their typical spending pattern. Anomaly detection works like a detective’s magnifying glass, spotting outliers that don’t belong in the regular flow of data.

Since anomaly detection and forecasting are vast topics with distinct methodologies and applications, we will explore them in detail in separate dedicated articles.

Why the Type of Data Matters in Time Series Analysis

In time series analysis, the importance of the dataset is often understated. While it’s common for data professionals to focus on the desired outcome or model results, the type of data they work with plays a critical role in the analysis process itself. Understanding whether you’re working with univariate or multivariate time series data is essential, as it directly influences the choice of analytical methods, model complexity, and the insights you can derive.

1. Univariate time series involves a single variable measured over time, allowing for a focused analysis of trends, seasonality, and other patterns within that single dimension. For example, tracking daily stock prices or temperature fluctuations. While it simplifies modeling, it can limit the breadth of insights, especially when external factors may influence the data.

• Multivariate time series, on the other hand, involves multiple variables measured simultaneously, offering a more comprehensive view of the system under study. For example, forecasting sales based on both product price and advertising spend, or weather forecasting using variables like temperature, humidity, and wind speed. Multivariate analysis allows for deeper insights into how variables interact with each other over time, but it also requires more sophisticated models and can lead to complex relationships that need to be carefully managed.

Therefore, recognizing whether your data is univariate or multivariate is not just an academic distinction — it directly impacts the kind of analysis you perform and the quality of insights you can derive from it. Choosing the right approach for your dataset is the first step in ensuring your time series analysis delivers meaningful, actionable results.

Univariate time series is like a solo performer on stage, while multivariate is the entire orchestra — more variables, more chaos, but also more harmony (good luck trying to manage it)!

Theoretically, both univariate and multivariate time series analysis involve similar techniques, but the level of complexity varies. For univariate data, simpler methods like trend analysis, seasonal analysis, and forecasting are most effective, as they focus on a single variable over time. Multivariate data, on the other hand, enables a deeper look at how multiple variables interact, making techniques like multivariate forecasting, autocorrelation analysis, and anomaly detection more useful for uncovering complex relationships.

Techniques For Manipulating Data

Once we identify the type of dataset we’re working with (univariate vs multivariate), the next step is data manipulation. How can we prepare the data so it’s ready for analysis? Real-life data is often messy and noisy. Depending on the characteristics of your data, various techniques can be applied to preprocess or transform it before you apply any complex modelling. These techniques help to remove noise, stabilize variance, and ensure that the data is in the right form for accurate analysis. Some common techniques are as follows:

• Smoothing: Smoothing techniques like moving averages and exponential smoothing are used to reduce noise in time series data, revealing underlying trends or patterns by averaging out short-term fluctuations.
• Differencing: This technique involves subtracting previous observations from the current data to remove trends and make the series stationary, helping to identify cyclical behaviour.
• Log Transformation: Log transformations stabilize the variance of time series data, making it more consistent and easier to model by reducing the impact of large fluctuations.
• Scaling and Normalization: Scaling and normalization adjust the range of data values, ensuring that they all have comparable magnitudes, which is essential for many machine learning models.
• Decomposition: Decomposition breaks down a time series into its components — trend, seasonality, and residuals — allowing for better analysis of individual patterns.
• Handling Missing Data: Missing data is addressed using interpolation or imputation techniques to fill in gaps, ensuring that models are built on complete datasets for accuracy.
• Outlier Detection and Treatment: Identifying and treating outliers helps improve model accuracy by removing data points that deviate significantly from the expected behaviour, which could skew results.

If you are interested in learning more about these techniques, check out: NIST Handbook on Time Series Analysis — https://www.itl.nist.gov/div898/handbook/pmc/section4/pmc4.htm.

Use Case

A retail company is analyzing its monthly sales data over the past decade to understand the following:

1. Long-term trends.
2. Seasonal fluctuations.
3. Cyclic patterns.
4. Whether the data is stationary for forecasting models.
5. The effect of data transformations, like smoothing and log transformation, on analysis.

Data Preprocessing

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from statsmodels.tsa.stattools import adfuller

# Simulate Monthly Sales Data (10 years)
np.random.seed(42)
months = pd.date_range(start='2014-01-01', periods=120, freq='M')
sales = np.linspace(500, 1500, 120) + np.random.normal(scale=100, size=120) + 200 * np.sin(np.linspace(0, 24, 120))
df = pd.DataFrame({'Month': months, 'Sales': sales})
df.set_index('Month', inplace=True)

#I have used LLMs to generate this code for better efficiency.

#Smoothing (Moving Average)
df['Smoothing_MA'] = df['Sales'].rolling(window=12).mean()

#Log Transform
df['Log_Transformed'] = np.log(df['Sales'])

#Differencing (To Remove Trend)
df['Differenced'] = df['Sales'] - df['Sales'].shift(1)

# Plot all the manipulated versions of the data
plt.figure(figsize=(14, 8))

# Original Data
plt.subplot(3, 1, 1)
plt.plot(df['Sales'], label='Original Sales', color='blue')
plt.title('Original Sales Data')
plt.legend()

# Smoothing (12-month Moving Average)
plt.subplot(3, 1, 2)
plt.plot(df['Smoothing_MA'], label='12-Month Moving Average', color='orange')
plt.title('Smoothed Sales Data (12-Month Moving Average)')
plt.legend()

# Log Transformation
plt.subplot(3, 1, 3)
plt.plot(df['Log_Transformed'], label='Log Transformed Sales', color='green')
plt.title('Log Transformed Sales Data')
plt.legend()

plt.tight_layout()
plt.show()

# Differencing Plot
plt.figure(figsize=(12, 5))
plt.plot(df['Differenced'], label='Differenced Sales Data', color='purple')
plt.title('Differenced Sales Data (To Remove Trend)')
plt.legend()
plt.show()

We performed the following:

• Smoothing (Moving Average):
  We applied a 12-month rolling average to smooth out the noise and visualize the trend over a longer period.
• Log Transformation:
  We used log transformation to stabilize variance, making the time series easier to model.
• Differencing:
  We performed differencing to remove trends in the data, making it stationary for further analysis.

Analysis

Now that the data is pre-processed, we can further analyze it.

Get Raghav Sharma’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Trend Analysis

plt.figure(figsize=(12, 5))
plt.plot(df['Sales'], label='Original Sales', color='blue')
plt.plot(df['Smoothing_MA'], label='Smoothed (12-Month MA)', color='orange', linestyle='dashed')
plt.title('Trend Analysis: Monthly Sales Trend')
plt.xlabel('Year')
plt.ylabel('Sales')
plt.legend()
plt.show()

Trend Analysis — Original vs Smoothed

By comparing the original sales data with the smoothed (12-month moving average) data, we can see the long-term upward trend in sales, even though there are short-term fluctuations.

Seasonal Analysis

from statsmodels.tsa.seasonal import seasonal_decompose

# Decompose the original data (additive model)
decomposition = seasonal_decompose(df['Sales'], model='additive', period=12)
decomposition.plot()
plt.show()

The decomposition plot shows the seasonal component, revealing recurring patterns in sales each year, which could be influenced by factors like holidays or sales promotions.

Cyclic Analysis

# Rolling Mean (24 months) for Cyclic Analysis
df['Rolling_Mean'] = df['Sales'].rolling(window=24).mean()

plt.figure(figsize=(12, 5))
plt.plot(df['Sales'], label='Actual Sales')
plt.plot(df['Rolling_Mean'], label='24-Month Rolling Mean', color='red', linestyle='dashed')
plt.title('Cyclic Analysis: Identifying Long-Term Cycles')
plt.legend()
plt.show()

The 24-month rolling mean helps identify long-term cycles that aren’t immediately obvious, smoothing out shorter fluctuations and revealing broader patterns in the data.

Stationary Analysis

from statsmodels.tsa.stattools import adfuller

# Augmented Dickey-Fuller (ADF) Test for Stationarity
result = adfuller(df['Sales'].dropna())
print("ADF Test Statistic:", result[0])
print("p-value:", result[1])
print("Critical Values:", result[4])

ADF Test Statistic: -1.043200881894232

p-value: 0.737183632586083

Critical Values: {‘1%’: np.float64(-3.4936021509366793), ‘5%’: np.float64(-2.8892174239808703), ‘10%’: np.float64(-2.58153320754717)}

If the p-value from the ADF test is less than 0.05, we reject the null hypothesis and conclude that the data is stationary. If it’s greater than 0.05, the series is non-stationary and may need further transformations (like differencing). In this case, further transformation is required.

ADF test is a form of hypothesis test used in time series analysis. It is specifically used to test for the presence of a unit root in a time series, which helps determine whether the series is stationary or non-stationary.

For more info on hypothesis testing, please review my article on Inferential Statistics — https://medium.com/@sharmaraghav644/statistical-analysis-with-python-part-2-inferential-statistics-cf30e1f12808.

Autocorrelation Analysis

from statsmodels.graphics.tsaplots import plot_acf, plot_pacf


fig, ax = plt.subplots(1, 2, figsize=(12, 5))
plot_acf(df['Sales'], ax=ax[0], lags=30)
plot_pacf(df['Sales'], ax=ax[1], lags=30)
ax[0].set_title("Autocorrelation Function (ACF)")
ax[1].set_title("Partial Autocorrelation Function (PACF)")
plt.show()

ACF and PACF plots help identify how past sales influence future sales and can inform the choice of model for forecasting (e.g., ARIMA).

Decomposition Analysis

decomposition = seasonal_decompose(df['Sales'], model='additive', period=12)
plt.figure(figsize=(12, 5))
plt.plot(df.index, decomposition.trend, label='Trend')
plt.plot(df.index, decomposition.seasonal, label='Seasonal')
plt.plot(df.index, decomposition.resid, label='Residual', linestyle='dashed')
plt.legend()
plt.title('Time Series Decomposition: Trend, Seasonal, Residuals')
plt.show()

Decomposition helps isolate the trend, seasonality, and residuals, making it easier to forecast future values and understand underlying patterns.

Key Business Decisions From Data

• Optimized Sales Forecasting and Resource Planning: By applying log transformation, differencing, and smoothing, we identified stable growth trends and reduced noise, enabling more accurate sales forecasts and better resource allocation. The company can now better plan inventory and staffing around forecasted sales.
• Strategic Inventory and Seasonal Adjustments: Seasonal fluctuations identified through analysis allow the company to anticipate high-demand months and adjust inventory levels and promotional campaigns accordingly, ensuring products are in stock during peak periods and reducing excess inventory during off-seasons.
• Long-term Growth and Expansion Planning: The upward trend in sales and identification of cyclic patterns provide the company with confidence in its long-term growth trajectory, guiding decisions on scaling operations, expanding product offerings, and investing in new market opportunities.
• Informed Risk Management: By understanding both the seasonality and economic cycles, the company can better prepare for downturns with adjusted budgets while capitalizing on periods of economic growth, ensuring stable operations through varying market conditions.

Key Takeaways

• Understanding Data Trends and Patterns: Time series analysis helps identify long-term trends, seasonal fluctuations, and cyclic patterns in data, providing essential insights for making informed business decisions, like resource allocation and strategic planning.
• Data Transformation for Better Analysis: Techniques like smoothing, log transformations, and differencing can make data more manageable and accurate for analysis, improving the reliability of forecasts and highlighting underlying trends and seasonality.
• Practical Business Applications: Time series analysis enables businesses to optimize inventory management, plan for seasonal demand fluctuations, and make long-term growth decisions, improving overall operational efficiency and reducing risk.

Next Up

In upcoming articles, we will dive deeper into forecasting and anomaly detection, exploring these topics in detail.

Follow for more: medium.com/@sharmaraghav

LinkedIn: https://www.linkedin.com/in/raghav-sharma-b7a87a142/

Thank you for being a part of the community

Before you go:

• Be sure to clap and follow the writer ️👏️️
• Follow us: X | LinkedIn | YouTube | Newsletter | Podcast | Twitch
• Start your own free AI-powered blog on Differ 🚀
• Join our content creators community on Discord 🧑🏻‍💻
• For more content, visit plainenglish.io + stackademic.com