DECODING MARKET SHIFTS: DETECTING STRUCTURAL BREAKS — I
Detecting breakpoints, shifts, or trend changes is critical, especially in fields where time series insights drive key decisions. This article boldly explores foundational structural change detection models, such as the Chow Test and Zivot-Andrews Test, and enhances the study with advanced approaches like Pre-Intervention and Post-Intervention Analysis, revealing concealed shifts and new patterns within market data.
Spotting structural breaks in the stock market is like uncovering hidden signals of shifting trends. Catch one early, and you’re primed to profit while others are still reacting. Take the 2008 financial crisis: markets were steady, but then everything changed as prices plunged, and behavior shifted almost overnight. The effects were global.
Turkey, one of the world’s top 20 economies, saw its own dramatic shift in June 2023 when Mr. Mehmet Şimşek took over as Minister of Treasury and Finance. After five years of unconventional, expansionary policies, his appointment signaled a major U-turn. Markets quickly sensed the pivot, anticipating a return to orthodox economic policies. As the new reforms rolled in, stock markets surged, cheering on this transformation with a rally that set the stage for a new economic chapter — a classic structural breakpoint in action.
Detecting these kinds of breaks is super helpful because it lets analysts adjust their models to better fit the new reality, making forecasts and strategies way more accurate after big events like this. This article will delve into powerful models and practical examples to reveal structural changes within financial markets.
Given the comprehensive nature of this study, it will be presented in two parts. The first article will delve into foundational models like the Chow Test and Zivot-Andrews Test, alongside sophisticated models such as Pre-Intervention and Post-Intervention Analysis. The second article will take a deeper dive into cutting-edge models — Dynp, Pelt, Binseg, BottomUp, Window, and KernelCPD — applying them to real financial data for robust analysis and insights.
The Effect of A U-turn on Stock Markets
Let’s take a closer look at the BIST-100 main index and see how the stock market reacted to this pivotal mid-2023 policy shift.
import yfinance as yf
import pandas as pd
import matplotlib.pyplot as plt
# Fetch XU100.IS data
data = yf.download("XU100.IS", start="2023-01-01", end="2024-01-01")
# Calculate 50 and 100-day moving averages and add them as new columns in the data DataFrame
data['50-Day MA'] = data['Close'].rolling(window=50).mean()
data['100-Day MA'] = data['Close'].rolling(window=100).mean()
# Plotting
plt.figure(figsize=(14, 7))
plt.plot(data['Close'], label='Closing Price', color='blue', linewidth=2)
plt.plot(data['50-Day MA'], label='50-Day Moving Average', color='orange', linewidth=2)
plt.plot(data['100-Day MA'], label='100-Day Moving Average', color='green', linewidth=2)
# Add titles and labels
plt.title('XU100.IS Closing Price with 50 & 100-Day Moving Averages')
plt.xlabel('Date')
plt.ylabel('Price')
plt.legend(loc='upper left')
# Show the plot
plt.grid(True)
plt.show()
It’s clear from the charts how the financial markets applauded the appointment of the new minister. Since June 2023, the stock market has surged into a bull run, showcasing renewed investor confidence.
The chart demonstrates a classic Golden Cross pattern, where the 50-day moving average (in orange) crosses above the 100-day moving average (in green) around mid-2023. This crossover is typically interpreted as a bullish signal, indicating a potential upward trend. Following the Golden Cross, the stock index indeed shows strong upward momentum, confirming market optimism.
Even someone unaware of the event could look at this chart and sense that something shifted dramatically around mid-year. Could this be a structural break? We’ll dig into the data using our analytical tools to uncover what really happened, starting with the Chow Test as our first approach.
The Chow Test
The Chow Test is like a financial detective tool that helps us figure out if a big change or “break” has occurred at a certain point in a dataset. Imagine we have data over time, and we’re wondering, “Did something significant happen here?” The Chow Test lets us split the data into two parts — before and after the suspected shift — and then compares them to see if the change is statistically meaningful. If we go into technical details, we can say that the critical point is the choice of a date at which a break occurs within the period under examination.
The testing process begins by selecting a suspected breakpoint — often tied to an event or policy change — then splitting the data into two segments: before and after this point. A separate regression model is fitted to each segment () , and another model is applied to the entire dataset as a single piece. By comparing the errors (sum of squared residuals) of these models, the test assesses whether the two segments behave differently enough to be considered separate trends. If the Chow Test statistic, calculated from these errors, exceeds a certain threshold, it suggests a meaningful break has occurred. Essentially, the Chow Test helps confirm whether an observed change is a real turning point or just a random fluctuation in the data.
import yfinance as yf
import pandas as pd
import statsmodels.api as sm
import matplotlib.pyplot as plt
from scipy.stats import f
from statsmodels.tsa.stattools import adfuller
# Downloading XU100.IS data
data = yf.download("XU100.IS", start="2023-01-01", end="2024-01-01")
# Selecting the 'Close' prices and resetting the index to keep 'Date' as a column
data = data[['Close']].dropna()
data.reset_index(inplace=True)
data['Date'] = data['Date'].dt.tz_localize(None) # Remove timezone information
# Dickey-Fuller test for stationarity
adf_result = adfuller(data['Close'])
print("ADF Statistic:", adf_result[0])
print("p-value:", adf_result[1])
if adf_result[1] > 0.05:
print("Series is not stationary. Differencing the series.")
# Differencing the series to make it stationary
data['Close_diff'] = data['Close'].diff().dropna()
data = data.dropna().reset_index(drop=True) # Drop first NaN created by differencing and reset index
series_used = 'Close_diff'
else:
print("Series is stationary. Proceeding with the original series.")
series_used = 'Close'
# Adding index for regression
data['Index'] = range(len(data))First, we download historical time series data for the XU100.IS index (BIST 100) from Yahoo Finance, covering January 1, 2023, to January 1, 2024. Next, we remove timezone information from the ‘Date’ column to ensure consistent date handling. Then, we add an ‘Index’ column with sequential numbers for each row, which will be useful for linear regression analysis.
At this point, we need to check whether the series is stationary using the Augmented Dickey-Fuller (ADF) test. Stationarity is essential in time series analysis, especially for methods like the Chow test, which assumes that the data has consistent statistical properties over time. If the ADF test shows a p-value greater than 0.05, this indicates the series is not stationary, meaning its statistical properties, such as mean and variance, change over time. To address this, we difference the series (calculate changes between consecutive values) to stabilize it, aiming to create a “difference stationary” series with constant mean and variance, making it suitable for further analysis. Here is the outcome!
Once the series is stationary (or difference stationary), we add an index column to the data to prepare it for the regression-based Chow test, which detects structural breaks.
# Suspected breakpoint (2023-06-03)
breakpoint_date = pd.to_datetime("2023-06-03")
# Find the exact or closest date if breakpoint_date is not in data
if breakpoint_date not in data['Date'].values:
breakpoint_date = data['Date'].iloc[(data['Date'] - pd.Timestamp("2023-06-03")).abs().argsort()[0]]
breakpoint_index = data[data['Date'] == breakpoint_date].index[0] # Find the index of the breakpoint dateAfter that, we define June 3, 2023, as our suspected structural break date, as it marks the appointment of the new minister. If this exact date isn’t available in the dataset (due to trading days), we automatically find the closest available date.
# Splitting the data into two parts
data1 = data.iloc[:breakpoint_index]
data2 = data.iloc[breakpoint_index:]
# Model fitting
X_full = sm.add_constant(data['Index'])
X1 = sm.add_constant(data1['Index'])
X2 = sm.add_constant(data2['Index'])
model_full = sm.OLS(data[series_used], X_full).fit()
model1 = sm.OLS(data1[series_used], X1).fit()
model2 = sm.OLS(data2[series_used], X2).fit()
# Sum of squared residuals
ssr_full = model_full.ssr
ssr1 = model1.ssr
ssr2 = model2.ssr
# Number of observations in each segment
n = len(data)
n1 = len(data1)
n2 = len(data2)
# Number of parameters (including intercept)
k = 2In this part, we first split the data into two segments: one for the period before the suspected breakpoint and another starting from the breakpoint onward. Then, we create three linear regression models: one for the entire dataset, one for the first segment, and one for the second segment. Each model helps us measure how well it fits the data by calculating the “error,” known as the sum of squared residuals (SSR). These SSR values for each model allow us to compare them and determine if there’s a meaningful change in trend at the breakpoint. Finally, we count the number of observations in each segment and set up the number of parameters, preparing everything for the Chow Test.
# Chow Test Statistic
chow_stat = ((ssr_full - (ssr1 + ssr2)) / k) / ((ssr1 + ssr2) / (n1 + n2 - 2 * k))
print(f"Chow Test Statistic: {chow_stat}")
# Critical value from F-distribution
f_critical = f.ppf(1 - 0.05, dfn=k, dfd=n1 + n2 - 2 * k)
print(f"Critical value (F-distribution): {f_critical}")
# p-value calculation
p_value = 1 - f.cdf(chow_stat, dfn=k, dfd=n1 + n2 - 2 * k)
print(f"P-value: {p_value}")
# Conclusion based on the Chow test statistic
if chow_stat > f_critical:
print("Structural break detected!")
else:
print("No significant structural break detected.")Now, we calculate the Chow Test statistic, which compares the fit of the combined model to the separate models, allowing us to detect a potential structural break. Then, we find the critical value from the F-distribution to use as our benchmark, setting a 5% significance level. Next, we calculate the p-value to assess the probability that the observed test statistic could occur by chance. Finally, we compare the Chow Test statistic to the critical value. If the test statistic exceeds this value, we conclude that there’s a structural break; otherwise, we find no significant structural break in the data. Here are the results!
# Plotting the data
plt.figure(figsize=(10, 6))
plt.plot(data['Date'], data[series_used], label=f'{series_used.replace("_", " ").title()} Price')
plt.axvline(x=breakpoint_date, color='red', linestyle='--', label=f'Breakpoint ({breakpoint_date.date()})')
plt.title(f'XU100.IS {series_used.replace("_", " ").title()} Price with Breakpoint on {breakpoint_date.date()}')
plt.xlabel('Date')
plt.ylabel(f'{series_used.replace("_", " ").title()} Price')
plt.legend()
plt.show()
The Chow Test results strongly indicate a structural break. The p-value is nearly zero, suggesting that the observed change is not due to random variation. This confirms a meaningful shift in the data, likely linked to the event we suspected.
The chart shows the XU100.IS closing prices with a clear upward shift following the marked breakpoint on June 3, 2023 (red dashed line). Before the breakpoint, the index fluctuates without a strong trend, but after this date, a significant upward movement is evident, suggesting a positive market response to events around early June 2023. This aligns with the findings of the Chow Test, confirming a structural break in the data.
Get Engin Sorhun’s stories in your inbox
Join Medium for free to get updates from this writer.
The Zivot-Andrews Test
The Zivot-Andrews Test is like a spotlight for spotting big turning points in a time series. It’s designed to detect “structural breaks” — those sudden changes where a trend might shift direction, often triggered by major events like economic policies or market shocks. Unlike typical tests that assume trends are smooth, the Zivot-Andrews Test assumes there could be a single, game-changing point. It searches through the data, trying to pinpoint that exact moment, and tells us if it’s statistically significant. In essence, it’s our go-to tool for saying, “Yes, something important happened here, and it changed the path forward.”
The Chow Test requires a known breakpoint and tests for a structural change at that specific point, making it useful for analyzing event-driven shifts. In contrast, the Zivot-Andrews Test searches for an unknown breakpoint, allowing it to identify unexpected shifts in a time series. Put simply, we use the Chow Test when we have a specific event and date we suspect caused a shift. But with the Zivot-Andrews Test, we don’t need a known event or date — this test digs into the data on its own to find the break point for us.
The Zivot-Andrews Test is a bit unique because it doesn’t just check for trends; it actively searches for a specific “breakpoint” where a trend may have shifted. Unlike other tests, it assumes there could be one sudden change in the data. The test has three options for spotting this shift: a change in the trend’s intercept (a sudden jump or drop in values), a change in the slope (a shift in the growth rate), or both. It runs through each potential breakpoint, calculates test statistics, and identifies the most likely break date. If the test statistic is high enough compared to critical values, it confirms a significant structural break, meaning the data likely shifted direction at that specific point. This makes the Zivot-Andrews Test particularly useful for detecting major shifts that regular trend tests might miss.
Additionally, Zivot-Andrews test doesn’t need the series to be stationary upfront. It checks if the data is stable while allowing for one structural break, making it more effective than basic tests like Dickey-Fuller when there’s a sudden change. This way, it accounts for non-stationarity and potential break points in one go.
import yfinance as yf
import pandas as pd
import matplotlib.pyplot as plt
from statsmodels.tsa.stattools import zivot_andrews
# Downloading XU100.IS data
data = yf.download("XU100.IS", start="2023-01-01", end="2024-01-01")
# Selecting the 'Close' prices and dropping any NaN values
data = data[['Close']].dropna()
# Perform the Zivot-Andrews test
za_result = zivot_andrews(data['Close'], maxlag=30, regression='ct')
# Display the results
print("Zivot-Andrews Test Results")
print(f"Test Statistic: {za_result[0]}")
print(f"P-value: {za_result[1]}")
print(f"Critical Values: {za_result[2]}")
print(f"Number of Lags: {za_result[3]}")
print(f"Break Date Index: {za_result[4]}")
print(f"Break Date: {data.index[za_result[4]]}") # Displaying the break dateWe apply the Zivot-Andrews test to the data, which allows us to detect any significant structural break in the time series without needing a predefined breakpoint. We set the maximum lag to 30 and use both a constant and trend in our regression. Finally, we display the test results, including the test statistic, p-value, critical values, the number of lags, and the break date index, allowing us to pinpoint where a potential structural shift may have occurred in the data. The actual break date is then extracted and displayed, giving us insight into when this turning point took place. Voilà the results!
First, by applying the Zivot-Andrews test, we detected June 21, 2023, as a potential breakpoint. Then, we compared the calculated test statistic for this date against the critical values at 1%, 5%, and 10% significance levels. And… it turns out, the Zivot-Andrews test results indicate that there is no statistically significant structural break in the data at the identified breakpoint (June 21, 2023). (The test statistic is much higher than the critical values at all significance levels and the p-value > 0.05).
Let’s look at the chart now!
# Plotting the data with the identified breakpoint
plt.figure(figsize=(10, 6))
plt.plot(data.index, data['Close'], label='Close Price')
plt.axvline(x=data.index[za_result[4]], color='red', linestyle='--', label=f'Breakpoint ({data.index[za_result[4]].date()})')
plt.title('XU100.IS Close Price with Zivot-Andrews Breakpoint')
plt.xlabel('Date')
plt.ylabel('Close Price')
plt.legend()
plt.show()
Pre-Intervention And Post-Intervention Analysis (PIPIA)
The PIPIA model is all about seeing how much of a difference an event made. First, we look at the data from before the event to understand the “normal” trend, then compare it with the data after the event to spot any changes. This way, we can measure the impact of things like a policy change, a new ad campaign, or a product launch. It’s a straightforward way to answer the question: “Did this event really change things, and if so, by how much?”
Developed by Google, the CausalImpact library is like a time-traveling detective for data. Allowing to apply the PIPIA, it helps us understand the effect of a specific event — like a policy change or a marketing campaign — by comparing what actually happened with what might have happened if that event had never taken place. Using a Bayesian structural time series model, it creates a “counterfactual” prediction, allowing us to see the difference the event made. The result? A clear view of the event’s impact over time, complete with confidence intervals to show the range of possible effects.
Unlike tests like the Chow Test or Zivot-Andrews Test, which look for single structural breaks or shifts in a time series, CausalImpact digs deeper by modeling the entire “before and after” story of an intervention. While the Chow and Zivot-Andrews tests are great for pinpointing change points, CausalImpact goes further by quantifying the impact and showing how long it lasts. This makes it especially powerful when we want not just to detect a shift but to measure its real-world effect over time.
What’s really impressive is that CausalImpact doesn’t just give you raw numbers; it actually explains the results for you, helping to interpret whether the intervention had a statistically significant impact. This reporting feature makes it incredibly user-friendly, leaving us with minimal work to do in terms of analysis and interpretation. With its visualizations and ready-to-use summary, CausalImpact essentially hands us the insights on a silver platter.
pip install CausalImpactimport yfinance as yf
import pandas as pd
import matplotlib.pyplot as plt
from causalimpact import CausalImpact
# Step 1: Fetch the data for XU100.IS
ticker = "XU100.IS"
data = yf.download(ticker, start="2023-01-01", end="2023-12-31")
# Step 2: Prepare the data
# We'll assume the intervention happens on 2023-06-02
pre_period_dates = ["2023-01-02", "2023-06-01"]
post_period_dates = ["2023-06-02", "2023-12-29"]
# Format the data as needed by CausalImpact (a time series DataFrame with a single column)
data = data[['Close']].reset_index()
data['Date'] = pd.to_datetime(data['Date'])
# Prepare the time series for CausalImpact
ts = data.set_index('Date').asfreq('B').ffill()
# Step 3: Define the pre and post periods
pre_period = [pre_period_dates[0], pre_period_dates[1]]
post_period = [post_period_dates[0], post_period_dates[1]]In this code, our goal is to analyze the impact of a specific event. We define the pre-intervention period (from January 2 to June 2) and the post-intervention period (from June 3 to December 29). Next, we format the data as a time series, with trading days filled in to ensure there are no gaps, preparing it for analysis with the CausalImpact library. This setup will let us clearly compare the time series before and after the intervention date.
In this part of the code, we run the CausalImpact analysis using the time series data (ts) along with the pre-intervention and post-intervention periods we defined earlier. Then we use impact.plot() to generate a visualization of the results, showing the actual data, predicted “what-if” scenario, and the difference between them. Finally, plt.show() displays the plot, allowing us to visually interpret the effect of the intervention.
# Step 4: Run the CausalImpact analysis
impact = CausalImpact(ts, pre_period, post_period)
# Step 5: Plot the results
impact.plot()
plt.show()
This plot reveals a strong positive impact of the intervention (policy change) on the XU100.IS index. In the top panel, we see that after the intervention date (marked by the dotted line), the actual values rise above the predicted “what-if” scenario, indicating a boost. The middle panel shows the pointwise effect, with consistently positive values after the policy change, confirming an ongoing positive influence. Finally, the bottom panel illustrates the cumulative effect, showing a steady increase, which highlights the significant overall gain attributed to the intervention over time.
And the detailed report… But subsequently, I’ll wrap up the report with a punchy summary that brings all the insights together in one clear takeaway.
The report from the CausalImpact analysis tells a clear story: the intervention had a strong and statistically significant impact: the appointment of the new minister and the resulting U-turn in macroeconomic policy had a strong and statistically significant impact on the BIST 100 index.
During the post-intervention period, the actual average of the BIST 100 index was 7341.11, compared to an expected average of 4920.04 if the policy shift hadn’t occurred. This difference — an impressive increase of 2421.08 points — is highly unlikely to be due to random chance (p-value = 0.0). Cumulatively, this effect sums up to a substantial gain of a 49.21% relative increase. The tight 95% confidence intervals confirm that this impact is real and meaningful. In short, the policy shift had a powerful and positive effect on the BIST 100, and the numbers clearly support it!
The Story Doesn’t End Here!
In this article, we applied the Chow Test, Zivot-Andrews Test, and Google’s CausalImpact model to financial markets, analyzing whether a policy shift caused a structural break. The models mostly point to a significant shift in the markets in June 2023.
In the next article, we’ll step it up with advanced models — Dynp, Pelt, Binseg, BottomUp, Window, and KernelCPD — using real financial data to uncover deeper insights and make our analysis even more powerful.
