Imagine trying to understand the behaviour of a hurricane after observing only one storm. You watch it form, evolve, change direction, strengthen, weaken, and eventually dissipate. You record everything: wind speed, pressure, temperature, satellite imagery. Every fluctuation is measured in extraordinary detail, producing an enormous amount of data.

Yet despite this wealth of information, a fundamental limitation remains. You have observed only one realised trajectory. You never get to rewind the storm and see what would have happened if the ocean had been slightly warmer, the pressure system slightly weaker, or the wind direction subtly different. The storm unfolds once, and history records the result.

Financial markets are a natural example of this same constraint. We observe enormous quantities of data — prices, volumes, macroeconomic conditions, and cross-asset behaviour — but everything is generated along a single unfolding history. One sequence of crises, recoveries, policy shifts, technological changes, and evolving market regimes.

The challenge is not that we lack data. The challenge is that we have only one realised path through a much larger space of possibilities.

Why This Matters

In practice, we rarely care about the past for its own sake. We care about what it implies for situations that have not yet occurred. Investors make decisions based on expectations of future market behaviour. Risk managers are concerned with rare but consequential events that may not resemble anything in the historical record. Policymakers prepare for crises that have not yet occurred in precisely the same form. In each case, the object of interest is not the realised history itself, but how the system might behave beyond it.

So the question becomes:

What kind of model could help us reason about outcomes that differ from the single history we have observed?

From Trajectories to Evolving Structure

Across many domains, the systems we care about are not static objects but evolving sequences of states over time. What matters is not just the system at a given moment, but how it changes from one moment to the next.

In biology, populations change through interaction and selection. In climate systems, local feedback effects accumulate into large-scale patterns. In social and economic systems, behaviour emerges from the interaction of agents, institutions, and external shocks over time.

What is common across these settings is simple: what we observe at any point in time is only a snapshot of a single unfolding sequence. Each snapshot is connected to the next, but we only ever see one realised chain of outcomes.

A useful model is one that represents how the system moves from one observed state to the next in a way that allows multiple plausible continuations from the same point.

Structure Within a Single History

Even though we may only ever observe a single realised trajectory of the system, this does not mean the trajectory is unstructured.

Within a single realised history, certain patterns recur with enough consistency that they are not easily dismissed as incidental. Volatility is not evenly distributed through time. It arrives in clusters. Large moves are more common than simple Gaussian models would suggest. Correlation structures are not fixed, but shift in response to changing market conditions.

These patterns are typically referred to as stylised facts. They are not properties of individual observations, and they are not fixed rules imposed on the system. They are patterns that emerge from the realised history itself, and which appear consistently enough to be treated as meaningful structure rather than noise. Their value comes not only from how often they appear, but from the fact that they persist across changes in context, sampling, and scale.

Why This Matters for Modelling

If we want to reason about outcomes that differ from the realised history, we need some way of distinguishing plausible trajectories from implausible ones. Stylised facts provide one such constraint: structural checks on any model of market behaviour. A model that generates future scenarios which systematically violate persistent patterns observed in historical data is unlikely to be useful.

Generating futures therefore does not mean extrapolating a single expected path, nor projecting a smooth continuation of past trends. Instead, it means starting from the current observed state and allowing for multiple continuations consistent with the structural patterns revealed by history.

From this perspective, a model is evaluated by whether the trajectories it produces preserve the structural features of the system as observed: clustering of volatility, distributional properties of large moves, and regime-dependent shifts in dependence.

Historical data is a source of constraints on admissible futures.

Conclusion

The core limitation is simple: we only ever observe one realised history of a system. Everything we measure, analyse, and model is drawn from that single unfolding trajectory. Yet within that limitation, we still find structure. The realised history is not featureless. It contains persistent regularities — stylised facts — that remain visible across time, regimes, and levels of aggregation.

Once this perspective is adopted, the role of historical data changes. It is no longer merely a sequence of observations to be analysed, but also a source of constraints that shape the space of futures we consider plausible. This marks the central shift: from a single realised path to a set of possible continuations shaped by constraint; from describing what has happened to reasoning about what could happen under those constraints.

Financial markets make this problem especially visible, but they do not uniquely define it. Any system observed through a single realised history — where we cannot rerun conditions or experiment with alternatives — faces the same fundamental limitation.

The question is not how to extract more information from the past. It is how to represent the constraints it imposes well enough to reason about futures that were never observed, but remain structurally possible.

That is what it means to learn from a single history.

Even a well-diversified portfolio can hide risks beneath the surface, like an iceberg drifting in calm waters. Most of the danger remains invisible until market stress hits and assets begin moving together. This subtle collapse of diversification dispersion — when previously uncorrelated or weakly correlated assets start moving in unison — often goes unnoticed, yet it carries crucial implications for risk managers, portfolio constructors, and anyone seeking forward-looking financial insight. Understanding this phenomenon, and being able to reproduce it in synthetic data, is central to building more resilient, scenario-aware risk models.

Challenges of Modeling Realistic Portfolios

Realistic portfolio simulation is challenging because markets operate at multiple interconnected levels. Individual assets have their own behaviors: returns are unpredictable, can experience extreme moves, often exhibit fat tails or skewed distributions, and may display mean-reverting tendencies. Volatility itself is clustered, with quiet periods followed by bursts of intense activity, and extreme events can hit multiple assets simultaneously. Assets also move in relation to each other: correlations shift over time, tail co-movements can occur, and both directional and nondirectional co-movements influence overall portfolio risk.

While some traditional approaches handle basic volatility or correlation patterns, the collapse of diversification is subtle, regime-dependent, and difficult to capture. Yet it is arguably one of the earliest and most important signals that portfolio risk is rising — showing when assets that normally hedge each other begin to move in unison, reducing the portfolio’s ability to absorb shocks. Capturing this behavior, alongside other stylized facts such as volatility clustering, tail clustering, mean reversion, and fat tails, is what makes portfolio modeling so challenging.

Evidence from Historical Portfolios

To see how diversification can break down in real markets, we examine an equal-weighted portfolio of 30 S&P 500 stocks spanning technology, financials, healthcare, and industrial sectors from June 2018 to December 2025 (Figure 1).

The top panel shows daily portfolio returns in black, cumulative portfolio value in gray, and extreme positive and negative returns highlighted in blue and red. The bottom panel plots the fraction of asset pairs with low correlation (<0.4), our metric for portfolio diversification breakdown. With 30 assets, there are 435 unique pairs, so a threshold of 0.4 corresponds to roughly 174 pairs behaving independently.

During calm periods, this fraction stays high, reflecting effective diversification. As structural stress builds — during the Trade War sell-off (2018), COVID crash (2020), and the transition away from quantitative easing (2022) — the fraction declines, indicating that previously uncorrelated or weakly correlated assets are beginning to move together. This decline often precedes peaks in portfolio volatility, making it a practical early-warning signal rather than a purely descriptive metric. The Tariff Turmoil in 2025 is an exception: here, diversification only collapsed after the main volatility surge, illustrating that the timing of breakdowns can vary across events.

Reproducing Market Dynamics with Synthetic Data

We generated synthetic returns for the same equal-weighted 30-stock portfolio over the same period using the Skanalytix platform (Figure 2). The synthetic portfolio closely reproduces the time-varying fraction of weakly correlated asset pairs (<0.4) and the overall return patterns seen historically. During calm periods, most asset pairs behave independently, but as structural stress builds, the fraction declines — capturing the early stages of portfolio diversification breakdown. This demonstrates that scenario-aware synthetic data can replicate not just individual asset behavior, but higher-order portfolio dynamics, providing a forward-looking tool for stress-testing, risk analysis, and exploration of potential crises before they materialize.

Quantitative Metrics that Validate Realism

To assess how closely the synthetic scenarios replicate real market behavior, we computed several single-number diagnostics commonly used by quants:

• Mean pairwise correlation (435 asset pairs):
  Historical = 0.395, Synthetic = 0.388
  → Average asset co-movement is preserved.
• Market factor strength (largest eigenvalue λ₁ of correlation matrix): Historical = 12.8, Synthetic λ₁ = 12.5 (≈ 42% variance explained)
  → Dominant market mode is captured.
• Annualized portfolio volatility:
  Historical = 19.5%, Synthetic = 24.0%
  → Elevated stress regimes are realistically preserved.
• Correlation by volatility regime:
  Low-vol: 0.226 vs 0.225
  High-vol: 0.571 vs 0.498
  → Correlations rise appropriately in stressed regimes, mirroring historical market behavior.

These metrics show that the synthetic scenarios faithfully reproduce both average co-movement and higher-order portfolio behaviors. By combining single-number summaries with time-varying diagnostics, the synthetic data provides risk managers and portfolio analysts with a forward-looking, scenario-aware lens to study both day-to-day dynamics and structural portfolio risks.

Managing Risk in a Converging Market

The collapse of diversification isn’t just an academic observation — it has real consequences for portfolio management. When previously uncorrelated assets start moving together, portfolios become more exposed to market shocks, even when traditional metrics continue to signal diversification. By reproducing these patterns in synthetic data, risk managers and portfolio analysts can explore a wider range of scenarios, including rare or extreme market events, stress-test strategies under conditions that mirror real market dynamics, and anticipate potential crises before spikes in volatility occur. Recognizing early declines in diversification allows decision-makers to adjust exposures or hedge positions proactively, turning historical insight into actionable foresight. In this way, synthetic data provides a forward-looking lens on portfolio vulnerability, helping teams prepare for risks that might otherwise remain hidden.

Conclusion

The collapse of diversification dispersion is a subtle but powerful signal of rising portfolio vulnerability. Historical data shows this clearly, and synthetic data demonstrates that it can be reproduced and studied systematically. By capturing these early-warning signals, scenario-aware models enable risk managers to anticipate crises, stress-test portfolios, and explore alternative futures before losses occur. In complex, interconnected markets, risks emerge quietly. Looking beyond simple volatility or correlation measures is essential to identifying them early. Metrics such as the largest eigenvalue, mean pairwise correlation, dispersion of weak correlations, and portfolio volatility provide a concise but powerful lens into these dynamics.

The most damaging episodes in financial markets rarely follow “normal” patterns. Black swans — rare, high-impact events — can strike as sudden, isolated shocks, like Black Monday in 1987, when a single day produced unprecedented losses. They can also unfold over weeks or months as structured sequences of extreme returns, as during the Global Financial Crisis, where large gains or losses followed patterns in time, building on what had come before rather than appearing randomly. Periods of intense volatility can persist, interspersed with calmer stretches, creating sequences that compound portfolio risk. These events matter not just because they are extreme, but because they often unfold in sequence, shaping long-term risk and challenging models designed for more typical market behavior.

What makes these events dangerous is not only their magnitude, but their structure. In real market data, extreme returns are not scattered randomly. They tend to organize themselves into bursts, producing sustained stress precisely when diversification, historical calibration, and volatility-based controls are expected to work. A lone shock may be absorbed; a sequence often overwhelms.

Volatility clustering — the tendency for periods of high or low variability to persist — is now a standard feature of financial modeling. Yet volatility alone does not explain why extreme gains and losses so often arrive close together. This deeper phenomenon, tail clustering, reflects temporal dependence in the extremes themselves. Such sequences can develop gradually within high-volatility regimes or erupt abruptly after long stretches of apparent calm. From a risk perspective, the distinction is decisive: survival depends less on whether an extreme occurs than on whether another is likely to follow. Treating black swans as isolated deviations misses precisely the dynamics that make them catastrophic.

Path-Dependent Modeling of Extremes

Many traditional approaches to modeling extremes — including regime-switching and jump-diffusion models — capture tail behavior by introducing predefined states or jump mechanisms calibrated to the full dataset. While effective at reproducing broad volatility patterns, they impose globally shared structure on dynamics that are, in practice, highly local, path-dependent, and regime-specific.

Sequence-aware generative models, such as LSTMs, TimeGAN, and RGAN, adopt a similar philosophy in a more flexible form. They learn temporal dependence through globally shared parametric or latent representations optimized to match average behavior across the data. The resulting paths may appear plausible at a high level, yet tend to miss clustered sequences of large moves that dominate realized losses. This highlights a fundamental tension: global model fitting struggles to capture market dynamics whose most consequential behavior is inherently local and conditioned on the recent path.

A more robust alternative is local, data-driven modeling. The guiding question is simple: When the market has looked like this before, what followed? Rather than imposing global structure, the model draws directly on historically similar sequences of recent returns to generate plausible continuations of the market path. When many historical analogues exist, outcomes are tightly grounded in observed experience; when analogues are scarce, confidence naturally diminishes.

Volatility and tail clustering arise endogenously under this approach, as properties of the historical sequences being conditioned upon. No latent regimes or hidden states are assumed — the recent realized path, together with historical sequences, provides the primary conditioning information for scenario generation, and extreme behavior emerges naturally from how similar situations actually unfolded.

Generating Plausible Black Swans

Under the Skanalytix framework, market returns are generated step by step. At each point, the model conditions on the recent realized path of returns, identifies historically similar sequences, and samples the next return from the corresponding conditional distribution. Repeating this process produces full market trajectories that evolve sequentially, with each outcome shaped by what has already occurred.

Extreme returns emerge naturally, reflecting the underlying historical structure rather than being artificially imposed. Key stylized features — including fat tails, mean-reverting tendencies, and volatility clustering — are preserved, ensuring synthetic trajectories remain statistically consistent with real markets. When the current trajectory resembles paths that historically preceded large moves, the conditional distribution widens to reflect that variability; during calmer periods, outcomes remain correspondingly stable. Clustered extremes therefore arise as conditional continuations of the path, rather than as exogenous shocks or predefined events.

Over sufficiently long simulations, this process generates a wide spectrum of behaviors — from extended calm periods to sequences that broadly resemble historical patterns in structure, while also exploring new combinations of extreme returns, clustering patterns, and volatility regimes. These dynamics set the stage for the figures that follow, which compare historical black swan episodes in the S&P 500 with extreme scenarios generated as described above.

By preserving the statistical features of the market while allowing novel sequences to emerge, the framework provides a realistic, historically grounded way to explore extreme events — without implying predictability.

Historical and Synthetic Black Swan Scenarios

Figure 1 shows the structure of three historic black swan episodes in the S&P 500: Black Monday 1987 (top), the Global Financial Crisis (center), and COVID-19 (bottom).

A comparison of these episodes highlights important differences in how extreme events manifest. Black Monday appears as a single, severe daily drop — a highly concentrated shock of unprecedented magnitude. The Global Financial Crisis period, by contrast, is characterized by prolonged sequences of negative returns accompanied by sustained high volatility over many months, reflecting extended market stress. The COVID-19 episode shows a sharp but relatively brief drawdown followed by a rapid recovery, illustrating how large losses can materialize over very short horizons.

These examples show that extreme events vary not only in magnitude but also in temporal structure, persistence, and duration. Understanding these differences is essential when interpreting synthetic scenarios: realistic trajectories must capture not just how large extremes can be, but also how they unfold — whether as single shocks, extended sequences, or fast-moving cycles.

Figure 2 shows three synthetic extreme scenarios generated by the Skanalytix framework, conditioned on historical S&P 500 returns from 1985 to 2025.

These scenarios demonstrate how extreme sequences can emerge naturally from historical market behavior, without duplicating any particular episode. Each trajectory blends magnitude, clustering, and temporal persistence, capturing realistic extremes while exploring new patterns — including multi-day drawdowns, volatility bursts, and partial recoveries.

Conclusion

The Skanalytix framework allows us to generate full market trajectories that naturally include extreme outcomes — without artificially inserting shocks. By conditioning sequentially on the recent realized path and historically observed sequences, the model produces scenarios that capture magnitude, clustering, and temporal persistence in a realistic, historically grounded way.

This approach produces a wide spectrum of behaviors — from extended calm periods to sequences that resemble historical patterns in broad structure — while also exploring new combinations of extreme returns and clustering patterns. The resulting synthetic scenarios illustrate how extreme sequences can plausibly arise from historical market structure, offering a data-driven way to stress-test portfolios and evaluate risk under conditions that may not have occurred in history.

Viewed this way, extreme events — the kind we might loosely describe as “plausible black swans” — emerge naturally from the unfolding market path rather than as exogenous, pre-defined shocks. This highlights the value of path-dependent scenario generation: it provides a framework for exploring extreme outcomes in context and helps quantify risks that traditional, static models may understate.

Note: This article was originally published on LinkedIn. You can view the original here: https://www.linkedin.com/pulse/tracking-tomorrows-risk-sp500-how-return-shift-day-andrew-skabar-eplwc/

What does tomorrow’s return distribution look like when markets are calm? How does it change when macroeconomic uncertainty rises sharply? How quickly does skew emerge, dissipate, or even reverse direction? And do these shifts happen suddenly, or quietly beneath the surface before they appear in volatility?

These are questions every risk manager cares about, yet most models cannot answer them. Traditional volatility models scale risk up or down, but leave the shape of the distribution largely untouched. Deep-learning sequence models generate plausible paths, but rarely reveal how the underlying distribution evolves day by day.

In this article, we examine that shape directly: the conditional probability density function (PDF) of next-day S&P500 returns, estimated using the Skanalytix scenario-aware generative modeling platform. By tracking the conditional PDF over time, we gain a more nuanced view of risk, revealing how market conditions influence not just volatility, but the evolving distribution of returns.

S&P500: From Calm to Turbulence and Back

To illustrate how the conditional PDF evolves, we look at the daily close of the S&P500 over 80 trading days beginning on 14 February 2025, just a couple of weeks before the market turbulence sparked by the Trump tariff announcements.

Watch the estimated PDF evolve over the 80-day window:

The video visualizes the estimated distribution of the next day’s return, including summary risk metrics such as VaR and CVaR. The realized return is also shown. For context, we also include the return series for the previous 1,000 trading days, which highlights extreme events and changing volatility regimes, as well as the price series itself, showing the broader trend environment.

At the start of this 80-day window, the market is relatively calm, reflected in narrow, nearly symmetric PDFs. But by early March, turbulence arrives abruptly. A cluster of large moves in the first week of the month causes the distribution to expand sharply. Importantly, this is not just an increase in variance: the tail mass grows faster than the center, and the distribution becomes noticeably skewed. The model picks up these changes well before they are fully reflected in realized volatility.

It’s also worth noting why the conditional PDFs sometimes appear multimodal during turbulent periods. In calm markets, there is ample historical precedent for returns, so the estimated distribution is narrow and unimodal. During extreme or fast-changing conditions, however, there are fewer comparable historical situations to draw on. The model therefore spreads probability across multiple plausible outcomes, producing multiple peaks in the PDF. This multimodality reflects genuine uncertainty about which path the market may follow, rather than a modeling artifact, and is a feature that scenario-aware generative methods can capture — unlike traditional volatility-based approaches that assume a single, smooth outcome.

Later in the period, immediately after the “Liberation Day” announcement, two consecutive large negative returns on April 3 and April 4 push the distribution to its widest point. During these episodes, the conditional PDF becomes heavily distorted, with shifts in asymmetry and tail weight that conventional volatility models cannot capture. In the days following these shocks, the distribution gradually contracts, though it often retains traces of skew or heavier-than-usual tails. An extreme positive return on April 9 briefly shifts the distribution, demonstrating that sudden upward moves can also affect skew and tail behavior. Volatility rises sharply during shocks but subsides more gradually afterward, and the conditional PDF captures this asymmetric path back to calmer conditions.

By the end of the window, the distribution has contracted from its widest point, reflecting a return toward calmer conditions. While volatility has subsided, the sequence of PDFs makes clear how rapidly market risk can shift and how day-to-day distributions capture these evolving patterns.

It’s important to note that these conditional PDFs are estimated from historical market behavior. They show how returns have typically evolved under similar conditions in the past, but the realized return on any given day will also reflect new macroeconomic announcements, sentiment shifts, geopolitical events, and other surprises. What the PDF captures is the shape of risk entering each day, not a precise prediction of the outcome.

What Conventional Models Miss — and Why It Matters

Most modeling approaches in finance are built around specific assumptions about how markets behave. That works well when conditions match the assumption, but markets rarely stay in one mode (e.g., low volatility or calm conditions) for long. Volatility ebbs and flows, correlations shift, macro conditions change, and sometimes prices simply jump in ways that defy the underlying model.

GARCH and related volatility models capture short-run volatility clustering but are fixed in structure, so they struggle when markets shift regimes or macro events change behavior. Jump-diffusion models layer sudden shocks on top of a diffusion process, yet the jump mechanism is usually predefined, making it hard to match markets where jumps vary and cluster. Neural network–based sequence models, including LSTMs, GANs, and VAEs, can generate plausible time series, but they often miss extremes, tail behavior, and regime changes, and their performance drops sharply with limited or noisy data.

The result is a patchwork of partial solutions: one model for volatility, another for jumps, another for long-term structure. Each captures one aspect of market behavior, but they don’t combine naturally into a single, coherent picture. Yet in real markets, all these behaviors interact continuously.

This is the gap Skanalytix aims to fill: a generative approach that models the full joint structure — returns, volatility, dependencies, and regime behavior — without forcing the data into a fixed parametric form. The result is a model that can follow the market as it moves, rather than one that requires the market to stay still.

Conclusion: Seeing Risk Through the Full Distribution

By following the conditional PDF of next-day returns over time, we gain a view of market behavior that goes beyond simple measures of volatility. We see how the entire distribution shifts — broadening under stress, skewing as uncertainty builds, and contracting as conditions stabilize. These changes are often subtle at first, then abrupt, and they carry information that conventional models either miss or flatten out.

This is where scenario-aware generative modeling becomes valuable. Instead of forcing the data into a fixed structure, it lets the distribution evolve alongside the market. The result is a richer understanding of risk: one that captures not just its scale, but also the evolving shape of uncertainty.

As markets move into a world of faster cycles, more frequent shocks, and tighter macro-financial links, this distribution-level insight becomes essential. It helps risk managers anticipate behaviors that traditional models overlook, and brings us closer to representing markets as they truly move.

Note: This article was originally published on LinkedIn. You can view the original here: https://www.linkedin.com/pulse/beyond-volatility-capturing-hidden-shape-market-risk-andrew-skabar-lm1nc/

Financial risk is often summarized by volatility — realized daily volatility, implied volatility, or indexes like the VIX. But volatility captures only the scale of typical moves. It says nothing about how risk manifests: whether returns arrive smoothly or in sudden bursts, whether extremes cluster, or whether calm markets are quietly harboring instability. Two periods with the same volatility can hide very different risk profiles: one may be calm and stable, another punctuated by sudden, concentrated bursts of large moves. Extreme returns also happen far more often than a normal distribution predicts, and these extremes tend to cluster rather than appear randomly. Low volatility does not necessarily mean low risk; often, it masks hidden risk. Understanding these temporal patterns is essential for realistic risk modeling, stress testing, and scenario analysis.

In this article, I explore these deeper structural features using both historical equity data and synthetic data generated by the Skanalytix generative modeling platform. The goal is not to replicate markets day-by-day but to show that key behaviors — heavy tails, clustering of extremes, and dynamic patterns — emerge naturally from the generative process.

Historical Data: Volatility Clustering and Tail Clustering

Figure 1 shows daily S&P500 returns from January 2015 to December 2024, the 3-month rolling standard deviation (σ), and markers for extreme returns (those exceeding twice the rolling σ). The extreme high-volatility region in the center corresponds to the onset of COVID.

Two patterns stand out. Volatility clustering appears as periods of sustained large-magnitude returns, apparent in the shaded elevated σ regions. Tail clustering, in contrast, occurs when extreme returns arrive in concentrated bursts, sometimes even when overall volatility is low. Tail clustering is particularly important: extreme events are rarely independent and instead form concentrated episodes of stress that conventional volatility measures fail to capture. These patterns provide the empirical baseline for evaluating whether generative models, such as Skanalytix, can reproduce the same temporal dynamics.

Synthetic Data: Capturing the Temporal Structure of Extremes

Next, we examine whether the patterns seen in historical data also appear in synthetic series. Unlike traditional forecasts, which predict the next return based on current conditions, synthetic returns generate their own internally consistent sequences of states, so we focus on structural properties rather than point-by-point matches. Figure 2 shows three synthetic S&P500 returns over a ten-year period, generated by a model trained on the historical data shown in Figure 1.

The synthetic series exhibit the same key behaviors observed in real markets: sustained periods of high or low variability, bursts of extreme moves, and shifts between calm and turbulent regimes. These temporal patterns — volatility clustering, tail clustering, and regime-like transitions — emerge naturally from the model rather than being explicitly imposed. This provides intuitive evidence that the Skanalytix framework captures not just the scale of daily returns but also the internal dynamics of how risk evolves, accumulates, and concentrates over time.

Building on this, Figures 1 and 2 together highlight these patterns clearly and raise a natural question: what must the underlying one-day return distributions look like for volatility clustering and bursts of extremes to emerge? Conditional PDFs give us that view.

Conditional PDFs: Risk Across Regimes

Each return in the synthetic series of Figure 2 was generated by first estimating the conditional distribution of returns given the current market state, then sampling a value from that distribution. To explore how risk evolves across market conditions, we can compare the shape of these conditional distributions under different regimes.

Figure 3 shows three distributions. The unconditional PDF is computed from a large set of synthetic returns generated by the model, summarizing the full range of outcomes it considers possible and serving as a reference. The PDF from a low-volatility period illustrates returns during a historically calm regime, which are narrower. In contrast, a PDF from a high-volatility period highlights returns during a turbulent regime, exhibiting wider, skewed, heavy-tailed behavior. In some high-volatility periods, the PDF may appear irregular or even multi-peaked, reflecting the presence of multiple overlapping market behaviors captured by the model. Each conditional PDF incorporates the most recent market information, so its exact shape can change day-to-day, capturing the dynamic, time-dependent nature of risk.

Conditional PDFs such as these reveal that risk is not constant. The likelihood of extreme returns, reflected in the fat tails of the distribution, varies across market conditions, remaining possible even during calm periods and rising sharply in turbulent regimes. These shifting distributions provide a dynamic, time-dependent picture of equity risk that traditional volatility measures cannot capture. Because the model can generate a new conditional distribution for every possible market state, it also lets us explore many day-ahead scenarios, providing a natural bridge to the multi-day risk examined in the next section.

N-day Scenarios: Risk over Longer Horizons

To explore how uncertainty accumulates over multiple days, we generate N-day-ahead synthetic return distributions using Monte Carlo sampling from the Skanalytix generative model. Unlike one-day-ahead conditional PDFs, which are tightly tied to the current market state, these multi-day distributions reflect the range of possible paths and how risk evolves over time.

Figure 4 shows 5-day, 10-day, and 30-day synthetic return distributions, each annotated with 95% Value-at-Risk (VaR) and Conditional Value-at-Risk (CVaR) to highlight potential extreme outcomes. As the horizon increases, the distributions widen, reflecting growing uncertainty. Skew and heavy tails emerge, demonstrating that extreme losses can accumulate over multiple days. The particularly large 30-day potential losses partly reflect extreme events present in the training data, including the COVID crisis. The model produces some synthetic paths that experience concentrated periods of extreme market stress, while other paths remain relatively calm. This variability mirrors the clustering of extreme events observed in historical markets and shows how losses can compound — features that simpler approaches often underestimate.

By combining these N-day scenarios with one-day-ahead conditional PDFs, we obtain a full picture of risk across both regimes and horizons. These synthetic distributions reveal not only the scale of potential returns, but also the temporal and structural dynamics of extreme events — patterns that conventional volatility-based measures alone cannot capture.

Conclusion

Volatility alone cannot capture the full structure of equity risk. Historical data show that markets exhibit volatility clustering, tail clustering, and regime shifts — patterns that simple distributional summaries miss. Synthetic data generated by the Skanalytix model reproduces these behaviors, capturing both the conditional dynamics and the multi-day accumulation of extreme outcomes that shape real episodes of stress.

By combining one-day conditional PDFs with multi-day scenario distributions, we obtain a coherent view of risk across both regimes and horizons. This approach reveals not just the scale of potential returns, but the underlying dynamics — fat tails, skew, the tendency of extremes to cluster over time, and the episodic nature of market stress. These structural insights support more realistic scenario generation, enhanced stress testing, and a deeper understanding of how risk evolves over time, far beyond what volatility-based measures alone can offer.

Note:

Originally published on LinkedIn: https://www.linkedin.com/pulse/beyond-volatility-capturing-hidden-shape-market-risk-andrew-skabar-lm1nc/

Financial forecasts are often presented as single numbers, offering a best guess of where the market might go next. But markets are rarely that simple. Returns are irregular, shifting, and dependent on the path that led there. Each moment carries its own context, shaped by volatility, momentum, and broader market conditions. Ignoring that structure can make even the most sophisticated models blind to how uncertainty actually evolves.

Tomorrow could be calm, slightly up, or sharply down. In high-volatility periods, extreme returns of either sign become more likely, producing fat tails and sometimes even bimodal patterns. The likelihood of different outcomes depends on the current market context. For example, a single large negative return in a low-volatility bull market would suggest a very different set of possible next moves than the same negative return in a high-volatility bear market. To truly capture this uncertainty, we need more than a single number; we need the full probability distribution. Instead of asking ‘what will the return be?’, we ask ‘what could the return be, and how likely is each outcome?”

From History to Scenarios

Note: NVIDIA’s historical prices are used purely for illustration. This example demonstrates how the Skanalytix framework generates scenarios and is not a recommendation to buy or sell any securities.

To see these ideas in action, we examine NVIDIA’s daily closing prices over the three-year period from October 3, 2022, to September 29, 2025 (Figure 1). The historical series exhibits periods of relative calm interspersed with episodes of heightened volatility, illustrating how returns can shift over time. For clarity and comparison, we normalize the series to the first price, creating a relative price series that emphasizes shape and dynamics rather than absolute levels.

This historical data forms the basis for generating synthetic scenarios. A scenario-aware generative framework can produce a range of plausible future paths by estimating conditional probability distributions for daily returns. “Scenario-aware” means that each distribution reflects both the current market context and the path that led there, capturing how volatility and momentum evolve through time, and how correlations evolve in the case of portfolios, rather than assuming these features remain fixed. The Skanalytix framework implements this approach in practice.

To generate simulated price paths, the model repeatedly estimates the conditional distribution for the next day’s return, samples a value from it, and then moves forward one day at a time. Each sampled return influences the starting point for the following day, so the sequence of draws reflects the evolving market state. Because the model learns these conditional patterns directly from historical data, the resulting trajectories are both plausible and dynamically consistent, revealing how risk and uncertainty unfold over time. These simulated paths, shown in Figure 2, illustrate how the model translates conditional distributions into coherent forward scenarios.

Some paths drift more steeply upward, others more gently; some remain relatively calm, while others exhibit bursts of volatility. Overall, the trajectories reflect the upward trend observed in the historical NVIDIA prices used to build the model, with the historical series roughly centered within the simulated envelope, anchoring the simulations in reality.

Stylized Facts in Synthetic Data

But realistic scenarios aren’t just about plausibility; they must exhibit the patterns observed in actual markets. Financial time series display several stylized facts, including fat tails, volatility clustering, and mean reversion, and a credible generative model should reproduce them.

Fat tails appear as extreme moves that occur more frequently than a Normal distribution would predict. These are visible in Figure 3, where the synthetic returns match the historical distribution of NVIDIA returns.

Volatility clustering shows up as consecutive periods of high or low volatility. Figure 4 demonstrates that stretches of turbulent returns alternate with calmer periods in both historical and synthetic series.

Mean reversion occurs when prices tend to return toward a long-term trend rather than drifting indefinitely in one direction. This is evident in the price series, where both historical and synthetic paths oscillate around a central trajectory, avoiding unrealistic persistent drifts, as shown in Figure 2.

By reproducing these features, the model does more than generate visually plausible paths; it mirrors the key statistical characteristics of real financial data. The reason lies beneath the surface, in the conditional probability distributions that drive these dynamics. Watching how these PDFs evolve over time reveals the market’s hidden structure and the unfolding of scenario-aware uncertainty.

The Market’s Hidden Shapes: Watching PDFs Evolve

Up to now, we’ve explored synthetic paths and assessed stylized facts using static figures. While those snapshots reveal important patterns, they only provide a partial view of how uncertainty evolves over time. The true power of a scenario-aware generative model becomes clear when we watch the conditional probability distributions themselves unfold day by day. These PDFs capture the full spectrum of possibilities, reflecting both the current market state and the outcomes of previous days.

Figure 5 presents a dynamic visualization of a three-year simulation of daily NVIDIA closing prices. Each frame shows the estimated PDF for the current day with the sampled return marked, the evolving sequence of returns up to that day, and the corresponding price series. The distributions shift dynamically in response to both market state and prior draws, illustrating how today’s outcomes influence tomorrow’s possibilities.

These evolving conditional PDFs reveal how the shape of risk changes over time. Fat tails expand or contract, asymmetry shifts, and extreme outcomes become more or less likely depending on recent market moves. Unlike traditional mean-variance measures, which ignore these dynamics, this approach provides a more realistic view of risk. The daily conditional distributions can be used to estimate single-day VaR or CVaR, while Monte Carlo simulations of the scenarios enable multi-day estimates, capturing both the magnitude and the changing likelihood of extreme events across different horizons.

For practitioners accustomed to standard assumptions of Normality, these irregular and shifting distributions and the way each sample influences the next may be surprising or even unsettling. Yet they reveal an essential truth about financial markets: uncertainty is never constant, and risk is rarely symmetric. By modeling conditional distributions and highlighting sequential sampling, we gain a richer, more realistic view of market behavior and a clearer sense of how uncertainty evolves over time.

Conclusion: The Future Isn’t a Number

Financial markets are inherently uncertain, and a single-point forecast can never capture the full range of possible outcomes. At Skanalytix, our modeling framework provides a richer, scenario-aware view of market dynamics. It reveals how possible outcomes evolve over time, not only for individual assets but also across portfolios, capturing inter-asset dependencies, volatility regimes, and extreme events.

This framework goes beyond visualization: it serves as the foundation for a range of downstream tasks. Analysts can generate realistic synthetic data for testing and validation, produce scenario-based forecasts, and perform robust risk analysis that reflects the complex interplay of multiple assets. By making uncertainty visible and actionable, the Skanalytix platform helps practitioners move from guessing what might happen to understanding what could plausibly occur, providing a more resilient basis for portfolio management, stress testing, and strategic decision-making.

Explore More About Skanalytix

For further information on our generative modeling platform and its applications in financial time series analysis, please visit our official website:

👉 skanalytix.com

To stay updated with our latest developments, research, and insights, connect with us on LinkedIn:

👉 Skanalytix on LinkedIn

Feel free to reach out for inquiries, demo requests, or exploratory conversations via email:

📧 admin@skanalytix.com

Synthetic financial time series are artificially generated sequences of data that mimic the statistical properties of real market data. These synthetic datasets play a pivotal role for traders and strategy designers, offering a controlled environment to test and refine trading strategies, ensuring model adaptability, and providing insights into performance under diverse market scenarios and economic conditions that may be difficult to observe in reality.

But creating ‘realistic’ synthetic financial time series is notoriously difficult, as historical data often contains specific, complex features that are difficult to replicate. For example, consider Figure 1, which shows the prices, p(t), returns, r(t) = (p(t) − p(t−1)) / p(t−1), and the distribution of returns for Apple (AAPL) over the three-year period from January 1, 2021, to January 1, 2024.

The distribution of returns (right) isn’t a standard bell curve; it has fatter tails and a sharper peak than a normal (or ‘Gaussian’) distribution (dashed curve). This is called ‘excess kurtosis’. Looking at the return series (center), we notice that large returns (positive or negative) tend to be followed by more large returns, while small returns tend to follow small returns. This pattern is known as ‘volatility clustering’. In the price series (left), there’s an overall upward trend over the three years, but we also see shorter ups and downs. These shorter trends often correct themselves, with prices returning to more ‘normal’ levels based on fundamentals like price/earnings ratios — a behavior known as ‘mean reversion’. Fat tails, volatility clustering, and mean reversion are examples of ‘stylized features’. To create convincing synthetic financial series, we MUST capture these features.

While parametric deep-learning models such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) have gained popularity in recent years, applying them in this domain can be challenging. Capturing long-term dependencies inherent in features like volatility clustering and mean reversion often requires large amounts of data. These models also come with practical considerations, including training complexity, high computational demands, and limited interpretability.

In this story, I take a completely different approach and describe a non-parametric model based on explicitly estimating the probability distribution of returns. I will begin by exploring how to model a distribution of returns without considering the temporal sequence in which they occur. We’ll see that capturing features like excess kurtosis and skew is fairly straightforward. Then, I’ll demonstrate how to extend this to estimate ‘conditional’ return distributions — distributions that take into account previous returns. This paves the way for more advanced modelling that can generate realistic synthetic series, accurately capturing the stylized features of financial time series and providing a much-needed alternative to GANs and VAEs.

Modeling financial returns using KDE

Kernel density estimation (KDE) is a useful method for modelling non-Normal distributions. It works by placing a kernel (often Gaussian, but it can be different) at each data point and normalizing so that when all these kernels are summed together, the area under the curve equals one, making it a valid probability distribution function (PDF).

The key factor in KDE is the kernel width, or ‘bandwidth’. If this is too narrow, the density estimate becomes overly localized, resulting in a squiggly and noisy density function. Conversely, if the bandwidth is too broad the estimate will over-smooth the data and fail to capture important features of the distribution. Although rules of thumb such as the Silverman bandwidth¹ often provide a good trade-off, a constant bandwidth is unlikely to adequately capture the fat-tailed nature of financial return distributions while simultaneously modelling the higher-density regions with the required accuracy.

Variable-width kernels

An intuitive way to tackle this problem is to use variable-width kernels where the bandwidth increases as we move away from the center (mean) of the distribution. We use the following to determine the bandwidth bₜ for data point rₜ :

where 𝜇̂ is the mean of the historical returns, A is the width of the ‘base’ kernel at the center of the distribution, and k controls how quickly the bandwidth increases with distance from the mean. We use an exponentially increasing bandwidth, but a linearly increasing one could work too.

Let’s apply this to some historical financial series. Figure 2 shows the empirical return distributions and PDF estimates for the daily close prices of Apple (AAPL), Google (GOOG), Facebook (META) and Microsoft (MSFT) over the same three-year period using variable-width kernels. In each case, the estimated PDF captures the sharp peaks and fat tails seen in the actual data. It also captures the asymmetry (or ‘skew’) in the historical distributions, which is perhaps most evident for the case of Google. Note that while KDE can model skew, if the skew is extreme, it might be better to handle the left and right sides separately using different k values.

But what about a ‘series’ of returns?

Drawing a single value from the estimated PDF is straightforward, but our aim is to generate a sequence, or ‘series,’ of returns. A naive approach would be to simply sample returns successively from the PDF, but this overlooks a critical fact:

Returns sampled from the same PDF will be independent and identically distributed (i.i.d.). However, returns from financial series that exhibit features such as volatility clustering and mean reversion are not independent — past returns influence future ones.

To capture these patterns, we need to estimate ‘conditional distributions’ — distributions of returns that take into account previous returns. Only then can our synthetic series accurately reflect the temporal dynamics of the historical data.

Modeling conditional returns using weighted KDE

The kernels we used above were centered at the historical returns rₜ, and we found that they could produce a sufficiently detailed estimate of the PDF of the historical return distribution. By weighting these same kernels it is possible to generate a multitude of PDFs with diverse characteristics (symmetrical versus skewed, single-mode versus multi-modal, etc.).

We would like to produce PDFs that represent the return distribution conditional on previous return values. The challenge lies in finding a set of weights that capture the relationship between past returns. In general, we seek a function of the form

where wₜ is the weight for the kernel centered at historical return rₜ. Let’s consider the simplest case: estimating the return distribution conditional solely on the previous day’s return.

Example: Conditioning on the previous day’s return

Let’s assume the previous day’s return has value c. The essential idea is that for each historical return rₜ, the closer that its 1-day lagged return rₜ ₋ ₁ is to c, the stronger should be the influence of the kernel centered at rₜ on the estimate of the conditional PDF. There are various ways of capturing this relationship. We will use the following:

where 𝜎̂ is the standard deviation of the historical returns and parameter k controls the rate of decay.

Figure 3 displays the three-year historical returns of Apple (X-axis) plotted against the 1-day lagged values (Y-axis). It also shows three specific values for the previous day’s return that we will condition on: −0.06, −0.03 and 0.07. Figure 4 shows the corresponding conditional PDFs.

The shape of the estimated PDF varies based on the conditioning value. Conditioning on a value of −0.03 results in a conditional distribution similar in shape to the historical distribution. In contrast, when the conditioned value has a relatively large magnitude (e.g., −0.06 or 0.07), the distribution becomes more complex in shape. Note that in this example we have used a large value of k to illustrate the complexity achievable in the shape of the conditional distributions. In practice, a smaller value is typically more appropriate.

We can generate a series of synthetic returns as follows, where we use primes (as in r’₀) to distinguish synthetic returns from historical returns:

The sequence of returns r’₀,…,r’ₙ forms the synthetic return series, which can then be converted into a price series.

‘Realistic’ synthetic financial time series

Estimating conditional distributions based solely on the previous day’s return is unlikely to capture the richness of features we observe in real financial data. To do so, we need to consider distributions conditional not just on the previous day’s return, but on a sequence of previous returns.

At Skanalytix², we have built on these concepts to produce a synthetic financial time series generator that can effectively model conditional distributions to produce realistic synthetic financial series. The generator estimates conditional PDFs of the form:

where Rₜ and its lags are either scalars representing the return of a single stock, or vectors representing the returns of multiple stocks, and Xₜ is either a scalar, or vector of ‘auxiliary’ features, which can be a mix of numerical and categorical and might include fundamentals, economic variables, sentiment, etc.

We have applied the time series generator to generating synthetic datasets from the three-year Apple (AAPL) data. Figure 5 displays the return series, distribution of returns, autocorrelation function (ACF) for returns, ACF for absolute returns, and price series for historical data (first row), together with three synthetic series (second to fourth rows). (The synthetic series were produced using lagged returns only, without any auxiliary features.) The synthetic return series exhibit fat tails and long-term volatility clustering similar to the historical data, and the prices fall within a comparable range to the historical prices, suggesting that the model has captured the mean reversion properties.

Figure 6 shows PCA³ and t-SNE⁴ plots of the original and synthetic returns. These are handy dimensionality reduction techniques that allow us to visualize high-dimensional data in two dimensions. (The returns data were expanded to include lagged values as additional dimensions). Both methods show that points from the historical and synthetic datasets are well interspersed, indicating that the synthetic data captures a broad range of variability similar to the historical data. Moreover, there are no single-colored clusters, indicating that there are no patterns present in one group but absent in the other. From Figures 5 and 6 we can be confident that the synthetic data has mimicked the overall structure of the historical data.

Correlated Stocks

In addition to generating synthetic series for individual stocks, we can also generate realistic series for multiple correlated stocks. Figure 7 shows the historical and synthetic price series for four technology stocks — Apple (AAPL), Google (GOOG), Facebook (META), and Microsoft (MSFT) — over the three-year period. The synthetic series exhibit the strong positive correlation observed in the historical values of these stocks, while simultaneously modelling the stylized features of each individual series. Negative correlations can be similarly modeled.

Conclusion

In this article, I have described a non-parametric method for generating synthetic financial time series based on explicitly estimating the distribution of returns. This method generates realistic synthetic time series that capture stylized features such as volatility clustering and mean reversion, while avoiding some of the practical challenges often associated with parametric approaches like GANs and VAEs. Importantly, the model is interpretable — the distribution from which a return is drawn is a sum of weighted kernels, making it directly visualizable.

We have been primarily interested in generating synthetic financial series — for each point generated, a conditional distribution was estimated, and a random value was then drawn from that distribution. However, this same distribution can be used for a variety of purposes. If the aim is to perform prediction (or ‘forecasting’), then it is simply a matter of determining the conditional mean (or median or mode) of the estimated distribution. Estimating Value at Risk (VaR) is also straightforward, as is computing the probability that a return falls within a defined interval.

The synthetic financial time series generator we have developed at Skanalytix² is based on our Unified Numeric/Categorical Representation and Inference (UNCRi) framework, which combines a graph-based data representation scheme with a flexible inference procedure that can be used to estimate and sample from the conditional distribution of any categorical or numeric variable. The generator is a valuable tool for organizations requiring realistic synthetic data to use for testing and refining trading strategies, model validation, and other tasks that involve decision-making based on financial time series. To stay updated with our latest news and insights, follow Skanalytix on LinkedIn (https://www.linkedin.com/company/97182209).

References

[1] Silverman, B.W. (1982). Applied Statistics, Royal Statistical Society, Vol. 33.

[2] http://skanalytix.com

[3] Jolliffe, I.T. (1986). “Principal Component Analysis”, Springer, New York.

[4] van der Maaten, L. and Hinton, G. (2008). “Visualizing Data using t-SNE” Journal of Machine Learning Research, 9(11).

Are my real and synthetic datasets random samples from the same parent distribution?

When we perform synthetic data generation, we typically create a model for our real (or ‘observed’) data, and then use this model to generate synthetic data. This observed data is usually compiled from real world experiences, such as measurements of the physical characteristics of irises or details about individuals who have defaulted on credit or acquired some medical condition. We can think of the observed data as having come from some ‘parent distribution’ — the true underlying distribution from which the observed data is a random sample. Of course, we never know this parent distribution — it must be estimated, and this is the purpose of our model.

But if our model can produce synthetic data that can be considered to be a random sample from the same parent distribution, then we’ve hit the jackpot: the synthetic data will possess the same statistical properties and patterns as the observed data (fidelity); it will be just as useful when put to tasks such as regression or classification (utility); and, because it is a random sample, there is no risk of it identifying the observed data (privacy). But how can we know if we have met this elusive goal?

In the first part of this story, we will conduct some simple experiments to gain a better understanding of the problem and motivate a solution. In the second part we will evaluate performance of a variety of synthetic data generators on a collection of well-known datasets.

Part 1 — Some Simple Experiments

Consider the following two datasets and try to answer this question:

Are the datasets random samples from the same parent distribution, or has one been derived from the other by applying small random perturbations?

The datasets clearly display similar statistical properties, such as marginal distributions and covariances. They would also perform similarly on a classification task in which a classifier trained on one dataset is tested on the other.

But suppose we were to plot the data points from each dataset on the same graph. If the datasets are random samples from the same parent distribution, we would intuitively expect the points from one dataset to be interspersed with those from the other in such a manner that, on average, points from one set are as close to — or ‘as similar to’ — their closest neighbors in that set as they are to their closest neighbors in the other set. However, if one dataset is a slight random perturbation of the other, then points from one set will be more similar to their closest neighbors in the other set than they are to their closest neighbors in the same set. This leads to the following test.

The Maximum Similarity Test

For each dataset, calculate the similarity between each instance and its closest neighbor in the same dataset. Call these the ‘maximum intra-set similarities’. If the datasets have the same distributional characteristics, then the distribution of intra-set similarities should be similar for each dataset. Now calculate the similarity between each instance of one dataset and its closest neighbor in the other dataset and call these the ‘maximum cross-set similarities’. If the distribution of maximum cross-set similarities is the same as the distribution of maximum intra-set similarities, then the datasets can be considered random samples from the same parent distribution. For the test to be valid, each dataset should contain the same number of examples.

Since the datasets we deal with in this story all contain a mixture of numerical and categorical variables, we need a similarity measure which can accommodate this. We use Gower Similarity¹.

The table and histograms below show the means and distributions of the maximum intra- and cross-set similarities for Datasets 1 and 2.

On average, the instances in one data set are more similar to their closest neighbors in the other dataset than they are to their closest neighbors in the same dataset. This indicates that the datasets are more likely to be perturbations of each other than random samples from the same parent distribution. And indeed, they are perturbations! Dataset 1 was generated from a Gaussian mixture model; Dataset 2 was generated by selecting (without replacement) an instance from Dataset 1 and applying a small random perturbation.

Ultimately, we will be using the Maximum Similarity Test to compare synthetic datasets with observed datasets. The biggest danger with synthetic data points being too close to observed points is privacy; i.e., being able to identify points in the observed set from points in the synthetic set. In fact, if you examine Datasets 1 and 2 carefully, you might actually be able to identify some such pairs. And this is for a case in which the average maximum cross-set similarity is only 0.3% larger than the average maximum intra-set similarity!

Modeling and Synthesizing

To end this first part of the story, let’s create a model for a dataset and use the model to generate synthetic data. We can then use the Maximum Similarity Test to compare the synthetic and observed sets.

The dataset on the left in the figure below is just Dataset 1 from above. The dataset on the right (Dataset 3) is the synthetic dataset. (We have estimated the distribution as a Gaussian mixture, but that’s not important).

Here are the average similarities and histograms:

The three averages are identical to three significant figures, and the three histograms are very similar. Therefore, according to the Maximum Similarity Test, both datasets can reasonably be considered random samples from the same parent distribution. Our synthetic data generation exercise has been a success, and we have achieved the trifecta — fidelity, utility, and privacy.

[Python code used to produce the datasets, plots and histograms from Part 1 is available from https://github.com/a-skabar/TDS-EvalSynthData]

Part 2— Real Datasets, Real Generators

The dataset used in Part 1 is simple and can be easily modeled with just a mixture of Gaussians. However, most real-world datasets are far more complex. In this part of the story, we will apply several synthetic data generators to some popular real-world datasets. Our primary focus is on comparing the distributions of maximum similarities within and between the observed and synthetic datasets to understand the extent to which they can be considered random samples from the same parent distribution.

The six datasets originate from the UCI repository² and are all popular datasets that have been widely used in the machine learning literature for decades. All are mixed-type datasets, and were chosen because they vary in their balance of categorical and numerical features.

The six generators are representative of the major approaches used in synthetic data generation: copula-based, GAN-based, VAE-based, and approaches using sequential imputation. CopulaGAN³, GaussianCopula, CTGAN³ and TVAE³ are all available from the Synthetic Data Vault libraries⁴, synthpop⁵ is available as an open-source R package, and ‘UNCRi’ refers to the synthetic data generation tool developed under the proprietary Unified Numeric/Categorical Representation and Inference (UNCRi) framework⁶. All generators were used with their default settings.

The table below shows the average maximum intra- and cross-set similarities for each generator applied to each dataset. Entries highlighted in red are those in which privacy has been compromised (i.e., the average maximum cross-set similarity exceeds the average maximum intra-set similarity on the observed data). Entries highlighted in green are those with the highest average maximum cross-set similarity (not including those in red). The last column shows the result of performing a Train on Synthetic, Test on Real (TSTR) test, where a classifier or regressor is trained on the synthetic examples and tested on the real (observed) examples. The Boston Housing dataset is a regression task, and the mean absolute error (MAE) is reported; all other tasks are classification tasks, and the reported value is the area under ROC curve (AUC).

The figures below display, for each dataset, the distributions of maximum intra- and cross-set similarities corresponding to the generator that attained the highest average maximum cross-set similarity (excluding those highlighted in red above).

From the table, we can see that for those generators that did not breach privacy, the average maximum cross-set similarity is very close to the average maximum intra-set similarity on observed data. The histograms show us the distributions of these maximum similarities, and we can see that in most cases the distributions are clearly similar — strikingly so for datasets such as the Census Income dataset. The table also shows that the generator that achieved the highest average maximum cross-set similarity for each dataset (excluding those highlighted in red) also demonstrated best performance on the TSTR test (again excluding those in red). Thus, while we can never claim to have discovered the ‘true’ underlying distribution, these results demonstrate that the most effective generator for each dataset has captured the crucial features of the underlying distribution.

Privacy

Only two of the seven generators displayed issues with privacy: synthpop and TVAE. Each of these breached privacy on three out of the six datasets. In two instances, specifically TVAE on Cleveland Heart Disease and TVAE on Credit Approval, the breach was particularly severe. The histograms for TVAE on Credit Approval are shown below and demonstrate that the synthetic examples are far too similar to each other, and also to their closest neighbors in the observed data. The model is a particularly poor representation of the underlying parent distribution. The reason for this may be that the Credit Approval dataset contains several numerical features that are extremely highly skewed.

Other observations and comments

The two GAN-based generators — CopulaGAN and CTGAN — were consistently among the worst performing generators. This was somewhat surprising given the immense popularity of GANs.

The performance of GaussianCopula was mediocre on all datasets except Wisconsin Breast Cancer, for which it attained the equal-highest average maximum cross-set similarity. Its unimpressive performance on the Iris dataset was particularly surprising, given that this is a very simple dataset that can easily be modeled using a mixture of Gaussians, and which we expected would be well-matched to Copula-based methods.

The generators which perform most consistently well across all datasets are synthpop and UNCRi, which both operate by sequential imputation. This means that they only ever need to estimate and sample from a univariate conditional distribution (e.g., P(x₇|x₁, x₂, …)), and this is typically much easier than modeling and sampling from a multivariate distribution (e.g., P(x₁, x₂, x₃, …)), which is (implicitly) what GANs and VAEs do. Whereas synthpop estimates distributions using decision trees (which are the source of the overfitting that synthpop is prone to), the UNCRi generator estimates distributions using a nearest neighbor-based approach, with hyper-parameters optimized using a cross-validation procedure that prevents overfitting.

Conclusion

Synthetic data generation is a new and evolving field, and while there are still no standard evaluation techniques, there is consensus that tests should cover fidelity, utility and privacy. But while each of these is important, they are not on an equal footing. For example, a synthetic dataset may achieve good performance on fidelity and utility but fail on privacy. This does not give it a ‘two out of three’: if the synthetic examples are too close to the observed examples (thus failing the privacy test), the model has been overfitted, rendering the fidelity and utility tests meaningless. There has been a tendency among some vendors of synthetic data generation software to propose single-score measures of performance that combine results from a multitude of tests. This is essentially based on the same ‘two out of three’ logic.

If a synthetic dataset can be considered a random sample from the same parent distribution as the observed data, then we cannot do any better — we have achieved maximum fidelity, utility and privacy. The Maximum Similarity Test provides a measure of the extent to which two datasets can be considered random samples from the same parent distribution. It is based on the simple and intuitive notion that if an observed and a synthetic dataset are random samples from the same parent distribution, instances should be distributed such that a synthetic instance is as similar on average to its closest observed instance as an observed instance is similar on average to its closest observed instance.

We propose the following single-score measure of synthetic dataset quality:

The closer this ratio is to 1 — without exceeding 1 — the better the quality of the synthetic data. It should, of course, be accompanied by a sanity check of the histograms.

References

[1] Gower, J. C. (1971). A general coefficient of similarity and some of its properties. Biometrics, 27(4), 857–871.

[2] Dua, D. & Graff, C., (2017). UCI Machine Learning Repository, Available at: http://archive.ics.uci.edu/ml.

[3] Xu, L., Skoularidou, M., Cuesta-Infante, A. and Veeramachaneni., K. Modeling Tabular data using Conditional GAN. NeurIPS, 2019.

[4] Patki, N., Wedge, R., & Veeramachaneni, K. (2016). The synthetic data vault. In 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA) (pp. 399–410). IEEE.

[5] Nowok, B., Raab G.M., Dibben, C. (2016). “synthpop: Bespoke Creation of Synthetic Data in R.” Journal of Statistical Software, 74(11), 1–26. doi:10.18637/jss.v074.i11.

[6] http://skanalytix.com/uncri-framework

[7] Harrison, D., & Rubinfeld, D.L. (1978). Boston Housing Dataset. Kaggle. https://www.kaggle.com/c/boston-housing. Licensed for commercial use under the CC: Public Domain license.

[8] Kohavi, R. (1996). Census Income. UCI Machine Learning Repository. https://doi.org/10.24432/C5GP7S. Licensed for commercial use under a Creative Commons Attribution 4.0 International (CC BY 4.0) license.

[9] Janosi, A., Steinbrunn, W., Pfisterer, M. and Detrano, R. (1988). Heart Disease. UCI Machine Learning Repository. https://doi.org/10.24432/C52P4X. Licensed for commercial use under a Creative Commons Attribution 4.0 International (CC BY 4.0) license.

[10] Quinlan, J.R. (1987). Credit Approval. UCI Machine Learning Repository. https://doi.org/10.24432/C5FS30. Licensed for commercial use under a Creative Commons Attribution 4.0 International (CC BY 4.0) license.

[11] Fisher, R.A. (1988). Iris. UCI Machine Learning Repository. https://doi.org/10.24432/C56C76. Licensed for commercial use under a Creative Commons Attribution 4.0 International (CC BY 4.0) license.

[12] Wolberg, W., Mangasarian, O., Street, N. and Street,W. (1995). Breast Cancer Wisconsin (Diagnostic). UCI Machine Learning Repository. https://doi.org/10.24432/C5DW2B. Licensed for commercial use under a Creative Commons Attribution 4.0 International (CC BY 4.0) license.

Evaluating Synthetic Data — The Million Dollar Question was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Graph data science methods are usually applied to data that has some inherent graphical nature, e.g., molecular structure data, transport network data, etc. However, graph methods can also be useful on data which does not display any obvious graphical structure, such as the tabular datasets used in machine learning tasks. In this article I will demonstrate simply and intuitively — without any mathematics or theory — that by representing tabular data as a graph we can open new possibilities for how to perform inference on this data.

To keep things simple, I will use the Credit Approval dataset below as an example. The objective is to predict the value of Approval based on the values of one or more of the other attributes. There are many classification algorithms that can be used to do this, but let’s explore how we might approach this using graphs.

Graph representation

The first thing to consider is how to represent the data as a graph. We would like to capture the intuition that the greater the number of attribute values that are shared between two instances, the more similar are those instances. We will use one node to represent each instance (we refer to these nodes as instance nodes), and one node for each possible attribute value (these are attribute value nodes). The connections between instance nodes and attribute value nodes are bi-directional and reflect the information in the table. To keep things really simple, we will omit attributes Employment and History. Here is the graph.

Message Passing

Given attribute values for some new instance, there are several ways in which we might use graph methods to predict some unknown attribute value. We will use the notion of message passing. Here is the procedure we will use.

Message-passing procedure

Initiate a message with value 1 at the starting node, and let this node pass the message to each node to which it is connected. Let any node that receives a message then pass on the message (dilated by a factor k, where 0 <k<1) to each other node to which it is connected. Continue message passing until either a target node is reached (i.e., a node corresponding to the attribute whose value is being predicted), or there are no further nodes to pass the message to. Since a message cannot be passed back to a node from which it was received the process is guaranteed to terminate.

When message passing has completed, each node in the graph will have received zero or more messages of varying value. Sum these values for each node belonging to the target attribute, and then normalize these (sum) values so that they themselves sum to 1. Interpret the normalized values as probabilities. These probabilities can then be used to either predict the unknown attribute value or to impute a random value drawn from the distribution. Dilating message values at each pass reflects the intuition that longer paths should contribute less to the probability estimate than shorter paths.

Example 1

Suppose that we wish to predict the value of Approval given that Income is Low. The arrows on the graph below illustrate the operation of the message-passing procedure, with the thickness of each arrow representing the message value (dilated by factor k = 0.5 at every hop).

The message is initiated at node Income:Low (colored green). This node passes messages of value 1 to both Instance 1 and Instance 2, which then each pass the message (with a dilated value of 0.5) to nodes Education:College and Approval:No. Note that since Education:College receives messages from both Instance 1 and Instance 2, it must pass each of these messages to Instance 3, with a dilated value of 0.25. The numbers at each node of the target variable show the sum of message values received and (in parentheses) the normalized values as percentages. We have the following probabilities for Approval, conditional on Income is Low:

• Prob (Approval is ‘Yes’ | Income is Low) = 20%
• Prob (Approval is ‘No’ | Income is Low) = 80%

These probabilities are different to what we would have obtained from a count-based prediction from the table. Since two of the five instances have Income Low, and both of these have Approval No, a count-based prediction would lead to a probability of 100% for Approval No.

The message-passing procedure has taken into account that the attribute value Education College, possessed by Instances 1 and 2, is also possessed by Instance 3, which has Approval Yes, thus contributing to the total message value received at node Approval:Yes. If we had incorporated the additional attributes Employment and History in our graph, this would likely have further increased the number of paths connecting the start node to the target nodes, thereby utilizing additional contextual information, and improving the estimate of the probability distribution.

Example 2

The message-passing procedure can also be used when conditioning on more than one attribute. In such a case we simply initiate a message at each of the nodes corresponding to the attribute values we are conditioning on and follow the same procedure from each. The graph below shows the result of predicting the value of Approval given Income is Low and Education is Graduate. The sequence of messages originating from each starting node are shown in different colors.

Instance 4 has Education value Graduate, and therefore contributes to the sum of message values received at node Approval:Yes. A further contribution to Approval:Yes is made by Instance 5, which shares Income High with Instance 4.

[Note: in each of these examples we have used Approval as our target variable; however, we could have estimated the probability distributions for Income or Education in exactly the same way].

These examples demonstrate that the simple notion of message passing (a fundamentally graph-based operation), coupled with an appropriate graph representation, allows us to perform inference of a general and flexible nature. Specifically, it can be used to estimate the probability distribution of ANY attribute, conditional on the values of ONE or MORE attributes.

The inference procedure is somewhat ad hoc, devised to make our argument as simply as possible. There are many variations that we could have used. The crucial point is that graph methods enable us to utilize the rich network of relationships between instances in a way that is difficult to capture using vector-based approaches. We could of course attempt to express the message-passing procedure in terms of vectors (i.e., with reference to the table as opposed to the graph) but this would be convoluted and cumbersome. The simplicity and elegance of the graph-based approach arise from the natural pairing of the graph representation and the inference procedure.

There is another interesting observation that we can make from our examples. In the analysis above we did not make any reference to the sum of message values received at the instance nodes (i.e., the nodes along the left-hand side of the graphs). Let’s see what these values can show us. In Example 1, at the completion of message passing the sum of message values at Instance nodes 1 to 5 are respectively 1.0, 1.0, 0.5, 0.0 and 0.0. Normalizing these values to sum to 1 results in 40%, 40%, 20%, 0% and 0%. The first two instances have Approval No, and the sum of their normalized values is 80%. The last three instances have Approval Yes, and the sum of their normalized values is 20%. But these are just the probabilities we obtained for attribute Approval in the original analysis. (You can verify the same for Example 2). This is no coincidence. The sum of message values at an instance node can be interpreted as representing the similarity of that instance to the attribute values we are conditioning on. Thus, the inference procedure can be considered a form of weighted nearest neighbor, in which the similarity measure is implicit in the message-passing procedure.

The UNCRi framework

At Skanalytix we have developed a graph-based computational framework named Unified Numerical/Categorical Representation and Inference (UNCRi). The framework combines a unique graph-based data representation with a flexible inference procedure and can be used to estimate and sample from the conditional distribution of any categorical or numeric variable. It can be applied to tasks such as classification and regression, missing value imputation, anomaly detection, and synthetic data generation from either the full joint distribution or some conditional distribution. The framework is robust to extremities in the data: categorical variables can range from binary to high cardinality; numerical variables can be multi-modal, highly skewed and of arbitrary scale; and the missing-value ratio can be high. You can find out more about UNCRi at http://skanalytix.com.

Conclusion

The Pattern Recognition and Machine Learning disciplines have from the outset been dominated by methods that operate on vectors. Vectors have been so ubiquitous that it can be difficult to conceive of any other approach. Graph methods provide a powerful and flexible alternative. When applied to tabular data, graph methods can be used to not only predict the value of an attribute or impute an attribute with a random value drawn from its estimated distribution, but — by using the chain rule of probability — to generate entire synthetic datasets of instances distributed in the same way as the source data. And all of this from one graph and one inference procedure! While in this article we have considered only categorical variables, it is possible to extend these ideas to datasets containing a mix of numeric and categorical attributes.

Graph Data Science for Tabular Data was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.