Synthetic data generation
A while ago I did some work on the generation of synthetic data from existing real data. This resulted in a small python package that allows to generate synthetic data from a given data set. Data generation is accomplished by first estimating the underlying (joint) pdf from the available samples and then drawing random samples from the estimated pdf. In this post I explain this process with examples. The synthetic data generator package is publicly available in my GitHub.
There are situations in which it is desired to generate synthetic data from existing data. Synthetic data can be used in place of real data where real data cannot be used, e.g., because they may contain sensitive and / or identifying information. Additionally, synthetic data can be used in machine learning applications to fill missing values, alleviate class imbalance, and increase the size of the training set.
Synthetic data can be generated by first estimating a probability density function (pdf) for the existing, i.e., real, data and then sampling from the estimated pdf. This approach enables the generation of synthetic data sets of arbitrary size.
There exist a variety of parametric and non-parametric approaches to pdf estimation and sampling. In the present work we use kernel density estimation, which is a non-parametric method, to estimate the pdf from the data and rejection sampling, which is a Monte Carlo method, to draw random samples from the estimated pdf.
Furthermore, it is noteworthy that another common approach to the generation of synthetic data is to use artificial neural networks, in particular, generative adversarial networks (GANs) and variational autoencoders (VAEs). These network structures consist of two serial blocks, namely, a “generator” and a “discriminator” in GANs, and an “encoder” and a “decoder” in VAEs. These networks are usually first trained on the available data sample as a whole, and then the generator or the decoder component is used to generate new data. In the case of VAEs, the encoder “encodes” the data into low-dimensional vectors in the “latent” space. The decoder “decodes” latent vectors to reconstruct the original data points. The objective of training is to reconstruct the original data as closely as possible. Once the network, including both the encoder and decoder, is trained, the decoder can be used to create new data by inputting new points in the latent space. A detailed description of these network structures is beyond the scope of this post but it is good to be aware of these techniques also!
Data often contain multiple variables. While in some (rare) cases all variables are independent from each other, more commonly, there is usually a correlation structure between these variables and it is often desired to preserve this correlation structure in the synthetic data. Therefore, it is needed to estimate the joint pdf of the data. Note that in practice we can often identify clusters of highly correlated variables in the data. In those cases, the joint pdf can be estimated for each cluster independently and synthetic data generated for each cluster to reduce the complexity and computational cost of the pdf estimation and sampling.
In the following, we first look at the process of estimating the pdf using kernel density estimation (KDE). We then discuss drawing random samples from the estimated pdf using rejection sampling.
Probability density function (pdf) estimation using kernel density estimation (KDE)
A common approach to estimating the pdf from an existing sample is to use kernel density estimation (KDE). KDE is non-parametric technique, in which the pdf is estimated using kernel smoothing from the observed samples.
Get Aras Kayvan’s stories in your inbox
Join Medium for free to get updates from this writer.
Consider the following toy data set consisting of two variables, x1 and x2. I created this toy data set for illustration purposes by drawing samples, at random, from a normal and a uniform distribution, corresponding to x1 and x2, respectively.
There are many packages that implement KDE. Let us python’s statsmodels KDE implementation for the present visualizations.
The estimated pdf is plotted below. Looking at this distribution, I can convince myself that the marginal distribution of x1 is normal and the marginal distribution of x2 is uniform, and therefore give this estimate a pass!
Sampling using rejection sampling
Once the pdf of the data is estimated, we use rejection sampling to generate random samples from the pdf. Rejection sampling is easily visualized for a one-dimensional (univariate) pdf but it is directly applicable to multivariate pdf’s of arbitrary dimensionality. To sample a one-dimensional pdf, one can uniformly sample the two-dimensional Cartesian space formed by plotting the pdf and only retain points under the curve by rejecting those above (hence the name rejection sampling). See the following illustration.
In the present work, we use rejection sampling to draw samples from the estimated distributions. The following is an implementation of rejection sampling in python. Note that when the data include discrete variables, their discrete nature must be taken into account to avoid creating invalid samples.
Summary
In this post I explained the process of generating synthetic data from available real data. This process consists of two main steps: (1) estimating the (joint) probability density function (pdf) of the data from the available samples using kernel density estimation (KDE), and (2) drawing random samples from the estimated pdf using rejection sampling.
I have created a small python package, called SynDat, for synthetic data generation. Check out the github repo for code and more details on the usage.
