<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ja-Keoung Koo on Medium]]></title>
        <description><![CDATA[Stories by Ja-Keoung Koo on Medium]]></description>
        <link>https://medium.com/@jakeoungkoo?source=rss-6265e2b8f4c2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*1rjgffW82oqMLpcj.</url>
            <title>Stories by Ja-Keoung Koo on Medium</title>
            <link>https://medium.com/@jakeoungkoo?source=rss-6265e2b8f4c2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 16:55:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jakeoungkoo/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[draft]]></title>
            <link>https://medium.com/@jakeoungkoo/draft-36b2d7d2faa3?source=rss-6265e2b8f4c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/36b2d7d2faa3</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Ja-Keoung Koo]]></dc:creator>
            <pubDate>Tue, 23 Jan 2018 22:44:10 GMT</pubDate>
            <atom:updated>2018-01-23T22:44:10.301Z</atom:updated>
            <content:encoded><![CDATA[<p>Gaussian process regression: Let the data explain our prior</p><p>In this post, we aim to understand the basics of Gaussian process in regression problem and derive the explicit formulas in detail. Given training data \(\{x_i,y_i\}\), we need to predict target values for any continuous input. This estimation is called Regression and cannot be achieved without assuming an underlying function. Then, the question is: how can we model the hidden function? As an approach, we can specify functions such as polynomial form before observing data. To make this assumption hold, we need reliable information on our data. However, in practice, it is difficult to know the trustworthy information in advance.</p><p><strong>In Gaussian process, Random variables represent the value of the function.</strong></p><p>Instead of restricting a function form, Gaussian process is more interactive to data by adapting its distribution. As shown in Figure 1, Gaussian process predicts more correctly, as we collect more training data. Gaussian process’s distribution is the joint distribution of (infinitely many) random variables representing the value of the function. Gaussian process regards a function as a vector form: \([f(x_1), f(x_2), f(x_3), \cdots]^T\).</p><p>[img1] Figure 1. Effect of the number of training data. As training data increases, the uncertainty in our posterior decreases. The shaded area denotes the 98% confidence region.</p><p>Gaussian process</p><p>Gaussian process is a collection of random variables which represent the function values and follow multivariate Gaussian distributions. We need to specify mean and covariance of its distribution. In this post, we choose the zero mean function because it is reasonable choice without observing data and makes our calculation easier. On the other hand, we impose on covariance function our prior assumption: if the input values are similar, their function values should be close to each other. This assumption leads to the following covariance function:</p><p>\[cov(f(x),f(x’)) = K(x,x’) = \exp( -\frac{1}{2} \| x — x’ \|²) \tag{✻}\]</p><p>With these mean and covariance, we denote Gaussian process as follows:</p><p>\[f \sim GP(m(x), cov(x, x’))\]</p><p>As we discussed before, Gaussian process regards a function as a high-dimensional vectors showing function values. When we sample a function from Gaussian process, we receive a sequence of function values. This is where I have difficulty understanding a weird concept of function.</p><p>Gaussian process employ the fact that if likelihood follows normal distribution and prior distribution follows Gaussian proess, then the posterior also follows Gaussian process.</p><p><strong>Prior</strong></p><p>In Gaussian process regression, we have two random variables of training outputs \(f\) and test outputs \(f_✻\). Their joint distribution is easily computed by the property of normal distribution. To be specific, the joint distribution is given by:</p><p>\[\begin{bmatrix}f \\ f_✻\end{bmatrix} \sim\mathcal{N}\left( \begin{bmatrix}m(x)\\ m(x_✻)\end{bmatrix}, \begin{bmatrix}K &amp; K_✻ \\ K_✻^T &amp; {K}{✻✻}\end{bmatrix} \right)\]</p><p>where the random variable \(f_✻\) is what we are interested in and its distribution also follows normal distribution (without prove):</p><p>\[p(\mathbf{f}✻\vert \mathbf{x}✻, \mathbf{x}, \mathbf{f})=\mathcal{N}(\mathbf{f}✻\vert \mu✻, \Sigma_✻).\]</p><p><strong>Posterior estimation</strong></p><p>With the help of conditional normal distribution formula, we compute the distribution of test outputs where its mean and covariance is given by:</p><p>\[\mu_✻ = m(x_✻) + K_✻^T K^{-1}(f — m(x))\]</p><p>\[\Sigma_✻ = K_{✻✻} — K_✻^T K^{-1} K_✻\]</p><p>We predict the test function values by sampling from the posterior distribution.</p><p><strong>Noisy observation</strong></p><p>In our covariance function in (✻), we have assumed that our data is perfect without noise. This assumption makes our predicted variance zero on the observed point. [With this assumption, Gaussian process makes our predicted variance zero on the observed point.] In practice, however, we live in a noisy world and we could not believe our data perfectly. Therefore, instead of assuming perfect data, we assume that our data is corrupted with Gaussian noise:</p><p>\[y_i = f(x_i) + \epsilon_i\]</p><p>Then, we need to adjust our covariance function for the random variable \(y\):</p><p>\[cov(x_i,x_j) = K(x_i,x_j) + \sigma²\delta_{ij}\]</p><p>, where \(x_i, x_j\) are the elements in training data. With this slight modification, our prediction becomes</p><p>\[\mu_✻ = m(x_✻) + K_✻^T (K + \sigma² I)^{-1}(y — m(x))\]</p><p>\[\Sigma_✻ = K_{✻✻} — K_✻^T (K + \sigma² I)^{-1} K_✻\]</p><p>As shown in Figure 2, if we impose noise in data process, the uncertainty becomes large around observation. That’s it! Our journey to Gaussian process is done.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36b2d7d2faa3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>