<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ronik Basak on Medium]]></title>
        <description><![CDATA[Stories by Ronik Basak on Medium]]></description>
        <link>https://medium.com/@ronikbasak?source=rss-600f37eccd68------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*RH42RgPALdPvShNDUI32Mg.jpeg</url>
            <title>Stories by Ronik Basak on Medium</title>
            <link>https://medium.com/@ronikbasak?source=rss-600f37eccd68------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 01:24:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ronikbasak/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Navigating the Uncertainty of Stock Price Prediction: A home-brewed instrument for prediction using…]]></title>
            <link>https://medium.com/@ronikbasak/navigating-the-uncertainty-of-stock-price-prediction-a-home-brewed-instrument-for-prediction-using-d2268594389b?source=rss-600f37eccd68------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2268594389b</guid>
            <category><![CDATA[stock-market]]></category>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[fintech]]></category>
            <category><![CDATA[investment]]></category>
            <category><![CDATA[prediction-markets]]></category>
            <dc:creator><![CDATA[Ronik Basak]]></dc:creator>
            <pubDate>Fri, 24 Mar 2023 17:45:01 GMT</pubDate>
            <atom:updated>2023-03-25T04:44:12.065Z</atom:updated>
            <content:encoded><![CDATA[<h3>Navigating the Uncertainty of Stock Price Prediction: A home-brewed instrument for prediction using Python</h3><p>The stock market is a marketplace where publicly traded companies’ stocks are bought and sold. It is also known as the equity market or the share market. The stock market provides a platform for companies to raise capital by issuing stocks or shares to the public. Investors, in turn, buy these shares in the hope of earning a return on their investment, either through capital gains or dividends.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q6_6YKiC1I2sGfI4P6sDgA.jpeg" /><figcaption>courtesy: <a href="https://www.thebalancemoney.com/outperform-the-market-3305874">FANATIC STUDIO/GETTY IMAGES</a></figcaption></figure><p>The stock market plays a crucial role in the finance industry as it enables companies to raise capital, provides a platform for investors to invest in companies, and helps facilitate the transfer of risk from investors to traders. The stock market is also an important indicator of the overall health of the economy, as it reflects investor sentiment about the prospects of the companies listed on the market and the broader economic environment.</p><p>There are two types of stock markets: primary markets and secondary markets. In the primary market, companies issue new stocks to raise capital, while in the secondary market, investors trade stocks among themselves.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uZAWwTuGewusDJbuNAbSAw.jpeg" /></figure><p>There are a few strategies that investors may use to attempt to profit from stock price prediction:</p><ol><li><strong><em>Fundamental analysis</em></strong>: This involves researching a company’s financial and economic factors, such as revenue, earnings, assets, and liabilities, to determine its potential for growth and profitability. If the company’s financials are strong, its stock price may increase, allowing investors to profit.</li><li><strong><em>Technical analysis</em></strong>: This involves analyzing past market trends and stock price patterns to predict future stock price movements. Technical analysts use various tools and techniques, such as chart patterns, indicators, and trading signals, to identify potential buying or selling opportunities.</li><li><strong><em>Quantitative analysis</em></strong>: This involves using mathematical and statistical models to analyze large amounts of financial data and identify patterns that may predict future stock price movements. This approach requires advanced programming and data analysis skills and is often used by institutional investors and hedge funds.</li></ol><p>It’s important to note that these strategies are not foolproof and carry a certain degree of risk. Investors should also be prepared to hold onto their investments for the long term and not be swayed by short-term market fluctuations. Additionally, it’s essential to have a well-diversified portfolio and to only invest money that you can afford to lose.</p><p>The scope of this blog post is mostly concentrated on touch-basing a set of components related to the quantitative analysis of stock price prediction and the definition of a few of those technical components and models involved in it. This post also includes and shares a <strong><em>tool</em></strong>(<a href="https://colab.research.google.com/drive/1TlOt2dP8mSzwUe4yy-fgtDypUwKZ4izR?usp=sharing">a google-colab notebook link</a>) with its readers which can be used to get the prediction of a stock price given the historical data of that stock.</p><h3>Call option:</h3><p>A call option is a financial contract that gives the holder the right, but not the obligation, to buy a specific underlying asset (such as a stock, commodity, or currency) at a predetermined price (the strike price) on or before a specific date (the expiration date). The holder of a call option is betting that the price of the underlying asset will increase in the future, allowing them to buy the asset at a lower price and sell it at a higher price, thus making a profit.</p><p>For example, suppose an investor holds a call option to buy 100 shares of Company XYZ at a strike price of $50 per share, with an expiration date of one month from now. If the price of Company XYZ shares increases to $60 per share before the expiration date, the investor can exercise their option and buy the shares for $50 each, then sell them on the market for $60 each, realizing a profit of $10 per share.</p><h3>Difference between the long-call option and short-call option:</h3><p>A long call option and a short call option are two different types of options contracts that give the holder the right to buy an underlying asset at a predetermined price, known as the strike price, on or before a specified date, known as the expiration date. The main difference between a long call option and a short call option is the position of the investor in the transaction.</p><p>A long call option is a type of option contract in which the investor, or buyer, has the right, but not the obligation, to buy the underlying asset at the strike price on or before the expiration date. If the price of the underlying asset increases above the strike price before the expiration date, the investor can exercise the option and buy the asset at the lower strike price, which can lead to a profit. However, if the price of the underlying asset decreases or remains below the strike price, the investor may choose not to exercise the option, in which case they would lose the premium paid to purchase the option.</p><p>On the other hand, a short-call option is a type of option contract in which the investor, or seller, has the obligation to sell the underlying asset at the strike price if the holder of the option exercises the right to buy the asset on or before the expiration date. In exchange for taking on this obligation, the seller receives a premium payment from the buyer of the option. If the price of the underlying asset remains below the strike price, the holder of the option is unlikely to exercise the option, and the seller keeps the premium as profit. However, if the price of the underlying asset increases above the strike price, the seller may be required to sell the asset at a loss, which could be much higher than the premium received.</p><blockquote><strong><em>P&amp;L (Long Call)</em></strong> upon expiry is calculated as: P&amp;L = Max [0, (Spot Price — Strike Price)] — Premium Paid</blockquote><blockquote><strong><em>P&amp;L (Short Call)</em></strong> option upon expiry is calculated as P&amp;L = Premium Received — Max [0, (Spot Price — Strike Price)]</blockquote><h3>Put option:</h3><p>A put option is a financial contract that gives the holder the right, but not the obligation, to sell a specific underlying asset at a predetermined price on or before a specific date. The holder of a put option is betting that the price of the underlying asset will decrease in the future, allowing them to sell the asset at a higher price and buy it back at a lower price, thus making a profit.</p><p>For example, suppose an investor holds a put option to sell 100 shares of Company XYZ at a strike price of $50 per share, with an expiration date of one month from now. If the price of Company XYZ shares decreases to $40 per share before the expiration date, the investor can exercise their option and sell the shares for $50 each, then buy them back on the market for $40 each, realizing a profit of $10 per share.</p><p>Both call options and put options are traded on exchanges and can be bought and sold like any other financial asset. They are often used by investors and traders to speculate on the price movements of underlying assets, hedge against potential losses, and generate income through option writing strategies.</p><h3>Difference between the long-put option and short-put option:</h3><p>A long put option and a short put option are two different types of options contracts that give the holder the right to sell an underlying asset at a predetermined price, known as the strike price, on or before a specified date, known as the expiration date. The main difference between a long put option and a short put option is the position of the investor in the transaction.</p><p>A long put option is a type of option contract in which the investor, or buyer, has the right, but not the obligation, to sell the underlying asset at the strike price on or before the expiration date. If the price of the underlying asset decreases below the strike price before the expiration date, the investor can exercise the option and sell the asset at the higher strike price, which can lead to a profit. However, if the price of the underlying asset increases or remains above the strike price, the investor may choose not to exercise the option, in which case they would lose the premium paid to purchase the option.</p><p>On the other hand, a short put option is a type of option contract in which the investor, or seller, has the obligation to buy the underlying asset at the strike price if the holder of the option exercises the right to sell the asset on or before the expiration date. In exchange for taking on this obligation, the seller receives a premium payment from the buyer of the option. If the price of the underlying asset remains above the strike price, the holder of the option is unlikely to exercise the option, and the seller keeps the premium as profit. However, if the price of the underlying asset decreases below the strike price, the seller may be required to buy the asset at a loss, which could be much higher than the premium received.</p><blockquote><strong><em>P&amp;L (Long Put)</em></strong> upon expiry is calculated as P&amp;L = [Max (0, Strike Price — Spot Price)] — Premium Paid</blockquote><blockquote><strong><em>P&amp;L (Short Put)</em></strong> option upon expiry is calculated as P&amp;L = Premium Received — Max (0, Strike Price — Spot Price)</blockquote><h3>Strike price:</h3><p>The strike price is the price at which the holder of an option has the right to buy or sell the underlying asset, such as a stock, currency, or commodity, depending on whether the option is a call option or a put option.</p><p>In the case of a call option, the strike price is the price at which the holder of the option has the right to buy the underlying asset. For example, if an investor holds a call option with a strike price of $50 on a stock, it means that they have the right to buy the stock at $50, regardless of its current market price.</p><p>In the case of a put option, the strike price is the price at which the holder of the option has the right to sell the underlying asset. For example, if an investor holds a put option with a strike price of $50 on a stock, it means that they have the right to sell the stock at $50, regardless of its current market price.</p><p>The strike price is an important factor in determining the price of an option, along with the current market price of the underlying asset, the time until expiration, and the volatility of the underlying asset. The higher the strike price, the lower the value of a call option and the higher the value of a put option. Conversely, the lower the strike price, the higher the value of a call option and the lower the value of a put option.</p><h3>Risk-free interest rate:</h3><p>The risk-free interest rate is the theoretical rate of return on an investment that carries no risk. It is the minimum rate of return that an investor expects to earn on an investment with zero risks.</p><p>In the financial markets, the risk-free interest rate is often based on the yield of a government bond, such as a U.S. Treasury bond, which is considered to be a risk-free investment since the government is highly unlikely to default on its debt.</p><p>The risk-free interest rate is used in many financial models and calculations, such as determining the fair value of an option or calculating the cost of capital for a company. In these cases, the risk-free rate serves as a benchmark to compare the expected rate of return of an investment with its level of risk.</p><ul><li>The real risk-free rate is the required return on zero-risk financial instruments with the rate of inflation taken into account.</li><li>The nominal risk-free rate refers to the yield on a risk-free asset without the effect of inflation.</li></ul><p>The risk-free rate is commonly considered to be equal to the interest paid on a 3-month government Treasury bill, generally the safest investment an investor can make.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7cfp0m1PDrBJ2mMOv_Wsjg.png" /></figure><p>However, it is important to note that while the risk-free rate is used as a benchmark, there is no investment that is truly risk-free in practice. All investments carry some degree of risk, even if it is small.</p><h3>Expected Return(mu):</h3><p>The expected return of a stock is the estimated rate of return that an investor expects to earn on the stock over a given time period. One way to calculate the expected return of a stock using historical data is to use the following formula:</p><p>Expected Return = (Historical Average Return) + (Expected Risk Premium)</p><p>where:</p><ul><li>Historical Average Return: This is the average rate of return that the stock has generated over a specific time period, such as the past year, 3 years, 5 years, or 10 years. The historical average return can be calculated using the following formula:</li></ul><p>Historical Average Return = (Sum of Historical Returns) / (Number of Historical Periods)</p><ul><li>Expected Risk Premium: This is the additional return that the investor expects to earn on the stock to compensate for the level of risk associated with the investment. The expected risk premium can be estimated based on factors such as the volatility of the stock, the company’s financial health, and the overall economic environment.</li></ul><p>The formula for expected return is just one approach, and there are other methods that can be used to estimate the expected return of a stock, such as the Capital Asset Pricing Model (CAPM) or the Dividend Discount Model (DDM). It is important to note that the expected return used in any investment analysis is just an estimate and may not accurately reflect the actual returns that the stock will generate in the future. Investors should always perform their own due diligence and consult with a qualified financial advisor before making any investment decisions.</p><p>Here’s an example Python program that demonstrates how we can calculate the expected return of a stock given the historical data:</p><pre>import pandas as pd<br>import numpy as np<br><br># Load historical stock data<br>df = pd.read_csv(&#39;historical_data.csv&#39;)<br><br># Calculate historical average return<br>historical_return = df[&#39;Return&#39;].mean()<br><br># Estimate expected risk premium<br>expected_risk_premium = 0.05<br><br># Calculate expected return using formula<br>expected_return = historical_return + expected_risk_premium<br><br>print(&#39;Historical Average Return: {:.2f}%&#39;.format(historical_return*100))<br>print(&#39;Expected Risk Premium: {:.2f}%&#39;.format(expected_risk_premium*100))<br>print(&#39;Expected Return: {:.2f}%&#39;.format(expected_return*100))</pre><p>In this example, the historical stock data is loaded from a CSV file named ‘historical_data.csv’, and the ‘Return’ column contains the historical returns of the stock. The historical average return is calculated using the mean() function, and an expected risk premium of 5% is assumed. The expected return is then calculated using the formula, and the results are printed to the console.</p><p>Note that this is just one example of how to calculate the expected return of a stock using Python, and there are many different approaches and variations that can be used depending on your specific investment strategy or analysis.</p><h3>Historical Volatility &amp; Implied Volatility(sigma):</h3><p>Historical volatility and implied volatility are two important concepts used in options trading to measure the expected volatility of the underlying asset.</p><p><strong><em>Historical volatility</em></strong> refers to the actual volatility of the underlying asset over a certain period of time, such as a year or a month. It is calculated using the standard deviation of the asset’s daily returns over the chosen period. Historical volatility is a backward-looking measure, as it reflects the actual volatility that has occurred in the past.</p><p><strong><em>Implied volatility</em></strong>, on the other hand, is a measure of the expected volatility of the underlying asset in the future, as implied by the current market price of an option. It is calculated by inputting the current market price of an option into an option pricing model, such as the <strong>Black-Scholes model</strong>(<strong><em>I have talked about this model in detail in the below section of this post</em></strong>), and solving for the volatility parameter. Implied volatility is a forward-looking measure, as it reflects the market’s expectation of how volatile the underlying asset will be in the future.</p><p>Implied volatility is often used by traders to assess whether an option is overpriced or underpriced relative to the expected future volatility of the underlying asset. If the implied volatility is higher than the historical volatility, it suggests that the market expects the asset to be more volatile in the future than it has been in the past, which could make the option more expensive. Conversely, if the implied volatility is lower than the historical volatility, it suggests that the market expects the asset to be less volatile in the future than it has been in the past, which could make the option less expensive.</p><p>Overall, historical volatility and implied volatility are both important measures of the expected volatility of the underlying asset and can help traders make informed decisions about option trading strategies.</p><h3><strong>Mathematical formula:</strong></h3><p>The to calculate the implied volatility of a stock in India is the same as the formula used in other countries:</p><blockquote>Implied Volatility = Market Price of the Option / Theoretical Price of the Option</blockquote><p>where the theoretical price of the option is calculated using an option pricing model, such as the Black-Scholes model. The implied volatility is the parameter that is solved in the option pricing model.</p><p>The Black-Scholes model uses the following formula to calculate the theoretical price of a call option:</p><blockquote>C = S * N(d1) — X * e^(-r*t) * N(d2)</blockquote><p>where:</p><ul><li>C is the theoretical price of the call option</li><li>S is the current stock price</li><li>X is the strike price of the option</li><li>r is the risk-free interest rate</li><li>t is the time to the expiration of the option (in years)</li><li>N() is the cumulative distribution function of the standard normal distribution</li><li>d1 and d2 are calculated as follows:</li></ul><blockquote><strong>d1 = [ ln(S/X) + (r + σ²/2) * t ] / ( σ * sqrt(t) ) d2 = d1 — σ * sqrt(t)</strong></blockquote><blockquote>where σ is the volatility of the underlying asset.</blockquote><p>Once the theoretical price of the option is calculated, the implied volatility can be obtained by rearranging the formula as follows:</p><blockquote><strong>σ = sqrt[2π/t] * (C/X) * e^(r*t) * N^-1[d2/Sqrt(t)]</strong></blockquote><blockquote>where N^-1() is the inverse of the cumulative distribution function of the standard normal distribution.</blockquote><p>This formula can be used to calculate the implied volatility of a call option or a put option on a stock in India, provided that the other inputs to the Black-Scholes model (such as the stock price, strike price, risk-free interest rate, and time to expiration) are known.</p><p>Let’s get our hands dirty to write a Python program that uses the Black-Scholes model to calculate the implied volatility of a call option on a stock:</p><p>Calculating implied volatility in Python requires several steps:</p><ol><li>Collect data on the stock price, option price, strike price, expiration date, and risk-free interest rate. This can be done using an API or by manually inputting the data.</li><li>Use an option pricing model, such as the Black-Scholes model, to calculate the theoretical price of the option.</li><li>Solve for the implied volatility by iteratively adjusting the volatility parameter until the theoretical price matches the market price of the option.</li></ol><pre>import numpy as np<br>from scipy.stats import norm<br>from scipy.optimize import bisect<br><br># Define the Black-Scholes formula for a call option<br>def black_scholes_call(S, K, T, r, sigma):<br>    d1 = (np.log(S / K) + (r + 0.5 * sigma**2) * T) / (sigma * np.sqrt(T))<br>    d2 = d1 - sigma * np.sqrt(T)<br>    N1 = norm.cdf(d1)<br>    N2 = norm.cdf(d2)<br>    call_price = S * N1 - K * np.exp(-r * T) * N2<br>    return call_price<br><br># Define the function to solve for implied volatility<br>def implied_volatility(S, K, T, r, C):<br>    def f(sigma):<br>        return black_scholes_call(S, K, T, r, sigma) - C<br>    return bisect(f, 0.01, 2)<br><br># Input the stock price, option price, strike price, expiration date, and risk-free rate<br>S = 100<br>C = 10<br>K = 105<br>T = 0.5<br>r = 0.05<br><br># Calculate implied volatility<br>iv = implied_volatility(S, K, T, r, C)<br>print(f&quot;Implied volatility: {iv}&quot;)</pre><p>Alternatively, To calculate the implied volatility of a stock using Python, you can use the black_scholes function from the py_vollib library, which is a library for calculating option prices and implied volatilities using various models.</p><p>Here’s an example Python program that uses the black_scholes function to calculate the implied volatility of a call option on a stock:</p><pre>import math<br>import scipy<br>from py_vollib.black_scholes import black_scholes<br><br># Define the option parameters<br>S = 192.90  # stock price<br>X = 200.0  # strike price<br>t = 0.083  # time to expiration (in years), considering the expiry date to be 27/04/2023, the duration would be almost 1 month i.e. 1/12 = 0.083 years<br>r = 0.05  # risk-free interest rate<br>C = 4.80  # market price of the call option<br><br># Define the function to calculate implied volatility<br>def implied_volatility(S, X, t, r, C):<br>    def f(sigma):<br>        return black_scholes(&#39;c&#39;, S, X, t, r, sigma) - C<br>    return scipy.optimize.brentq(f, 1e-6, 1.0)<br><br># Calculate the implied volatility<br>sigma = implied_volatility(S, X, t, r, C)<br>print(&quot;Implied volatility:&quot;, sigma)</pre><h3>Monte Carlo Simulation:</h3><p>Monte Carlo Simulation is a statistical method used to model and predict the behaviour of complex systems, including stock prices. In the context of stock price prediction, Monte Carlo Simulation involves generating a large number of possible future stock price paths based on a set of assumptions and inputs and then analyzing the resulting distribution of outcomes to estimate the probability of different stock price scenarios.</p><p>To perform a Monte Carlo Simulation for stock price prediction, a model is developed that incorporates various factors that influence the stock price, such as historical price trends, volatility, interest rates, economic indicators, and company-specific news and events. The model typically uses a stochastic process, such as Geometric Brownian Motion, to simulate the random fluctuations in the stock price over time.</p><p>Once the model is developed, a large number of simulations are run, each of which generates a possible future path of the stock price based on the assumptions and inputs used in the model. The resulting set of simulated paths can be analyzed to calculate various statistics and probabilities, such as the expected value of the stock price, the range of possible price outcomes, and the likelihood of the stock price crossing certain thresholds or hitting certain targets.</p><p>Monte Carlo Simulation can be a useful tool for stock price prediction because it allows for a probabilistic approach that captures the uncertainty and complexity of stock price movements. However, it should be noted that Monte Carlo Simulation is based on various assumptions and inputs that may not accurately reflect the true behaviour of the stock market, and the results of the simulation should be interpreted with caution.</p><h3>Deep dive on Geometric Brownian Motion:</h3><p><strong><em>Geometric Brownian Motion</em></strong> is a mathematical model that is often used in finance to describe the dynamics of stock prices over time. The model is based on the assumption that stock prices follow a random walk with constant drift and volatility, where the drift is the expected rate of return and the volatility is the standard deviation of returns.</p><p>One of the key applications of Geometric Brownian Motion in finance is the pricing of options, including call options and put options. An option is a financial contract that gives the holder the right, but not the obligation, to buy or sell an underlying asset (such as stock) at a predetermined price (the strike price) on or before a specific date (the expiration date).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_n5cPsBOIO5kiyLU7PiMGg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cmETp-XNwvzrTZiUf-yE0g.png" /></figure><h3>Black-Scholes model:</h3><p>The <strong><em>Black-Scholes model</em></strong> is a widely used option pricing model that is based on the assumption of Geometric Brownian Motion. This model takes into account the expected return, volatility, time to expiration, and the strike price of the underlying asset, as well as the prevailing risk-free interest rate, to calculate the fair price of an option.</p><p>By using the Black-Scholes model, traders and investors can estimate the price of a call option or put option for a particular stock, based on current market conditions and other relevant factors. This information can be used to make informed investment decisions and manage risk in a portfolio of assets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H81M6-jonZTQ5-B7VjBXUA.png" /></figure><p>Here’s an example Python program that uses Monte Carlo Simulation with Geometric Brownian Motion to predict the future price of a stock:</p><pre>import numpy as np<br>import pandas as pd<br>import matplotlib.pyplot as plt<br><br># Set parameters for the simulation<br>T = 252 # number of trading days<br>N = 10000 # number of simulations<br>S0 = 100 # initial stock price<br>mu = 0.1 # expected annual return<br>sigma = 0.2 # annual volatility<br><br># Calculate drift and diffusion coefficients<br>dt = 1 / T<br>mu_dt = (mu - 0.5 * sigma**2) * dt<br>sigma_dt = sigma * np.sqrt(dt)<br><br># Generate stock price paths using Monte Carlo Simulation with GBM<br>S = np.zeros((T+1, N))<br>S[0, :] = S0<br>for t in range(1, T+1):<br>    eps = np.random.normal(0, 1, N)<br>    S[t, :] = S[t-1, :] * np.exp(mu_dt + sigma_dt * eps)<br><br># Calculate statistics of the simulated stock price paths<br>S_mean = np.mean(S, axis=1)<br>S_std = np.std(S, axis=1)<br>S_upper = S_mean + 1.96 * S_std<br>S_lower = S_mean - 1.96 * S_std<br><br># Plot the simulated stock price paths and confidence intervals<br>fig, ax = plt.subplots()<br>ax.plot(S_mean, color=&#39;blue&#39;, label=&#39;Mean&#39;)<br>ax.fill_between(range(T+1), S_upper, S_lower, color=&#39;gray&#39;, alpha=0.2, label=&#39;95% Confidence Interval&#39;)<br>ax.set_xlabel(&#39;Trading Days&#39;)<br>ax.set_ylabel(&#39;Stock Price&#39;)<br>ax.set_title(&#39;Monte Carlo Simulation with GBM&#39;)<br>ax.legend()<br>plt.show()</pre><p>This program uses the numpy library to generate random numbers, the pandas library to organize the data, and the matplotlib library to plot the results. The program begins by setting the parameters for the simulation, including the number of trading days, the number of simulations to run, the initial stock price, the expected annual return, and the annual volatility.</p><p>Next, the program calculates the drift and diffusion coefficients for the Geometric Brownian Motion model and then uses a loop to generate a series of simulated stock price paths using Monte Carlo Simulation with GBM. The program stores the results in a 2D NumPy array, where each row represents a different trading day and each column represents a different simulation.</p><p>After the simulations are complete, the program calculates the mean and standard deviation of the simulated stock price paths and uses these values to calculate a 95% confidence interval for the future stock price. Finally, the program uses matplotlib to plot the mean stock price and confidence interval over the simulated trading days.</p><p>This is a simple example, and in practice, more complex models and additional factors may need to be incorporated to improve the accuracy of the stock price predictions.</p><p><strong><em>An alternative implementation</em></strong>:</p><p>Here is another example Python program to predict stock price using Monte Carlo Simulation with Geometric Brownian Motion:</p><pre>import numpy as np<br>import matplotlib.pyplot as plt<br><br># Set up simulation parameters<br>S0 = 192.90  # initial stock price<br>mu = -0.53  # expected return<br>sigma = 0.334  # volatility<br>T = 1  # time horizon (in years)<br>N = 252  # number of trading days in a year<br>dt = T / N  # time step<br>n_sim = 10000  # number of simulations<br><br># Generate stock price paths using Monte Carlo Simulation with Geometric Brownian Motion<br>t = np.linspace(0, T, N + 1)<br>dB = np.random.normal(0, np.sqrt(dt), size=(n_sim, N))<br>B = np.cumsum(dB, axis=1)<br>S = S0 * np.exp((mu - 0.5 * sigma ** 2) * t[1:] + sigma * B)<br><br># Calculate statistics and plot results<br>mean_S = np.mean(S, axis=0)<br>upper_95 = np.percentile(S, 97.5, axis=0)<br>lower_95 = np.percentile(S, 2.5, axis=0)<br><br>plt.plot(t[1:], mean_S, label=&#39;Mean&#39;)<br>plt.plot(t[1:], upper_95, &#39;--&#39;, label=&#39;95% Confidence Interval&#39;)<br>plt.plot(t[1:], lower_95, &#39;--&#39;)<br>plt.xlabel(&#39;Time (years)&#39;)<br>plt.ylabel(&#39;Stock Price&#39;)<br>plt.title(&#39;Monte Carlo Simulation with Geometric Brownian Motion&#39;)<br>plt.legend()<br>plt.show()</pre><p>This program generates 10,000 simulated paths of a stock price using Monte Carlo Simulation with Geometric Brownian Motion. The stock price is assumed to have an initial value of 100, an expected return of 5%, and a volatility of 20% over a time horizon of 1 year. The program then calculates the mean and 95% confidence interval of the simulated stock price paths and plots the results.</p><p>Please note again, that this is a fairly simple example and there are many factors and assumptions that can be added to make the simulation more realistic and accurate.</p><h3>Determine call-option and put-option price of a stock using Monte Carlo Simulation with GBM:</h3><p>Here’s a Python program to determine the price of a <strong>call option</strong> with a given strike price for a stock using Monte Carlo Simulation with Geometric Brownian Motion:</p><pre>import numpy as np<br><br>def call_option_price(S, K, T, r, sigma, n):<br>    &quot;&quot;&quot;<br>    This function computes the price of a call option using Monte Carlo simulation with GBM.<br><br>    Parameters:<br>    S: float, the initial stock price<br>    K: float, the strike price<br>    T: float, the time to maturity in years<br>    r: float, the risk-free interest rate<br>    sigma: float, the volatility of the stock returns<br>    n: int, the number of simulations<br><br>    Returns:<br>    The estimated price of the call option based on the simulation.<br>    &quot;&quot;&quot;<br>    dt = T / 252.0<br>    N = int(T / dt)<br>    S_t = np.zeros((n, N+1))<br>    S_t[:, 0] = S<br>    for i in range(1, N+1):<br>        Z = np.random.normal(0, 1, size=n)<br>        S_t[:, i] = S_t[:, i-1] * np.exp((r - 0.5 * sigma**2) * dt + sigma * np.sqrt(dt) * Z)<br>    payoff = np.maximum(S_t[:, -1] - K, 0)<br>    call_price = np.exp(-r * T) * np.mean(payoff)<br>    return call_price<br><br># Example usage<br>S = 100   # Initial stock price<br>K = 110   # Strike price<br>T = 1     # Time to maturity in years<br>r = 0.05  # Risk-free interest rate<br>sigma = 0.2  # Volatility<br>n = 10000  # Number of simulations<br><br>call_price = call_option_price(S, K, T, r, sigma, n)<br>print(f&quot;Call option price: {call_price:.2f}&quot;)</pre><p>This program uses the Monte Carlo Simulation with Geometric Brownian Motion to generate a set of simulated stock prices over the time to maturity of the option. It then computes the payoff of the call option at expiration, which is the difference between the stock price and the strike price, if positive, or zero otherwise. Finally, it takes the average of the payoffs, discounted to the present value using the risk-free interest rate, to obtain an estimate of the call option price.</p><p>The program allows you to customize the input parameters, such as the initial stock price, the strike price, the time to maturity, the risk-free interest rate, the volatility, and the number of simulations. You can run this program with different inputs to explore how the option price changes with these factors and to compare the simulation results with other pricing methods, such as the Black-Scholes formula or the Binomial Model.</p><p>Here’s a Python program to determine the price of a <strong>put option</strong> with a given strike price for a stock using Monte Carlo Simulation with Geometric Brownian Motion:</p><pre>import numpy as np<br><br>def put_option_price(S, K, T, r, sigma, n):<br>    &quot;&quot;&quot;<br>    This function computes the price of a put option using Monte Carlo simulation with GBM.<br><br>    Parameters:<br>    S: float, the initial stock price<br>    K: float, the strike price<br>    T: float, the time to maturity in years<br>    r: float, the risk-free interest rate<br>    sigma: float, the volatility of the stock returns<br>    n: int, the number of simulations<br><br>    Returns:<br>    The estimated price of the put option based on the simulation.<br>    &quot;&quot;&quot;<br>    dt = T / 252.0<br>    N = int(T / dt)<br>    S_t = np.zeros((n, N+1))<br>    S_t[:, 0] = S<br>    for i in range(1, N+1):<br>        Z = np.random.normal(0, 1, size=n)<br>        S_t[:, i] = S_t[:, i-1] * np.exp((r - 0.5 * sigma**2) * dt + sigma * np.sqrt(dt) * Z)<br>    payoff = np.maximum(K - S_t[:, -1], 0)<br>    put_price = np.exp(-r * T) * np.mean(payoff)<br>    return put_price<br><br># Example usage<br>S = 100   # Initial stock price<br>K = 90   # Strike price<br>T = 1     # Time to maturity in years<br>r = 0.05  # Risk-free interest rate<br>sigma = 0.2  # Volatility<br>n = 10000  # Number of simulations<br><br>put_price = put_option_price(S, K, T, r, sigma, n)<br>print(f&quot;Put option price: {put_price:.2f}&quot;)</pre><p>This program is similar to the previous program, but it computes the payoff of a put option instead of a call option. The payoff of a put option is the difference between the strike price and the stock price at expiration, if positive, or zero otherwise. The rest of the program is the same as before: it generates a set of simulated stock prices using GBM, computes the payoffs of the put option, takes the average of the payoffs, and discounts them to the present value using the risk-free interest rate.</p><p>You can customize the input parameters and run this program with different values to explore the effect of changing the strike price, the time to maturity, the risk-free interest rate, the volatility, and the number of simulations on the put option price. You can also compare the results with other pricing methods, such as the Black-Scholes formula or the Binomial Model.</p><p>Now, it is time to stitch all the components mentioned above to form the tool in the google colab notebook which we will use to predict the <strong><em>call and put option price</em></strong> of one of the renowned stocks in the current Indian stock market i.e. <a href="https://www.tatapower.com/"><strong>Tata Power</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*JqcbMT0fGdxKhSnOnR8YMQ.jpeg" /></figure><p><a href="https://colab.research.google.com/drive/1TlOt2dP8mSzwUe4yy-fgtDypUwKZ4izR?usp=sharing">Google Colaboratory</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iLifdUi1JzXU3fuyEd2v1A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*440-0fnlw3sYouShsrGhSA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6j4ql4AV3JNPKZIM74egxw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*teW_ocmN1af-TvPx9urP-A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mgRbNDD7dpSMhM3-kQpVDw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HhdW_IVrSEP6z_fAB4A6bQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QzO7dgW2jQwoLCSM-8NthQ.png" /></figure><p>It is essential to highlight that be it any model/technology, the stock price prediction cannot always be accurate. The stock market is highly complex and influenced by numerous factors, many of which are unpredictable, such as global events, economic changes, political decisions, natural disasters, and technological advancements. These factors can cause sudden and significant changes in stock prices, making it difficult to predict the future performance of a stock with certainty.</p><p>Moreover, stock price prediction models, such as those based on technical analysis or machine learning, are inherently probabilistic and rely on assumptions and historical data to make forecasts. These models are not foolproof and can generate errors, biases, and uncertainties. For example, they may fail to capture new information or trends, overfit to the past data, or suffer from data quality issues or model misspecification.</p><p>Therefore, while stock price prediction can provide valuable insights and guidance for investment decisions, it should be taken with caution and scepticism, and should not be relied upon as the sole basis for investment strategies. It is important to complement stock price prediction with other sources of information, such as fundamental analysis, market research, and risk management techniques, and to diversify investments across different stocks, sectors, and asset classes to minimize risks and maximize returns.</p><p>Happy Reading, Happy Learning!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2268594389b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Guided Tour of gRPC with Protocol Buffers]]></title>
            <link>https://blog.devgenius.io/a-guided-tour-of-grpc-with-protocol-buffers-a8400b1ab361?source=rss-600f37eccd68------2</link>
            <guid isPermaLink="false">https://medium.com/p/a8400b1ab361</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[protocol-buffers]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[grpc]]></category>
            <dc:creator><![CDATA[Ronik Basak]]></dc:creator>
            <pubDate>Thu, 28 Apr 2022 12:08:56 GMT</pubDate>
            <atom:updated>2022-04-29T12:47:38.635Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction to gRPC and Protobuf:</h3><p>gRPC is a modern open-source high-performance Remote Procedure Call (RPC) framework initially developed at Google in 2015. gRPC uses HTTP 2.0 as its underlying transport protocol.</p><p>Protocol Buffer a.k.a Protobuf is the most widely used Interface Definition Language as well as the message interchange format for gRPC where we conventionally store the data and function contracts in the form of a proto file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*9fzPTvrTtsJhcpP8M-Uy-A.png" /><figcaption>source: <a href="https://golangexample.com/">https://golangexample.com/</a></figcaption></figure><p>Proto compiler(protoc) generates the necessary code using the serialization format for the data defined in the .proto files along with the language-specific runtime libraries to let the system behave as an interface to expose the procedures(like any other RPC model) to be called by the external systems.</p><p>Protocol buffers provide a language-neutral, platform-neutral, extensible mechanism for serializing structured data in both forward-compatible and backward-compatible ways.</p><p>One important fact we should keep in mind before designing our solution with gRPC and Protobuf is that: Protocol buffers tend to assume that entire messages can be loaded into memory at once and are not larger than an object graph. While working with larger data(greater than a few megabytes in size), Protobuf can cause surprising spikes in memory usage due to multiple serialized copies of the data.</p><p>Okay! Enough of theory. Let us dive into implementing one small hands-on project using gRPC and Protocol Buffers.</p><h3>Set up guide:</h3><p>There are certain steps to be followed to set up the Protobuf in our local environment. I am using macOS Mojave 10.14.6 and I already had go1.17.1 installed on my machine. I used homebrew to install the Protocol Buffers and Go. The commands for that were:</p><blockquote>brew install go<br>brew install protobuf</blockquote><p>We should ensure that the PATH mappings are done correctly in the <em>.bash_profile </em>file so that <em>protoc</em> compiler can find the required plugins. I used the below four lines of code.</p><blockquote>export GOROOT=/usr/local/go<br>export GOPATH=$HOME/go<br>export GOBIN=$GOPATH/bin<br>export PATH=$PATH:$GOROOT:$GOPATH:$GOBIN</blockquote><p>And then, execute the below command to allow the above changes to reflect.</p><blockquote>source ~/.bash_profile</blockquote><p>Alternatively, we can also use the below command on terminal to set and export the new value of $PATH variable with GOPATH:</p><blockquote>export PATH=”$PATH:$(go env GOPATH)/bin”</blockquote><p>We need to install the main <em>grpc</em> package and the <em>protoc-gen-go-grpc</em> package using the below two commands respectively to use the protocol buffers with grpc.</p><blockquote>go install google.golang.org/protobuf/cmd/protoc-gen-go@latest<br>go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest</blockquote><h3>Hands-on:: Order Billing Service:</h3><p><strong><em>Problem Statement</em></strong>: Build a service which can be used to perform the calculations to derive the total amount for an order considering the below conditions regarding the delivery-fee:</p><blockquote><em>If the order is a “</em><strong><em>FOOD</em></strong><em>” order, we should consider the delivery-fee as 25.0 INR</em></blockquote><blockquote><em>If the order is a “GROCERY” order, we should consider the delivery-fee as 50.0 INR</em></blockquote><blockquote><em>If the order is a “FASHION” order, we should consider the delivery-fee as 15.0 INR</em></blockquote><blockquote><em>And, for all other types of orders, we should consider the delivery-fee as 10.0 INR</em></blockquote><p>We also need to follow a constraint where we should restrict the system to prepare the bill with orderTotal amount until the order is delivered to the customer.</p><p>The use-case mentioned in the above problem statement is completely superficial. I am considering it as a simple business use-case, so that it becomes easy to explain.</p><p><strong>Solution</strong>: I have created a folder called <em>/go-orders </em>to hold the<em> </em>implementation of the codes needed to build this project. This folder is created in a place in my local system except under the GOROOT location (/usr/local/go).</p><p>Now, we will create the <em>protos</em> package within it and then a .proto file called orderBillingService.proto to declare the Request/Response data structures(message) and the declaration of the rpc function named CalculateOrderTotalForBilling.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sZ6_6831I7teUU2SNBI2JQ.png" /><figcaption>Snapshot of the implementation of <strong>orderBillingService.proto</strong></figcaption></figure><p>Please remember to use the below line in your proto file. Otherwise, you may face issues with compiling it. <em>go_package</em> variable should contain the full import path of the Go package which will hold the generated code for the .proto file.</p><blockquote>option go_package = “github.com/go-orders/protos”;</blockquote><p>Now, as we have finished writing the definition of our proto file(<strong>orderBillingService.proto</strong>), we can compile it to generate the <strong>.pb.go</strong> files using the below command:</p><blockquote><em>protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative protos/orderBillingService.proto</em></blockquote><p>Now, we can see that after the successful compilation of the orderBillingService.proto file two new <em>pb.go </em>files named <strong>orderBillingService_grpc.pb.go</strong> and <strong>orderBillingService.pb.go </strong>are automatically generated under the protos package.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/1*znfAZTDmqZc5jENsv87kWg.png" /><figcaption>Project Structure of the go-orders</figcaption></figure><p>Now, we will implement the code inside the <em>server </em>package to build the gRPC server. This gRPC server will be running on <strong><em>localhost:4040</em></strong><em> (as we have created a listener on that port) </em>to consume the requests sent by client(s). This will also contain the business logic of the rpc function declared in the above proto file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rWXM0RfrhC35l2mKn3N3eA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J2vpCYndzwWndhPNEdDzJA.png" /><figcaption>gRPC Server Implementation Code</figcaption></figure><p>Below is the screenshot of my terminal while developing this project. I am sharing it here because it may become useful to refer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Al7G8_csRgihz6Qbd9Fo4Q.png" /><figcaption>Snapshot of the list of commands being run in terminal</figcaption></figure><p>Now, we are all set to run our gRPC server using the command “<strong><em>go run server/main.go</em></strong>”. We will notice that our system will ask for the permission to allow accepting incoming network on <em>localhost:4040</em>. We should allow it so that client(s) can now call the gRPC function to get the necessary output as per our problem statement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*GQ01FIc1KJTr8wH4uZHMxQ.png" /><figcaption>Snapshot of the terminal when we run our <strong>gRPC Server</strong></figcaption></figure><p>Now, as our gRPC Server is running successfully. We will try to call the RPC function named <strong>CalculateOrderTotalForBilling</strong> by passing the required attributes mentioned in our data structure(message) named <strong>OrderBillingRequest </strong>in the proto file. And, we should also observe that we are getting the response from the gRPC server as per the data structure of <strong>OrderBillingResponse </strong>defined in our proto file.</p><p>I am going to use <a href="https://github.com/bloomrpc/bloomrpc">BloomRPC</a>, an open-source GUI Client for GRPC services to trigger the calls to the gRPC server running on localhost:4040 by passing two different values of <strong>OrderBillingRequest</strong>. We can see below that the corresponding responses are generated as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b7Rb0HZgEScRJ0LMo-FbtA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_C4n8eeTJARsVuRO4ZleVA.png" /><figcaption>BloomRPC Screenshot of the Request/Response where the <strong>OrderTotal</strong> is calculated for <strong>FOOD</strong> &amp; <strong>GROCERY</strong> order</figcaption></figure><p>As per the problem-statement, in the implementation of the gRPC function <strong>CalculateOrderTotalForBilling</strong>, we have kept a constraint where we are refusing to calculate the OrderTotal if the order is not <strong>DELIVERED</strong> yet, instead we are returning the default output 0 as orderTotal along with the failure message in the response.</p><p>Below is the example of the use-case where we have marked the <strong>isOrderDelivered</strong> flag as “<strong>false</strong>”, and we can see in the output that the <strong>orderTotal</strong> is zero and the message tells us the reason behind the failure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zdn8_EnhZvdyg0TyL71w3Q.png" /><figcaption>As the order was <strong>Undelivered</strong>, the system refused to calculate billing for this order</figcaption></figure><p>Now, let us see how we can call this gRPC server programmatically. We have declared a package called “<strong>client</strong>” and also defined a method called <strong>runCalculateOrderTotalForBilling</strong> which will eventually call the <strong>CalculateOrderTotalForBilling</strong> function of the gRPC server to fetch the result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-IYKerP3zRNyfvR5QSec3Q.png" /><figcaption>Definition of the <strong>runCalculateOrderTotalForBilling</strong> function in the client</figcaption></figure><p>This call to the gRPC server will happen via calling the <strong>NewOrderBillingServiceClient </strong>function generated during the compilation of the proto file. This function exists in the <strong>orderBillingService_grpc.pb.go</strong> file and receives an object of ClientConnInterface as input.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TbWj9rq03gnzC9XMlJl0Hw.png" /><figcaption>Snapshot of orderBillingService_grpc.pb.go where the <strong>NewOrderBillingServiceClient </strong>function is highlighted</figcaption></figure><p>We also implemented the logic in the main() function of the client to call the gRPC server for the below two sets of input parameters.</p><blockquote>runCalculateOrderTotalForBilling(client, 614352, 500.0, “FOOD”, true)</blockquote><blockquote>runCalculateOrderTotalForBilling(client, 987654, 1000.0, “GROCERY”, true)</blockquote><p>Now, let us run the client using the command “<strong><em>go run client/main.go</em></strong>”, and we can see that the output is generated and printed in order for the above two function calls.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IZwhiLHsX9yhniKkLaF_3w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WB2qKT1d5XZlHhHL0tVg6g.png" /><figcaption>Definition of the client/main.go file &amp; the output after running it</figcaption></figure><p>Okie Dokie!! Hope I was able to expound the process how we can build our first simple gRPC server where the Protocol Buffers are being used during the serialization/de-serialization of data between the client and the server.</p><p>Thanks for reading. Hope you enjoyed the journey with me. Please feel free to share your feedback on this article.</p><h3>References:</h3><ul><li><a href="https://developers.google.com/protocol-buffers/docs/overview">Overview | Protocol Buffers | Google Developers</a></li><li><a href="https://en.wikipedia.org/wiki/GRPC">gRPC - Wikipedia</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8400b1ab361" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/a-guided-tour-of-grpc-with-protocol-buffers-a8400b1ab361">A Guided Tour of gRPC with Protocol Buffers</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A cookbook to deal with throttling issues in Amazon DynamoDB]]></title>
            <link>https://levelup.gitconnected.com/a-cookbook-to-deal-with-throttling-issues-in-amazon-dynamodb-f953c4ea4785?source=rss-600f37eccd68------2</link>
            <guid isPermaLink="false">https://medium.com/p/f953c4ea4785</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[dynamodb]]></category>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[throttling]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Ronik Basak]]></dc:creator>
            <pubDate>Tue, 23 Jun 2020 10:30:28 GMT</pubDate>
            <atom:updated>2020-07-08T02:36:40.719Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>According to Amazon DynamoDB documentation, it is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability. DynamoDB lets us offload the administrative burdens of operating and scaling a distributed database so that we don’t have to worry about hardware provisioning, setup and configuration, replication, software patching, or cluster scaling.</blockquote><p>DynamoDB also offers encryption at rest, which eliminates the operational burden and complexity involved in protecting sensitive data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q1SqGhnaUzNmntqLJlR-Rg.jpeg" /><figcaption>Source: <a href="https://foxutech.com/how-to-create-a-dynamodb-table-on-aws/">https://foxutech.com/how-to-create-a-dynamodb-table-on-aws/</a></figcaption></figure><p>In a DynamoDB table, items are stored across many <em>partitions</em> according to each item’s partition key. Each partition has a share of the table’s provisioned <strong>RCU</strong> (read capacity units) and <strong>WCU</strong> (write capacity units).</p><p>When a request is made, it is routed to the correct partition for its data, and that partition’s capacity is used to determine if the request is allowed, or will be <em>throttled</em> (rejected). Some amount of throttling should be expected and handled by the application.</p><p><strong><em>Excessive throttling is caused by</em></strong>:</p><blockquote><strong>Hot partitions:</strong> throttles are caused by a few partitions in the table that receive more requests than the average partition</blockquote><blockquote><strong>Not enough capacity:</strong> throttles are caused by the table itself not having enough capacity to service requests on many partitions.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o8E6CvH9EFOnxL5IWge6XQ.png" /></figure><p><strong><em>Impact of write throttle</em></strong> — We might end-up missing out events which could lead to cascading failure in downstream systems.</p><p><strong><em>Impact of read throttle</em></strong> — We might end-up processing double/duplicate events if the previously processed record could not be retrieved.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*69iNlaxq6RmlbS8cBjNqNw.png" /></figure><p>Amazon DynamoDB is a NoSQL cloud database service that is designed to provide low-latency and high-throughput performance for applications and services running at scale. Example use cases include:</p><ul><li><em>Massively multiplayer online games (MMOGs)</em></li><li><em>Virtual and augmented reality</em></li><li><em>Checkout and order processing in e-commerce</em></li><li><em>Real-time stock pricing and trading</em></li></ul><p>When we operate such systems globally, we can occasionally experience latency spikes. These spikes can occur because of retries from transient network interruptions, service-side and network-side issues, or overloaded and slow clients.</p><blockquote>DynamoDB is designed to ensure that our provisioned capacity is available on a per-second basis. If we provision a table for ten 1kB reads per second then DynamoDB will give us enough capacity to handle that throughput rate. In addition, DynamoDB will sometimes allow us to achieve limited bursting above our provisioned throughput for a short period of time. This is intended to absorb natural variations in customer workloads. This bursting is not guaranteed and it is not always available (and the nature of the available bursting may change over time). As is currently described in the best practices documentation, in order to get the best performance we should have an evenly distributed workload that does not exceed our provisioned capacity and distributes the load evenly over the key space.</blockquote><p>The provisioned capacity and auto-scaling capacity can be tweaked from AWS console to fine tune the performance according to the need of the application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L8viB7r1d32G-Flsxayzbw.png" /></figure><p>Regardless of the root causes, an application that interacts with the DynamoDB service should be tuned to follow a <strong>retry strategy</strong> that helps avoid latency spikes.</p><p>Depending on the AWS SDK in use, the underlying HTTP client behaviour can be reconfigured from the default settings to help ensure that low-level client-server communications over HTTP honour the application’s latency requirements.</p><p>But in this blog, I am going to discuss about the control over the HTTP client behaviour and retry strategies which AWS Java SDK offers us.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2yleMfQAu85M-PrBTbgH7w.jpeg" /></figure><p>Below is the more specific configurations that I created to build a latency-aware DynamoDB application client. I build an asynchronous DynamoDB client in Java from scratch, and show how to use the ClientConfiguration implementation from the AWS SDK to define the application-specific latency requirements.</p><p>In this example, I create an asynchronous DynamoDB client that can make multiple sequential DynamoDB API calls to the service endpoint, without waiting for the responses to return before issuing the next API call. Because I want our DynamoDB application to be latency sensitive, an asynchronous client application is a good choice. It can prepare and process increasing numbers of API calls from different modules or micro-services to the backend, and thus decouple individual execution processes.</p><p>First, I have created a Java class called DynamoDBConfig with functions createDynamoDBClient() and createDynamoDBClientConfiguration().</p><p>The createDynamoDBClient() function returns an asynchronous DynamoDB client object that uses the low-level HTTP client configurations from a ClientConfiguration object that are returned by the createDynamoDBClientConfiguration() private API operation. As we can see in the following code example, five HTTP client configuration parameters are set during the ClientConfiguration object creation:</p><blockquote><strong><em>ConnectionTimeout</em></strong>: ConnectionTimeout is the maximum amount of time that the client waits for the underlying HTTP client in the SDK to establish a TCP connection with the DynamoDB endpoint. The connection is an end-to-end, two-way communication link between the client and server, and it is used and reused to make API calls and receive responses.</blockquote><blockquote><strong><em>ClientExecutionTimeout</em></strong>: ClientExecutionTimeout is the maximum allowed total time spent to perform an end-to-end operation and receive the desired response, including any retries that might occur. Essentially, this is the SLA of your DynamoDB operation — the timeframe for completion of all HTTP requests, including retries.</blockquote><blockquote><strong>RequestTimeout</strong>: RequestTimeout is the time it takes for the client to perform a single HTTP request. RequestTimeout is measured from the moment that a DynamoDB API call (such as PutItem or GetItem) is made until a response is received from the service. Logically, the value of this timeout should be less than the ClientExecutionTimeout. As with ClientExecutionTimeout, this setting is disabled by default.</blockquote><blockquote><strong><em>SocketTimeout: </em></strong>SocketTimeout defines the maximum amount of time that the HTTP client waits to read data from an already established TCP connection. This is the time between when an HTTP POST ends and the entire response of the request is received, and it includes the service and network round-trip times.</blockquote><blockquote><strong><em>DynamoDB default retry policy</em></strong>: The default retry policy available in the AWS Java SDK for DynamoDB is a good starting point to define client-side retry strategies for the underlying HTTP client. The default policy starts with a maximum of 10 retries with a predefined base delay of 25 milliseconds for any 5XX server-side exceptions (such as “HTTP status code — 500 Internal Server Error” or “HTTP status code — 503 Service Unavailable”), and 500 milliseconds for any 4XX client-side exceptions (such as “HTTP status code 400 — ProvisionedThroughputExceededException”). PredefinedBackoffStrategiesincludes two backoff strategies that are used in these retries. For the non-throttled 5XX requests, the FullJitterBackoffStrategy is picked up and uses the base delay of 25 milliseconds with a maximum delay of 20 seconds. For the throttled 4XX requests, EqualJitterBackoffStrategy is used. It starts with a 500-millisecond base delay, and can reach up to a maximum of 20 seconds, exponentially growing by 500 milliseconds, 1,000 milliseconds, 2,000 milliseconds, and so on, until reaching 20,000 milliseconds.</blockquote><p>I have assigned the values of above mentioned parameters by taking it as input through the config file (i.e. application.properties in my spring-boot application) and injecting their values using @Value annotation into corresponding variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*aXtCryQKuiQT8-r7ybuQ7g.png" /><figcaption>Assignment of Configuration Variables Required for DynamoDB Client Configuration</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VyBZx3bsnDGpgqcReSQVqg.jpeg" /><figcaption>Source code to create DynamoDBClient</figcaption></figure><ul><li><strong><em>DynamoDB Mapper</em></strong>: If we use the DynamoDBMapper class in our application, it internally initiates the client with the default ClientConfiguration options that I mentioned previously. Additionally, this class also uses its own retry mechanism, DefaultBatchWriteRetryStrategy, for the unprocessed items from a BatchWriteItem API call. It includes a one-second minimum delay and a maximum delay of three seconds, and an exponential backoff strategy. Therefore, the use of DynamoDBMapper class with its default settings can add unexpected extra latency to your requests. In other words, in order to maintain maximum control, you should consider using the low-level DynamoDB API operations first wherever possible, and then tweak the SDK-level settings to define the application’s behaviour in production.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ym-ks0sRNWZTFB2YoIz-NA.jpeg" /><figcaption>Source code to create DynamoDBMapper using Default Configuration</figcaption></figure><p>We can also overwrite the DefaultDynamoDBMapperConfig which can be passed as an argument along with while calling DynamoDBMapper methods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ozaKVPvtso9mN7av8WDhtg.png" /><figcaption>DynamoDBMapperConfig with customized RetryStrategy</figcaption></figure><p>Finally, if the default strategies do not address our use case, disable the default retry option with NO_RETRY_POLICY by specifying it with RetryPolicy (PredefinedRetryPolicies.NO_RETRY_POLICY) when creating the ClientConfiguration. We can also implement our own retry logic by extending the <em>V2CompatibleBackoffStrategyAdapter</em> class.</p><p>That’s all for now. See you later. :-)</p><h3>References:</h3><ul><li><a href="https://stackoverflow.com/questions/16980678/when-does-dynamodb-throttle-request">When does DynamoDB throttle request?</a></li><li><a href="https://aws.amazon.com/blogs/database/tuning-aws-java-sdk-http-request-settings-for-latency-aware-amazon-dynamodb-applications/">Tuning AWS Java SDK HTTP request settings for latency-aware Amazon DynamoDB applications | Amazon Web Services</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f953c4ea4785" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/a-cookbook-to-deal-with-throttling-issues-in-amazon-dynamodb-f953c4ea4785">A cookbook to deal with throttling issues in Amazon DynamoDB</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Druid — The sine qua non of contemporary Big-Data analytics]]></title>
            <link>https://medium.com/swlh/apache-druid-the-sine-qua-non-of-contemporary-big-data-analytics-cc82a081e?source=rss-600f37eccd68------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc82a081e</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[big-data-analytics]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[big-data-technology]]></category>
            <category><![CDATA[druid]]></category>
            <dc:creator><![CDATA[Ronik Basak]]></dc:creator>
            <pubDate>Tue, 24 Mar 2020 07:52:30 GMT</pubDate>
            <atom:updated>2020-03-24T14:15:04.198Z</atom:updated>
            <content:encoded><![CDATA[<h3>Apache Druid — The sine qua non of contemporary Big-Data analytics</h3><p>The term “big data” refers to the digital stores of data that have a high volume, velocity and variety. Big data analytics is the process of using software to uncover trends, patterns, correlations or other useful insights in those large stores of data. Apache Druid is substantially capable to prove itself as one of the backbone to build a platform for Big-Data Analytics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iSToRzvkbFaIEUyp-TO9lA.jpeg" /></figure><p>According to Wikipedia, <em>Druid is a column-oriented, open-source, distributed data store written in Java</em>. Druid is designed to quickly ingest massive quantities of event data, and provide low-latency queries on top of the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/1*H7LLNO-NRmo76Srgk_KjrA.png" /></figure><p>Druid’s core design combines ideas from <em>data warehouses, time-series databases, and search systems</em> to create a unified system for real-time analytics for a broad range of use cases.</p><p>Druid merges key characteristics of each of the 3 systems into its <em>ingestion layer, storage format, querying layer, and core architecture</em>.</p><h3>Key Features of Druid:</h3><blockquote><em>Column-Oriented Storage</em>:</blockquote><p>Druid stores and compresses each column individually, and only needs to read the ones needed for a particular query, which supports fast scans, rankings, and group-By.</p><blockquote>Native search indexes:</blockquote><p>One important aspect regarding Druid’s index implementation is, it uses inverted index. An inverted index is an index data structure storing a mapping from content, such as words or numbers, to its locations in a document or a set of documents. The purpose of an inverted index is to allow fast full-text searches, at a cost of increased processing when a document is added to the database.</p><p>There are <em>two types of inverted indexes</em>: A <strong>record-level inverted index</strong> contains a list of references to documents for each word. A <strong>word-level inverted index</strong> additionally contains the positions of each word within a document.</p><p>Druid creates inverted indexes for string values for faster search and filter operations.</p><blockquote>Streaming and batch ingest:</blockquote><p>Druid allows its users to utilise the working principles of Lambda Architecture. <strong>Lambda architecture</strong> is a data-processing architecture designed to handle massive quantities of data by taking advantage of both batch and stream-processing methods.</p><p>Druid provides connectors to Apache Kafka for real-time data ingestion through streams and HDFS, AWS S3 for batch ingestion. It also supports other streaming and batch pipeline for data ingestion.</p><blockquote>Time-optimised partitioning:</blockquote><p>Similar to time-series databases, Druid intelligently partitions data by time to enable fast time-oriented queries.</p><p>Unlike many traditional systems, Druid can optionally pre-aggregate data as it is ingested. This pre-aggregation step is known as rollup, and can lead to dramatic storage savings. <strong><em>Roll-up</em></strong> is a first-level aggregation operation over a selected set of columns that reduces the size of stored data. During roll-up activity, the input rows are grouped by the timestamp and dimension columns with sum aggregations on the metric columns.</p><p>When rollup is enabled, then any rows that have identical dimensions and timestamp to each other (after <em>queryGranularity </em>— based truncation) can be collapsed, or <em>rolled up</em>, into a single row in Druid.</p><blockquote>Flexible Columns and SQL Support:</blockquote><p>Druid gracefully handles evolving schemas and nested data. In the legacy dataSchema, the flattenSpec is located in dataSchema → parser → parseSpec → flattenSpec and is responsible for bridging the gap between potentially nested input data (such as JSON, Avro, etc) and Druid&#39;s flat data model.</p><p>In addition to its native JSON based language, Druid speaks SQL over either HTTP or JDBC.</p><h3><strong><em>Druid Components and Working Principles</em></strong>:</h3><p>Druid has a multi-process, distributed architecture that is designed to be cloud-friendly and easy to operate. Each Druid process type can be configured and scaled independently, giving you maximum flexibility over your cluster. This design also provides enhanced fault tolerance: an outage of one component will not immediately affect other components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*PThAOpct62vvRVxy_SuUFQ.png" /></figure><p>Druid has several process types, briefly described below:</p><blockquote><strong><em>Coordinator</em></strong> :</blockquote><p>Responsible to manage data availability on the cluster. The Druid Coordinator process is primarily responsible for segment management and distribution. More specifically, the Druid Coordinator process communicates to Historical processes to load or drop segments based on configurations.</p><p>The Druid Coordinator is responsible for loading new segments, dropping outdated segments, managing segment replication, and balancing segment load and Segment Compaction.</p><p>The Druid Coordinator maintains a connection to the Zookeeper cluster for current cluster information.</p><p>The Coordinator also maintains a connection to a database containing information about available segments and rules. Available segments are stored in a segment table and list all segments that should be loaded in the cluster. Rules are stored in a rule table and indicate how segments should be handled.</p><h4>Segment Compaction:</h4><p>During each run, the Druid Coordinator compacts data-segments by merging small segments or splitting a large one.</p><p>This is useful when segments are not optimised in terms of segment size which may degrade query performance.</p><p>The Coordinator first finds the segments to compact based on the segment search policy. Once some segments are found, it issues a <strong>compaction task</strong> to compact those segments.</p><p>Compaction tasks might fail due to the following reasons.</p><ul><li>If the input segments of a compaction task are removed or overshadowed before it starts, that compaction task fails immediately.</li><li>If a task of a higher priority acquires a <strong><em>time chunk lock</em></strong> for an interval overlapping with the interval of a compaction task, the compaction task fails.</li></ul><p>Once a compaction task fails, the Coordinator simply checks the segments in the interval of the failed task again, and issues another compaction task in the next run.</p><blockquote><strong><em>Overlord:</em></strong></blockquote><p>Responsible to control the assignment of data ingestion workloads. The Overlord process is responsible for accepting tasks, coordinating task distribution, creating locks around tasks, and returning statuses to callers.</p><p>Overlord can be configured to run in one of two modes — local or remote (local being default).</p><p>In local mode Overlord is also responsible for creating Peons for executing tasks.</p><p>When running the Overlord in local mode, all MiddleManager and Peon configurations must be provided as well. Local mode is typically used for simple workflows.</p><p>In remote mode, the Overlord and MiddleManager are run in separate processes and can run individually on a different server. This mode is recommended if we intend to use the <strong>indexing service</strong> as the single endpoint for all Druid indexing.</p><blockquote><strong>MiddleManager:</strong></blockquote><p>The MiddleManager process is a worker process that executes submitted tasks. Middle Managers forward tasks to Peons that run in separate JVMs. The reason we have separate JVMs for tasks is for resource and log isolation.</p><p>Each Peon is capable of running only one task at a time, however, a MiddleManager may have multiple Peons.</p><h4><strong>Indexing Service</strong>:</h4><p>The Apache Druid indexing service is a highly-available, distributed service that runs indexing related tasks.</p><p>Indexing tasks create (and sometimes destroy) Druid segments. The indexing service has a master/slave like architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6mVEBjDYV35yGmbOZkwWOw.png" /></figure><p>The indexing service is composed of three main components:</p><ul><li><em>a Peon component that can run a single task.</em></li><li><em>a Middle Manager component that manages Peons.</em></li><li><em>an Overlord component that manages task distribution to MiddleManagers. Overlords and MiddleManagers may run on the same process or across multiple processes while MiddleManagers and Peons always run on the same process.</em></li></ul><blockquote><strong>Broker:</strong></blockquote><p>Responsible to handle druid queries from external clients. The Broker is the process to route queries when we run Druid on a distributed cluster. It understands the metadata published to ZooKeeper about what segments exist on what processes and routes queries such that they hit the right processes. This process also merges the result sets from all of the individual processes together.</p><h4>Query Forwarding:</h4><p>Druid broker determines which processes to forward queries, the Broker process first builds a view of the world from information in Zookeeper. Zookeeper maintains information about Historical and streaming ingestion Peon processes and the segments they are serving.</p><p>For every datasource in Zookeeper, the Broker process builds a timeline of segments and the processes that serve them. When queries are received for a specific datasource and interval, the Broker process performs a lookup into the timeline associated with the query datasource for the query interval and retrieves the processes that contain data for the query. The Broker process then forwards down the query to the selected processes.</p><h4>Caching</h4><p>Broker processes employ a cache with an LRU cache invalidation strategy. The Broker cache stores per-segment results. The cache can be local to each Broker process or shared across multiple processes using an external distributed cache such as memcached.</p><p>Each time a broker process receives a query, it first maps the query to a set of segments. A subset of these segment results may already exist in the cache and the results can be directly pulled from the cache. For any segment results that do not exist in the cache, the broker process will forward the query to the Historical processes.</p><p>Once the Historical processes return their results, the Broker will store those results in the cache. Real-time segments are never cached and hence requests for real-time data will always be forwarded to real-time processes. Real-time data is perpetually changing and caching the results would be unreliable.</p><blockquote><strong>Historical:</strong></blockquote><p>Druid historical processes are responsible to store queryable data. Each Historical process maintains a constant connection to Zookeeper and watches a configurable set of Zookeeper paths for new segment information. Historical processes do not communicate directly with each other or with the Coordinator processes but instead rely on Zookeeper for coordination.</p><h4><strong>How it loads and serves segments:</strong></h4><p>The Coordinator process is responsible for assigning new segments to Historical processes. Assignment is done by creating an ephemeral Zookeeper entry under a load queue path associated with a Historical process.</p><p>When a Historical process notices a new load queue entry in its load queue path, it will first check a local disk directory (cache) for the information about segment. If no information about the segment exists in the cache, the Historical process will download metadata about the new segment to serve from Zookeeper. This metadata includes specifications about where the segment is located in deep storage and about how to decompress and process the segment. Once a Historical process completes processing a segment, the segment is announced in Zookeeper under a served segments path associated with the process. At this point, the segment is available for querying.</p><h4>Segment Cache and How it works:</h4><p>When a Historical process notices a new segment entry in its load queue path, the Historical process first checks a configurable cache directory on its local disk to see if the segment had been previously downloaded. If a local cache entry already exists, the Historical process will directly read the segment binary files from disk and load the segment.</p><p>The segment cache is also leveraged when a Historical process is first started. On startup, a Historical process will search through its cache directory and immediately load and serve all segments that are found. This feature allows Historical processes to be queried as soon they come online.</p><h3>Druid Ingestion Methods</h3><p>The table below lists Druid’s most common data ingestion methods, along with comparisons to help us choose the best one for our need. Each ingestion method supports its own set of source systems to pull from.</p><h4>Streaming</h4><p>The most recommended, and most popular, method of streaming ingestion is the Kafka Indexing Service that reads directly from Kafka. The Kinesis indexing service also works well if we prefer Amazon Kinesis.</p><p>This table compares the major available options:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mDB7KhX_mlJ548_4JRQoNA.png" /></figure><h4>Batch</h4><p>When doing batch loads from files, you should use one-time <a href="https://druid.apache.org/docs/latest/ingestion/tasks.html">tasks</a>, and you have three options: index_parallel (native batch; parallel), index_hadoop (Hadoop-based), or index (native batch; single-task).</p><p>In general, we recommend native batch whenever it meets your needs, since the setup is simpler (it does not depend on an external Hadoop cluster). However, there are still scenarios where Hadoop-based batch ingestion might be a better choice, for example when you already have a running Hadoop cluster and want to use the cluster resource of the existing cluster for batch ingestion.</p><p>This table compares the three available options:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cf8kkmCRE1JzF7PMa4tljA.png" /></figure><h3>Druid Ingestion Spec:</h3><p>No matter what ingestion method we use, data is loaded into Druid using either one-time tasks or ongoing “<strong><em>supervisors</em></strong>” (which run and supervises a set of tasks over time). In any case, part of the task or supervisor definition is an <em>ingestion spec</em>.</p><p>Ingestion specs consists of three main components:</p><ul><li>dataSchema, which configures the datasource name, primary timestamp, dimensions, metrics, and transforms and filters (if needed).</li><li>ioConfig, which tells Druid how to connect to the source system and how to parse data. For more information, see the documentation for each ingestion method.</li><li>tuningConfig, which controls various tuning parameters specific to each ingestion method.</li></ul><p>I have shared two types of Ingestion-Spec below where we are ingesting data into the same datasource using both real-time Kafka streaming and batch ingestion through a file from AWS S3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*dJ8phaI77SS8GYmm28CklA.png" /><figcaption>Ingestion-Spec sample for Realtime Druid Ingestion for Kafka Indexing Service</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2PqqRB07aXHu3wHB2Gv6Iw.png" /><figcaption>Ingestion-Spec for Native Batch Ingestion into Druid from a file stored on AWS S3</figcaption></figure><h3>Druid’s data model</h3><h4>Datasources</h4><p>Druid data is stored in datasources, which are similar to tables in a traditional RDBMS. Druid offers a unique data modelling system that bears similarity to both relational and time-series models.</p><h4>Primary timestamp</h4><p>Druid schemas must always include a primary timestamp. The primary timestamp is used for partitioning and sorting the data. Druid queries are able to rapidly identify and retrieve data corresponding to time ranges of the primary timestamp column. Druid is also able to use the primary timestamp column for time-based data management operations such as dropping time chunks, overwriting time chunks, and time-based retention rules.</p><p>The primary timestamp is parsed based on the timestampSpec. In addition, the granularitySpec controls other important operations that are based on the primary timestamp.</p><h4>Dimensions</h4><p>Dimensions are columns that are stored as-is and can be used for any purpose. We can group, filter, or apply aggregators to dimensions at query time in an ad-hoc manner.</p><p>If we run with rollup disabled, then the set of dimensions is simply treated like a set of columns to ingest, and behaves exactly as we would expect from a typical database that does not support a rollup feature.</p><p>Dimensions are configured through the dimensionsSpec.</p><h4>Metrics</h4><p>Metrics are columns that are stored in an aggregated form. They are most useful when rollup is enabled. Specifying a metric allows us to choose an aggregation function for Druid to apply to each row during ingestion.</p><p>Metrics are configured through the metricsSpec.</p><h4>Partitioning</h4><p>Optimal partitioning and sorting of segments within your datasources can have substantial impact on footprint and performance.</p><p>Druid datasources are always partitioned by time into <em>time chunks</em>, and each time chunk contains one or more segments. This partitioning happens for all ingestion methods, and is based on the segmentGranularity parameter of your ingestion spec&#39;s dataSchema.</p><p>The segments within a particular time chunk may also be partitioned further, using options that vary based on the ingestion type you have chosen. In general, doing this secondary partitioning using a particular dimension will improve locality, meaning that rows with the same value for that dimension are stored together and can be accessed quickly.</p><h4>Segments</h4><p>Apache Druid stores its index in <em>segment files</em>, which are partitioned by time. In a basic setup, one segment file is created for each time interval, where the time interval is configurable in the segmentGranularity parameter of the granularitySpec.</p><p>For Druid to operate well under heavy query load, it is important for the segment file size to be within the recommended range of 300MB-700MB. If our segment files are larger than this range, then we should consider either changing the granularity of the time interval or partitioning our data and tweaking the targetPartitionSize in our partitionsSpec (a good starting point for this parameter is 5 million rows).</p><h4><strong>Core Data Structure of Segment:</strong></h4><p>Here we describe the internal structure of segment files, which is essentially <em>columnar</em>: the data for each column is laid out in separate data structures. By storing each column separately, Druid can decrease query latency by scanning only those columns actually needed for a query.</p><p>There are three basic column types: the timestamp column, dimension columns, and metric columns, as illustrated in the image below:</p><p>The timestamp and metric columns are simple: behind the scenes each of these is an array of integer or floating point values compressed with LZ4. Once a query knows which rows it needs to select, it simply decompresses these, pulls out the relevant rows, and applies the desired aggregation operator. As with all columns, if a query doesn’t require a column, then that column’s data is just skipped over.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HjdmN0iUdq00J-_e3tqVUw.png" /><figcaption>Data-Structure of the implementation strategy behind a Druid Segment</figcaption></figure><p>Dimensions columns are different because they support filter and group-by operations, so each dimension requires the following three data structures:</p><ol><li>A dictionary that maps values (which are always treated as strings) to integer IDs,</li><li>A list of the column’s values, encoded using the dictionary in 1, and</li><li>For each distinct value in the column, a bitmap that indicates which rows contain that value.</li></ol><p>Why these three data structures? The dictionary simply maps string values to integer ids so that the values in (2) and (3) can be represented compactly. The bitmaps in (3) — also known as <em>inverted indexes</em> allow for quick filtering operations (specifically, bitmaps are convenient for quickly applying AND and OR operators). Finally, the list of values in (2) is needed for <em>group by</em> and <em>TopN</em> queries. In other words, queries that solely aggregate metrics based on filters do not need to touch the list of dimension values stored in (2).</p><h3>Druid Cluster Set-Up:</h3><p>Apache Druid is designed to be deployed as a scalable, fault-tolerant cluster. Druid processes can be deployed any way we like, but for ease of deployment we suggest organising them into three server types: <strong>Master</strong>, <strong>Query</strong>, and <strong>Data</strong>.</p><ul><li><strong><em>Master</em></strong>: Runs Coordinator and Overlord processes, manages data availability and ingestion.</li><li><strong><em>Query</em></strong>: Runs Broker and optional Router processes, handles queries from external clients.</li><li><strong><em>Data</em></strong>: Runs Historical and MiddleManager processes, executes ingestion workloads and stores all queryable data.</li></ul><h3>Druid Architecture:</h3><p>Druid has a micro-service-based architecture can be thought of as a disassembled database. Each core service in Druid (ingestion, querying, and coordination) can be separately or jointly deployed on commodity hardware.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7EMQ2YDof0cH5LIct4cDPA.png" /></figure><p>Apache Druid is a real-time analytics database designed for fast slice-and-dice analytics (“OLAP” queries) on large data sets. Druid is most often used as a database for powering use cases where real-time ingest, fast query performance, and high uptime are important. As such, Druid is commonly used for powering GUIs of analytical applications, or as a backend for highly-concurrent APIs that need fast aggregations. Druid works best with event-oriented data.</p><p>Common application areas for Druid include:</p><ul><li><em>Clickstream analytics (web and mobile analytics)</em></li><li><em>Network telemetry analytics (network performance monitoring)</em></li><li><em>Server metrics storage</em></li><li><em>Supply chain analytics (manufacturing metrics)</em></li><li><em>Application performance metrics</em></li><li><em>Digital marketing/advertising analytics</em></li><li><em>Business intelligence / OLAP</em></li></ul><h3>Use-Cases where Druid can be a good fit:</h3><p>Druid is likely a good choice if our use case fits a few of the following descriptors:</p><ul><li><em>Insert rates are very high, but updates are less common.</em></li><li><em>Most of our queries are aggregation and reporting queries (“group by” queries). We may also have searching and scanning queries.</em></li><li><em>We are targeting query latencies of 100ms to a few seconds.</em></li><li><em>Our data has a time component (Druid includes optimisations and design choices specifically related to time).</em></li><li><em>We may have more than one table, but each query hits just one big distributed table. Queries may potentially hit more than one smaller “lookup” table.</em></li><li><em>We have high cardinality data columns (e.g. URLs, user IDs) and need fast counting and ranking over them.</em></li><li><em>We want to load data from Kafka, HDFS, flat files, or object storage like Amazon S3.</em></li></ul><h3>References:</h3><p>Apache Druid has a very elaborated documentation already on its architecture and working principles. I have experience in using Druid as a backbone for three of our Analytical Products in Data Platform team of my current organisation. Based upon my experience and need, i have tried to jot down all the relevant details in the above blog.</p><p><a href="https://druid.apache.org/docs/latest/design/index.html">Introduction to Apache Druid · Apache Druid</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc82a081e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/apache-druid-the-sine-qua-non-of-contemporary-big-data-analytics-cc82a081e">Apache Druid — The sine qua non of contemporary Big-Data analytics</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The rudiments of Obsessive Compulsive Disorder(OCD)]]></title>
            <link>https://medium.com/@ronikbasak/the-rudiments-of-obsessive-compulsive-disorder-ocd-6d461a2ed1b8?source=rss-600f37eccd68------2</link>
            <guid isPermaLink="false">https://medium.com/p/6d461a2ed1b8</guid>
            <category><![CDATA[psychiatry]]></category>
            <category><![CDATA[psychology]]></category>
            <category><![CDATA[mental-health]]></category>
            <category><![CDATA[ocd]]></category>
            <category><![CDATA[obsessive-compulsive]]></category>
            <dc:creator><![CDATA[Ronik Basak]]></dc:creator>
            <pubDate>Wed, 17 Jul 2019 20:47:38 GMT</pubDate>
            <atom:updated>2020-12-24T08:15:11.431Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*PfAHqO49kPWFqW9c11pAQA.png" /></figure><h3>Definition of OCD:</h3><p>According to the definition of <strong>American Psychiatric Association</strong>,</p><blockquote><strong>Obsessive-compulsive disorder</strong> (OCD) is an anxiety disorder in which time people have recurring, unwanted thoughts, ideas or sensations (obsessions) that make them feel driven to do something repetitively (compulsions). The repetitive behaviours, such as hand washing, checking on things or cleaning, can significantly interfere with a person’s daily activities and social interactions.</blockquote><p>According to the definition of <strong>International OCD Foundation</strong>,</p><blockquote><strong>Obsessive compulsive disorder</strong> (OCD) is a mental health disorder that affects people of all ages and walks of life, and occurs when a person gets caught in a cycle of <a href="https://iocdf.org/about-ocd/#OC">obsessions and compulsions</a>. Obsessions are unwanted, intrusive thoughts, images, or urges that trigger intensely distressing feelings. Compulsions are behaviours an individual engages in to attempt to get rid of the obsessions and/or decrease his or her distress.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9KFe2VN_gfcjXYRgyX644g.png" /></figure><p>Research suggests that OCD involves problems in communication between the front part of the brain and deeper structures of the brain. These brain structures use a neurotransmitter (basically, a chemical messenger) called serotonin.</p><p>OCD can be so debilitating and disabling that the <a href="https://www.ocduk.org/ocd/world-health-organisation/">World Health Organisation (WHO)</a> ranked OCD in the top ten of the most disabling illnesses of any kind, in terms of lost earnings and diminished quality of life. The same report also said OCD was the fifth leading causes of disease burden for women aged 15–44 in the developed world.</p><h3>OCD Cycle:</h3><p>The <a href="https://www.louisvilleocdclinic.com/index.php">Louisville OCD Clinic</a> has pictorially represented the vicious cycle of conversion of mental states of an OCD impacted person.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zSXZ5rTa-uTGMsbVY-CL-g.png" /><figcaption>Source: <a href="https://www.louisvilleocdclinic.com/worry-doubt-ocd.php">www.louisvilleocdclinic.com</a></figcaption></figure><h3>Symptoms of OCD:</h3><p>The <a href="https://www.betterhealth.vic.gov.au/health/conditionsandtreatments/obsessive-compulsive-disorder">BetterHealth channel of Victoria State Government</a> has precisely jotted down the symptoms and probable causes of OCD on their website. Let me mention few of them here on my blog for your reference.</p><h3>obsessions:</h3><p>Obsessions are usually exaggerated versions of concerns and worries that most people have at some time. Common obsessions include:</p><ul><li>fear of contamination from germs, dirt, poisons, and other physical and environmental substances</li><li>fear of harm from illness, accidents or death that may occur to oneself or to others. This may include an excessive sense of responsibility for preventing this harm</li><li>intrusive thoughts and images about sex, violence, accidents and other issues</li><li>excessive concern with symmetry, exactness and orderliness</li><li>excessive concerns about illness, religious issues or morality</li><li>needing to know and remember things.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JPFzMg7VR1qQwcR4EjrzCw.png" /></figure><p>Obsessions may be constantly on a person’s mind. They may also be triggered by physical objects, situations, smells or something heard on television, radio or in a conversation.</p><p>Obsessive fears usually move beyond a specific trigger — for example, anytime you were crossing a road, and suddenly a thought came up in your mind, which scares you in a way that if you do not stop walking now, it will result into the death of any of your near and dear ones.</p><p>Obsessions can change in nature and severity and do not respond to logic.</p><p>Obsessional anxiety leads to vigilance for possible threats, and a compelling need for certainty and control. Obsessions can produce feelings ranging from annoyance and discomfort to acute distress, disgust and panic.</p><h3>compulsions:</h3><p>Compulsions can be behavioural (actions) or mental (thoughts). Compulsions are repetitive actions that are often carried out in a special pattern or according to specific rules.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*puvQPXFc20Sohiag8i_kpQ.png" /></figure><p>Compulsions are usually performed to try and prevent an obsessive fear from happening, to reduce the anxiety which the obsessive thought creates, or to make things feel ‘<em>just right</em>’.</p><p>To be precise, Compulsions are repetitive behaviours or mental acts that a person feels driven to perform in response to an obsession.</p><p>Common compulsions include:</p><ul><li>excessive hand washing, showering and tooth brushing</li><li>excessive cleaning and washing of house, household items, food, car and other areas</li><li>excessive checking of locks, electrical and gas appliances, and other things associated with safety</li><li>repeating routine activities and actions such as reading, writing, walking, picking up something or opening a door</li><li>applying rigid rules and patterns to the placement of objects, furniture, books, clothes and other items</li><li>touching, tapping or moving in a particular way or a certain number of times</li><li>needing to constantly ask questions or confess to seek reassurance</li><li>mentally repeating words or numbers a certain number of times, or concentrating on ‘good’ or ‘safe’ numbers</li><li>replacing a ‘bad thought’ with a ‘good thought’.</li></ul><p>Usually compulsions become like rituals; they follow specific rules and patterns, and involve constant repetitions. Compulsions give an illusory sense of short-term relief to anxiety. However, they actually reinforce anxiety and make the obsessions seem more real, so that the anxiety soon returns.</p><h3>Cause of OCD:</h3><p>The causes of OCD are not fully understood yet. There are several theories about the causes of OCD, including:</p><ul><li>Compulsions are learned behaviours, which become repetitive and habitual when they are associated with relief from anxiety.</li><li>OCD is due to genetic and hereditary factors.</li><li>Chemical, structural and functional abnormalities in the brain are the cause.</li><li>Distorted beliefs reinforce and maintain symptoms associated with OCD.</li></ul><p>It is possible that several factors interact to trigger the development of OCD. The underlying causes may be further influenced by stressful life events, hormonal changes and personality traits.</p><h3>Types of OCD:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uUFkUAF1KFVwkE-Pzzpa4w.png" /><figcaption>Source: <a href="http://www.verywellmind.com/">www.verywellmind.com</a></figcaption></figure><h3>Treatments:</h3><blockquote><strong>Cognitive Behavioural Therapy:</strong></blockquote><p><a href="https://www.betterhealth.vic.gov.au/health/conditionsandtreatments/cognitive-behaviour-therapy"><strong>Cognitive behavioural therapy</strong></a> aims to change patterns of thinking, beliefs and behaviours that may trigger anxiety and obsessive compulsive symptoms. This therapy uses education to promote control over symptoms. The education includes information that helps to expose myths about the causes of OCD.</p><p>Part of the therapy involves gradually exposing the person to situations that trigger their obsessions and, at the same time, helping them to reduce their compulsions and avoidance behaviours. This process is gradual and usually begins with less feared situations. The exposure tasks and prevention of compulsions are repeated daily and consistently until anxiety decreases. Over time, this allows the person to rebuild trust in their capacity to manage and function, even with anxiety. This process is known as exposure and response prevention (ERP).</p><p>Cognitive behaviour therapy should be undertaken with a skilled, specialist mental health professional. Over-use of alcohol, drugs and some medications may interfere with the success of this type of treatment.</p><p>Three major components can be mentioned which can help improve a OCD-impacted person’s mental health:</p><p><strong><em>Time Schedule</em></strong>: The person should create a timetable of his daily activities and should start to implement that in his lifestyle in a strict manner.</p><p><strong><em>Time Rationality</em></strong>: As the logical thinking of a person is highly influenced by OCD, most of the times the impacted person becomes unable to figure out the rationality of time taken by him for a specific daily activity like brushing, taking bath etc.</p><p>Hence, the above mentioned timetable should be verified by others to enforce that the routine is prepared following logical timeline of a healthy lifestyle as well as it is getting enforced properly.</p><p><strong><em>Delayed Response</em></strong>: The person should start a practice to enforce a delay on the compulsive response. e.g. If some obsession inside a person is generating a compulsion within him to wash his hands frequently, he might intentionally delay in doing the same rather than doing it immediately. This habit can earn certain level of confidence inside him to overcome the vicious impact of OCD.</p><blockquote><strong>Anxiety management techniques for OCD:</strong></blockquote><p>Anxiety management techniques can help a person to manage their own symptoms. Such techniques can include relaxation training, <a href="https://www.betterhealth.vic.gov.au/health/healthyliving/breathing-to-reduce-stress"><strong>slow breathing techniques</strong></a>, mindfulness <a href="https://www.betterhealth.vic.gov.au/health/conditionsandtreatments/meditation"><strong>meditation</strong></a> and hyperventilation control. These techniques require regular practice and are most effective if used together with a cognitive behaviour therapy treatment program.</p><blockquote><strong>Medication with Fluoxetine:</strong></blockquote><p><strong>Fluoxetine</strong>, sold under the brand names <strong>Prozac</strong> and <strong>Sarafem</strong> among others, is an <a href="https://en.wikipedia.org/wiki/Antidepressant">antidepressant</a> of the <a href="https://en.wikipedia.org/wiki/Selective_serotonin_reuptake_inhibitor">selective serotonin reuptake inhibitor</a> (SSRI) class. It is used for the treatment of <a href="https://en.wikipedia.org/wiki/Major_depressive_disorder">major depressive disorder</a>, <a href="https://en.wikipedia.org/wiki/Obsessive%E2%80%93compulsive_disorder">obsessive–compulsive disorder</a> (OCD), <a href="https://en.wikipedia.org/wiki/Bulimia_nervosa">bulimia nervosa</a>, <a href="https://en.wikipedia.org/wiki/Panic_disorder">panic disorder</a>, and <a href="https://en.wikipedia.org/wiki/Premenstrual_dysphoric_disorder">premenstrual dysphoric disorder</a>.</p><h3><strong>Usefulness of Fluoxetine on Obsessive-Compulsive Disorder (Statistically)</strong>:</h3><p>The <a href="https://en.wikipedia.org/wiki/Efficacy">efficacy</a> of fluoxetine in the treatment of <a href="https://en.wikipedia.org/wiki/Obsessive%E2%80%93compulsive_disorder">obsessive–compulsive disorder</a> (OCD) was demonstrated in two randomized multicenter <a href="https://en.wikipedia.org/wiki/Clinical_trial#Phases">phase III clinical trials</a>.</p><p>The pooled results of these trials demonstrated that 47% of course-finishers treated with the highest dose were “much improved” or “very much improved” after 13 weeks of treatment, compared to 11% in the placebo arm of the trial.</p><p>The <a href="https://en.wikipedia.org/wiki/American_Academy_of_Child_and_Adolescent_Psychiatry">American Academy of Child and Adolescent Psychiatry</a> stated that <a href="https://en.wikipedia.org/wiki/SSRI">SSRIs</a>, including fluoxetine, should be used as first-line therapy in children, along with <a href="https://en.wikipedia.org/wiki/Cognitive_behavioral_therapy">cognitive behavioural therapy</a> (CBT), for the treatment of moderate to severe OCD.</p><h3>International OCD Awareness Week:</h3><p><strong>OCD Awareness</strong> Week is an international effort taking place during the second week in October each year to raise <strong>awareness</strong> and understanding about <strong>obsessive compulsive</strong> disorder and related disorders, with the goal of helping more people to get timely access to appropriate and effective treatment.</p><h3>References:</h3><ul><li><a href="https://www.psychiatry.org/patients-families/ocd/what-is-obsessive-compulsive-disorder">What Is Obsessive-Compulsive Disorder?</a></li><li><a href="https://iocdf.org/about-ocd/">What is OCD? | International OCD Foundation</a></li><li><a href="https://en.wikipedia.org/wiki/Fluoxetine">Fluoxetine - Wikipedia</a></li><li><a href="https://designtaxi.com/news/376013/Watch-A-Cute-Easy-To-Understand-Animation-About-OCD-Misconceptions/">Watch: A Cute, Easy-To-Understand Animation About OCD Misconceptions - DesignTAXI.com</a></li><li><a href="https://www.betterhealth.vic.gov.au/health/conditionsandtreatments/obsessive-compulsive-disorder">Obsessive compulsive disorder</a></li><li><a href="https://www.verywellmind.com/what-are-the-different-types-of-ocd-2510663">The 5 Types of Obsessive-Compulsive Disorder</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d461a2ed1b8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CompletableFuture : A new era of asynchronous programming]]></title>
            <link>https://levelup.gitconnected.com/completablefuture-a-new-era-of-asynchronous-programming-86c2fe23e246?source=rss-600f37eccd68------2</link>
            <guid isPermaLink="false">https://medium.com/p/86c2fe23e246</guid>
            <category><![CDATA[java8]]></category>
            <category><![CDATA[completablefuture]]></category>
            <category><![CDATA[asynchronous]]></category>
            <category><![CDATA[asynchronous-programming]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Ronik Basak]]></dc:creator>
            <pubDate>Thu, 04 Jul 2019 08:36:12 GMT</pubDate>
            <atom:updated>2020-06-24T21:27:46.050Z</atom:updated>
            <content:encoded><![CDATA[<h3>CompletableFuture : A new era of asynchronous programming</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*YyPo83ZRL0eklClnLcZXXA.jpeg" /></figure><p><em>Asynchronous programming</em> is a form of parallel programming that allows a unit of work to run separately from the primary application thread. When the work is complete, it notifies the main thread (as well as whether the work was completed or failed).</p><p>On the other hand, In a <em>synchronous</em> programming model, things happen one at a time. When we call a function that performs a long-running action, it blocks our program for the time that action takes and returns as well as unblocks the primary application thread only when the action has finished and it can return the result.</p><p>Waiting for actions to finish is <em>implicit</em> in the synchronous model, while it is <em>explicit</em>, under our control, in the asynchronous one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7eWf_rkKbBHiXFXnQFnvbA.png" /></figure><p>There are different ways to implement asynchronous programming in Java using any of the below mechanisms e.g. Futures, ExecutorService, CompletionService, Callback interfaces, ThreadPools, etc.</p><p>But Java 8 has introduced a new feature called <strong>CompletableFuture </strong>which can be extensively used in case of implementation of asynchronous programming in Java.</p><h3><strong>What is CompletableFuture and how is it different from Future:</strong></h3><p>According to Oracle documentation, a CompletableFuture is a <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/Future.html">Future</a> that may be explicitly completed (setting its value and status), and may be used as a <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletionStage.html">CompletionStage</a>, supporting dependent functions and actions that trigger upon its completion.</p><p>When two or more threads attempt to <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html#complete-T-">complete</a>, <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html#completeExceptionally-java.lang.Throwable-">completeExceptionally</a>, or <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html#cancel-boolean-">cancel</a> a CompletableFuture, only one of them succeeds.</p><p>CompletableFuture class belongs to the java.util.concurrent package.</p><blockquote><strong>Signature</strong>:</blockquote><pre>public class <strong>CompletableFuture&lt;T&gt;</strong><br>extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html">Object</a><br>implements <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/Future.html">Future</a>&lt;T&gt;, <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletionStage.html">CompletionStage</a>&lt;T&gt;</pre><p>CompletableFuture implements Future and CompletionStage interfaces and provides a huge set of convenience methods for creating, chaining and combining multiple Futures. It also has a very comprehensive exception handling support. CompletableFuture overcomes below limitations of Future:</p><ul><li>Futures can not be explicitly completed even when it has encountered an exception scenario.</li><li>Future provides a get() method which blocks until the result is available. further action can not be performed on a Future’s result without blocking the primary application thread.</li><li>Asynchronous workflows can not be created by chaining multiple Futures together.</li><li>Futures which are running in parallel, can not be combined together.</li><li>Future API does not have any exception handling construct.</li></ul><h3><strong>Implementation of CompletableFuture</strong>:</h3><blockquote><strong>How to start</strong>:</blockquote><p>We can create an instance of CompletableFuture class with a no-arg constructor to represent some future result. The simplest forward step would be to hand it out to the consumers and complete it at some time in the future using the CompletableFuture.complete() method.</p><p>We can try to get the result of this CompletableFuture using CompletableFuture.get() method, the get() method blocks until the Future is complete.</p><blockquote><strong>CompletableFuture.runAsync()</strong> :</blockquote><p>If we want to run some background task asynchronously and <strong><em>do not want to return</em></strong> anything from that task, that is an ideal use-case of CompletableFuture.runAsync() method. It takes a Runnable object and returns CompletableFuture&lt;Void&gt;.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wGTk7VrdUl42dEuolCpDew.png" /></figure><p>The Runnable object can also be passed in the form of a lambda expression:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*31svExkFH_nlHjYiUEqNsA.png" /></figure><blockquote><strong>CompletableFuture.supplyAsync()</strong> :</blockquote><p>On contrary to the above use-case, if we want to run some background task asynchronously and <strong><em>want to return</em></strong> anything from that task, we should use CompletableFuture.supplyAsync(). It takes a <a href="https://docs.oracle.com/javase/8/docs/api/java/util/function/Supplier.html">Supplier&lt;T&gt;</a> and returns CompletableFuture&lt;T&gt; where T is the type of the value obtained by calling the given supplier.</p><p>A <a href="https://docs.oracle.com/javase/8/docs/api/java/util/function/Supplier.html">Supplier&lt;T&gt;</a> is a simple functional interface which represents a supplier of results. It has a single get() method where we can write our background task and return the result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YLVmP5b5UTO6da3NwFkq_g.png" /></figure><p>Now, we know that the runAsync() and supplyAsync() methods execute their tasks in a separate thread which was never explicitly created by us.</p><p>CompletableFuture executes these tasks in a thread obtained from the global <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ForkJoinPool.html#commonPool--">ForkJoinPool.commonPool()</a>.</p><p>But we can also create a <a href="https://www.callicoder.com/java-executor-service-and-thread-pool-tutorial/">Thread Pool</a> and pass it to runAsync() and supplyAsync() methods to let them execute their tasks in a thread obtained from our thread pool.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zxxXgi-VExv7ETMnE9epkQ.png" /></figure><h3><strong><em>Callback</em></strong>:</h3><p>Callbacks are the provision for transforming and acting on the result of a CompletableFuture. For building asynchronous systems, we should be able to attach a callback to the CompletableFuture which should automatically get triggered and executed when the Future completes its task.</p><p>That way, we won’t need to wait for the result of the future(non-blocking), and we can write the logic that needs to be executed after the completion of the Future inside our callback function.</p><p>We can attach a callback to the CompletableFuture using thenApply(), thenAccept() and thenRun() methods.</p><blockquote><strong>thenApply()</strong>:</blockquote><p>We can use thenApply() method to process and transform the result of a CompletableFuture when it arrives. It takes a <a href="https://docs.oracle.com/javase/8/docs/api/java/util/function/Function.html">Function&lt;T,R&gt;</a> as an argument. <a href="https://docs.oracle.com/javase/8/docs/api/java/util/function/Function.html">Function&lt;T,R&gt;</a> is a simple functional interface representing a function that accepts an argument of type T and produces a result of type R.</p><p>We can also write a sequence of transformations on the CompletableFuture by attaching a series of thenApply() callback methods. The result of one thenApply() method is passed to the next in the series.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MkbxtoFDER5x6G_m1qr3-w.png" /></figure><blockquote><strong>thenAccept()</strong>:</blockquote><p>If you do not want to return anything from the callback function and just want to run some piece of code after the completion of the CompletableFuture, then we can use thenAccept() method.</p><p>CompletableFuture.thenAccept() takes a <a href="https://docs.oracle.com/javase/8/docs/api/java/util/function/Consumer.html">Consumer&lt;T&gt;</a> and returns CompletableFuture&lt;Void&gt;. It has access to the result of the CompletableFuture on which it is attached.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-AYPe36j5M6AH_oTfZrk5A.png" /></figure><blockquote><strong>thenRun()</strong>:</blockquote><p>thenRun() method is also used as callback function which does not return anything but executes the logic written inside this block. thenAccept() and thenRun()methods are consumers and are often used as the last callback in the callback chain.</p><p>While thenAccept() has access to the result of the CompletableFuture on which it is attached, thenRun() doesn’t even have access to the Future’s result. It takes a Runnable and returns CompletableFuture&lt;Void&gt;.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZIxx9SDmUqlR-2en_YCIUw.png" /></figure><h3>Asynchronous Callback:</h3><p>In the above examples, the task inside thenApply() is executed in the same thread where the supplyAsync() task is executed, or in the main thread if the supplyAsync() task completes immediately. If we want to increase the parallelism of our program furthermore and want to run the callback task on a separate thread — asynchronous callbacks are the solution.</p><p>We can use thenApplyAsync() callback in place of thenApply(), then this task will be executed in a different thread obtained from ForkJoinPool.commonPool().</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rAPYxjFecabnHnNJnd1wbQ.png" /></figure><p>We could also pass an Executor to the thenApplyAsync() callback, then the task will be executed in a thread obtained from the Executor’s thread pool.</p><h3>Combining CompletableFutures together :</h3><blockquote><strong>Combine dependent completableFutures using thenCompose():</strong></blockquote><p>In earlier examples, the Supplier function passed to thenApply() callback would return a value. What if our use-case is to return a CompletableFuture instead. That is the primary use-case of thenCompose() method.</p><p><em>If our callback function returns a CompletableFuture, and we want a flattened result from the CompletableFuture chain, we need to use </em><em>thenCompose().</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AYG5p4VHBnkwM0JShSltrw.png" /></figure><blockquote><strong>Combine independent completableFutures using thenCombine()<em>:</em></strong></blockquote><p>While thenCompose() is used to combine two CompletableFutures where one future is dependent on the other, thenCombine() is used when we want two Futures to run independently and do something after both are complete. The callback function passed to thenCombine() will be called when both the CompletableFutures are complete.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LfiS6uc-b6ok4BuTn7MCaw.png" /></figure><blockquote><strong>Combine n number of completableFutures together</strong>:</blockquote><p>CompletableFuture.allOf() is used in scenarios when we have a List of independent completableFutures and we want to run in parallel and do something after all of them are complete.</p><p>CompletableFuture.anyOf() as the name suggests, returns a new CompletableFuture which is completed when any of the given CompletableFutures completes, with the same result.</p><h3>Exception handling in CompletableFuture :</h3><blockquote><strong>Handle exceptions using exceptionally() callback</strong> :</blockquote><p>The exceptionally() callback gives us a chance to recover from errors generated from the original CompletableFuture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zefUJbE9rZIp0tSpVq7bng.png" /></figure><blockquote><strong>Handle exceptions using the generic handle() method</strong> :</blockquote><p>The API also provides a more generic method — handle() to recover from exceptions. It is called irrespective of whether or not an exception occurs. If an exception occurs, the value of ex argument will be non null, otherwise the value of res argument is non null in a success scenario.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ChMeJR7k7eE-yRJkFIt8gw.png" /></figure><h3>Conclusion:</h3><p>In this article, I have tried to put some light on different use-cases of asynchronous programming and how these scenarios can be implemented using Java 8’s CompletableFuture. If you enjoyed this post, I would be very grateful if you’d help it spread. Thank you very much for reading!</p><h3><strong>References:</strong></h3><ul><li><a href="https://eloquentjavascript.net/11_async.html">Asynchronous Programming :: Eloquent JavaScript</a></li><li><a href="https://stackify.com/when-to-use-asynchronous-programming/">When to Use (and Not to Use) Asynchronous Programming</a></li><li><a href="https://technology.amis.nl/2009/02/19/asynchronous-processing-in-java-applications-leveraging-those-multi-cores/">Asynchronous processing in Java applications - leveraging those multi-cores - AMIS Oracle and Java Blog</a></li><li><a href="https://www.callicoder.com/java-8-completablefuture-tutorial/">Java CompletableFuture Tutorial with Examples</a></li><li><a href="https://www.baeldung.com/java-completablefuture">Guide To CompletableFuture | Baeldung</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=86c2fe23e246" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/completablefuture-a-new-era-of-asynchronous-programming-86c2fe23e246">CompletableFuture : A new era of asynchronous programming</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Artificial Neural Network and it’s contribution to Machine Learning — A beginner’s hand-book]]></title>
            <link>https://medium.com/good-audience/artificial-neural-networks-and-its-contribution-to-machine-learning-a-beginner-s-hand-book-ab7f4e7b230e?source=rss-600f37eccd68------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab7f4e7b230e</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-neural-network]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Ronik Basak]]></dc:creator>
            <pubDate>Sat, 15 Sep 2018 10:44:30 GMT</pubDate>
            <atom:updated>2018-09-16T15:45:49.366Z</atom:updated>
            <content:encoded><![CDATA[<p>Artificial Neural Networks(ANN) is a <em>computational nonlinear model</em> which is widely used in Machine Learning and is considered to be a prominent component of futuristic Artificial Intelligence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*su6pnh1fTyHd8a47oXFXSw.png" /></figure><p>The <em>neural</em> part of the name aptly suggests that these are <em>brain-inspired systems</em> which are intended to replicate the way humans learn through their biological nervous system.</p><p>Thus, such systems “<em>learn</em>” to perform tasks by considering examples(training data), generally without being programmed with any task-specific rules.</p><blockquote>It is composed of a large number of <em>highly interconnected</em> processing elements (neurones) working in unison with their remarkable ability to derive meaning from complicated or imprecise data to solve a variety of problems in the area of pattern recognition, trend detection, classification, prediction, optimisation, visualisation, associative memory and many more.</blockquote><p>A trained neural network can be thought of as an “<em>expert</em>” in that specific category of information it has been given to analyse. This expert can then be used to provide projections given new situations of interest and answer “<em>what if</em>” questions.</p><h3>ANN Architecture Overview:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*Zg55_95DWMmfRw-fBfZs_A.jpeg" /><figcaption>Sample ANN architecture</figcaption></figure><blockquote>ANN consists of multiple layers and often called as <em>MLP (Multi Layer Perceptron)</em>. This network architecture includes an <em>input layer</em>, one or more <em>hidden layer</em>(s) and the <em>output layer</em>.</blockquote><p>The hidden layers are also known as “<em>Distillation Layer</em>” as it extracts(distills) the important features / patterns from the input layer removing the redundant information and forwards those features to the next layer for further derivation.</p><h3>Working Principles:</h3><p>In a typical ANN, the input layer contains input nodes(neurons) that transmits information to the hidden layer(s). If there are multiple hidden layers, every layer accepts the pre-processed information from the previous layer, that information is then processed further with specified logic, an intermediate output is generated and forwarded to the corresponding neuron of next hidden layer. The final hidden layer transmits the data to the output layer.</p><p>Every input to a neuron has an <em>amplitude(weight),</em> an <a href="https://en.wikipedia.org/wiki/Activation_function">activation function</a> (<em>to convert a input signal of a node in a ANN to an output signal</em>), and one output. The weights of the inputs are also known as <em>Synapses</em>. These are the adjustable parameters that convert a neural network to a parameterised system.</p><blockquote>The weighted sum of the inputs produces the <em>activation signal</em> that is passed to the <em>activation function</em> to obtain one output from the neuron.</blockquote><p>The commonly used activation functions are linear, step, sigmoid, tanh, and rectified linear unit (ReLu) functions.</p><p><strong>Linear function:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/136/1*UaL09lAmVvmkI1nfSqFacg.png" /></figure><p><strong>Step function:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/183/0*Vx0yjLP7VgvPIe88." /></figure><p><strong>Logistic (Sigmoid) Function :</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/127/0*TzwfGagVeF43Kiks." /></figure><p><strong>Tanh Function — <em>Hyperbolic tangent</em></strong><em>:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/253/0*4ljHOiE3Z60ivC5G." /></figure><p><strong>Rectified linear units (ReLu) function :</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/0*D2lOXl28ZoL52blt." /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/1*Vip0qXplMUEeYRr6T5--UQ.png" /></figure><p><strong>Leaky ReLu function</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*UF_1DKO-gk76KtF_Ia0bZw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*9wwi7CAXFWhZTzIyCvefeg.png" /></figure><blockquote><strong>ReLu</strong> has become the most popular activation function among them. Because it has been able to rectify the <a href="https://en.wikipedia.org/wiki/Vanishing_gradient_problem"><em>Vanishing Gradient problem</em></a> which was a challenge with <em>Sigmoid</em> and <em>Hyperbolic tangent </em>function. But ReLu should only be applied to the hidden layers. And if the model suffers form dead neurons during training using ReLu, we may use <strong>leaky ReLu</strong> function in that case.</blockquote><p>The weights(synapses) are optimized during the training process.</p><blockquote>Below is a flow diagram which simplifies the entire workflow of an Artificial Neural Network where we are trying to build a model which derives the <em>probability of rainfall</em> at a particular geographic location based upon it’s <em>humidity, atmospheric pressure</em> and <em>temperature</em>.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4W5D7O42ev9CvO_JdRbUew.png" /></figure><p>It is possible to make an ANN system more flexible and more powerful by using additional hidden layers. Artificial neural networks with multiple hidden layers between the input and output layers are called <a href="https://en.wikipedia.org/wiki/Deep_learning#Deep_neural_networks">deep neural networks (DNNs)</a>, and they can model complex nonlinear relationships.</p><h3>How an ANN learns:</h3><p>The learning process of a typical Artificial Neural Network consists of the below steps:</p><ol><li><strong>Random initialization</strong> of the weights / model</li><li>Picking appropriate <strong>activation function</strong></li><li><strong>Feed forward</strong> : We pass the input data through the network layer (input -&gt; hidden -&gt; output) and calculate the actual output of the model.</li><li><strong>Calculate Loss Function</strong>: Loss function is nothing but the absolute error on the performance considering the difference between the desired output and the actual output of our model.</li></ol><blockquote>loss = Absolute value of (desired — actual)</blockquote><p>5. Our ultimate objective would be to <strong>minimize the loss function</strong> by optimizing the weights. Generally, we use derivative of the loss function (gradient) to take decision on weight-update. In mathematics, the derivative of a function at a certain point, gives the rate or the speed of which this function is changing its values at this point. Here,</p><blockquote>we check the derivative.<br>- If it is positive, meaning the error increases if we increase the weights, then we should decrease the weight.<br>- If it’s negative, meaning the error decreases if we increase the weights, then we should increase the weight.<br>- If it’s 0, we do nothing, we reach our stable point.</blockquote><p>6. <strong>Back-propagation</strong>: The error contribution of each neuron is calculated starting from the gradient of the loss function, and the same is back-propagated to the previous layer(s) and so forth step-by-step based upon the derivative of the function used in every particular layer. The technique of <a href="https://en.wikipedia.org/wiki/Automatic_differentiation">auto-differentiation</a> can be used here.</p><p>7. Update and optimize the weights based upon the gradient calculations so far. We should use a small constant as “learning rate” while optimizing the weights.</p><p>8. Iterate the same steps from 1–7 till we achieve the specified accuracy i.e. till the model converges(Minimized loss function).</p><h3>Benefits of putting an ANN into action?</h3><p>A few advantages of choosing ANN over other ML paradigms are:</p><ol><li><strong>Non Linear</strong>: ANNs have the ability to learn and model non-linear and complex relationships, which is really important because in real-life, many of the relationships between inputs and outputs are non-linear as well as complex.</li><li><strong>Adaptive learning</strong>: An ability to learn how to do tasks based on the data given for training or initial experience.</li><li><strong>Self-Organisation</strong>: An ANN can create its own organisation or representation of the information it receives during learning time. It can infer unseen relationships on unseen data as well, thus making the model “generalize” and predict on unseen data.</li><li><strong>Having a distributed memory</strong>: The information used by the ANN are distributed throughout the network.</li><li><strong>Interpolating</strong>: Because of the continuous nature of the ANN, unknown input patterns are processed according to the curve fitted through the trained examples.</li><li><strong>Real Time Operation</strong>: ANN works in massively parallel distributed structure and computations can be carried out in <strong>parallel.</strong></li><li><strong>Fault Tolerance</strong>: A small amount of signal corruption or a partial destruction of a network has only a limited influence on the quality of the output, because of the distributed information storage and real time computation.</li><li><strong>Support Heteroskedasticity</strong>: Studies have shown that ANNs can better model heteroskedasticity i.e. data with high volatility and non-constant variance, given its ability to learn hidden relationships in the data without imposing any fixed relationships in the data. This is something very useful in regression analysis like in financial time series forecasting (e.g. stock prices) where data volatility is very high.</li></ol><h3>ANN can be suitable if:</h3><p>Where ANN is used, we try to achieve the specified level of accuracy using Machine Learning by back propagating and trying to mimic the known results over and over and over, by adjusting the weights until we come close to the known results.</p><ol><li>An ANN can be best fitted if you have a <em>large set</em> (tens of thousands of cases, or more) of data for training and testing, including all of the possible inputs along with the corresponding correct (desired) outputs.</li><li>If the input data correlates to the output data in a way that is somewhat <em>linearly separable</em> (is directly or indirectly related to its impact on the output) or that “can be made to be linearly separable” (i.e. via kernel trick) then an ANN may be called for.</li></ol><p>The trick is to know the domain of competence (or sweet spot) for every family of machine learning algorithms. It is not always mandatory that we will build a model using only one ML algorithm paradigm. We may include multiple models to design the final ML solution for a particular problem.</p><h3>Conclusion:</h3><p>In this blog, I tried to introduce the readers to the high-level architecture and workflow of a basic artificial neural network. There are already plenty of study materials and research papers available online on the same topic. I just tried to jot down the fundamental theoretical aspects which can help a beginner to understand how ANN works and when to choose an ANN in the field of machine learning.</p><p>In my next blogs, I will try to illustrate over ANN using further example and to discuss about the classifications and implementations of multiple ANN algorithms.</p><h3>References:</h3><ul><li><a href="http://www.thepinsta.com/happy-cartoon-nerve-cells_wXTe48qLEGjDoTwo0v%7CkbuVE96D*1edMCQlq%7CKmlojU/">Happy Cartoon Nerve Cells Pictures to Pin on Pinterest - ThePinsta</a></li><li><a href="https://www.quora.com/What-is-the-difference-between-CNN-and-RNN">What is the difference between CNN and RNN?</a></li><li><a href="https://medium.com/datathings/neural-networks-and-backpropagation-explained-in-a-simple-way-f540a3611f5e">Neural networks and backpropagation explained in a simple way</a></li><li><a href="https://towardsdatascience.com/introduction-to-neural-networks-advantages-and-applications-96851bd1a207">Introduction to Neural Networks, Advantages and Applications</a></li><li><a href="https://medium.com/@datamonsters/artificial-neural-networks-for-natural-language-processing-part-1-64ca9ebfa3b2">7 types of Artificial Neural Networks for Natural Language Processing</a></li><li><a href="https://www.linkedin.com/pulse/artificial-neural-networks-advantages-disadvantages-maad-m-mijwel/">Artificial Neural Networks Advantages and Disadvantages</a></li><li><a href="https://www.quora.com/What-are-examples-of-problems-where-training-an-ANN-is-the-optimal-solution">What are examples of problems where training an ANN is the optimal solution?</a></li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2F7b0181%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2F7b0181%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/748a06e9019942f2677bba917ef55804/href">https://medium.com/media/748a06e9019942f2677bba917ef55804/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab7f4e7b230e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/good-audience/artificial-neural-networks-and-its-contribution-to-machine-learning-a-beginner-s-hand-book-ab7f4e7b230e">Artificial Neural Network and it’s contribution to Machine Learning — A beginner’s hand-book</a> was originally published in <a href="https://medium.com/good-audience">Good Audience</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>