<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vibhav Inna Kedege on Medium]]></title>
        <description><![CDATA[Stories by Vibhav Inna Kedege on Medium]]></description>
        <link>https://medium.com/@ikvibhav?source=rss-c7c0df462c19------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ZhzJrMu132E-5OCGR5K_fw.png</url>
            <title>Stories by Vibhav Inna Kedege on Medium</title>
            <link>https://medium.com/@ikvibhav?source=rss-c7c0df462c19------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 10:11:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ikvibhav/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[MLOps Systems: Feature Engineering: Moving Average Convergence Divergence (MACD) in Python]]></title>
            <link>https://ikvibhav.medium.com/mlops-systems-feature-engineering-moving-average-convergence-divergence-macd-in-python-532fd467c120?source=rss-c7c0df462c19------2</link>
            <guid isPermaLink="false">https://medium.com/p/532fd467c120</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Vibhav Inna Kedege]]></dc:creator>
            <pubDate>Sun, 28 Jun 2026 17:51:38 GMT</pubDate>
            <atom:updated>2026-06-28T17:51:38.696Z</atom:updated>
            <content:encoded><![CDATA[<p>Moving Average Convergence Divergence (MACD) is popular because it compresses multiple pieces of market behavior into a compact set of features: trend direction, momentum, and momentum acceleration. In a feature-engineering pipeline, MACD becomes a structured representation of how short-term movement compares with the longer-term baseline. In this implementation, I explain MACD and abstract it into code that is suitable for analytics pipelines and any further downstream modelling. his article is part of the MLOps Systems: Feature Engineering track, where we move from indicator mechanics to production-ready feature workflows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jX8YprgCapMtdfBX6flzzg.png" /><figcaption>The above image shows the close price of the S&amp;P500 index along with MACD over the first 6 months of 2026. The histograms above 0 suggest a bullish momentum, while histograms below 0 suggest bearish momentum</figcaption></figure><h3><strong>Problem</strong></h3><p>A raw close-price series tells you where price is, but not how short-term behavior compares to the broader trend. When making decisions on whether buying of selling an asset, it is often important to understand how the short term stock compares to the broader term</p><h3><strong>MACD as a solution</strong></h3><p>MACD addresses the above problem by comparing two exponential moving averages (EMA) and then smoothing their difference again to create a signal line.</p><p>The result is three related features:</p><ol><li>The MACD line for trend spread</li><li>The signal line for smoothed momentum confirmation</li><li>The histogram for momentum acceleration</li></ol><h3><strong>Solution (Code)</strong></h3><p>The core function validates inputs, computes fast and slow EMAs, derives the MACD line, and then computes both the signal line and histogram.</p><pre>def compute_macd(<br>    data: pd.DataFrame,<br>    fast: int = 12,<br>    slow: int = 26,<br>    signal: int = 9,<br>    close_column: str = &quot;Close&quot;,<br>) -&gt; pd.DataFrame:<br>    if data.empty:<br>        raise ValueError(&quot;Input data is empty.&quot;)<br>    if close_column not in data.columns:<br>        raise ValueError(f&quot;Missing required column: {close_column}&quot;)<br>    if not (fast &gt; 0 and slow &gt; 0 and signal &gt; 0):<br>        raise ValueError(&quot;fast, slow, and signal must all be positive integers.&quot;)<br>    if fast &gt;= slow:<br>        raise ValueError(&quot;fast window must be smaller than slow window.&quot;)<br><br>    engineered = data.copy()<br>    close = engineered[close_column]<br><br>    ema_fast = close.ewm(span=fast, min_periods=fast, adjust=False).mean()<br>    ema_slow = close.ewm(span=slow, min_periods=slow, adjust=False).mean()<br><br>    macd_line = ema_fast - ema_slow<br>    signal_line = macd_line.ewm(span=signal, min_periods=signal, adjust=False).mean()<br><br>    engineered[&quot;macd_line&quot;] = macd_line<br>    engineered[&quot;macd_signal&quot;] = signal_line<br>    engineered[&quot;macd_histogram&quot;] = macd_line - signal_line<br><br>    return engineered</pre><h4><strong>Fast and Slow EMA</strong></h4><p>Fast and Slow EMA are the foundation to MACD signals. The fast EMA reacts more quickly to recent price movement, while the slow EMA acts as a longer baseline. Their difference is a compact way to measure whether short-term price action is outperforming or underperforming the broader trend.</p><h4><strong>Signal line</strong></h4><p>The MACD line alone can be jittery. The Signal Line Smooths out the trend with another EMA. The Signal line is then used to confirm directional shifts.</p><h4><strong>Histogram</strong></h4><p>Subtracting the signal line from the MACD line produces the histogram line. This is often the most interesting of the three features because it indicates whether momentum is strengthening or weakening.</p><h3><strong>Design Decisions and Edge Case Handling</strong></h3><p>The implementation makes several notable design decisions:</p><ol><li>It validates that all windows are positive.</li><li>It explicitly rejects `fast &gt;= slow`, which would undermine the indicator’s meaning.</li><li>It returns all three outputs in a copied dataframe, making the function composable in a larger pipeline.</li><li>The use of `min_periods` on the EMA calculations is also important. It preserves warmup nulls rather than inventing values too early, which makes the resulting features more honest and easier to reason about.</li></ol><h3><strong>Tradeoffs and Pitfalls</strong></h3><ol><li>Short datasets may produce many `NaN` values because MACD needs enough history for both EMAs and the signal EMA.</li><li>If `fast` and `slow` are too close together, the indicator may become less informative.</li><li>MACD is scale-sensitive to the close-price input. So, adjusted versus unadjusted data can change interpretation.</li><li>Like other technical features, MACD should be validated against leakage and resampling behavior in the surrounding pipeline.</li></ol><h3><strong>Usage and Considerations</strong></h3><ol><li>Use the histogram as a standalone feature in addition to the MACD and signal lines; it often surfaces momentum changes earlier.</li><li>Tune `fast`, `slow`, and `signal` windows to your trading horizon instead of assuming 12/26/9 is always appropriate</li><li>Keep warmup nulls visible until a deliberate downstream cleaning step so you do not blur feature provenance.</li></ol><h3>Conclusion</h3><p>This MACD implementation is aims to convert a trader-facing concept into pipeline-grade feature code. It is explicit, validated, and structured for reuse. That makes it useful not just for charts, but for any system that needs compact indicators of trend and momentum.</p><p>Key takeaways:</p><ol><li>MACD compresses trend direction, momentum, and momentum acceleration into three composable features that are easier to reason about than raw price movement.</li><li>Built-in validation and immutable output make it a reliable primitive in production feature pipelines.</li><li>Production use should include a clear warmup-row policy so training and scoring pipelines handle early `NaN` values consistently.</li></ol><h3><strong>Further Reading</strong></h3><p>1. Exponential smoothing in technical indicators</p><p>2. MACD histogram interpretation in momentum systems</p><p>3. Feature provenance and warmup handling in time-series pipelines</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=532fd467c120" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MLOps Systems: Feature Engineering: Relative Stength Index (RSI) in Python]]></title>
            <link>https://ikvibhav.medium.com/mlops-systems-feature-engineering-building-a-leakage-safe-rsi-feature-in-python-for-stock-1b72c88492e6?source=rss-c7c0df462c19------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b72c88492e6</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Vibhav Inna Kedege]]></dc:creator>
            <pubDate>Sat, 20 Jun 2026 01:49:07 GMT</pubDate>
            <atom:updated>2026-06-28T17:20:02.837Z</atom:updated>
            <content:encoded><![CDATA[<p>The Relative Strength Index (RSI) is one of the most widely used momentum indicators in technical analysis, but its real engineering value shows up when it is implemented as a reusable, pipeline-safe feature. In this implementation, I explain RSI and abstract it into a deterministic transformation over market data, which makes it suitable for analytics pipelines and any further downstream modeling. By the end, you will know how to build and interpret RSI output that is safe to integrate into a broader pipeline. This article is part of the MLOps Systems: Feature Engineering track, where we move from indicator mechanics to production-ready feature workflows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LKeY-_C-CHXngNfdeqmkYQ.png" /><figcaption>The above image shows the close price of the S&amp;P500 index along with RSI over the first 6 months of 2026. The dotted lines at 30 and 70 serve as critical thresholds to indicate potential market movement.</figcaption></figure><h4>Problem</h4><p>Raw price levels rarely say enough on their own. If raw values are used directly in alerts, dashboards, or models, one can obtain jittery outputs and misleading triggers. What often matters more is whether recent gains are consistently stronger than recent losses.</p><h4>Solution Candidate (RSI)</h4><p>RSI captures the balance between recent gains and losses by converting directional price movement into a bounded momentum score from 0 to 100. This bounded scale is easier to compare across dates and instruments than raw returns or price deltas.</p><p>Common interpretations are:</p><ol><li><strong>**RSI above 70 (overbought):**</strong> Price may have risen too quickly and could be vulnerable to a pullback or short-term reversal.</li><li><strong>**RSI below 30 (oversold):**</strong> Price may have fallen too sharply and could be positioned for a short-term bounce.</li></ol><p>These levels are not standalone buy/sell signals, but they are useful context when combined with trend, volume, and risk controls.</p><h4>Solution (Code)</h4><p>The following function validates input, computes daily price changes, separates gains from losses, and then applies Wilder-style smoothing before deriving the final RSI column.</p><pre>def compute_rsi(<br>    data: pd.DataFrame,<br>    window: int = 14,<br>    close_column: str = &quot;Close&quot;,<br>) -&gt; pd.DataFrame:<br>    if data.empty:<br>        raise ValueError(&quot;Input data is empty.&quot;)<br>    if close_column not in data.columns:<br>        raise ValueError(f&quot;Missing required column: {close_column}&quot;)<br>    if window &lt;= 0:<br>        raise ValueError(&quot;window must be a positive integer.&quot;)<br><br>    engineered = data.copy()<br>    close = engineered[close_column]<br>    delta = close.diff()<br>    gain = delta.clip(lower=0)<br>    loss = (-delta).clip(lower=0)<br><br>    avg_gain = gain.ewm(alpha=1 / window, min_periods=window, adjust=False).mean()<br>    avg_loss = loss.ewm(alpha=1 / window, min_periods=window, adjust=False).mean()<br><br>    rs = avg_gain / avg_loss<br>    engineered[f&quot;rsi_{window}&quot;] = 100 - (100 / (1 + rs))<br><br>    return engineered</pre><p><strong>### Step 1: Validate the input</strong></p><p>The function rejects empty data, missing close-price columns, and non-positive windows. That matters because technical indicators fail in subtle ways when fed malformed data, and bad validation usually surfaces later as misleading features rather than immediate exceptions.</p><p><strong>### Step 2: Convert prices into directional changes</strong></p><p>The dataframe operator `diff()` transforms the close series into day-over-day movement. That creates the raw data needed for momentum analysis.</p><p><strong>### Step 3: Split movement into gains and losses</strong></p><p>Positive deltas are retained in a new series called gain and negative deltas are inverted into a loss series. This is the core abstraction behind RSI: not how much price moved in total, but how much of that movement was upward versus downward.</p><p><strong>### Step 4: Apply Wilder smoothing</strong></p><p>The implementation uses exponential weighting with `alpha = 1 / window`, which is a standard way to model Wilder’s smoothing in pandas. This makes the indicator responsive without being overly noisy.</p><p><strong>### Step 5: Normalize to a bounded oscillator</strong></p><p>The final score is derived from the relative strength ratio:</p><p>- `RS = average gain / average loss`</p><p>- `RSI = 100 — (100 / (1 + RS))`</p><p>This yields a bounded feature that is easy to interpret and safe to join into a broader feature table.</p><p><strong>Design Decisions and Edge Case Handling</strong></p><p>The implementation makes several notable design decisions:</p><ol><li>Copies the dataframe instead of mutating the caller’s object.</li><li>Parameterizes the close column and window size for reusability.</li><li>Names the output column dynamically as `rsi_&lt;window&gt;`, keeping multiple RSI windows composable.</li><li>Rejects edge cases immediately at the entry point — a `window` of zero or less is invalid, and a missing close-price column raises right away, preventing silent data corruption.</li></ol><h4>Tradeoffs and Pitfalls</h4><ol><li>Warmup rows remain `NaN` because the function delegates missing-value policy to the caller. That is usually the right choice in pipelines, but it means downstream steps must decide whether to drop or preserve early rows.</li><li>If average loss becomes zero during a sustained uptrend, RSI approaches 100, which is mathematically expected but should be interpreted carefully in models</li></ol><h4>Pro Tips</h4><ol><li>Generate multiple RSI windows, such as 7, 14, and 21, when you want short- and medium-term momentum views in the same dataset.</li><li>Keep RSI as one feature among many. On its own, RSI is descriptive, not predictive.</li><li>Document your warmup-row policy explicitly so training and scoring pipelines treat `NaN` periods consistently.</li></ol><h4>Conclusion</h4><p>This RSI implementation is more than a technical-analysis helper. It turns market behavior into a reusable, pipeline-friendly feature with clear validation and predictable output semantics.</p><p>Key takeaways:</p><ol><li>RSI is a bounded, momentum-based feature that is easier to interpret and compare across instruments than raw price deltas.</li><li>Built-in validation and immutable output make it a reliable primitive in production feature pipelines.</li><li>Production use should include a clear warmup-row policy so training and scoring pipelines handle early `NaN` values consistently.</li></ol><h4>Further Readings</h4><ol><li>Wilder-style smoothing versus simple rolling averages</li><li>Momentum oscillators in time-series feature sets</li><li>Handling warmup nulls in financial feature engineering</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b72c88492e6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MLOps Systems: Feature Engineering: Moving Averages for Noise-Resilient Trend Signals in Python]]></title>
            <link>https://ikvibhav.medium.com/mlops-systems-feature-engineering-moving-averages-for-noise-resilient-trend-signals-in-python-68b85e2365cf?source=rss-c7c0df462c19------2</link>
            <guid isPermaLink="false">https://medium.com/p/68b85e2365cf</guid>
            <category><![CDATA[stock-market]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[finance]]></category>
            <dc:creator><![CDATA[Vibhav Inna Kedege]]></dc:creator>
            <pubDate>Mon, 15 Jun 2026 11:12:29 GMT</pubDate>
            <atom:updated>2026-06-15T12:57:24.335Z</atom:updated>
            <content:encoded><![CDATA[<p>In stock analytics, moving averages are essential because they convert noisy close-price streams into stable, model-friendly trend signals. Without this smoothing step, teams often face jittery alerts, unstable features, and weaker decision quality in downstream systems. In this post, I implement and explain a simple moving average (SMA) using Python and pandas as a practical feature-engineering primitive. By the end, you will know how to build and interpret SMA output in a way that is safe to integrate into a broader pipeline. This article is part of the MLOps Systems: Feature Engineering track, where we move from indicator mechanics to production-ready feature workflows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k_hFKJPEnlE-9wC1vTVLFw.png" /><figcaption>The above image shows the close price, 50 &amp; 200 day SMA of the S&amp;P500 index</figcaption></figure><h4><strong>Problem</strong></h4><p>Stocks Price series are full of small, random movements. If raw values are used directly in alerts, dashboards, or models, one can obtain jittery outputs and misleading triggers. The SMA provides a cleaner baseline by averaging values over a rolling window.</p><h4>Solution</h4><p>Here is the full implementation:</p><pre>def moving_average(data, window_size):<br>    return data.rolling(window=window_size).mean()</pre><p>This function does three things:</p><p>1. Accepts a sequence-like pandas object in `data`.</p><p>2. Builds rolling windows using `window_size`.</p><p>3. Computes mean for each window and returns the smoothed series.</p><pre>import pandas as pd<br><br>prices = pd.Series([100, 102, 101, 105, 107])<br>sma_3 = moving_average(prices, 3)<br>print(sma_3)<br><br># Expected output:<br># 0           NaN<br># 1           NaN<br># 2    101.000000<br># 3    102.666667<br># 4    104.333333</pre><p>The first `window_size — 1` rows are `NaN` because a full window is required before the mean can be computed.</p><h4>Design Decision</h4><p>This implementation is intentionally minimal and idiomatic for pandas workflows. It is easy to read, fast enough for most feature pipelines, and composable with other transformations.</p><h4><strong>Tradeoffs</strong></h4><p>The tradeoff is that input validation is not included. If `window_size` is invalid or larger than the dataset, behavior may not match business expectations unless handled upstream.</p><ul><li>`window_size &lt;= 0` should be guarded against.</li><li>`window_size &gt; len(data)` can yield all `NaN` values.</li><li>Non-numeric values in `data` may fail or produce unusable output.</li><li>Missing values in input can propagate into rolling means.</li></ul><h4><strong>Usage and Considerations</strong></h4><ul><li>Use multiple windows (for example 10, 20, 50) to capture short-, medium-, and long-term trend layers.</li><li>Combine SMA with momentum or volatility features for more reliable signals.</li><li>Define a consistent startup policy for initial `NaN` rows (drop, delay, or impute carefully).</li></ul><h4>Conclusion</h4><p>A one-line moving average is a simple, foundational building block in stock data systems. It improves signal clarity, reduces reactive noise, and prepares data for more robust downstream analysis.</p><p>Key takeaways:</p><ol><li>SMA is a low-cost, high-value smoothing primitive.</li><li>Rolling means are easy to integrate in pandas pipelines.</li><li>Production use should include clear edge-case and `NaN` handling policy.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68b85e2365cf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Leetcode Patterns | Leetcode 75 — Sort Colours]]></title>
            <link>https://ikvibhav.medium.com/leetcode-patterns-leetcode-75-sort-colours-3f2b1b33c19c?source=rss-c7c0df462c19------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f2b1b33c19c</guid>
            <category><![CDATA[two-pointers]]></category>
            <category><![CDATA[dsa-problem]]></category>
            <category><![CDATA[arrays]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[dutch-national-flag]]></category>
            <dc:creator><![CDATA[Vibhav Inna Kedege]]></dc:creator>
            <pubDate>Sun, 14 Dec 2025 02:08:38 GMT</pubDate>
            <atom:updated>2025-12-18T16:14:51.629Z</atom:updated>
            <content:encoded><![CDATA[<h3>Leetcode Patterns | Leetcode 75 — Sort Colours</h3><h3>Problem</h3><p>Given an array nums with n objects colored red, white, or blue, sort them <a href="https://en.wikipedia.org/wiki/In-place_algorithm"><strong>in-place</strong></a><strong> </strong>so that objects of the same color are adjacent, with the colors in the order red, white, and blue.</p><p>Integers 0,1 &amp; 2 are red, white &amp; blue respectively</p><h3>Intuition</h3><p>Use three pointers to divide the nums array into four regions. The regions are confirmed 0s, confirmed 1s, confirmed 2s and elements yet to be processed. This is also called the Dutch National Flag Algorithm.</p><h3><strong>Approach</strong></h3><ol><li>Initialize 3 pointers, i, j, k where -</li><li>i is where 0 (red) will be inserted (All confirmed 0s). Initialize this to 0.</li><li>j is where 2(blue) will be inserted (All confirmed 2s). Initialize this to the end of the array</li><li>k is where the current nums array element is being processed. Initialize this to 0</li><li>Start a list traversal until k crosses j (k &lt; = j ). The = is important because this ensures that the element k gets processed when k and j meet.</li><li>For each element in the list, there are three cases as below</li><li><em>CaseA:</em> When nums[k] is 2. In this case we swap the element from nums[j] (As j represents where 2 will be placed). Once done, nums[j] has 2, so we decrement j by 1. We do not know what element arrives from j as it is unprocessed. So, we do not increment k and deal with the element swapped into nums[k] in the next loop iteration</li><li><em>CaseB:</em> When nums[k] is 1. In this case, the 1 is already in its correct position, so no swaps are made and only increment k to the next element</li><li><em>CaseC:</em> When nums[k] is 0. In this case, we swap the element from nums[i] (As i represents where 1 will be placed). Once done, nums[i] has 1, so we increment i by 1. <strong>Now, as k is moving in the same direction of i (from 0 to nums array end), the elements at i are being simultaneously processed. Thus, we can be confident that i represents a processed value. In this case the processed value is 1. The value 1 because elements before i are 0, and after it until k (and at the end j) are 1. Thus, due to the existence of processed elements property at i, we also move/increment k by 1.</strong></li><li>Do the above until the k&lt; = j loop condition</li></ol><h3>Code</h3><pre><br>def sortColors(self, nums: List[int]) -&gt; None:<br>    # Intuition - Keep 3 pointers<br>    # i -&gt; The pointer before which all elements are 0<br>    # j -&gt; The pointer after which all elements are 2<br>    # k -&gt; The pointer before which all elements are 1, upto and including i<br>    <br>    # 1. Initialisation<br>    i = 0<br>    j = len(nums) - 1<br>    k = 0<br><br>    # 2. Iterate for k in len(nums)<br>    while k &lt;= j:<br><br>        #2.1 If nums[k] is 2, swap with nums[j]<br>        if nums[k] == 2:<br>            nums[j], nums[k] = nums[k], nums[j]<br>            j -= 1<br>        <br>        #2.2 If nums[k] is 1, increase k by 1<br>        elif nums[k] == 1:<br>            k += 1<br>        <br>        # 2.3 If nums[k] is 0, swap nums[i] and nums[k] and increment each pointer<br>        else:<br>            nums[i], nums[k] = nums[k], nums[i]<br>            i += 1<br>            k += 1</pre><h3>Example Walkthrough</h3><pre>1. nums = [2,2,1,0,1,0,1] (CaseA)<br>           i           j<br>           k<br><br>2. nums = [1,2,1,0,1,0,2] (CaseB)<br>           i         j<br>           k<br><br>3. nums = [1,2,1,0,1,0,2] (CaseA)<br>           i         j<br>             k<br><br>4. nums = [1,0,1,0,1,2,2] (CaseC)<br>           i       j<br>             k<br><br>5. nums = [0,1,1,0,1,2,2] (CaseB)<br>             i     j<br>               k<br><br>6. nums = [0,1,1,0,1,2,2] (CaseC)<br>             i     j<br>                 k<br><br>7. nums = [0,0,1,1,1,2,2] (CaseB)<br>               i   j<br>                   k<br><br>8. nums = [0,0,1,1,1,2,2] (Loop Break when k &gt; j)<br>               i   j<br>                     k</pre><h3>Complexity Analysis</h3><p>There is only one loop/iteration that takes place. In the worse case this iteration is len(nums) in size (N). Thus Time Complexity — O(N)</p><p>There is no additional space being used during processing, apart from the pointers. The size occupied by the pointers do not vary with the size of the input and are thus constant. Thus the space complexity — O(1)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f2b1b33c19c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LLM Application — QnA Assistant | Ollama, LangChain, VectorDB, RAG]]></title>
            <link>https://ikvibhav.medium.com/llm-application-qna-assistant-ollama-langchain-vectordb-rag-5949be026189?source=rss-c7c0df462c19------2</link>
            <guid isPermaLink="false">https://medium.com/p/5949be026189</guid>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ollama]]></category>
            <category><![CDATA[vector-database]]></category>
            <dc:creator><![CDATA[Vibhav Inna Kedege]]></dc:creator>
            <pubDate>Sun, 06 Jul 2025 10:49:56 GMT</pubDate>
            <atom:updated>2025-07-06T10:49:56.963Z</atom:updated>
            <content:encoded><![CDATA[<h3>LLM Application — QnA Assistant | Ollama, LangChain, VectorDB, RAG</h3><p>The motivation of this write-up is to -</p><ol><li>Document the working of a Question and Answer Assistant application.</li><li>Describe/point to the tech stacks that are used to create the application.</li><li>Provide a base application that can be used as reference for other LLM based applications</li></ol><h4>Code</h4><p>The working code is written in python on a linux system and is available on github over <a href="https://github.com/ikvibhav/datascience101/tree/main/llm/qna_assistant">here</a></p><h4>Idea</h4><ol><li>The idea of this application is an interactive command-line Q&amp;A assisstant. The specific application chosen is a pizza resteraunt.</li><li>The application uses an Ollama Large Language Model (LLM) &amp; Vector Database to answer user questions through a retriever augmented generation (RAG) approach.</li><li>When a user asks a question, the assistant retrieves relevant reviews from the vector store and generates a context aware answer using a locally running LLM model.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1021/1*Rw8_zovbuVvH3UQI7SzWiw.png" /><figcaption>QnA Application Working</figcaption></figure><h4>Background</h4><ol><li><a href="https://ollama.com/">Ollama</a> — A platform designed to run and manage LLMs locally on a machine that focuses on privacy, efficiency and offline capabilities.</li><li><a href="https://www.langchain.com/">Langchain</a> — An open-source python framework that makes it easier to build Large Language Model (LLM) applications by providing services like connections to external data sources, models, Retrieval Augmented Generation (RAG) capabilities etc</li><li><a href="https://blogs.nvidia.com/blog/what-is-retrieval-augmented-generation/">RAG</a> — Retrieval-Augmented Generation (RAG) is the process of extending LLM capabilities to application specific information, so that an LLM can reference authoritative knowledge base outside its training data sources and generate contextually accurate, credible and relevant information.</li><li><a href="https://aws.amazon.com/what-is/vector-databases/">VectorDB</a> — A type of database that is designed to store &amp; index data in high dimensional vector embeddings, and makes use of similiarity search algorithms to search for data.</li></ol><h4>Software Design</h4><ol><li>The simple software design is composed of two files — qna_assistant.py and vector_db.py</li><li>The vector_db.py has utility functions used to create a vector database of customer reviews</li><li>The qna_assistant.py has a class named QnAAssistant. This class makes use of the vector database and has a handler to answer user questions</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*0YHOHiPyVKgvdZw-0vXBnA.png" /><figcaption>QnAAssistant Architecture</figcaption></figure><h4>Working</h4><p>Using the steps provided in the above code link to</p><ol><li>Install Ollama locally in the system</li><li>Run the code</li></ol><p>I have personally found <a href="https://www.youtube.com/watch?v=UtSSMs6ObqY">this</a> video to be the best for installing Ollama</p><h4>Conclusion</h4><ol><li>The simple application provides a good learning oppurtunity to develop an LLM agent that is hosted locally on a system and act a question and answer bot.</li><li>The RAG approach that can add application domain information and create a contextually relevant assistant makes it a very adaptable application</li><li>An Object Oriented approach of wrapping code within a class makes it very easy to extend the code to other domains</li></ol><p>This small application extends on the base code provided by <a href="https://www.youtube.com/watch?v=E4l91XKQSgw">TechForTim</a>. Do check out the complete code and provide me with any valuable feedback.</p><p>Thank you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5949be026189" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deep Learning in TensorFlow | Part 1 — The Perceptron]]></title>
            <link>https://ikvibhav.medium.com/deep-learning-in-tensorflow-part-1-the-perceptron-7340ab823a7d?source=rss-c7c0df462c19------2</link>
            <guid isPermaLink="false">https://medium.com/p/7340ab823a7d</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Vibhav Inna Kedege]]></dc:creator>
            <pubDate>Sun, 20 Oct 2024 19:22:15 GMT</pubDate>
            <atom:updated>2026-06-15T10:17:24.184Z</atom:updated>
            <content:encoded><![CDATA[<h3>Deep Learning in TensorFlow — The Perceptron</h3><p>The motivation of this write-up is to -</p><ol><li>Translate a fundamental neural network perceptron model into simple tensorflow code.</li><li>Vary and visualize <strong>input layers</strong> and <strong>hidden layer units</strong></li><li>Understand the relation of input layers &amp; hidden layers to the number of trainable parameters in a neural network</li></ol><h4>Resources</h4><ol><li><a href="http://introtodeeplearning.com/">http://introtodeeplearning.com/</a></li><li><a href="https://dev.mrdbourke.com/tensorflow-deep-learning/">https://dev.mrdbourke.com/tensorflow-deep-learning/</a></li></ol><h4>Background</h4><ol><li>Perceptron</li></ol><p>A Perceptron is the simplest type of an artificial neuron (<a href="https://deepai.org/machine-learning-glossary-and-terms/perceptron">source</a>). A perceptron takes in inputs, applies weights to them, makes a sum and finally applies non-linearity to the sum.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/397/1*W8iWWvrlLoHKqQAzMhycsA.png" /><figcaption>Source — <a href="http://introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L1.pdf">http://introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L1.pdf</a></figcaption></figure><p>Usually the bias term is removed in simpler representations thereby resulting in the following</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/1*JNG-nX0T2vII3AUqprF0Eg.png" /><figcaption>A simple perceptron with 3 inputs</figcaption></figure><p>2. Weights</p><p>Weights are the variables whose value gets modified when the neural network is trained. In the figure above, these as the “w” terms. For every unit of the neural network there is an additional bias (b) or w0 term.</p><h4>Implementation</h4><ol><li>From the above diagram of a simple perceptron, there are 3 inputs and a single neuron. This can be represented in tensorflow as below,</li></ol><pre>model = tf.keras.Sequential([<br>        tf.keras.layers.Dense(units = 1, input_shape=[3])<br>    ])<br><br>print(model.count_params()) # Output is 4</pre><p>2. The number of trainable parameters as seen in the previous image and the output of the code is 4.</p><p>3. To modify the number of input layers, modify the input shape parameter of the Dense Layer. For example, in the below image and code the number of trainable parameters is 5.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/1*9ki4b90gUqSetyJUVP35xQ.png" /><figcaption>Simple perceptron with 4 inputs</figcaption></figure><pre>model = tf.keras.Sequential([<br>        tf.keras.layers.Dense(units = 1, input_shape=[4])<br>    ])<br><br>print(model.count_params()) # Output is 5</pre><p>3. Apart from varying the input layers, the number of output layers can also be varied. This results in a <strong>multi-output perceptron</strong>. Below is an image of an architecture with 2 input layers and 2 perceptrons -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*ViHjQrMO-miS_kdxodv9FA.png" /></figure><pre>model = tf.keras.Sequential([<br>        tf.keras.layers.Dense(units = 2, input_shape=[2])<br>    ])<br><br>print(model.count_params()) # Output is 6</pre><p>4. For larger neural networks, the number of trainable parameters follows the same trend. It finally generalises into a formula that is dependent on the number of inputs and number of hidden neurons. This generalised relation is,</p><p>Number of Trainable Parameters = Number of Input Layers X Number of Hidden Neurons + Number of Hidden Neurons</p><p>ie, the number of trainable parameters in a neural network is the product of number of inputs and number of hidden neurons plus the number of Hidden Neurons.</p><p>The additional sum with the hidden neurons is because of the bias terms. This is seen from the results of the code link at the end -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*F_iY0T6FlapwFs67lfWEjw.png" /></figure><h4>Conclusion</h4><p>In this simple blogpost, single layer perceptrons were visualised, the number of trainable parameters were computed for varying input sizes and single neural network units and the intution behind the formula for computing the total trainable parameters was visualised.</p><p>Do check out the <a href="https://github.com/ikvibhav/datascience101/blob/main/medium_writeups/deep_learning_part1/perceptron.py">complete code</a> and provide me with any valuable feedback!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7340ab823a7d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Open AIGym|Simple SARSA and Q-Learning Reinforcement Learning Implementations]]></title>
            <link>https://ikvibhav.medium.com/open-aigym-simple-sarsa-and-q-learning-reinforcement-learning-implementations-7d5ea6f1ff9?source=rss-c7c0df462c19------2</link>
            <guid isPermaLink="false">https://medium.com/p/7d5ea6f1ff9</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[hyperparameter-tuning]]></category>
            <dc:creator><![CDATA[Vibhav Inna Kedege]]></dc:creator>
            <pubDate>Fri, 25 Sep 2020 23:18:12 GMT</pubDate>
            <atom:updated>2024-10-02T19:02:36.199Z</atom:updated>
            <content:encoded><![CDATA[<h3>Open AIGym | Simple SARSA and Q-Learning Reinforcement Learning Implementations</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FQzgSXa2m4ps%3Fstart%3D2%26feature%3Doembed%26start%3D2&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DQzgSXa2m4ps&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FQzgSXa2m4ps%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9aedac9b369a02f64e15ea37aa02d467/href">https://medium.com/media/9aedac9b369a02f64e15ea37aa02d467/href</a></iframe><p>In this blogpost, Reinforcement Learning has been used in order to solve OpenAI Gym problems (These environments can be found <a href="https://gym.openai.com/">here</a>). More specifically the algorithms of SARSA and Q-Learning have been used to solve the problems of,</p><ol><li>Balancing a pole on a moving cart</li><li>Driving a car up a mountain.</li></ol><p>The aim of solving multiple problems using two reinforcement learning algorithms is to compare each algorithm in terms of the episodes taken for learning and the resulting behaviour of the agent in the environment.</p><p>To facilitate such an analysis in a methodical manner, this report has been written such section 1 talks about the related work that has been done and that has helped with implementation and analysis. An overview on the important aspects of each algorithm has been mentioned in algorithms, in section 2. Following this an analysis on the environment/problems being solved is presented in section 3. Section 4 presents the results of the learning and important inferences made, followed by section 5 on a sensitivity analysis which aims at studying the algorithm a deeper by varying important hyper-parameters. This is followed by section 6 on conclusions consisting of the links to the code and videos.</p><h3>1 | Literature References</h3><p>OpenAI gym based environments were chosen to compare the algorithms. Such a choice was mainly because of the ease of interfacing with the agents and also because it supports multiple environments. Ever since its inception in 2016, the gym has been widely used and there have been many implementations on this. In the specific case of this report, the method of SARSA and Q-Learning is similar to the implementation of Q-Learning made by Sentdex in this <a href="https://pythonprogramming.net/q-learning-algorithm-reinforcement-learning-python-tutorial/?completed=/q-learning-reinforcement-learning-python-tutorial/">link</a>. For the analysis and conclusions drawn, the major reference book used is Sutton and Barto which can be found in this <a href="http://www.incompleteideas.net/book/the-book-2nd.html">link</a> .</p><h3>2 | Algorithms</h3><p>In order to solve the 2 mentioned problems, the reinforcement learning algorithms of SARSA and Q-Learning have been utilised. We explain both in detail in this section.</p><h4>SARSA</h4><p>State, Action, Reward, Next State and Next Aciton which is abbreviated as SARSA is an On-Policy temporal difference algorithm. Temporal difference learning is a type of model-free learning that performs learning by bootstrapping from the current value function estimate. Further, an on-policy algorithm implies that the agent learns the Q-values based on the policy being carried out by the agent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/1*P2S4-ZS1-bEHs_I3kCrGEQ.png" /><figcaption>SARSA Algorithm (<a href="http://www.incompleteideas.net/book/the-book-2nd.html">source</a>)</figcaption></figure><p>In the above algorithm, the learning rate (alpha), discount factor (gamma), exploration rate (epsilon), the number of episodes and a Q table of the size of state times action is first set up. Following this multiple episodes of training are run within which various steps of the episode are executed. At each step, when the agent is in a particular state, an action is taken and the corresponding reward and state that is reached (next state) are recorded. The corresponding action value is selected from the Q-table and the Q table is updated using the Q learning update equation as shown in line 9 of the figure. Following this the state and actions are updated with the new state and action pairs and the step is continued again. A step terminates after a predefined maximum number of steps or when the goal of the environment has been reached.</p><h4>Q-Learning</h4><p>Q-Learning is an off-policy temporal difference learning algorithm. The term off-policy refers to the fact that at each step the optimal policy/Q-value is learnt independently from the agents actions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*mID0FfOx0GpK_EqWZB3INw.png" /><figcaption>Q-Learning (<a href="http://www.incompleteideas.net/book/the-book-2nd.html">source</a>)</figcaption></figure><p>The working of of the algorithm is shown in the above figure. The initialisation of the hyper-parameters (alpha, delta etc) is similiar to SARSA. When comparing figures for sarsa and qlearning, it can be seen that the main difference comes in the selection of the action and updating of the Q-table. In Q-Learning the action from the current step is not retained for the next step and at each step a new action is selected. In line 8 of Q-Learning, the Q-table is updated by selecting an action that maximises the next state instead of using the action that would have corresponded to this state in the Q-value. By doing this the agent is learning using a different action value than what the agent has taken and thus going <strong><em>off</em></strong> the current policy and thus being classified as an <a href="https://stats.stackexchange.com/questions/184657/what-is-the-difference-between-off-policy-and-on-policy-learning"><strong><em>off</em></strong><em> </em>policy</a> algorithm. The Q-table update is continued further for a finite number of times following which the next episode of training starts again where a new state is initialised while retaining the same Q-table.</p><p>Some additional good sources to learn about Q-Learning are on S<a href="https://pythonprogramming.net/q-learning-reinforcement-learning-python-tutorial/">implilearn</a>, <a href="https://pythonprogramming.net/q-learning-reinforcement-learning-python-tutorial/">Sentdex</a> and <a href="https://www.datacamp.com/tutorial/introduction-q-learning-beginner-tutorial">DataCamp</a></p><h3>3 | Environment Analysis</h3><p>To implement the algorithms it is required to note the important aspects of the environments in below figures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/1*6ejPXBO4QY4dssOrs6_3Tg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*MSrOzymume4vtmsV8dO_2w.png" /><figcaption>CartPole (left) and MountainCar (left) environments</figcaption></figure><p>The size and limits of the action and state space have been captured in below table</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*7EASAO91iDLyY7kU9fzCsQ.png" /></figure><h4>Cart Pole</h4><p>The state space of the cart pole problem is a 4 x 1 array consisting of the values of cart position, cart velocity, pole angle and pole angular velocity. The action space consists of 2 values, 0 which is to push the cart to the left and 1 to push the cart to the right. A reward of +1 is given for every step taken and an episode terminates when<br> 1) Pole angle ≥12 degrees<br> 2) The cart centre reaches display edge<br> 3) Episode length is ≥ 200</p><h4>Mountain Car</h4><p>The state space of the mountain car problem is a 2 x 1 array containing the car position and car velocity. The 3 action values are 0 denoting leftward acceleration, 1 for no acceleration and 2 for accelerating towards to the right. A reward of 0 is received if the agent reaches the flag (position = 0.5) and -1 is rewarded if the agent position is less than 0.5. The episode terminates when:-<br>1) Car position ≥ 0.5<br>2) Episode length is ≥ 200</p><h3>4 | Implementation and Results</h3><h4>CartPole</h4><p>To implement a Q-Learning solution as well as SARSA, a Q-table was first constructed that was of the size, <em>state size x action size</em>. This corresponded to the number of divisions in the pole position and velocity space which was continuous. In order to have an element of exploration in the agent, a constant value of the exploration constant (epsilon) was used. The values of eahc hyperparamter that was used for both SARSA and Q-Learning are shown in the below table,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/226/1*Fjz-HNodVxGc_0qDcXqXrA.png" /><figcaption>Cartpole Hyperparameters</figcaption></figure><p>The training was run for a number of episodes and at every episode the Q value corresponding to the agents pole position and pole velocity was updated. The environment was also rendered/displayed every 500 episodes. The corresponding graphs can be seen below. In each case, the graph is a plot between the maximum reward, minimum reward and average reward per episode, versus the episode. The average reward per episode is computed as the sum of total rewards until a particular episode divided by the total number of episodes until that point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*PPnrhBa1uD-3DQNjmEnuJg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*aCkYopCPpH55sN4WrIlegQ.png" /><figcaption>SARSA and Q-Learning behaviour for Cartpole</figcaption></figure><h4>Mountain Car</h4><p>When implementing the algorithms on this environment, it was found that using a constant value of epsilon was not able to solve this problem. Instead, a dynamic epsilon was used that was of a high value in the beginning and was decreased as the episodes progressed. This had the effect of ensuring a higher probability of exploration in the beginning and a higher probability of exploitation or a greedy action at higher episodes. The hyperparameter table being,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/226/1*ykNf34bNfG94J80txIuelw.png" /><figcaption>Mountain Car Hyperparameters</figcaption></figure><p>The corresponding graphs can be seen in the below figures, which shows the trend of the maximum, average and minimum rewards at every episode.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*ucSU4TCEDfXvDbYW0Cjizg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*9Nzi5wj0yzIo6z2so_K4PQ.png" /><figcaption>SARSA and Q-Learning behaviour for MountainCar</figcaption></figure><h4>Inferences</h4><p>On comparing the graphs of SARSA and Q-Learning for each case we observe that:-</p><ol><li>In both the cartpole and mountain car problems, the reward converges to a larger value in the case of Q-Learning than in the case of SARSA. This is possibly due to the action selection step. In Q-Learning, the action corresponding to the largest Q-value is selected. This therefore can cause a higher reward value to be obtained in the longrun.</li><li>The maximum reward is obtained by the agent in 2500 episodes in the case of cart pole and 7500 episodes in the case of mountain car. This points towards the cart pole example being more easy to learn than the mountain car example, for the same algorithm used. This is also the reason why a dynamic epsilon has been used in the mountain car case.</li><li>The average reward almost converges to the maximum reward after 30000 episodes in the case of Cart-pole but not in the case of mountain car. This also thus implies that more training is required by the agent if it needs to converge to the maximum value.</li></ol><h3>5 | Sensitivity Analysis</h3><p>In order to critically analyse the effect of each hyper-parameter, a single environment was considered and a particular hyper-parameter was varied across 4 different values while keeping other values constant. For this analysis, we consider the cart-pole environment on which we apply Q-Learning following an epsilon-greedy strategy. Thus, we consider the below update equation and select the hyper-parameters for the sensitivity analysis that can potentially change the updated values at each iteration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*kRpotPBvbGAfXa6UDkw5bg.png" /><figcaption>Q Learning Update Equation (<a href="https://wiki.pathmind.com/deep-reinforcement-learning">source</a>)</figcaption></figure><p>In each case a plot has been made between the average reward per episode versus the episode for 4 values of the same hyper-parameter.</p><h4>Exploration constant sensitivity</h4><p>Epsilon as mentioned previously is used to give the agent an element of exploration. When epsilon=0, the algorithm only considers actions corresponding to the highest Q-value. When epsilon = 1, the algorithm only selects random action values. The below graph shows the training results for each of the 4 values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*1Q9siVKDXlzERPdT0SseIA.png" /><figcaption>Epsilon Sensitivity Analysis</figcaption></figure><p>We make the following observations:-</p><ol><li>A higher epsilon value (in this case 0.5 and 0.75) causes the agent to obtain average reward values of only around 48 even after 30000 episodes. This is expected as With a high value of epsilon the agent has a higher probability of exploration and there are very less chances that the agent actually takes actions that give a higher reward at that moment.</li><li>A decreasing of epsilon to a mid range value (like 0.25) causes training to improve further, but also causes the agent to begin exploring again. This therefore appears to cause the agent to obtain lower rewards again.</li><li>A low value of epsilon (around 0.1) actually helps with the training for this example. Due to the simplicity of the example, having almost a low probability of exploration is beneficial.</li></ol><p>It is important to note that for more complicated examples, the value of epsilon must be first kept high and must be decremented at each episode, as has been done in the mountain car example. This would help in exploration in the beginning following which the probability of exploration decreases and only actions corresponding to the highest Q-value is selected.</p><h4>Discount Factor sensitivity</h4><p>The discount factor (delta) is used to measure how far ahead in time the algorithm must look. When delta =0, then none of the future rewards are considered in the Q-Learning equation and when delta = 1, future rewards are given a high weight. The results for running the training for various values of delta is shown below,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*erJeUeVhaYVZuYsL6zBTyg.png" /><figcaption>Discount Factor Sensitivity Analysis</figcaption></figure><p>We make the following observations:-</p><ol><li>With Lower delta values (like 0.5) there is no improvement ion the average reward per episode that the agent receives even in the 30000 episodes. With low delta values, future average rewards are given a very less weight and therefore do not impact the update of the q-value in each step of an episode. Therefore this result is expected.</li><li>When the delta value is increased (to around 0.5), the average reward per episode increases. This is due to a better update of the Q-table values and the agent learning more from the future Q-values.</li><li>With values of delta closer to 1 like 0.95 and 1 itself, the agent performs the best because each update step updated the Q-table by a larger factor and can therefore consider the effects of future Q-values better when choosing action values the next time.</li></ol><h4>State window sensitivity</h4><p>When working out the problems, it was found that the selection of the number of states between the minimum and maximum values of a particular state (also called window/bucket) was important in order to have good learning. This is primarily associated to the dimensions of the Q-table while learning and therefore a higher window value implies a higher Q-table size and a higher resolution across each state. The buckets were varied across 4 values and it has been shown below,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*_a-5S0_b3Cu2j5B25-fbjQ.png" /><figcaption>State Window Sensitivity Analysis</figcaption></figure><p>We make the following observations:-</p><ol><li>A low state window value like 5 leads to a lower resolution of the state space. Due to this for larger groups of state space values, the same action is taken and this leads to a poorer learning for an agent which gets an almost constant average reward per episode for a large number of Episodes (30000 in this case).</li><li>When the number of windows increases, then the state space is differentiated better and therefore when the number of windows is 25 or 50, it results in faster and better learning at 30000 episodes.</li></ol><h4>Learning rate sensitivity</h4><p>The learning rate (alpha) is set between 0 and 1. It is used to facilitate the Q-value update at a desired rate. When alpha = 0 then the Q-values are never updated and so nothing is learnt. When alpha = 1 then nothing is added to the current Q-value. Varying alpha between these values and running the simulations we get the trend as shown below,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*1UX1PESINp6JOzHOl3gKJg.png" /><figcaption>Learning Rate Sensitivity Analysis</figcaption></figure><p>From the figure, we make the following observations:-</p><ol><li>Setting alpha = 1 causes no learning to taken place. This is due to the nature of update equation (shown at the beginning of the section) which is a difference and hence keeping such a value causes the existing Q-value to vanish.</li><li>Setting alpha = 0.25 appears to be an optimum level of learning which provides a balance between very slow learning as seen in alpha = 0.1 and no learning with alpha=1.</li><li>The behaviour of setting alpha to very high values (like 0.75) can appear to converge to a maximum average reward at the beginning but has the possibility to reduce upon further training.</li></ol><h3><strong>6 | Conclusion</strong></h3><p>In this blogpost, SARSA and Q-Learning has been implemented in order to solve the cart pole and mountain car problems of the OpenAI gym environment. The algorithms have been compared collectively and a sensitivity analysis of varying one of the hyper-parameters and looking at its effect on the learning has also been performed.</p><p>The code has been made freely available and the recorded videos of its working have been made and these can be found below. Further, please reach out to me if anything is unclear or any corrections/modifications are needed</p><ol><li>Code (<a href="https://github.com/ikvibhav/reinforcement_learning">link</a>)</li><li>Cart-Pole SARSA (<a href="https://www.youtube.com/watch?v=3hItuvLUzvs">link</a>)</li><li>Cart-Pole Q-Learning (<a href="https://www.youtube.com/watch?v=ODpZ6IdsPBg">link</a>)</li><li>Mountain car SARSA (<a href="https://www.youtube.com/watch?v=QzgSXa2m4ps">link</a>)</li><li>Mountain car Q-Learning (<a href="https://www.youtube.com/watch?v=LT696W3mzdo">link</a>)</li></ol><h3>Thank you!</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7d5ea6f1ff9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[End-to-End Autonomous Driving for Embedded Agents using CNNs]]></title>
            <link>https://ikvibhav.medium.com/end-to-end-autonomous-driving-for-embedded-agents-using-cnns-e0e9836782d?source=rss-c7c0df462c19------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0e9836782d</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[embedded-systems]]></category>
            <category><![CDATA[autonomous-cars]]></category>
            <dc:creator><![CDATA[Vibhav Inna Kedege]]></dc:creator>
            <pubDate>Fri, 03 Jul 2020 10:20:49 GMT</pubDate>
            <atom:updated>2024-10-02T19:11:00.115Z</atom:updated>
            <content:encoded><![CDATA[<h3>End-to-End Autonomous Driving for Embedded Agents using Deep Learning</h3><p>The working code can be found <a href="https://github.com/ikvibhav/DeepLearning_Autonomous_Embedded_Agents">here</a> and the model working videos can be found <a href="https://www.youtube.com/channel/UCIDoj4S9lDs_YkHe3O_lbCA?view_as=subscriber">here</a>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FbtGmuH96UdU%3Fstart%3D75%26feature%3Doembed%26start%3D75&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DbtGmuH96UdU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FbtGmuH96UdU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/96827d7cdf2fb0112cfa297bebdf15f8/href">https://medium.com/media/96827d7cdf2fb0112cfa297bebdf15f8/href</a></iframe><p>Over the years, Deep Learning has been utilised in various applications. Autonomous driving is one such example. Alongside the development of deep learning techniques in the automotive field for commercial vehicles, there is also research being done on the deployment of such models onto various embedded automotive platforms. Inspired by this, our main motive in this project is to design and compare end to end light deep neural networks that are capable of performing autonomous driving on a predefined track when deployed on a low resource hardware platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n2Xdsw_YzkSOfEJkBSjQrQ.png" /><figcaption>Figure 2: Autonomous System main blocks (take from <a href="https://community.arm.com/developer/ip-products/system/b/embedded-blog/posts/7-key-challenges-impacting-the-mass-deployment-of-autonomous-vehicles">here</a>)</figcaption></figure><p>In general, autonomous systems usually have four main blocks — sensing, perception, planning, and control. As seen in figure 2 the flow of the blocks is such that the environment is first perceived by the sensors. This information is then converted into meaningful information in the perception block. Based on this information, the planning block decides on a path which the system would need to follow. Finally, the control system of the vehicle executes actions and actuates the vehicle to move according to the path. This approach has been considered in this project too where the deep neural network combines the perception, planning and control blocks in order to output steering angle predictions (thus control the vehicle) by using sensed camera images obtained as input. This results in end-to-end learning that enables a system/vehicle to drive automatically, by learning important details of the path.</p><p>In this project two different datasets are considered. One is the Yellow track dataset which is short and does not have too many sharp turns or steep terrains. The other is the green track dataset which is comparatively more difficult to train on as the path consists of steep terrains and sharp corners. To learn the paths, the approach is to implement various architectures that have been created since the beginning of 2019. More specifically the results of the yellow track dataset for the models of JNet, JNet_ELU, Pilotnet, and AIRSIM (will be discussed in the later sections) will be analysed. Along with this, the performance of the architectures on the green track dataset will also be investigated.</p><p>In the following sections, we go deeper into the related work done, the training methods, the data collected, experiments, comparison of results and the implementation of the autonomous driving feature for all four neural networks.</p><h3>Related Work</h3><p>Most of the work done in this paper is based on analysing different neural networks on the lightweight Udacity simulator which is available <a href="https://github.com/udacity/self-driving-car-sim">here</a>. This simulator was made open source on GitHub in January 2019 and from then on, it has proved to be an easy tool to use and test out various architectures. There has been various architectures implemented like the NVIDIA model as in <a href="https://developer.nvidia.com/blog/deep-learning-self-driving-cars/">here</a>, the AlexNet and VGG-Net architectures as in <a href="http://dspace.calstate.edu/bitstream/handle/10211.3/207785/Project_Report_fall_aditya_1.pdf?sequence=1">this study</a>. Another architecture on JNET has been implemented in the <a href="https://www.mdpi.com/1424-8220/19/9/2064">study by kocic</a>. This paper was one of the first ones to bring into the idea of making the architecture smaller thereby allowing such a solution to be uploaded in a lightweight manner. Apart from using the UdaCity simulator, there are also other platforms on which architectures have been implemented. One such simulator is AIRSIM, which is developed by <a href="https://github.com/microsoft/AutonomousDrivingCookbook/blob/master/AirSimE2EDeepLearning/TrainModel.ipynb">microsoft</a> . The architecture is also referred to as the hello world program of this platform and we will refer to as AIRSIM model.</p><p>In this project, the work done is to reproduce the results of the NVIDIA (also called PilotNet) and the <a href="https://www.mdpi.com/1424-8220/19/9/2064">JNET</a> architectures. Instead of using an ALexNet based method as presented in the paper, the AIRSIM model has been used instead. The original JNET model as presented in the paper has also been modified and compared with here. Following this a qualitative and quantitative analysis of the performance has been made for all the 4 models on a flat terrain and hilly terrain dataset. The comparison metrics and training methods used are similar to those presented in the paper and while the paper and most of the resources online test their solutions on the flat terrain track, this project aims at extending the training and testing to the more difficult hilly terrain.</p><h3>Methods</h3><h4><em>Overview</em></h4><p>The essential questions that this project aims at answering are:-</p><ol><li>Can the size of the model used in autonomous driving be reduced?</li><li>Can the reduced model perform well on difficult tracks?</li></ol><p>To answer the above questions, we first selected 4 different architectures, based on the number of trainable parameters. We trained these models on the yellow dataset having the same training hyper-parameters (EPOCH, batches and learning rate) and compared the performance to investigate if lighter models would give a similar or even better performance than large sized models. Therefore, by doing this the first question is answered. Following this, we trained the architectures on the more difficult green dataset and noted down its performance. A larger model has a higher capability to learn more complex features of the images and therefore can perhaps track the path better for the same input images. Keeping this in mind we moved onto investigating if lighter models too can learn to traverse more difficult paths thereby helping us to investigate the second question. The architectures chosen were:-</p><ol><li>AIRSIM</li><li>PilotNet</li><li>J-Net</li><li>J-Net_ELU</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PeFrk6oP4J9vs55xgXEdzA.png" /><figcaption>Figure 3: Architectures studied</figcaption></figure><p>In the above, AIRSIM, PilotNet and JNet are directly taken from previous works. In our project, we modified the JNet architecture to have an ELU instead of ReLU activation and a dropout layer. Its simple to remember that JNet\_ELU = JNet — ReLU + ELU. The choice of using ELU was because it can produces negative output values which are present in the steering values, which cannot be done by ReLU. The reason for using dropout is to avoid overfitting. Now, Consider the number of parameters from each architecture as in figure 3. We observe that each model has parameters in the order of 10^5. These parameters are directly proportional to the computations performed. Thus, the computations are different between the case of JNET_ELU and PilotNet. For each architecture, the following table presents the number parameters, type of layers and the size of the “.h5” model file (which will be explained later) generated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*48-UB6E2QcCxzIwxRrR-DA.png" /><figcaption>Figure 4: Numeric comparison of models</figcaption></figure><p>It can be seen that JNET_ELU has 157k trainable parameters as shown above. This network has nearly half the number of trainable parameters compared to the PilotNet having nearly 354K parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*HG_TBUvS154kNMx0aCew6w.png" /><figcaption>Figure 5: Training overview</figcaption></figure><p>The idea used in the training is as described in the figure 5 and has been taken from this <a href="https://www.mdpi.com/1424-8220/19/9/2064">paper</a> where we first obtain the left, center and right camera data and the corresponding steering angle values. The data that is obtained is first passed through a data augmentation step following which it is used to train a CNN. The training method used in the CNN was to use 80% of the obtained data for training and 20% for validation. The training method compares the predicted output to the ground truth (Actual steering angle Y) using a mean squared error function, following which an adam optimiser is used to update the weights of the CNN. The choice of using ADAM was because it is computationally efficient, suited for problems with a large amount of data (like this one) and also combines the benefit of Adaptive gradient descent algorithm and RMS Propagation. More information on ADAM is explained by Jason Brownlee <a href="https://ml-cheatsheet.readthedocs.io/en/latest/activation_functions.html">here</a>. At the end of the training we observe that by making the CNN learn the weight parameter, we obtain a controller that readily makes steering angle predictions based on the input data from the camera.</p><h4>Dataset</h4><p>The dataset were the images of the tracks taken by the left, center and right cameras. The simulator that we used for this was the Udacity’s Self-Driving Car Simulator. There are 2 tracks that are present in this simulator, the yellow track and the green track.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wVdi8qL6DTEr45xWc3nqrw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I8MEasftALDKu3THVr4fxA.png" /><figcaption>Figure 6: Target maps</figcaption></figure><p>It can be seen from the above images that the yellow track is flat and has very few sharp turns. This is in contrast to the green track which has hilly terrain and many sharp turns. Due to this, we refer to the yellow track as the easy one and the green track as the difficult one. As mentioned in the introduction, the original paper referred to has made a comparative study on only the yellow dataset. In our study, we have done the same for the green dataset and have drawn conclusions based on a comparison of the models on both the datasets.</p><p>As seen from the figure 5, before feeding the data into the neural network, there is a data augmentation block. The block is required to convert the data into a size that is compatible with the architecture and also that consists of pixel values that are relevant to the steering angle prediction. The process that is followed in the original paper (<a href="https://www.mdpi.com/1424-8220/19/9/2064">kocic</a>) is to convert an input image of size 320*160*3 into a cropped up image of size 320*65*3 such that the portions of the front car bonnet and the sky are cropped out, as shown in the below figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o_FTMu1WoL_pcqPF3E70sQ.png" /><figcaption>Figure 7: Original cropping</figcaption></figure><p>While the above works for straightforward tracks, in the case of green tracks the situation is more different primarily because there are many moments when the car is not horizontal and the path in front of it lies at the upper region of the camera image. This can be seen in one such part of the map below,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*TuiUC_gCB_oDvn578StjrA.jpeg" /><figcaption>Figure 8: The nature of the green track</figcaption></figure><p>In order to deal with such scenarios, the data sets were only cropped in order to remove the car bonnet pixel data thus resulting in:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D5tup9vGcuhPF8hgnkLKZQ.png" /><figcaption>Figure 8: Modified cropping on green track</figcaption></figure><p>Further, the images were then resized into the size of 320*65*3, as per the requirements of the architecture in the paper. We also note that this dimension worked well for the AIRSIM model that was taken from another source. This type of data augmentation is also used for the yellow dataset, and it did not hamper the final functionality of predicting reasonable steering angle values (as will be seen later).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pkd3O1MEQ0W48M5w31VcGQ.png" /><figcaption>Figure 9: Modified cropping on yellow track</figcaption></figure><h4>Training Algorithm</h4><p><em>Original Approach</em></p><p>Along with the training images, the corresponding values of steering angles, throttle, reverse and speed were obtained. All this was automatically generated into an excel sheet format by the simulator, as can be seen in the below figure,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*vy1sPaaSma4nnRRyL9w4Bg.png" /><figcaption>Figure 9: Excel formatted dataset generated</figcaption></figure><p>For our experiments, we used the CENTER, LEFT, RIGHT as the training data and only the steering values for the output value. It is important to note that in a forward pass of training only one amongst the 3 images is used. Initially the strategy was that we used all the training fields across the entire rows and then trained the model. However there were 3 main demerits with this way for training:-</p><ol><li>The time taken to load all the images into the code even before training could begin was very high. An example of this is that for the green dataset where 58K image data points were present, it took 10–15 minutes to load the data itself.</li><li>Using the left and right training images was not that necessary for the driving because in the final working only CENTER images were to be taken. The LEFT and RIGHT images were only used to help improve the steering by ensuring that such difficult steering scenarios are possible to maneuver.</li><li>When we compared the final values of training and validation loss with another method (which will be mentioned below) that instantly began the training instead of loading the data, we found that the performance was very similar. Therefore the additional time taken would be unnecessary in the long term goal of predicting the correct steering value.</li></ol><p><em>Modified Approach</em></p><p>The method explained here has been originally created by <a href="https://github.com/naokishibuya">Naokishibuya</a>. In this project, we have modified this code as per our requirements. The fundamental aspect of the algorithm used in that it does not train over all the images in an EPOCH. Rather there is a parameter called samples per EPOCH that can be used to determine the number of samples to be considered per training EPOCH. Further, mini-batch gradient descent was used where for each gradient descent step, batch_size value of images was taken. The selection of such images was done by generating a random number and comparing it with a selected threshold (In our case 0.75). If the number was lesser than this then LEFT or RIGHT images would be taken, else only CENTER camera images. Once the image was selected, it would be sent to the data augmentation step (explained earlier) following which the images (of size batch_size) and its corresponding steering angle values would be sent for training. This is implemented in the code as in the below image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VGLhnNE6X6ojc0oivdZJig.png" /><figcaption>Figure 10: Modified training method</figcaption></figure><p>Upon trying different values, the best values were obtained by setting batch_size = 100, EPOCH=10 and samples_per_EPOCH = 10100 for the yellow dataset and samples_per_EPOCH = 46500 and EPOCH=20 for the green dataset. The samples_per_EPOCH was chosen based on the amount of data that was used to train the model. For both the yellow and green datasets, we ran the car for 10 laps to obtain the required data present in the excel figure 9. In the case of the yellow dataset, we obtained around 30K data points and setting the training to only train on 10100 samples per epoch with 10 EPOCH values was found to work OK. However, while using the green dataset the training example was 46K and therefore we decided to use more training samples corresponding to this number and for a higher value of EPOCH = 20. A higher value of batch size of 100 was taken because of higher convergence rate to the optimum value of weights. We ensured that it did not go beyond this value as this would possibly lead to overshooting the optimal value of gradient. A learning rate of 10^-4 was chosen and found to work okay for most of for architectures in the yellow dataset.</p><p>After every EPOCH, the code generated a “.h5” file based on a lower value of val_loss produced by the training. A “.h5” file is basically a hierarchical data format file that consists of the model in a binary format and is used to drive the car in the simulator. More on .h5 files can be found <a href="https://support.hdfgroup.org/HDF5/doc/H5.intro.html">here</a>. It is important to note that while we trained the data set such that a “.h5” was generated only when a better val_loss is obtained, we later realised that many times the models giving a lower training value proved to give better steering value. Due to this, we set the training such that it would generate a model after every EPOCH was complete. Thus, we obtained 10 “.h5” model files when EPOCH=10.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/1*skl59YiIJApm4IPeNrUSZA.png" /><figcaption>Figure 11: Hyperparameters chosen</figcaption></figure><p>The summarise values of hyper-parameters used have been mentioned in table</p><h3>Results</h3><p>After the training of the networks was complete a quantitative analysis of the loss value and the steering angles predicted as well as a qualitative analysis of the nature of successful driving on the representative track in the autonomous mode was performed. This type of analysis was chosen because the loss values would give us the extent to which the model was trained. The predicted steering values are obtained from the trained model, where the model gives out the output based on the image from the front camera. This can be seen from the below figure</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*xXplonacaKyX-HJraHDqqg.png" /><figcaption>Figure 12: Field testing methodology</figcaption></figure><p>We recorded the steering angle information in order to compare the output for different models for the same track. Further, a qualitative analysis on the driving has been done in order to compare the smoothness of the drive when following the road.</p><h4>Final Loss value comparison</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8S8eWsQ4BmVk3tfaoh-WrA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*5QXoeAcX1MiSBEJ3yiCsBw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R9cZpXWnixI32mAG9i1PNw.png" /><figcaption>Figure 13: Yellow dataset loss trend (Left), Final loss values (middle) and Green dataset loss trend (right)</figcaption></figure><p>The table specifying the loss values and the graphs are presented. It is important to note that the aim in each case is to reduce the mean square error values, which is indicated by the training and validation losses. We make the following observations from the figure 13:</p><ol><li>The training and validation loss for green track dataset is higher than the corresponding losses of yellow track for all the models. This can be attributed to the level of complexity in the track consisting of sharp turns and also of adjacent paths which can be challenging for the model to learn.</li><li>In the case of the yellow track, the losses for the 5.55MB sized AIRSIM model is higher than JNet_ELU. This is possibly due to the requirement for more training EPOCHS for the AIRSIM, which has more parameters to train.</li><li>In the case of the yellow track, if we consider the values of the losses to the 2nd decimal place we find that the losses are similar.</li><li>In the case of the yellow track, 10 EPOCHS is enough for enabling a smooth drive in all models (as will be seen) but it is not enough for asymptotic convergence of the loss values.</li><li>In the case of the green track, JNet_ELU and JNet appear to give lower values of training and validation loss when compared to PilotNet and AIRSIM.</li><li>There is a downward trend of the loss for each model, which indicates that the model is learning the parameters and not stuck at a constant value.</li></ol><p>Considering the observations, particularly 2 and 4 we can infer that using a lighter JNet architecture and JNet_ELU, we can get similar and even better loss values for a straightforward as well as difficult track.</p><h4>Driving Quality comparison</h4><p>To compare the driving quality or the behaviour of the trained model on a real scenario where the agent is left to drive on its own, we first focus on the models ability to complete 1 lap which can take either yes or no values. When the model is able to complete 1 lap, we move onto determine the curve handling capability and keeping to the middle of the road, or the centre trajectory as it is called in the analysis. These factors can take either good, medium and bad parameter values. Thus by doing this, we compare the models based on its capability to complete the track in a satisfactory manner for 1 lap, which is also representative of the fact that it can complete the track for more laps as has been done in <a href="https://www.mdpi.com/1424-8220/19/9/2064">kocic</a>.</p><p><em>Yellow track</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/1*_Rq_NsLxXQF1VCc4YaFo9w.png" /><figcaption>Figure 14: Yellow track driving quality</figcaption></figure><p>We make the following observations:-</p><ol><li>All the tracks were able to complete the lap after being trained for only 10 EPOCHS.</li><li>The curve handling of the most models was better when compared to keeping the center trajectory.</li><li>AIRSIM gave the best performance amongst all the models. This is probably due to the high number of parameters.</li><li>JNet and JNet_ELU gave similiar driving behaviour.</li></ol><p><em>Green track</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/1*CJqTsKZnwK_PZmO7odDAVQ.png" /><figcaption>Figure 15: Green track driving quality</figcaption></figure><p>We make the following observations:-</p><ol><li>For a training of 10 EPOCHS and a learning rate of 10^-4, only JNet and JNet_ELU were able to complete the track for 1 lap.</li><li>JNet was able to keep to center of the trajectory better as compared to JNet_ELU.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*V4CZAlCzncpv_qtxJ-NoMw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*EAs8pYP0ajuwToX7uHjRlw.png" /><figcaption>Figure 16: More difficult regions of the green dataset</figcaption></figure><p>Further, it was observed that AIRSIM and PilotNet model agents would get stuck and not run at 2 points of the track as shown in figure 16. In these cases, we find that the common element is that there are 2 paths that are sensed by the CENTER camera. Due to this we infer that possibly the agent is not able to predict the right steering angle value. To tackle this occurence, we increased the number of EPOCHs to 50, however this did not improve the scenario and required the learning rate to be tuned. However as these 2 models were larger and could not be trained for the hyper-parameters of figure 11, we decided to continue with the analysis only for JNet and JNet_ELU.</p><h4>Steering value comparison</h4><p>To quantitatively compare the steering value predictions of the 4 models, we adopt 2 separate methods for yellow and green track respectively. This has been done due to the nature of the tracks and will be explained further below. It is important to note that a negative value of angle corresponds to steering to the left (and therefore positive for right).</p><p><em>Yellow track</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Ve8PVFg0I_onjLP_tCUXw.png" /><figcaption>Figure 17: Histogram plots of predicted steering values for yellow dataset</figcaption></figure><p>The comparison methodology used here relies on the fact that there are more instances of having a straight path in the yellow track and the car spends most of the time in a state where its steering value needs to be 0 or almost 0. Considering this idea, the histogram of the predicted steering values for each model has been plotted. We make the following observations:-</p><ol><li>In all cases, most of the predicted steering values lie in the centre of the approximately -2 to +1 region which indicates that all the 4 models were able to traverse the straight paths for a significant amount of time.</li><li>The steering values produced are in the range of -25 to 21 in the case of PilotNet, which is higher than the case of the other models which roughly lie in the range of -17 to 18. This also indicates that 10 EPOCHS is not enough to train the PilotNet and certainly more epochs are needed for a smooth journey.</li><li>The steering angle values and the numbers of such angles produced by JNET is similar to AIRSIM, which has more trainable parameters. This implies the functionality of traversing a flat track can be done by the lighter model.</li></ol><p><em>Green track</em></p><p>As the track had sharper turns and did not require the agent to maintain a zero prediction angle for majority of the time, therefore the previous quantitative analysis could not be performed. Instead, the generated predicted steering angle values for 1 lap was plotted as a function of when they arrive. This meant that the x-axis represents the progress of time. The result was only plotted for JNet and JNet_ELU as these models were able to complete the track based on the hyper-parameters mentioned in figure 11.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SSqeCfCvWMxZAm4A3cRdhw.png" /><figcaption>Figure 18: Sequence of generated steering values for JNet and JNet_ELU</figcaption></figure><p>The resultant plot is shown on above and we make the following observations:-</p><ol><li>The behaviour of the models is similiar for the first 1000 values generated.</li><li>If we observe the highlighted values carefully, it is observed that in the case of JNet_ELU, similiar values of predicted steering occur before the JNet value.</li></ol><p>The above implies that in the case of JNet, more steering prediction values are generated as compared to JNet_ELU. This helps JNet stick to the trajectory more and make it complete the track faster which can be seen in the graph, whereas more JNet_ELU prediction values are necessary to complete 1 lap.</p><h4>Inferences</h4><p>From the qualitative and quantitative study (loss values and predicted steering values) of the models we make the following inferences:-</p><ol><li>10 EPOCH is sufficient for all the models to learn simple tracks, like the yellow one. While 10 EPOCH’s works for lighter model, on more complicated tracks a higher number of EPOCHS or higher learning rate is required for larger models.</li><li>If space is the constraint, then models of a lesser size (JNet and JNet_ELU) can be used instead of (AIRSIM and PilotNet) for simple as well as complicated tracks.</li><li>JNet gives a smoother and more controlled trajectory than JNet_ELU in terms of curve handling and keeping center trajectory. Further, it also appears to complete the lap faster.</li></ol><h3>Ablation Study</h3><p>In order to get an in depth understanding of the possible improvements of modifying the JNet model, we include this ablation study which compares the performance of 4 variants of the Jnet model on the same dataset having the same hyper-parameters values of number of epochs, samples per epoch and the batch size during training. This study was conducted by using the yellow track dataset and having the batch size as 40, the samples per epoch as 12000, the learning rate as 10^-4 and the number of epochs for training as 10. It is important to recall that while the original JNet architecture consists of the ReLU activation with no dropout, the modified JNet_ELU is basically JNet + dropout + ELU activation function as can be seen from the architecture in figure 3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1007/1*-4ryv3sBuxZvJv8Zw_RU1Q.png" /><figcaption>Figure 19: Ablation study final loss and accuracy values</figcaption></figure><p>From the table, we observe that</p><ol><li>JNet_ELU has a very similar result to the JNet architecture with the relu activation.</li><li>As can be seen in JNet_ELU v/s JNet+ELU and JNet v/s JNet+dropout, when dropout is used the training loss increases, which indicates that the dropout is indeed preventing overfitting.</li><li>The validation and training accuracy values are almost similiar for each case</li></ol><p>From the above results, we get that the performance of the JNet_ELU is very similar to the JNet, with the dropout. It has a slight increase in training loss but shows a very similar validation loss and performs well in the autonomous mode in the simulator. Moreover the JNet model has a lesser training loss when compared to JNet_ELU. This is possibly the reason why this results in a better driving experience for the green dataset as seen in figure 15.</p><h3>Shortcomings</h3><p>In our implementation of the models, we noticed some shortcomings which can possibly be tackled as a further research study. These are:-</p><ol><li>Only steering value has been used as the output predicted value. As seen in the excel sheet snapshot in figure 9, a collection is Steering, throttle, reverse and speed is obtained along with the images. Thus, instead of only using the values of steering, a weighted combination can be used in order to get lower loss values.</li><li>The controller designed through this method can only be used in applications where the path is predefined and known. Applications like warehouse robots and cleaning robots that follow a set trajectory are examples of this. For scenarios where the agent is required to learn the environment on its own, path planning algorithms and possibly self-supervised learning need to be used.</li><li>In order to deploy actual autonomous vehicles into the field, this type of controller is only one part of the picture and needs to work in conjunction with other algorithms like image segmentation and object detection in order to detect pedestrians and other obstacles.</li><li>The code used to implement this model is quite old. In fact, for the deployment of files onto embedded devices, “.h5” files is not as popular as using frameworks like TensorFlow Lite. Updating the code from Keras to TensorFlow 2 and Pytorch would be needed in the future.</li><li>As deployment in resource constraint environments is the aim, the use of Binary Neural Networks can possibly be used in this application. This is quite possible as BNN’s have been used for developing size constraint sensors previously, like <a href="https://www.ceva-dsp.com/product/ceva-senspro/">Ceva SensPro</a>.</li></ol><h3>Conclusion</h3><p>The JNet model and its modification JNet_ELU of size 1.83MB has been proved to be a possible solution for low resourced hardware platforms in order to give perform autonomous driving over easy and difficult tracks. Moreover, when compared to models of higher complexity such as PilotNet (4.10MB) and AIRSIM (5.55MB), it achieves similar performance (and even better as seen for the difficult dataset) when trained for EPOCH values of 10 and 20. Overall, we were able to get very similar training loss and driving results of <a href="https://www.mdpi.com/1424-8220/19/9/2064">kocic</a> and extend it further to make it work on a harder scenario (green dataset).</p><p>Moreover, the project also introduces an easy to use training simulator that can be installed in order to test various models. The link to this installer and all the important code and results can be found in this <a href="https://github.com/ikvibhav/DeepLearning_Autonomous_Embedded_Agents">GitHub Link</a>. Further, we recorded the driving of the working models on the yellow and green datasets and it can be found here,</p><ol><li>JNet_ELU Green Dataset- <a href="https://www.youtube.com/watch?v=OvItOVO2eDI">https://www.youtube.com/watch?v=OvItOVO2eDI</a></li><li>JNet Green Dataset- <a href="https://www.youtube.com/watch?v=btGmuH96UdU">https://www.youtube.com/watch?v=btGmuH96UdU</a></li><li>JNet\_ELU Yellow Dataset- <a href="https://www.youtube.com/watch?v=eVxBlSb8NZ0&amp;t=62s">https://www.youtube.com/watch?v=eVxBlSb8NZ0&amp;t=62s</a></li><li>JNet Yellow Dataset- <a href="https://youtu.be/aq5Lt-jD0Tss">https://youtu.be/aq5Lt-jD0Tss</a></li><li>AIRSIM Yellow Dataset- <a href="https://youtu.be/hLaC4e5PjPg">https://youtu.be/hLaC4e5PjPg</a></li><li>PilotNet Yellow Dataset- <a href="https://youtu.be/qZGBDkntrx8">https://youtu.be/qZGBDkntrx8</a></li></ol><h3>References</h3><p>[1]: 7 key challenges impacting the autonomous vehicles. <a href="https://community.arm.com/developer/ip-products/system/b/embedded-blog/posts/7-key-challenges-impacting-the-mass-deployment-of-autonomous-vehicles">https://community.arm.com/developer/ip-products/system/b/embedded-blog/posts/7-key-challenges-impacting-the-mass-deployment-of-autonomous-vehicles</a></p><p>[2]: Udacity/self-driving-car-sim. <a href="https://github.com/udacity/self-driving-car-sim">https://github.com/udacity/self-driving-car-sim</a></p><p>[3]: End-to-End Deep Learning for Self-Driving Cars. <a href="https://developer.nvidia.com/blog/deep-learning-self-driving-cars/">https://developer.nvidia.com/blog/deep-learning-self-driving-cars/</a></p><p>[4]: Self-driving Car Using Udacity’s Car Simulator Environment And Trained By Deep Neural Networks. <a href="http://dspace.calstate.edu/bitstream/handle/10211.3/207785/Project_Report_fall_aditya_1.pdf?sequence=1">http://dspace.calstate.edu/bitstream/handle/10211.3/207785/Project_Report_fall_aditya_1.pdf?sequence=1</a></p><p>[5]: An End-to-End Deep Neural Network for Autonomous Driving Designed for Embedded Automotive Platforms. <a href="https://www.mdpi.com/1424-8220/19/9/2064">https://www.mdpi.com/1424-8220/19/9/2064</a></p><p>[6]: microsoft/AutonomousDrivingCookbook. <a href="https://github.com/microsoft/AutonomousDrivingCookbook/blob/master/AirSimE2EDeepLearning/TrainModel.ipynb">https://github.com/microsoft/AutonomousDrivingCookbook/blob/master/AirSimE2EDeepLearning/TrainModel.ipynb</a></p><p>[7]: Gentle Introduction to the Adam Optimization Algorithm for Deep Learning. <a href="https://machinelearningmastery.com/adam-optimization-algorithm-for-deep-learning/">https://machinelearningmastery.com/adam-optimization-algorithm-for-deep-learning/</a></p><p>[8]: Naokishibuya — Overview. <a href="https://github.com/naokishibuya">https://github.com/naokishibuya</a></p><p>[9]: Introduction to HDF5. <a href="https://support.hdfgroup.org/HDF5/doc/H5.intro.html">https://support.hdfgroup.org/HDF5/doc/H5.intro.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0e9836782d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>