<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Faraz Gerrard Jamal on Medium]]></title>
        <description><![CDATA[Stories by Faraz Gerrard Jamal on Medium]]></description>
        <link>https://medium.com/@farazlfc?source=rss-b0394b4821b5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*75Dh2653kI0IRtDD9oKpHQ.jpeg</url>
            <title>Stories by Faraz Gerrard Jamal on Medium</title>
            <link>https://medium.com/@farazlfc?source=rss-b0394b4821b5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 13:11:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@farazlfc/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Interview Questions (Quant Research)]]></title>
            <link>https://farazlfc.medium.com/interview-questions-quant-research-de85049be482?source=rss-b0394b4821b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/de85049be482</guid>
            <dc:creator><![CDATA[Faraz Gerrard Jamal]]></dc:creator>
            <pubDate>Wed, 19 Jul 2023 01:51:29 GMT</pubDate>
            <atom:updated>2023-07-19T02:28:14.639Z</atom:updated>
            <content:encoded><![CDATA[<p>P(X + 3Y &gt; 0 | X &gt; 0) where x and y are standard normal random variables. Solve via diagram, as bivariate has cylindrical symmetry so just take the required region!</p><p>what is the significance of Jacobian, what is the joint pdf of 2 iid Normals signifying -&gt; a circle and angle pdf multiplication basically.</p><p>Double money on every head and if tail you take away all the money, game ends. assumptions? infinite money for casino. How to take care of these assumptions, how to make it a fair game?</p><p>pricing of bonds -&gt; x dollars in x years, which one to pick. How to get cashflow or value for current year, of the bond!</p><p>KL Divergence -&gt; log weighted by p(x).</p><p>Logistic Regression Convex Optimization -&gt; Always goes to global minima, because of loss function&gt; What if we take MAE as loss function? If being off by 4 is twice as bad as being off from 2, then use MAE. MAE is less sensitive to outliers basically.</p><p>For linear regression, Look at the gradients for numerical stability! Probably standardize them to a unit vector and then do the updates.</p><p>What are the assumptions for MLE? I.I.D</p><p>The main difference is that MLE assumes that all solutions are equally likely beforehand, whereas MAP allows prior information about the form of the solution to be harnessed. With a uniform prior, MLE = MAP.</p><p><strong>Why is correlation related to linearity, the intuition?</strong> The product of regression coefficients is equal to correlation coefficient squared for simple linear regression.</p><p>Sum of errors for regression <strong>with an intercept term</strong> is always equal to zero.</p><p>Autoencoder project -&gt; We train an autoencoder on good credit card transactions, so reconstruction error on good credit card data is low while on high credit card is high.</p><p>Linear fleet of cars, keep adding one car after other, the expected number of fleets? If the new car is slowest out of the current available cars, then the number of fleets increases by 1 else it catches up and the number of fleets remain the same. E(n) = (1/n)*(E(n-1)) + (n-1)/n*(E(n-1) + 1) =&gt; E(n) = 1 + 1/2 + 1/3 + 1/4 + 1/5 +….</p><p>Double chess? Okay.</p><p>Expected draws until we see an ace? 52 cards 4 aces, so 5 partitions</p><p>In each partition there are 48/5 cards =&gt; 9.6 =&gt; 10.6th card is ace. This is equivalent to after you pick the last ace, how many cards will be remaining.</p><p>Simulate event of probability 1/3 using a coin? Easy :)</p><p>Probability of forming a triangle using 2 random cuts on rod? 1/4 :)</p><p>Same but second cut on larger part now? Take max concept, solve?</p><p>Inverse CDF method, the CDF -&gt; F(x) -&gt; U(0,1) as each value is seen once only, so it is a Uniform distribution with values between (0,1) due to the probability.</p><p>PCA -&gt; Projection of vectors and then calculating mean and variance for them, |a|cos(theta) 0&lt;cos(theta)&lt;1</p><p>One 18 sided dice vs three 6 sided dice -&gt; thin tailed is 6 sided dice.</p><p>Calculate mean and variance of U(0,1) -&gt; 0.5 is mean, variance is …E(X²) — E(X)²</p><p>Central limit theorem? When does it not hold? For samples &lt; 30, For not IIDs, for distributions with infinite variance (Cauchy distribution).</p><p>CLT is great for hypothesis testing. Because stats is all about estimating population parameters from samples. And given a hypothesis about the population, given a sample can we say the null hypothesis is true or no. if p value is less, then reject null hypothesis.</p><p>Law of Large Numbers ? sigma/sqrt(n)</p><p>SVM -&gt; Kernel trick! It helps with <strong>dot products for between points</strong> in a high dimensional space</p><p>ML -&gt; Trees (<strong>Do one side project on tabular data!!</strong>) What happens as we keep increasing the trees on random subsets of data!</p><p>For GB -&gt; It will go down eventually. In boosting, we form an ensemble of the learners.</p><p>For LinR -&gt; Goes up and saturates, effectively duplicating the data, just the confidence intervals will shrink, parameter estimates remain the same(from normal equation)! standard error reduces basically.</p><p>95% Confidence Interval for different samples -&gt; 95% of time it will contain the true value!</p><p>P value -&gt; reject the null hypothesis if it is lower than a threshold!</p><p>Why Cross validation does not work on Finance(time series) data? out of context</p><p>Binomial Distribution example -&gt; n coin flips with X being number of heads!</p><h3>Why Is Autocorrelation Problematic?</h3><p>Most statistical tests assume the independence of observations. In other words,<strong> the occurrence of one tells nothing about the occurrence of the other. Autocorrelation is problematic for most statistical tests because it refers to the lack of independence between values.</strong></p><p>simulate irrational probability using fair dice?</p><p>how to check for stationarity in time series?</p><p>Visually, global vs local…</p><p>Augmented dicky dueller?</p><h3><strong>Potential questions</strong></h3><blockquote>A fair die is rolled n times. What is the probability that the largest number rolled is r, for each r in 1..6?P(r =1) = 1/6^n</blockquote><blockquote>P(r ≤2) = (2/6)^n</blockquote><blockquote>P(r =2) = (2/6)^n — (1/6)^n and so on</blockquote><p>A fair coin is tossed n times. Given that there were k heads in the n tosses, what is the probability that the first toss was heads?</p><p>P(first heads|k tosses) = P(k heads|first head).P(first head)/P(k heads) =&gt; *n-</p><p>[(n-1ck-1)]/[n-1ck + (n-1ck-1)] =&gt; (n-1)!/(n-k)!(k-1)![(n-1)!/k!(n-1-k)!) + (n-1)!/(n-k)!(k-1)!] =&gt;</p><p>Probability that x³ ends with 11? 1/100 :) (a + 10b)³</p><p>the variance of the x-axis of points distributed uniformly on a unit ball in R3?</p><p>There are n points on the plane and then find three points to form the largest angle.</p><p>Markov chains!!! -&gt; You are on “0” index of a X line at the beginning. There is a black hole on 2N and another black hole on -N. At every position there’s equal probability of going left or right. When you arrive one of the black hole, the process is end. What’s the probability that you end up at the black hole at index 2N?</p><p>When to use cosine similarity and when to use Pearson correlation?</p><p>How to exponentiate a matrix? X = PDP^-1 so X^n = PD^nP^-1 as all the other terms cancel out :) <strong>Verify! And implement the nth power of a real number in O(logn) time using Divide and conquer.</strong></p><p><strong>@lru_cache</strong></p><p>def recurse(x,n):</p><p>if dp[n] is not None:</p><p>return dp[n]</p><p>if n == 0:</p><p>return 1</p><p>a = n//2</p><p>b = n — a</p><p>return dp[n] = recurse(x, n//2)*recurse(x,n//2)</p><p>Distribution with infinite variance? Pareto, cauchy!</p><p>Black Scholes- &gt;</p><p>covariance of linear combination of random variables -&gt; cov(ax,by) = abcov(x,y)</p><p>How would you detect outliers in a dataset? box plots, plot histograms, quantile plots.</p><p>Compare Lasso and Ridge regression.</p><p>Math behind l1 and l2, why l1 gives sparsity?</p><ol><li>Write python code to find max drawdown of trading strategy?</li><li>2) Compare two lotteries based on their expected payoff and standard deviation</li><li>3) Probability that n points on a circle are in one semicircle -&gt; n/2^(n-1)</li></ol><p>Explanation -&gt; select one point and rest is 1/2^n-1, now each of them can be the leading point.</p><ol><li>4) How to extract features using linear regression -&gt; Take example of least angle regression, first one with highest correlation to y. Then next with highest correlation to residuals ( as if residuals show a pattern, they are botched basically)</li><li>The St Petersburg paradox -&gt;</li><li>1. Sum of list of list — find sum of elements in cols and rows in all previous index.</li><li>2. if event X is getting 2 in a dice and if event Y is getting 3 in a dice, what is the covariance of X and Y if the dice is rolled 6 times. (-n/36 =&gt; -1/6)</li><li>3. If the head is thrown it is 1 and if the tail is thrown it is 2. After every throw, if you sum the values . What is the probability of getting 100 as sum at one point? Solve! Take cases and solve</li></ol><p>Calculate probability one of two events occurs!</p><p>Gaussian processes, how to address non-stationarity (not asked in those terms but probably what they want you to do), lag analysis (auto-correlation, partial auto-correlation), correlation analysis between series. They don’t ask things clearly like this but it’s what they expect candidates to do I think.</p><p>Give an example of a Poisson distribution.</p><p>ROC/AUC -&gt; Area under curve calculated for FPR-TPR (TPR on y axis, FPR on x axis) at different probability thresholds. <strong>TPR = Recall! (TP/(TP + FN)) -&gt; it should BR H</strong></p><p><strong>FPR = FP/(FP + TN)</strong></p><p><strong>Precision-recall curve </strong>is used for evaluating the performance of binary classification algorithms. <strong>It is often used in situations where classes are heavily imbalanced.</strong></p><p>SVMs are very good when you have tabular data with many features and few datapoints. They got quite famous on biomedical and gene research, which often has an overwhelming number of features and 20–50 datapoints per dataset.</p><p>More generally, think of SVMs of a “very good algorithm that is cheap to train and use”. They won’t be good at things like images or audio, but they can be your go-to tool for tabular data. The same goes for XGBoost and other similar algorithms.</p><blockquote>Mathematically, in support vector machines, the non-linear decision boundary is constructed as a linear combination of all the training data, so that a new point is classified using the following function:</blockquote><blockquote>y=sign(∑mi=1αiyiK(x,xi))�=����(∑�=1������(�,��))</blockquote><blockquote>where K� is a kernel function which measures the similarity between points x� and xi�� and αi�� are the dual variables.</blockquote><blockquote>Now, it turns out that often, only a few αi��’s are non-zero. Therefore, you only need to keep those αi��’s and the corresponding training points in memory.</blockquote><p>RBF kernel vs linear kernel?</p><p>Random forest is an extension of bagging! [incorporate random feature selection also]</p><p>Spectral clustering?</p><p>consecutive sum (Leetcode)</p><p>3 sum in python (Leetcode)</p><p>How to construct alphas using daily data? Can you list five factors that can predict the return?</p><p>Predictive power of model?</p><p>What is the GMM model?</p><p>What ‘s the difference between bagging and boosting? random forest is built on top of bagging.</p><p>OLS by hand? OLS assumption? double derivative &gt; 0</p><p>Simulate Monty hall in python?</p><p>3 doors, say car == 1, we select a door, then one door must be having a goat. monty opens that door, what is probability of winning after switching?</p><p>try out for 4 doors.</p><p>How to write a function that return 4 times the input, but can not use multiply and add operators?</p><p><a href="https://www.quora.com/If-we-generate-a-uniform-random-number-X-in-0-1-then-generate-a-uniform-random-number-Y-in-X-1-are-the-pairs-X-Y-uniformly-distributed">If we generate a uniform random number X∈[0,1]�∈[0,1], then generate a uniform random number Y∈[X,1], are the pairs (X,Y)uniformly distributed?</a></p><p>Not at all. If we take X and Xin (0.9, 1.0) that is all values are within 0.9,1.0 so dense within right corner.</p><p>But for X around lower (0,0.1) only 10% of those values are here.</p><p>Expected number of rolls of a fair six-sided die to get a 6, given all rolls leading up to that 6 are even numbered? 3/2</p><p>P(of getting 1 or 3) = 1/3 rest is 2/3 which is 6, better is to calculate updated probabilites and proceed.</p><p>Given three time series data X, Y, Z, build a model of X in terms of Y and Z? Can we use greater lag?</p><p>You are given an urn with 100 balls (50 black and 50 white). You pick balls from urn one by one without replacements until all the balls are out. A black followed by a white or a white followed by a black is “a color change”. Calculate the expected number of color changes if the balls are being picked randomly from the urn.</p><ul><li>&gt; expectation is linear, so there can be 99 color change slots basically. We have so 99*E(x1) = 99*50/99 = 50 color changes.</li><li>There are 51 ants sitting on top of a square table with side length of 1. If you have a square card with side 1/5, can you put your card at a position on the table to guarantee that the card encompasses at least 3 ants? <strong>Yes, Pigeonhole principle</strong></li><li>(updated: square card was originally disk of radius 1/7) ?</li></ul><p>A hedge fund has 70 employees. For any two employees <em>X</em> and <em>Y</em> there is a language that <em>X</em> speaks but <em>Y</em> does not, and there is a language that <em>Y</em> speaks but <em>X</em> does not. At least how many different languages are spoken by the employees of this hedge fund? (8 -&gt; 8C4 = 70) max of NCr occurs at r = n//2.</p><p>for svm margin is calculated only using data points having alpha &gt; 0 which are called as the support vectors. It is a LaGrange problem, solved for maximizing the margin equivalent to minimizing ||w||² which is convex in nature subject to the data points class labels.</p><p>Why are other kernels slower, because of creating n*n matrix to store similarity of data points as we are calculating dot products effectively.</p><p>What is put call parity?</p><p>future vs forward, <strong>futures are regulated and traded on an exchange</strong> unlike forwards which are highly customized.</p><p>all options in india are european options.</p><p>what is stop loss?</p><p>how are ipos issued?</p><p>A limit order <strong>is used to buy or sell a security at a pre-determined price and will not execute unless the security’s price meets those qualifications</strong>. more.What is a stop limit order example?</p><p>For example, if the current price per share is $60, the trader can set a stop price at $55 and a limit order at $53. <strong>The order is activated when the price falls to $55, but not below $53. Below $53, the order will not be fulfilled.[limit loss but not that desperate]</strong></p><p>limit order is order will not execute until it meets a standard.</p><p>adani shares plumetted.</p><p>what happened to silicon valley bank? credit suisse?</p><p>Let the grid be the same then how many paths are there from (0,0) to (8,6) with the same constraints of moving either UP or RIGHT. But add another constraint that the total number of turns taken should be even.</p><p>In a deck of 52 cards, 26 are reds and 26 are blacks. After shuffling the cards, the deck is divided into two stacks. What is the probability of the number of red cards in stack 1 is equal to the number of black cards in the stack 2 -&gt; <strong>Answer is 1, lol</strong></p><p>There are 26 black(B) and 26 red(R) cards in a standard deck. A run is number of blocks of consecutive cards of the same color. For example, a sequence RRRRBBBRBRB of only 11 cards has 6 runs; namely, RRRR, BBB, R, B, R, B. Find the expected number of runs in a shuffled deck of cards.</p><p><strong>answer is 27 , 51 slots given that last card always represents a run. So, 51*26/51 + 1 =&gt; 27!</strong></p><p>Intuition behind linearity of expectation/ independence?(IMP)</p><p>A stick is broken into 3 pieces, by randomly choosing two points along its unit length, and cutting it. What is the expected length of the middle part?(Derive this again, imp)</p><p>Given the set of numbers from 1 to n: { 1, 2, 3 .. n } We draw n numbers randomly (with uniform distribution) from this set (with replacement). What is the expected number of distinct values that we would draw?</p><p>Each number becomes an indicator variable basically and we see if it is picked in any draw or not.</p><p>Note that not all Markov chains have a stationary distribution. Some Markov chains may have absorbing states where the system eventually settles and remains in a specific state indefinitely, resulting in a lack of a stationary distribution.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de85049be482" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Let E(x) be the expected value of a game given that the last rolled i.e.,]]></title>
            <link>https://farazlfc.medium.com/let-e-x-be-the-expected-value-of-a-game-given-that-the-last-rolled-dice-i-e-51194a8e6b73?source=rss-b0394b4821b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/51194a8e6b73</guid>
            <category><![CDATA[probability]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[expected-value]]></category>
            <category><![CDATA[gambling]]></category>
            <category><![CDATA[quantitative-finance]]></category>
            <dc:creator><![CDATA[Faraz Gerrard Jamal]]></dc:creator>
            <pubDate>Fri, 05 May 2023 11:47:48 GMT</pubDate>
            <atom:updated>2023-05-05T12:05:20.965Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Say you have an n-sided die. You keep rolling and sum the values as long as the current roll is larger than the previous. For example, if n = 6 then you might roll 1, 3, 2 and stop or 1, 4, 5, 3 and stop. In the first case, your total sum is 1 + 3 = 4, and in the second case your total sum if 1 + 4 + 5 = 10. What is the expected value of the sum?</blockquote><p>Let E(x) be the expected value of a game given that the last rolled i.e., the max roll was x.<br>E(n) = 0 [As no number is greater than n, hence once we roll an n the game ends]<br>E(n-1) = 1/n*(E(n) + n)<br>E(n-2) = 1/n*(E(n-1) + n-1) + 1/n*(E(n) + n) and so on…<br>E(n-1) = 1<br>E(n-2) = 2</p><p>E(n-3) = 3</p><p>so E(1) = n-1<br>and <strong>E(0) = n. [Expected value of game with zero rolls]</strong></p><p>Let’s verify via a small simulation in Python :-</p><pre>import random<br>n_dice = 10<br>n_trials = 10000<br>mega = 0<br><br>for _ in range(n_trials):<br>  last = None<br>  while True:<br>   curr = random.randint(1,n_dice)<br>   if last is None or curr &gt; last:<br>     mega+= curr<br>     last = curr<br>   else:<br>     break<br>print(mega/n_trials) #9.9x for n_dice = 10</pre><p>And there is our verification :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=51194a8e6b73" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Jane street interview question]]></title>
            <link>https://farazlfc.medium.com/jane-street-interview-question-ab047e8337da?source=rss-b0394b4821b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab047e8337da</guid>
            <category><![CDATA[quantitative-research]]></category>
            <category><![CDATA[probability]]></category>
            <category><![CDATA[expected-value]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[quant]]></category>
            <dc:creator><![CDATA[Faraz Gerrard Jamal]]></dc:creator>
            <pubDate>Thu, 16 Mar 2023 13:08:34 GMT</pubDate>
            <atom:updated>2023-03-16T13:08:34.478Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Q. Suppose you are given the opportunity to bid for a treasure chest, which you know with 100% confidence to be priced anywhere between $0-$1000. If you bid equal to or above the price, you win the treasure chest (at the cost of your bid). If you bid below the price, you do not earn the treasure chest. Now, also suppose you have a friend who is willing to buy the treasure chest from you for one and a half times the price of the treasure chest (should you obtain the chest). What should your bid be?</blockquote><p>Let’s say we bid at a price ‘y’.</p><p>The payoff function at bid price y i.e. <strong>f(y)</strong>-&gt;</p><blockquote>1.5*x -y (for y ≥x)</blockquote><blockquote>0 (for x&gt;y)</blockquote><p>So Expected payoff at bid price <strong>y</strong> is =&gt; E(f(y))</p><p>=&gt; f(y,x).p(x) =&gt;</p><blockquote>integral from x = 0 to x = y of [(1.5*x — y)*(1/1000)*dx] (<strong>We know x is from U(0,1000)</strong>)</blockquote><p>=&gt; (0.75*y² — y²)*(1/1000) = -y²/4000</p><p>So no matter what the bid price, the expected payoff is going to be ≤ 0.</p><p>This is because in 1.5*x — y the negative region from x = 0 to x = 2/3 y <strong>overpowers</strong> the positive region from x = 2/3y to x = y.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab047e8337da" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Copying in Python]]></title>
            <link>https://farazlfc.medium.com/copying-in-python-860e17ff8216?source=rss-b0394b4821b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/860e17ff8216</guid>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Faraz Gerrard Jamal]]></dc:creator>
            <pubDate>Wed, 08 Mar 2023 12:44:19 GMT</pubDate>
            <atom:updated>2023-03-08T12:44:19.698Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Case 1</strong></h3><p>Say we have a list, x = [1, 2, 3, 4]</p><p>If we use y = copy(x) and y = deepcopy(x) for ‘x’, it will have the same effect.</p><h3><strong>Case 2</strong></h3><p>But if we have, x = [1, 2, [3], 4]</p><p>If we use y = copy(x) and y = deepcopy(x) for this, only deepcopy will have the desired effect.</p><p>For y = copy(x) in this case,</p><p>if we do y[2].append(1)</p><p>then x = [1, 2, [3, 2], 4] and y = [1, 2, [3, 2], 4]</p><p><strong>While for y = deepcopy(x),</strong></p><p>if we do y[2].append(1)</p><p>then x = [1, 2, [3], 4] and y = [1, 2, [3, 2], 4] which is the desired effect.</p><p>This is because if we use copy, the 2nd index of y still holds a reference to x’s object (the list at the 2nd index), while for deepcopy, even the list gets copied and a new address in memory is created for it during the deepcopy.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=860e17ff8216" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Variance of the number of coin flips until we see a head]]></title>
            <link>https://farazlfc.medium.com/variance-of-the-number-of-coin-flips-until-we-see-a-head-5978bb686ef3?source=rss-b0394b4821b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/5978bb686ef3</guid>
            <category><![CDATA[probability]]></category>
            <category><![CDATA[quantitative-research]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Faraz Gerrard Jamal]]></dc:creator>
            <pubDate>Mon, 06 Feb 2023 15:37:19 GMT</pubDate>
            <atom:updated>2023-02-06T15:37:19.113Z</atom:updated>
            <content:encoded><![CDATA[<p>[This article is for a fair coin, p (head) = 0.5]</p><p>There are different techniques we can use to calculate the variance of the number of coin flips until we see a head :-</p><p><strong>Technique #1 -&gt;</strong></p><p>We know, Var(n) = E(n²) — (E(n))²<strong> [:-n is number of flips until we see a head]</strong></p><p>We know, E(n) = 2 so we want to estimate E(n²)</p><p>E(n²) = 0.5*1² + 0.5*<strong>E((n+1)²)</strong> [<strong>:- As we have wasted one flip, we need to estimate expected value of E((n+1)²)</strong>]</p><p>In case of E(n) we do,</p><p>E(n) = 0.5*1 + 0.5*(E(n+1)) =&gt; 0.5 + 0.5*(E(n) + 1))</p><p>=&gt; E(n) = 2</p><p>Similarly, we have for the E(n²) case,</p><p>E(n²) = 0.5*¹² + 0.5*E((n+1)²)</p><p>x = 0.5 + 0.5*x + 0.5*1 + 2*0.5*1*y<strong> [Here x -&gt; E(n²), y-&gt; E(n) | (n+1)² = n² + 1 + 2*n and expectation is linear]</strong></p><p>0.5*x = 1 + 1*y</p><p>0.5*x = 3</p><p>x = 6</p><p>=&gt; Variance(n) = E(n²) — [E(n)]² = 6–4 = 2</p><p><strong>Technique #2 -&gt;</strong></p><p>Second way is that this is a geometric distribution ~ geom(p = 1/2)</p><p>We know variance of a geometric distribution is =&gt; (1-p)/(p²)</p><p>Therefore, variance is 2.</p><p><strong>Technique #3 -&gt;</strong></p><p>Third way is through writing it out as a series and then taking the difference twice =&gt;</p><p>E(n²) = 0.5*1² + (0.5)²*2² + (0.5)³*3²…. -&gt; (i)</p><p>0.5*E(n²) = 0.5²*1² + 0.5³*4 + 0.5⁴*9…… 0&gt; (ii)</p><p>(i) — (ii),</p><p>0.5*E(n²) = 0.5 + 0.5²(1)(3) + 0.5³*(1)(5) + 0.5⁴(1)(7) + 0.5⁵(1)(9)… -&gt; (iii)</p><p>0.5*0.5*E(n²) = 0.5² + 0.5³(3) + 0.5⁴(5) + 0.5⁵(7)… -&gt; (iiii)</p><p>(iii) — (iiii),</p><p>0.5*0.5*E(n²) = 0.5 + 2*(0.5² + 0.5³ + 0.5⁴…)</p><p>0.5*0.5*y = 0.5 + 2*(0.5²/0.5) <strong>[:- y =&gt; E(n²)]</strong></p><p>0.5*0.5*y = 1.5</p><p>y = 6 =&gt; E(n²) = 6</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5978bb686ef3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Probability that we see the pattern HHT before HTH in tossing a coin]]></title>
            <link>https://farazlfc.medium.com/probability-that-we-see-the-pattern-hht-before-hth-in-tossing-a-coin-ac096c9124a4?source=rss-b0394b4821b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac096c9124a4</guid>
            <category><![CDATA[probability]]></category>
            <category><![CDATA[quantitative-research]]></category>
            <category><![CDATA[quantitative-finance]]></category>
            <dc:creator><![CDATA[Faraz Gerrard Jamal]]></dc:creator>
            <pubDate>Mon, 02 Jan 2023 14:41:56 GMT</pubDate>
            <atom:updated>2023-01-02T14:42:52.650Z</atom:updated>
            <content:encoded><![CDATA[<p>Let p is the probability HHT comes before HTH. (STATE ZERO)</p><p>p = 1*(0.5)² (HH -&gt; WINNING SCENARIO) + 0*(0.5)³ (HTH -&gt; LOSING SCENARIO) + (0.5)³<em>p (HTT -&gt; GAME RESETS) + 0.5</em>P (T -&gt; GAME RESETS)</p><p>AFTER SOLVING, p = 2/3</p><p>A simple way of looking at it is this way, Once we flip an H,</p><p>Possible outcomes for next two tosses are HT -&gt; WINNING SCENARIO</p><p>HH -&gt; WINNING SCENARIO</p><p>TH -&gt; LOSING SCENARIO</p><p>TT -&gt; GAME RESETS</p><p>Therefore, 2 out of 3 outcomes(which give a result) are winning scenarios. Hence, p -&gt; 2/3</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac096c9124a4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Calculate values for a column in a row based on previous row’s column’s value for a PySpark…]]></title>
            <link>https://farazlfc.medium.com/calculate-values-for-a-column-in-a-row-based-on-previous-rows-column-s-value-for-a-pyspark-7dd92bb58d91?source=rss-b0394b4821b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/7dd92bb58d91</guid>
            <category><![CDATA[competitive-programming]]></category>
            <category><![CDATA[dataframes]]></category>
            <category><![CDATA[pyspark]]></category>
            <dc:creator><![CDATA[Faraz Gerrard Jamal]]></dc:creator>
            <pubDate>Wed, 17 Aug 2022 20:17:11 GMT</pubDate>
            <atom:updated>2022-08-19T18:44:40.009Z</atom:updated>
            <content:encoded><![CDATA[<h4>Calculate values for a column in a row based on previous row’s column’s value for a PySpark Dataframe</h4><p>Say, I have a column ‘val’ whose value gets calculated at each row and then the next row takes in that value and applies some logic on it, and then value for that row also gets updated. It can be shown as follows:-</p><p>val(x) = f(val(x-1), col_a(x), col_b(x)) where x is the row number (indexed at 0)</p><p>val(0) = f(col_a(0), col_b(0)) {some fixed value calculated based on two columns}</p><p>val(0) represents the first value in a <strong>partition</strong>.</p><p><strong>[ f here represents some arbitrary function]</strong></p><p>Let’s try using lag function as follows(for a sample dataframe):-</p><pre>windowSpec  = Window.partitionBy(&quot;department&quot;)<br>+-------------+----------+------+------+------+<br>|employee_name|department|   a  |   b  |  val |   <br>+-------------+----------+------+------+------+<br>|James        |Sales     |3000  |2500  |5500  | #val(0) =  (a(0) + b(0)) = 5500 [first value within a partition]<br>|Michael      |Sales     |4600  |1650  |750   | #val(1) = (a(1) + b(1) - val(0)) =  750<br>|Robert       |Sales     |4100  |1100  |4450  | #val(2) = (a(2) + b(2) - val(1)) =  4450 <br>|Maria        |Finance   |3000  |7000  |xxxx  | #....... and so on, this is how I want the calculations to take place.<br>|James        |Finance   |3000  |5000  |xxxx  |<br>|Scott        |Marketing |3300  |4300  |xxxx  |<br>|Jen          |Marketing |3900  |3700  |xxxx  |</pre><pre>df = df.withColumn(&quot;val&quot;,col(&quot;a&quot;) + col(&quot;b&quot;) - lag(&quot;val&quot;,1).over(windowSpec)) #Does not have the desired result.</pre><p>Tracking the previously calculated value from the same column is hard to do in spark — I’m not saying it’s impossible, and there certainly are ways (hacks) to achieve it. One way to do is using array of structs and aggregate function.</p><p>Consider two assumptions in your data :-</p><ul><li>There is an ID column that has the sort order of the data — spark does not retain dataframe sorting due to its distributed nature.</li><li>There is a grouping key for the processing to be optimized.</li></ul><p>The assumptions can be implemented using <strong>‘monotonically_increasing_id’</strong> and <strong>‘lit’</strong> functions respectively.</p><p># input data with aforementioned assumptions<br>data_sdf.show()</p><p># +---+---+-------+---------+----+----+<br># | gk|idx| name| dept| a| b|<br># +---+---+-------+---------+----+----+<br># | gk| 1| James| Sales|3000|2500|<br># | gk| 2|Michael| Sales|4600|1650|<br># | gk| 3| Robert| Sales|4100|1100|<br># | gk| 4| Maria| Finance|3000|7000|<br># | gk| 5| James| Finance|3000|5000|<br># | gk| 6| Scott|Marketing|3300|4300|<br># | gk| 7| Jen|Marketing|3900|3700|<br># +---+---+-------+---------+----+----+</p><p># create structs with all columns and collect it to an array<br># use the array of structs to do the val calcs<br># NOTE - keep the ID field at the beginning for the `array_sort` to # #work as required.<br>arr_of_structs_sdf = data_sdf. \<br> withColumn(&#39;allcol_struct&#39;, func.struct(*data_sdf.columns)). \<br> groupBy(&#39;gk&#39;). \<br>agg(func.array_sort(func.collect_list(&#39;allcol_struct&#39;)).alias(&#39;allcol_struct_arr&#39;))</p><p>This piece of code will turn it into a dataframe with a single column having a single row composed of a <strong>list of struct objects</strong>, where each struct object corresponds to a record in the original dataframe. func.array_sort enables us to keep the order same by using <strong>‘idx’</strong> as the key for sorting.</p><p># a function to create struct schema string<br>struct_fields = lambda x: &#39;, &#39;.join([str(x)+&#39;.&#39;+k+&#39; as &#39;+k for k in data_sdf.columns])</p><pre># use `aggregate` to do the val calc<br>arr_of_structs_sdf. \<br>    withColumn(&#39;new_allcol_struct_arr&#39;,<br>               func.expr(&#39;&#39;&#39;<br>                         aggregate(slice(allcol_struct_arr, 2, size(allcol_struct_arr)),<br>                                   array(struct({0}, (allcol_struct_arr[0].a+allcol_struct_arr[0].b) as val)),<br>                                   (x, y) -&gt; array_union(x, <br>                                                         array(struct({1}, ((y.a+y.b)-element_at(x, -1).val) as val))<br>                                                         )<br>                                   )<br>                         &#39;&#39;&#39;.format(struct_fields(&#39;allcol_struct_arr[0]&#39;), struct_fields(&#39;y&#39;))<br>                         )<br>               ). \<br>    selectExpr(&#39;inline(new_allcol_struct_arr)&#39;). \<br>    show(truncate=False)</pre><p>The ‘aggregrate’ function takes in an initial value and performs operations on it as we iterate through the list. In this case we take the <strong>intial value as the sum of column‘a’ and column‘b’ for the first row as column‘val’</strong>. Using <strong>slice</strong>, we make sure we start the calculation from the <strong>second row</strong> of the dataframe (after initializing the value using the first row of the dataframe)</p><p>Then we keep adding new elements to the list using <strong>array_union</strong> and refer to the most recent ‘val’ column’s value via -&gt;</p><pre>element_at(x, -1).val #&#39;-1&#39; points to the last element in the array</pre><p>Finally we use selectExpr(&#39;inline(new_allcol_struct_arr)&#39;)</p><p>to explode the list of structs and convert it into a dataframe as below :-</p><pre># +---+---+-------+---------+----+----+----+<br># |gk |idx|name   |dept     |a   |b   |val |<br># +---+---+-------+---------+----+----+----+<br># |gk |1  |James  |Sales    |3000|2500|5500|<br># |gk |2  |Michael|Sales    |4600|1650|750 |<br># |gk |3  |Robert |Sales    |4100|1100|4450|<br># |gk |4  |Maria  |Finance  |3000|7000|5550|<br># |gk |5  |James  |Finance  |3000|5000|2450|<br># |gk |6  |Scott  |Marketing|3300|4300|5150|<br># |gk |7  |Jen    |Marketing|3900|3700|2450|<br># +---+---+-------+---------+----+----+----+</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7dd92bb58d91" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Testing the waters of Bayesian Neural Networks(BNNs)]]></title>
            <link>https://becominghuman.ai/testing-the-waters-of-bayesian-neural-networks-bnns-6a1fb5261228?source=rss-b0394b4821b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a1fb5261228</guid>
            <category><![CDATA[bayesian-machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Faraz Gerrard Jamal]]></dc:creator>
            <pubDate>Wed, 18 Nov 2020 22:08:59 GMT</pubDate>
            <atom:updated>2020-11-25T15:29:48.043Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*nTNJB0MkUtTJ62Qj2Rk61Q.jpeg" /></figure><p>Intelligence boils down to two things for me -&gt;</p><p>1. <strong>Acting when certain/necessary.</strong></p><p><strong>2. Not acting/staying pensive when uncertain.</strong></p><p>Point <strong>(2.)</strong> is what we are going to dive in!</p><p>Uncertainty is inherent everywhere, nothing is error free. So, it is frankly quite surprising that for most Machine Learning projects, gauging uncertainty isn’t what’s aimed for!</p><h3>Trending AI Articles:</h3><blockquote><a href="https://becominghuman.ai/how-to-automatically-deskew-straighten-a-text-image-using-opencv-a0c30aed83df">1. How to automatically deskew (straighten) a text image using OpenCV</a></blockquote><blockquote><a href="https://becominghuman.ai/explaining-yolov4-a-one-stage-detector-cdac0826cbd7">2. Explanation of YOLO V4 a one stage detector</a></blockquote><blockquote><a href="https://becominghuman.ai/5-best-artificial-intelligence-courses-for-beginners-in-2020-38db08617b12">3. 5 Best Artificial Intelligence Online Courses for Beginners in 2020</a></blockquote><blockquote><a href="https://becominghuman.ai/a-non-mathematical-guide-to-the-mathematics-behind-machine-learning-4b71d9e16dcd">4. A Non Mathematical guide to the mathematics behind Machine Learning</a></blockquote><p>As a not-so-real-world example, consider a <strong>treatment recommendation </strong><a href="http://becominghuman.ai/"><strong>algorithm</strong></a>. We are given a patient’s medical data that we feed into our network. For a plain <a href="https://becominghuman.ai/cheat-sheets-for-ai-neural-networks-machine-learning-deep-learning-big-data-678c51b4b463">NN</a>(<a href="https://becominghuman.ai/keras-cheat-sheet-neural-networks-in-python-738c0d170c2e">Neural Network</a>), it would just output a single class, say treatment type ‘C’. While for BNNs you would be able to see the whole distribution of the output, and gauge the confidence of your output label based on that. If the <strong>standard deviation is low </strong>of your output distribution<strong>, </strong>we are good to go with that output label. Otherwise, chuck it, we need human intervention.</p><p>So, BNNs are different from plain neural networks in the sense that <strong>their weights are assigned a probability distribution instead of a single value or point estimate</strong>. Hence, we can assess the uncertainty in weights to estimate the uncertainty in predictions.<strong> If your input parameters are not stable, how can you expect your output to be?!</strong> Makes sense, eh.</p><figure><a href="https://aijobsboard.com/?fbclid=IwAR0GcsLs8A6CFM-fynrZMM3sTgo_Zpbto2CjglJm0Dmi6otC6YWc7CDFpQk"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dAlwXfVrtYspdCzmC8iADw.png" /></a><figcaption>Artificial Intelligence Jobs</figcaption></figure><p>Say you have a parameter, you try to estimate its distribution and using the <strong>distribution’s high probability points </strong>you estimate the output value of your neural network. By high probability, I mean the more probable points. <strong>(Mean is the most probable point in a Normal distribution)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/1*Neh0aKQny7k6cOhyx5CdKw.png" /><figcaption><strong>Equation #1</strong></figcaption></figure><p>Looking at <strong>Equation #1,</strong></p><p>On the left hand side, we have the <strong>probability distribution of the output classes</strong>, which we get after feeding in our data ‘x’ to our model which has been trained on Dataset ‘D’.</p><p>On the right hand side, we have an <strong>integral</strong>. The <strong>middle term</strong> inside the integral is the <strong>posterior</strong>, which is a <strong>probability distribution of the weights GIVEN we have seen the data.</strong></p><p>Think of this integral as an ensemble, <strong>where each unique model is identified by a unique set of weights (because different weights means different models). </strong>Greater the posterior probability for a unique set of weights(therefore that unique model), greater weight will be given to that prediction. Hence, <strong>each model’s respective prediction is weighed by the posterior probability of it’s unique set of weights!</strong></p><p>Sounds good, eh? Just search the whole weight space and weigh in the <strong>good(high probability)</strong> parts. Wondering why people don’t do this? It is because even a simple 5–7 layer NN has around a <strong>million weights</strong>, so it is just not computationally feasible to construct an analytical solution for the <strong>posterior</strong>(p(<strong>w</strong>|D)) in <a href="https://becominghuman.ai/making-a-simple-neural-network-2ea1de81ec20">NNs</a>.</p><p>So the next step is, we need to approximate the posterior distribution. We cannot get the exact posterior distribution, but we surely can choose another distribution that replicates it to a good extent.</p><p>We can do this using a variational distribution whose <strong>functional form is known</strong>! By <strong>‘functional form is known’</strong>, I mean it is one of the standard statistical distributions which can be <strong>denoted</strong> using just a <strong>few parameters</strong>, like the <strong>Normal distribution</strong>( <em>We just need </em><strong><em>two</em></strong><em> parameters, i.e. the </em><strong><em>mean</em></strong><em> and </em><strong><em>variance</em></strong><em>, to denote a Normal distribution</em>). So, <strong>we are essentially trying to form a Normal distribution replica of the posterior distribution</strong>. Even though the actual posterior distribution might not be a Normal distribution, we are still going to replicate it as well as we can using the variational distribution, which is a Normal distribution in our case. <strong><em>You can pick any standard statistical distribution for the variational distribution.</em></strong></p><p>How do we go about replicating it? I’ll cover that and more in the next article. Hope this works up an appetite for the world of Bayesian Machine Learning! It’s a beautiful topic and one that has got a lot of exploring to do.</p><p>So yeah, see you!</p><h3>Don’t forget to give us your 👏 !</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*2lvCls4yjxVMfZSR" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2F8f5f8b%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2F8f5f8b&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/c43026df6fee7cdb1aab8aaf916125ea/href">https://medium.com/media/c43026df6fee7cdb1aab8aaf916125ea/href</a></iframe><figure><a href="https://becominghuman.ai/artificial-intelligence-communities-c305f28e674c"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*2f7OqE2AJK1KSrhkmD9ZMw.png" /></a></figure><figure><a href="https://upscri.be/8f5f8b"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*v-PpfkSWHbvlWWamSVHHWg.png" /></a></figure><figure><a href="https://becominghuman.ai/write-for-us-48270209de63"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*Wt2auqISiEAOZxJ-I7brDQ.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a1fb5261228" width="1" height="1" alt=""><hr><p><a href="https://becominghuman.ai/testing-the-waters-of-bayesian-neural-networks-bnns-6a1fb5261228">Testing the waters of Bayesian Neural Networks(BNNs)</a> was originally published in <a href="https://becominghuman.ai/">Becoming Human: Artificial Intelligence Magazine</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>