<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mohd Amaluddin Yusoff on Medium]]></title>
        <description><![CDATA[Stories by Mohd Amaluddin Yusoff on Medium]]></description>
        <link>https://medium.com/@amaluddin11?source=rss-90c0357d2562------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*pww2MfOFO78w8x9y.jpg</url>
            <title>Stories by Mohd Amaluddin Yusoff on Medium</title>
            <link>https://medium.com/@amaluddin11?source=rss-90c0357d2562------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 03:52:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@amaluddin11/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why Are We So Tired?]]></title>
            <link>https://medium.com/@amaluddin11/why-are-we-so-tired-ffde43d75658?source=rss-90c0357d2562------2</link>
            <guid isPermaLink="false">https://medium.com/p/ffde43d75658</guid>
            <dc:creator><![CDATA[Mohd Amaluddin Yusoff]]></dc:creator>
            <pubDate>Fri, 16 Oct 2026 07:41:43 GMT</pubDate>
            <atom:updated>2026-10-16T07:41:43.154Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PptuNKKUiwIx8GxydbAw2Q.png" /><figcaption><strong>Why Are We So Tired?</strong></figcaption></figure><p><em>Perhaps we have been looking for the solution in the wrong places.</em></p><p>After spending eight hours working in front of a computer, we often feel exhausted.</p><p>But why?</p><p>We have hardly done any physical activity compared with, say, a farmer working in the field all day.</p><p>Perhaps tiredness is just a feeling, like happiness or sadness. We feel tired, but that does not necessarily mean we have depleted our physical energy.</p><p>Yet we behave as though we have.</p><p>We eat to replenish our energy, sometimes consuming more calories than we have expended. We rest, watch television, or seek entertainment to make ourselves feel better.</p><p>But what if we have misunderstood the problem?</p><p>Perhaps our tiredness has less to do with physical energy and more to do with what occupies our mind.</p><p>And perhaps the solution is not simply to disengage from work, but to reconsider why we work in the first place.</p><p>There is a familiar saying:</p><p><em>Choose a job you love, and you will never have to work a day in your life.</em></p><p>But I would rephrase it.</p><p><strong>First, identify your deepest desire. Then find work that aligns with it.</strong></p><p>Not your immediate desires. Not your career ambitions. Not even what makes you happy today.</p><p>Your <strong>deepest</strong> desire.</p><p>Keep asking yourself why.</p><p>Why do you want wealth? Perhaps for freedom.</p><p>Why do you want freedom? Perhaps for peace.</p><p>Why do you want peace?</p><p>Keep asking, until you arrive at something you desire for its own sake.</p><p>Perhaps your deepest desire is knowledge. Or power. Or love. Or peace. Or life itself.</p><p>Now imagine someone whose deepest desire is knowledge.</p><p>This person obtains ten PhDs from MIT, Cambridge, LSE, and other prestigious universities. Reads every book in the ancient Library of Alexandria.</p><p>Would this person finally be satisfied?</p><p>Perhaps not. There is always more to know.</p><p>What if the desire for knowledge can only be ultimately fulfilled by the Most Knowledgeable?</p><p>Similarly, imagine someone who desires power. This person eventually controls the entire world.</p><p>Would that be enough?</p><p>Or would the desire remain, pointing towards the Most Powerful?</p><p>What about love, peace, or life itself?</p><p>Could it be that all these apparently different desires are pointing towards the same ultimate desire? <strong>The One. Only One.</strong></p><p>Perhaps our deepest desire has always been the same, although we express it through different pursuits.</p><p>And perhaps the question is no longer whether we love our work.</p><p>It is whether our work aligns with our deepest desire.</p><p>This does not mean we will never experience physical or mental fatigue. But it may fundamentally change how we experience our responsibilities, difficulties, and daily activities.</p><p>We began by asking why we feel tired after working all day.</p><p>Perhaps the more important question is:</p><p><strong>Do we truly know why we are working in the first place?</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ffde43d75658" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Hidden Bottleneck in Data Science (Part 2)]]></title>
            <link>https://medium.com/@amaluddin11/the-hidden-bottleneck-in-data-science-part-2-b6fb32e281db?source=rss-90c0357d2562------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6fb32e281db</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-management]]></category>
            <dc:creator><![CDATA[Mohd Amaluddin Yusoff]]></dc:creator>
            <pubDate>Tue, 28 Jul 2026 09:17:28 GMT</pubDate>
            <atom:updated>2026-07-28T09:17:28.366Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>How Data Management Fixes the SME Dependency Problem</strong></p><p>In <a href="https://medium.com/@amaluddin11/the-hidden-bottleneck-in-data-science-part-1-74bbfe58179b">Part 1</a>, we looked at a pattern: a business use case comes in, the data sources are known, but the relevant data points aren’t, so the team leans entirely on a Subject Matter Expert to point the way. We walked through three problems this creates: the SME’s mental model sets a <em>ceiling</em> on what can be found, the process narrows the data before exploration even starts (the chicken-and-egg problem), and the SME becomes a single point of dependency with no independent way to verify their word. The uncomfortable conclusion was that without something to counterbalance or move beyond this, the initiative isn’t really data-driven, no additional gained from the wealth of existing data. It is just digitizing SME’s opinion or manual processes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O-c-VVrp-dqgJt76E-cPEQ.png" /><figcaption>How Data Management Fixes the SME Dependency Problem</figcaption></figure><p>None of this is a modeling problem. It’s a data management problem. Here’s how real data management practices address it.</p><p><strong>1. Data Cataloging &amp; Metadata Management — directly addresses the SME ceiling</strong></p><p>The core issue in Part 1 was that “relevant” data only meant “data the SME happens to be using.” A data catalog — a living, searchable record of what data exists, what each field actually means, who owns it, and how it’s been used across the organization — breaks that ceiling.</p><p>Instead of relying on one person’s mental model of the past, the data scientist can search and explore the <em>organization’s</em> collective usage history. Data points that are relevant but never came up in an SME conversation — because no one individual had a reason to connect them to this use case — become discoverable on their own. This doesn’t remove the SME from the process. This gives the data scientist an independent way to explore beyond SME’s knowledge.</p><p><strong>2. Data Profiling — directly addresses the chicken-and-egg problem</strong></p><p>Part 1’s second issue was that the SME is asked to define “relevant” <em>before</em> any exploration has happened, which pre-narrows the search. Data profiling — automatically surfacing the distribution, null rate, cardinality, correlations, and anomalies of a data point — lets a data scientist start exploring the data <em>independently and in parallel</em> with the SME conversation, rather than waiting on it.</p><p>When profiling already exists as routine infrastructure rather than a one-off task, the data scientist doesn’t arrive at the SME conversation empty-handed asking “what’s relevant?” — they arrive with their own early findings and can ask, “here’s a pattern I noticed, does this line up with what you know?” Exploration and expertise now run side by side. This is how a data scientist can truly contribute to a business use case, not just coding SME’s knowledge.</p><p><strong>3. Data Lineage — directly addresses the single point of dependency</strong></p><p>Part 1’s third issue was that the SME’s word becomes the dataset’s boundary with no way to independently verify it. Data lineage solves exactly this: it traces a field back through its source systems and every transformation applied to it, turning “I believe this is what it means” into “here is exactly where this came from and what happened to it along the way.”</p><p>With lineage in place, an SME being unavailable, mistaken, or in conflict with another SME is no longer a dead end — there’s a second, independently verifiable source of truth sitting alongside their input.</p><p>The three practices above map directly onto the three risks from Part 1. But mature data management also brings benefits that Part 1 didn’t raise directly — worth mentioning because they compound the effect above rather than standing apart from it.</p><p><strong>Additional benefit: Master Data Management — makes “relevant” data actually trustworthy</strong></p><p>Even a correctly identified, genuinely relevant field can quietly produce misleading insights if the underlying entities are inconsistent — the same equipment, customer, or staff name appearing as multiple different records across systems. Resolving these entities at the source means that once relevant data is identified, it can actually be relied on.</p><p><strong>Additional benefit: Governance with a feedback loop — knowledge compounds instead of resetting</strong></p><p>One risk we touched on but didn’t dwell on in Part 1: whatever a data scientist discovers during a project — gaps, ambiguities, a newly relevant data point nobody flagged — usually dies with that project if there’s no mechanism to record it. Mature governance builds in a path for that discovery to flow back into shared documentation. Each project can then make the <em>next</em> one less dependent on finding and interviewing the right SME from scratch.</p><p><strong>The Shift This Creates</strong></p><p>Put together, these practices change the fundamental shape of the process — from insight discovery bottlenecked by one person’s knowledge, to insight discovery bounded by what the organization collectively knows and can systematically expose.</p><p>The SME still matters — tacit, hard-won operational knowledge will never be fully captured in a catalog. But with strong data management underneath, the SME becomes one strong input, not the single ceiling on what a project can discover. That’s the difference between digitizing an opinion and actually doing data science.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6fb32e281db" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Hidden Bottleneck in Data Science (Part 1)]]></title>
            <link>https://medium.com/@amaluddin11/the-hidden-bottleneck-in-data-science-part-1-74bbfe58179b?source=rss-90c0357d2562------2</link>
            <guid isPermaLink="false">https://medium.com/p/74bbfe58179b</guid>
            <category><![CDATA[data-driven-insights]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-management]]></category>
            <dc:creator><![CDATA[Mohd Amaluddin Yusoff]]></dc:creator>
            <pubDate>Wed, 15 Jul 2026 14:28:20 GMT</pubDate>
            <atom:updated>2026-07-15T14:28:20.028Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Why “Just Ask the SME” Isn’t a Strategy</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AFpd4jyoYfahSi3a_Iw3fw.png" /><figcaption>Digitizing SME’s expertise is not a data-driven initiative.</figcaption></figure><p>Every data science project starts with a version of the same conversation.</p><p>The business has a use case. Maybe it’s predicting production rate, optimizing a supply chain, or flagging anomalous operations. Everyone agrees it’s valuable. The data sources are known — the CRM, the production system historian, scheduling system. What’s not known is exactly which data points, buried inside those sources, are actually relevant to the question at hand.</p><p>So the team does what teams almost always do: they find a Subject Matter Expert (SME) and ask them to point the way.</p><p>This is not only a normal step but many times is the only feasible step. Let’s look at what’s actually happening here, because this routine move quietly shapes everything that follows.</p><p><strong>The SME Is Doing You a Favor — and Setting a Ceiling</strong></p><p>SMEs are valuable for a real reason: they carry tacit knowledge that no schema or data dictionary captures.</p><p>But there’s a quieter problem hiding underneath that value: <strong>an SME can only point you to what they already know is relevant.</strong></p><p>Think about what that actually means. The SME’s mental model of “relevant data” was built from their own experiences, from their historical workflows, from their manual workflows — the analysis they have done, the reports they’ve run, the decisions they’ve had to make. That’s a real and useful mental model. But it’s a model of the past, not a model of the use case’s full possibility space, not a model based on full spectrum of the available data. Data points that are relevant but have never been used before simply don’t appear on it — not because they don’t matter, but because no one has ever had a reason to ask or explore.</p><p><strong>The Chicken-and-Egg Problem</strong></p><p>There’s a deeper structural issue here too. The team is asking the SME to define “relevant data” before the analysis has actually started — before anyone has explored the data, tested a hypothesis, or found a pattern worth chasing.</p><p>For well-established use cases, this is usually fine. But for anything genuinely exploratory — anything where the insight itself is the point of the project — this approach quietly narrows the data. You’re defining the boundaries of the search before you’ve searched. No new discovery. No additional insights from the data.</p><p><strong>A Single Point of Dependency</strong></p><p>There’s also a simple operational risk: the SME becomes a bottleneck, and worse, a single point of failure.</p><p>What happens if the SME is unavailable? What happens if they’re wrong about something — not maliciously, just because memory is imperfect and systems change faster than any one person can track? In most projects, there’s no independent way to check. The SME’s word becomes the dataset’s boundary, becomes the most authoritative source, might even more than the data itself! Nobody has the visibility to challenge it.</p><p><strong>The Real Cost: You Haven’t Done Data Science, You’ve Digitized an Opinion</strong></p><p>When the data boundary for a project is drawn entirely by what one SME already believes is relevant, the resulting model can’t discover anything the SME didn’t already know. At best, it quantifies their intuition, expressed more precisely and applied more consistently than a human could by hand.</p><p>That’s not nothing — consistency and precision have real value. But it’s worth being honest about what it actually is: <strong>you haven’t built a data-driven initiative. You’ve digitized an SME’s expertise.</strong> The data pipeline, the model, the dashboard — all of it becomes an elaborate translation layer, converting one person’s existing mental model into code. The initiative looks data-driven because it involves data, algorithms, and a technical team. But the ceiling on what it can find was defined by the SME. No additional insights gained.</p><p>Genuinely data-driven work is supposed to do the opposite: let the data surface things nobody has articulated yet — a pattern the SME never noticed because it never showed up in the reports, a relationship between two columns that the SME never had reason to connect. If the entire discovery process is pre-filtered through one person’s existing map, that possibility is closed off before it can ever occur.</p><p>The good news is that this isn’t an unsolvable problem — it’s a <strong>data management problem</strong>.</p><p>In Part 2, we’ll look at exactly how strong data management practices — cataloging, profiling, lineage, governance — replace this fragile, SME-dependent process with something far more robust<strong>.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=74bbfe58179b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Credit Card Fraud Detection]]></title>
            <link>https://medium.com/@amaluddin11/credit-card-fraud-detection-7e47750db863?source=rss-90c0357d2562------2</link>
            <guid isPermaLink="false">https://medium.com/p/7e47750db863</guid>
            <category><![CDATA[fraud-detection]]></category>
            <category><![CDATA[autoencoder]]></category>
            <category><![CDATA[anomaly-detection]]></category>
            <category><![CDATA[credit-cards]]></category>
            <category><![CDATA[one-class-svm]]></category>
            <dc:creator><![CDATA[Mohd Amaluddin Yusoff]]></dc:creator>
            <pubDate>Fri, 01 Nov 2019 13:50:52 GMT</pubDate>
            <atom:updated>2019-11-01T13:50:52.641Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0s-4yieqpC3FXzgFCfZBsw.jpeg" /><figcaption>Image from cardozalawcorp.com</figcaption></figure><p>An investigation on One-class SVM and Autoencoders</p><p><strong><em>Did you know that global card losses are expected to exceed $35 billion by 2020?</em></strong></p><p>Do I have to worry? Yes! because 65% of the time, credit card fraud results in a direct or indirect financial loss to you, the user. By 2020, we are losing $22.75 billion. This is our hard-earned money!</p><p>Let’s dive into two techniques to detect fraud namely <strong>one-class SVM</strong> and <strong>autoencoders</strong>. We want to compare their performances in terms of <strong>Recall</strong>, <strong>Accuracy</strong>, <strong>Precision </strong>and <strong>F1-score</strong>.</p><p><strong>The data<br></strong>The <a href="https://www.kaggle.com/mlg-ulb/creditcardfraud">datasets </a>contains transactions made in September 2013 by European cardholders. This dataset presents transactions that occurred in two days, where we have <strong>492</strong> frauds out of <strong>284,807</strong> transactions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/258/1*9-QwQJnj-dqZwLzw7OC6Aw.png" /><figcaption>Highly unbalanced dataset</figcaption></figure><p>The dataset is highly unbalanced, the positive class, Class 1 (frauds) accounts for only <strong>0.173%</strong> of all transactions. Because of this, <strong>Recall </strong>score is given more emphasize than the other scores i.e., we want to capture as many frauds as possible. Of course, in doing this we would mistakenly classify normal and valid transactions as fraudulent.</p><h3><strong>One-class SVM</strong></h3><p>One-class SVM model is trained using a single class information unlike the traditional classification problem where the model is trained with labelled data with at least two classes. In this case, the model is trained with Class 0 data, normal transactions of the training data which was 80% of the whole dataset. The remaining 20% is the testing data, reserved to test the model.</p><p>In this particular investigation, although we have more than 200,000 data points to train the model, only 50,000 data points were used due to limitation of my computing power.</p><p>Here are the performance scores when tested with testing data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/1*Uz9t44idrZZZSz-DGzv_Vw.png" /></figure><ul><li>Although Recall for one-class SVM is 86%, the precision is only 1.5%. In fact, although it correctly identified 82 out of 95 frauds, it wrongly identified 5542 normal transactions as fraudulent.</li></ul><h3><strong>Autoencoder</strong></h3><p>Autoencoder is a neural networks that reconstructs input signal as its output. It has encoder and decoder to encode features from the inputs and decode the features to produce the output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*CRP98rEvxpcdo6gJI3tXWA.png" /><figcaption>Image from curiousily.com</figcaption></figure><p>Of course, there will be some differences between original inputs and the outputs, the reconstructed inputs (see original and represented mushrooms in the picture above). This difference is called reconstructed error.</p><p>In this investigation, 2 layers of encoder of size 32 and 16, and 2 layers of decoder of size 16 and 32 were used.</p><p><strong><em>How to make prediction using an encoder?</em></strong></p><p>Prediction is made based on the reconstruction error i.e., if the error is above a certain threshold the data is considered an anomaly or in this case a fraud. If the error is lower than the threshold, then the transaction is normal. Thus, we have to carefully choose a threshold to achieve good performance.</p><p>Let’s first look at error distribution for the two classes, fraud and normal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/292/1*KlQzet90IlCggW61DGzaGw.png" /><figcaption>Distribution of Reconstruction Error</figcaption></figure><ul><li>From the error distribution, all values for normal are lower than of for fraud which is intuitively correct, except for the max. The value for max is higher probably because of the highly unbalanced dataset.</li></ul><p>Based on the error distribution above, threshold value of 3 was selected and here is the performance of the autoencoder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*RK55BToMqEL09V-8535Tow.png" /></figure><ul><li>Although the Recall score is lower than that of one-class SVM, the other metrics are higher.</li></ul><p>Since the metrics of the autoencoder depend on the threshold of the reconstruction error, let’s investigate the relationship between the scores and the error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*N8VUFsj9G1V9SVmh9iaCfg.png" /></figure><p>Ideally we would like to have high precision and high recall. I wonder if achieving this is possible at all for the autoencoder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*aYbj0QX2DPkSBjQqES-PTw.png" /></figure><p>We can see from the plot that …</p><ul><li>Precision and recall depend on the error threshold.</li><li>As the threshold increases, the recall decreases and precision increases.</li><li>At a very low threshold, although recall is high, precision is low.</li><li>At a very high threshold, although precision is high, recall is low.</li></ul><h3>Can we make prediction using z-score of the error instead of the threshold?</h3><p>Let’s consider a transaction a fraud when its z-score is smaller then normal’s z-score i.e., we first compute two z-scores based on mean and standard deviation of each error distributions, for both fraud and normal. Then, if z-score for fraud is smaller than z-score for normal, the transaction will be considered fraudulent.</p><p>Here is the performance of the autoencoder using z-score, and previous results for easier comparison.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*HG2ovq97SzHdQrtJi7R73g.png" /></figure><ul><li>The prediction scores obtained when using z-score are similar with the scores obtained using error threshold with threshold value equals 3.</li></ul><h3>DISCUSSION</h3><ul><li>From the table above, although autoencoders perform better than one-class SVM except for Recall score, their performance in general are similar.</li><li>To make predictions, One-class SVM is more straightforward than antoencoder to use. This is because an autoencoder reconstructs the input as its output, hence decision has to be made how to use the reconstruction error to make predictions.</li><li>Comparing the two antoencoder models, their performance is quite similar. It is interesting that the method using z-score of the reconstruction error has similar performance with manually selected threshold error method. This would eliminate the need for experimenting with the threshold values for making predictions.</li><li>Note that not all data is used to train the one-class SVM model due to high computing power required whereas the entire training data was used to train the neural networks. It would be interesting to see if the performance of the one-class SVM would be different if we use all training data.</li></ul><h3>CONCLUSION</h3><p>We have explored credit card transactions data and use it to develop predictive model to detect fraudulent transactions. Two models have been investigated namely one-class SVM and antoencoder. The performance of the models have been compared and discussed. The relationship of Precision against Recall for the autoencoder and their dependencies on the reconstruction errors have been investigated.</p><p>The most interesting point from this project is that z-score of the reconstruction error can be used to make prediction for the autoencoder with similar performance compared with the method using a carefully and manually selected error threshold.</p><p><em>More detailed analysis is available at my </em><a href="https://github.com/Amaluddin11/Credit-Card-Fraud"><em>Github</em></a><em>.</em></p><p>Links:<br><a href="https://www.idexbiometrics.com/top-7-alarming-facts-and-statistics-on-card-fraud/">https://www.idexbiometrics.com/top-7-alarming-facts-and-statistics-on-card-fraud/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e47750db863" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Best Indicators for Day Traders]]></title>
            <link>https://medium.com/@amaluddin11/best-indicators-for-day-traders-e029d526f336?source=rss-90c0357d2562------2</link>
            <guid isPermaLink="false">https://medium.com/p/e029d526f336</guid>
            <category><![CDATA[apc]]></category>
            <category><![CDATA[support-vector-machine]]></category>
            <category><![CDATA[stock-market]]></category>
            <category><![CDATA[day-trading]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Mohd Amaluddin Yusoff]]></dc:creator>
            <pubDate>Mon, 05 Aug 2019 14:52:20 GMT</pubDate>
            <atom:updated>2019-08-05T14:52:20.355Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*CgOUlMRPJnU7NLQI4A8Orw.jpeg" /></figure><p><strong><em>Will tomorrow’s price go up or down? It is possible to predict?</em></strong></p><p>I was initially motivated to identify what are the most important technical indicators that can be used to predict tomorrow’s price movement, either going up or not … of course, assuming this is possible. This is especially important for a day trader and for a novice in stock market like me.</p><p><strong><em>Question 1: Which indicators are useful for Day Traders?</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/1*vIn8UDfv5Cb5IrKO7teMWQ.jpeg" /></figure><p>In an attempt to answer this question, historical stock market data with 21 indicators including momentum, moving averages, moving average convergence/divergence (MACD), return on investment (ROI), relative strength index (RSI), Stochastic RSI, Average True Range (ATR), Average Directional Index (ADX), Williams %R, Commodity Channel Index (CCI) and Ultimate Oscillator (UO) has been investigated. The meaning of these indicators is explained <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5219605/">here</a>.</p><p><em>It is not easy for a normal human being to monitor all these indicators, so which ones should we focus?</em></p><p>Our analysis shows that we can transform the original data of 21 indicators to a much simpler data consists of only 2 variables, and more importantly with less than 1% lost of information. Isn’t this brilliant?</p><p>These 2 variables are derived from the original indicators but with varying strength or importance. The plot below shows top 5 out of 21 indicators from the first variable. The length of the blue horizontal bar shows their strength, the longer the more important.</p><p>We can see that <strong>moving averages for 12 and 26 days </strong>are the two most important indicators from the first variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/0*O_TA7RxWc5XFjSZC.png" /><figcaption>Top 5 indicator of the first variable of transformed data.</figcaption></figure><p>From the plot below, we can see that <strong>Commodity Channel Index </strong>is the most important indicator from the second variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/503/0*al31f8C9NdTBznaZ.png" /><figcaption>Top 5 indicator of the second variable of transformed data.</figcaption></figure><p>To conclude, if I were to choose 5 indicators, they are …<br><strong>- moving averages for 12 days<br>- moving averages for 26 days<br>- Commodity Channel Index<br>- Williams %R<br>- Momentum for 5 days</strong></p><p><strong><em>Question 2: How predictable is tomorrow’s price? Will it go up or not?</em></strong></p><p>Since we are to predict one of two cases i.e., price goes up or not, this problem is called binary classification problem. There are plenty of algorithms that can be used to solve this problem. In this investigation, I have developed 3 models based on two algorithms called Support Vector Machines (SVM) and Neural Networks.</p><p>To assess the accuracy of the model, the data was split into two i.e., training and testing data. Training data is used to build the model and testing data to validate the model.</p><p>So, how well the model predict tomorrow’s stock price? Here is the results …</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*zZtvYS7wA4LXCPn5zOguXA.png" /><figcaption>The accuracy of Machine Learning models</figcaption></figure><p><em>They are all slightly better than flipping a coin!</em></p><p>SVM with PCA is the model that uses the 2 variables of transformed data (instead of the original 21 indicators). Even with 2 variables, the model could not achieve high accuracy. This is quite surprising.</p><p>One may think that once we reduce high dimensional data to 2-dimensions, the problem at hand becomes simple and we should be able to classify the data into two classes. Is this true?</p><p><strong><em>Question 3: Is it really hard to predict?</em></strong></p><p>To answer this question, please first take a look at below example plot. Can we visually separate the blue and red dots? It is clear that we can do it easily.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/0*3xxTHo5lSFdkiYGT" /><figcaption>Example plot</figcaption></figure><p>Now, let’s visualize the two variables of the transformed data. Remember that original data consists of 21 variables/indicators, so we can’t really visualize them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/549/0*XTdaCjjMEMASDwy2.png" /><figcaption>Plot of 2 variables of transformed data.</figcaption></figure><p>Figure above shows the plot of the 2 variables. The blue dot indicates price will go up tomorrow and red dot otherwise. Can we separate the blue and red dots? Very difficult!</p><p>There is no clear separation between the blue and red dots, so we can’t easily separate them, unlike in the example plot. This is the reason the accuracy obtained is around 50% even when using 2 variables.</p><h4>Conclusion</h4><p>In short, bad news to novice traders who are looking for shortcuts … stock price can’t really be predicted based on these technical indicators alone.</p><p>Simple classification algorithm is not really useful to predict stock market. More sophisticated time-series algorithms should be used with all factors considered such as news and Presidential announcements!</p><p>Here is the link to <a href="https://github.com/Amaluddin11/DayTraders">my Github</a> to get the Jupyter notebook on this investigation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e029d526f336" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>