<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Zain Khan on Medium]]></title>
        <description><![CDATA[Stories by Zain Khan on Medium]]></description>
        <link>https://medium.com/@xzknz?source=rss-494843a3cdbe------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*mRLC2pARGrs0xpmcHC25YA.jpeg</url>
            <title>Stories by Zain Khan on Medium</title>
            <link>https://medium.com/@xzknz?source=rss-494843a3cdbe------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 09:43:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@xzknz/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Make the Professions Fun Again]]></title>
            <link>https://medium.com/i-rr/make-the-professions-fun-again-85a388442922?source=rss-494843a3cdbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/85a388442922</guid>
            <category><![CDATA[digital-transformation]]></category>
            <category><![CDATA[professional-services]]></category>
            <category><![CDATA[investing]]></category>
            <dc:creator><![CDATA[Zain Khan]]></dc:creator>
            <pubDate>Mon, 04 Jul 2022 08:32:36 GMT</pubDate>
            <atom:updated>2022-07-04T08:32:36.007Z</atom:updated>
            <content:encoded><![CDATA[<p><em>It’s not an easy job convincing the professional sector to transform but it’s a worthy job if done right.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*Mp1vj5TXq7nsl0b_FrMuEA.jpeg" /></figure><p>We’ve had a simple hypothesis since our (<a href="http://mandeleo.co.uk/">Mandeleo</a>) inception: professional services will require serious business and digital transformation to navigate through (and beyond) the pandemic.</p><p>All three partners have worked extensively in or around the professional sector in our prior jobs so it was relatively easy for us to identify key areas of concern within the sector from our perspectives. Let’s dive in.</p><ol><li>Lack of differentiation: With the increase in competition, many professional service businesses find it difficult to differentiate themselves from their competitors. Beyond reputation and previous experience, there isn’t much that existing operators showcase to stand out amongst a sea of competitors.</li><li>Commoditisation: Another problem is commoditisation. When businesses offer services that become indistinguishable from the same services offered by another company. While differentiation speaks more to the business, team and brand, commoditisation refers to the services themselves. Most services in the market look and feel the same.</li><li>Low barriers to entry: The professional services market also has surprisingly low barriers to entry, which means that there are a plethora of new businesses popping up every week, offering the same list of professional services. These days anyone can set up a pretty looking no-code website and outsource the actual services offshore. With the three issues stated above, how is anyone supposed to make an informed decision to pick an experienced right team with bespoke services suited to your needs?</li><li>Fragmented market: The market is heavily fragmented. Independent operators live in their own space while bigger conglomerates inflate prices and reduce almost any ownership for their staff and professionals.</li><li>Lack of trust: Trust is an issue that gets compounded when you add all of the above. Should you trust an independent firm or a conglomerate? Will you get a better price with the former or the latter? What about trusted reviews? Eventually, as a client, you’ll leave with decision fatigue without making a choice at all.</li></ol><p>With all that taken into account, we needed to address each point one by one and create a playbook for future acquisitions and portfolio companies. There is no one-size-fits-all answer to this question since the professional services sector is composed of many different types of businesses, each with its own unique needs and challenges. However, there are some common themes and strategies that we can use to help digitally transform the professional services sector.</p><ol><li>Focusing on customer experience: One of the most important aspects of digitally transforming the professional services sector is focusing on improving the customer experience. We will be investing in technologies that make it easier for customers to interact with your business, such as online booking systems, online quoting tools, and live chats and support. It’s low-hanging fruit, we’re not reinventing the wheel but, for lack of a better word, transforming it.</li><li>Enhancing communication and collaboration: Another important aspect is enhancing communication and collaboration between businesses and their clients. We aim to invest in technologies that make it easier to communicate and collaborate. Leveraging proper video conferencing, project management software, and online collaboration tools. You’d be surprised (or not surprised at all) about the lack of proper processes in larger organisations. Their existing motto is ‘if it isn’t broken, don’t fix it.’ I see the value in that but the sundial wasn’t a broken tool either yet we definitely moved away from that.</li><li>Automating processes: Our ethos is centred around investing in technologies that automate repetitive tasks, such as customer relationship management (CRM) software, enterprise resource planning (ERP) software, and workflow management software. It’s easy to say but hard to accomplish and our choice of platform or software will change and evolve as we scale. It’s important to implement scalable solutions from the start so that we focus on the delivery of our services rather than the operational burden of creating new processes.</li><li>Improving data analytics: The final aspect of digitally transforming the professional services sector is improving data analytics. This can be done by investing in technologies that make it easier to collect, analyse, and use data, such as business intelligence (BI) software, data mining software, and data visualisation software. The tools and dashboards available today are basically plug-and-play. From PowerBI, Tableau, Airtable, you name it. The ability to communicate key metrics within your organisation and to external parties/clients has never been easier.</li></ol><p>As with all areas of business and life, the theory is often much simpler than the execution. Our efforts in the early days, as we continue down this journey, will be to convince the existing stakeholders that our solutions can improve their lives and performances. In order to grow and nurture alignment, we need individuals to buy into our strategy, ideas and, most importantly, our team.</p><p>Until next time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85a388442922" width="1" height="1" alt=""><hr><p><a href="https://medium.com/i-rr/make-the-professions-fun-again-85a388442922">Make the Professions Fun Again</a> was originally published in <a href="https://medium.com/i-rr">I,RR</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Follow The Market Breadcrumbs]]></title>
            <link>https://medium.com/i-rr/follow-the-market-breadcrumbs-2385d0a35c61?source=rss-494843a3cdbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/2385d0a35c61</guid>
            <category><![CDATA[private-equity]]></category>
            <category><![CDATA[investing]]></category>
            <category><![CDATA[buy-and-build]]></category>
            <dc:creator><![CDATA[Zain Khan]]></dc:creator>
            <pubDate>Mon, 27 Jun 2022 06:52:22 GMT</pubDate>
            <atom:updated>2022-06-27T06:52:22.401Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Traditional businesses are overlooked because they aren’t sexy but sexy is just about perspective.</em></p><p>I’m changing tune this week to stick closer to the I,RR remit: investing and finding opportunities.</p><p>Let’s get into it.</p><p>My firm, <a href="http://mandeleo.co.uk/">Mandeleo Partners</a>, did not invent acquisition entrepreneurship or private equity. Yet, we look to enhance it.</p><p>Our goal is to transform sectors in the United Kingdom through digital transformation and an infinite time horizon. These days, however, everyone seems to have an infinite time horizon.</p><p>We formed our thesis in the midst of the pandemic. Most businesses were struggling to navigate the changing economic environment. People were losing jobs and businesses were losing customers. It wasn’t an issue of demand but rather flexibility and adaptability. The local high street was banking on furlough schemes and job retention programs rather than introspecting on their existing model and the new generation of customers and clients that believe businesses can still run remotely. In fact, they demand that things run remotely with more touch points online rather than in-person. At least I do.</p><p>If I have to call my dry cleaner rather than select my service and time of booking on their website, I might cry. I’m not being petty, I just want to be able to handle tasks in this manner. Calls exhaust me and, let’s be honest, whenever the line is busy, I end up forgetting and the task goes into a bottomless pit of my incomplete actions.</p><p>Naturally, technology and transformation were always at the forefront of our thinking. Specifically in professional services and to build the next generation of professional services, we need to acquire businesses and transform them from within.</p><p>The opportunity is massive. <a href="https://www.openinvest.com/articles-insights/how-to-prepare-for-the-great-wealth-transfer-to-millennials">Over the next 25 years, it is estimated that $65 trillion will shift from Baby Boomers to Millennials</a>. This comes from inheritance, retiring professionals, family-owned operations and more. One way or another, Millennials, and the generations that follow, will guide change.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/1*MNV-T1jR8FqLUagwDnOL5g.png" /></figure><p>All three partners at Mandeleo fall into that demographic category. We are digital natives (one of the last portion of Millennials that still remember dial-up internet), open to alternative investment opportunities and focus on the environment. Businesses will have to adapt to these demands, fast. Consumers will spend their dollars on businesses that understand technology’s benefits. Bank accounts that require printed documents will be left behind. Appointments that can’t be scheduled online will not be scheduled. And on, and on..</p><p>This transition is not going to happen overnight nor is it going to be a shock to the system. We are already moving in that direction. Yet, businesses across the United Kingdom, have still not made those adjustments. There is a concerted effort across the board, but one that is slower than required.</p><p>That is why Mandeleo Partners exists. We believe in the strength and utility of traditional businesses. They must and will never die. However, they must adapt.</p><p>We focus on investing in traditional businesses that VCs often overlook, in fact, most B2C SaaS products are focussed on taking professionals out of the equation completely. We believe by facilitating digital transformation and implementing modern governance structures we can navigate the changing landscape and build the next generation of professional services.</p><p>Our investment criteria is simple.</p><ul><li>Large market or sector with relatively inelastic demand</li><li>Over 3 years of profitability</li><li>Strong team and management</li><li>Ripe for digital transformation</li></ul><p>Our first sector platform is <a href="http://mandeleoanimalhealth.co.uk/">Mandeleo Animal Health Group</a>. We are building the next generation in animal healthcare. We partner with clinics across the United Kingdom, build digital platforms for clients and internal operations, and focus on employee ownership and wellbeing.</p><p>The reception and acceptance of our model and strategy is market validation. The next step is to build on our promises and execute for the long haul (the easy part some say.. that was sarcasm).</p><p>We are always looking for new partners and ideas to join us on this journey. If you believe in what we are building, please reach out. Everyone is welcome here.</p><p>Until next time.</p><p><a href="http://iareare.substack.com/"><em>Subscribe to my newsletter I,RR</em></a><em> for similar content, every week or visit </em><a href="http://khanzain.com/"><em>khanzain.com</em></a><em> for more.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2385d0a35c61" width="1" height="1" alt=""><hr><p><a href="https://medium.com/i-rr/follow-the-market-breadcrumbs-2385d0a35c61">Follow The Market Breadcrumbs</a> was originally published in <a href="https://medium.com/i-rr">I,RR</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Anti-Goals and Knowing What You Don’t Want]]></title>
            <link>https://medium.com/i-rr/anti-goals-and-knowing-what-you-dont-want-6437bd658939?source=rss-494843a3cdbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/6437bd658939</guid>
            <category><![CDATA[strategy]]></category>
            <category><![CDATA[goals]]></category>
            <category><![CDATA[personal-development]]></category>
            <category><![CDATA[life-lessons]]></category>
            <dc:creator><![CDATA[Zain Khan]]></dc:creator>
            <pubDate>Thu, 19 May 2022 00:00:00 GMT</pubDate>
            <atom:updated>2022-06-14T16:35:36.911Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XOpmoMVZVcZN8zy0" /><figcaption>Photo by <a href="https://unsplash.com/@nickbartos?utm_source=medium&amp;utm_medium=referral">Nicholas Bartos</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Our likes and desires can change and our idea of success and growth can alter as we are presented new information but, more often than not, especially as time ticks along, we are quite set in our dislikes. I always thought I would eventually develop into a morning person, that once I reached a certain age (in my head that age was 21), I would start to wake up naturally at 6 or 7am and become a productive member of society.</p><p>That’s just never going to happen.</p><p>This isn’t to say I sleep in until noon but I need time to myself in the mornings to get the engine running. I need to stretch, drink a double espresso, read something, think about the day ahead and then slowly but surely get cracking until I hit a mental and emotional peak in the late morning or early afternoon. My wife is the opposite. She is up and at it by 7:30am and then slowly but surely settles into the evening.</p><p>One of my anti-goals is to not have any calls before 10am if I can help it. The truth is, I can almost always help it, no matter how important the call or meeting, there is almost always room to manoeuvre. That’s the point of anti-goals. Manifest the life and business you want by not putting yourself in situations that you want to avoid. Anti-goals are as aspirational and important as any other goal in your life.</p><p><strong>Set your own anti-goals:</strong></p><p>The rules are exactly the same as any other goal or target:</p><p><strong>Specific</strong></p><p>Make sure your goals are specific and clear.</p><p>Good example: Do no consume sugar or sugary foods during the workweek. Limit to the weekends.</p><p>Bad example: Cut sugar</p><p><strong>Measurable</strong></p><p>What’s the point of goals if we can’t track our progress?</p><p>Good example: Only allow a maximum of 3 calls or meetings a month before 10am.</p><p>Bad example: Keep meetings and calls for the afternoon or evening</p><p><strong>Achievable</strong></p><p>How achievable or realistic are your goals? It has to be something that is challenging but achievable and realistic given your current circumstances. If you create a goal of having a net worth of £1m in 6 months, well, it is possible, but quite unlikely. Thus if you fail to reach that target, you will be demoralised and that’s not the point.</p><p>Good example: Never sacrifice my bi-annual family vacation for work</p><p>Bad example: Holidays are non-negotiable</p><p>Like most things in life, it isn’t the action itself but your lens and perspective that defines it. Anti-goals don’t solve everything but they can help develop a framework to guide your work or life and internalise your ideal balance.</p><p>Until next time.</p><h3>The Back Pocket</h3><p><strong>From the source:</strong></p><ul><li><a href="https://awilkinson.medium.com/the-power-of-anti-goals-c38f5f46d23c">Andrew Wilkinson: The Power of Anti-Goals</a></li></ul><p><strong>Inverting problems:</strong></p><ul><li>Sahil Bloom | <a href="https://sahilbloom.substack.com/">The Curiosity Chronicle</a>: <a href="https://sahilbloom.substack.com/p/mind-blowing-facts-anti-goals-and?s=r">Mind-Blowing Facts and Anti-Goals</a></li></ul><p><em>Originally published at </em><a href="https://www.khanzain.com/irr/anti-goals-and-knowing-what-you-dont-want"><em>https://www.khanzain.com</em></a><em> on May 19, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6437bd658939" width="1" height="1" alt=""><hr><p><a href="https://medium.com/i-rr/anti-goals-and-knowing-what-you-dont-want-6437bd658939">Anti-Goals and Knowing What You Don’t Want</a> was originally published in <a href="https://medium.com/i-rr">I,RR</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your ‘Big’ Launch Is Anything But Big]]></title>
            <link>https://medium.com/i-rr/your-big-launch-is-anything-but-big-f6047eb8ef96?source=rss-494843a3cdbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/f6047eb8ef96</guid>
            <category><![CDATA[strategy]]></category>
            <category><![CDATA[planning]]></category>
            <category><![CDATA[launch]]></category>
            <category><![CDATA[execution]]></category>
            <dc:creator><![CDATA[Zain Khan]]></dc:creator>
            <pubDate>Thu, 12 May 2022 00:00:00 GMT</pubDate>
            <atom:updated>2022-06-14T16:36:56.222Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*j_-xGYpdWnn6eBOG" /></figure><p>One of my biggest beliefs in business and life is that you should build an audience. This is a superpower. Think about the celebrities that have the ability to invest in some of the best companies and ideas in the world. Some public figures launch their own businesses without any marketing or sales. Why? Because they already attract attention. For the rest of us, we need to build that trust. That’s another reason why I chose to build in public.</p><p>If you already have an audience, great, here is what you can do to craft a successful launch.</p><ol><li>Notify your audience about what you are building (build in public)</li><li>Gather feedback and interest via sign-up forms or waitlists</li><li>Tease the benefits and value of your product/service</li><li>Begin a countdown to the inevitable launch</li></ol><p>Bonus: Add referral programs to the mix so that your existing audience can work for you</p><p>It all seems so simple when one has an audience. Yet, many entrepreneurs and creators assume that they can follow that same outline. If you don’t have an audience, your process must be different.</p><p>To build an audience you need to gain people’s trust and establish yourself as an individual or company that can deliver value. Here is how to get started:</p><ol><li>Find a niche or topic that is important to you. This could be anything from database management to real estate development. Whatever it is, ensure that you have enough material or thoughts to share</li><li>Start writing or creating pieces of content to establish yourself as a voice in that sector/area</li><li>Share your work with the public (I know, very daunting)</li><li>Continue to find your voice via feedback and self-analysis</li><li>Consistency is key here, audiences can take months to build so it is important that you keep feeding it</li><li>Collaborate with other creators in similar or tangential sectors</li><li>Build a list of interested readers/consumers for bespoke content</li><li>Transform that list into a community that you can feed or feeds itself</li><li>Deliver value to your new community through products and services exclusive to them</li></ol><p>The steps above aren’t comprehensive but an outline. You can build a community on any topic of your interest. The trick is to have genuine passion and commitment. That enthusiasm needs to be electric and contagious. Millions of people out there might be looking for what you are selling. Your goal is to find them but, more importantly, to put yourself in a position to reach product-market fit quickly and cheaply. A lot of startups waste an extensive amount of time and money financing customer acquisition with a product or business that isn’t suited for their target audience. If you have an audience, you remove the guesswork and can get straight down into delivering your promise.</p><p>That’s my big launch plan for <a href="http://themoonshot.digital/">Moonshot Digital</a> and the ethos behind <a href="http://mandeleo.co.uk/">Mandeleo Partners</a>. Our thesis is to acquire audiences and build products for them.</p><p>Until next time.</p><h3>The Back Pocket</h3><h4><strong>Athletes becoming billionaires:</strong></h4><ul><li><a href="https://afrotech.com/kevin-durant-businesses-net-worth">Afrotech: Kevin Durant Is Dominating The Boardroom and The Court — And His $75M Net Worth Proves It</a></li></ul><p><strong>Modern-day businesspeople:</strong></p><ul><li><a href="https://thred.com/culture/ksi-and-logan-paul-announce-new-business-venture-prime/">Thred: KSI and Logan Paul announce new business venture ‘Prime’</a></li></ul><p><strong>Take it from the suits:</strong></p><ul><li><a href="https://www.morganstanley.com/articles/athlete-turned-entrepreneur-transition-tips">Morgan Stanley: The Path From Athlete To Entrepreneur</a></li></ul><p><em>Originally published at </em><a href="https://www.khanzain.com/irr/your-big-launch-is-anything-but-big"><em>https://www.khanzain.com</em></a><em> on May 12, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6047eb8ef96" width="1" height="1" alt=""><hr><p><a href="https://medium.com/i-rr/your-big-launch-is-anything-but-big-f6047eb8ef96">Your ‘Big’ Launch Is Anything But Big</a> was originally published in <a href="https://medium.com/i-rr">I,RR</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pre-Empt Your Bottlenecks And Headaches]]></title>
            <link>https://medium.com/i-rr/pre-empt-your-bottlenecks-and-headaches-e3a5109080ca?source=rss-494843a3cdbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/e3a5109080ca</guid>
            <category><![CDATA[agency]]></category>
            <category><![CDATA[management]]></category>
            <category><![CDATA[strategy-execution]]></category>
            <dc:creator><![CDATA[Zain Khan]]></dc:creator>
            <pubDate>Thu, 05 May 2022 00:00:00 GMT</pubDate>
            <atom:updated>2022-06-14T16:37:45.895Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*er4QW8mPt1jLf_kb" /><figcaption>Photo by <a href="https://unsplash.com/es/@octoberroses?utm_source=medium&amp;utm_medium=referral">Aubrey Odom-Mabey</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The journey of building a second agency in my lifetime while learning from my past experiences but not letting the associated biases cloud my judgement is an interesting task. All of this under the umbrella of our private equity firm is all the more interesting.</p><p>While you shouldn’t count your chickens before they hatch, I think there is some merit in pre-empting future bottlenecks which, with almost all agencies, is managing scale. How can you continue to onboard new clients sustainably while growing your team and ensuring everyone hits the ground running?</p><p><strong>Finding the right partners</strong></p><p>The answer is to find the right partnerships with other agencies or development/design partners. While it seems counterintuitive to work with another agency, it is an interesting way to split the workload and manage client satisfaction and quality of output. Partnering with a team of developers/designers/managers is dependent on finding the right fit for your needs and model. For our purposes, we need someone who understands our subscription-based model with unlimited iterations for clients. More importantly, since I was going to be running point on the development of Moonshot Digital, I was responsible for the structure, culture and experience of the new team.</p><p>Thus, I created the following criteria for any future partners that we bring on board:</p><ol><li>Fixed or capped pricing based on services provided</li><li>Validated knowledge and experience executing projects for a variety of clients</li><li>Clear and consistent communication</li><li>Capacity and ability to work within my workflow</li><li>Our ability to align on goals and targets</li></ol><p>Price caps are easy to resolve with a partner agency assuming you can give them the guarantee that you will bring in clients. If you bring people customers you often have a lot more leverage than you’d assume. I talked to potential partners about my model, worked out what their maximum fees would be for different offerings and added a nice 10–20% margin on top as a buffer in case some projects or tasks go over.</p><p>Validating their knowledge was also relatively easy given the fact that I can personally execute across all areas of the business. I asked poignant questions about digital marketing frameworks, SEO, deep dives into their preferred tech stack and project management.</p><p>This brings me to how I assessed their communication. I spent time with them on Zoom calls, Slack messages and task management tools to gauge what their average response time would be and if I can trust their words and the associated actions. I’ve worked with many developers and often they promise more than they can deliver.</p><p>The new-ish agency subscription model is quite dependent on a watertight workflow. My internal teams are going to be distributed across the world and in various time zones so our ability to work smoothly and efficiently without constantly checking in with each other is vital. The truth is this is the hardest area to assess without being in the weeds. I don’t believe you can simulate what it is like in the trenches when the work starts to pile up and we are working at capacity but with the right frameworks, you can soften the blows.</p><p><strong>Troubleshooting framework:</strong></p><ol><li>Create support tickets for any issues or problems on Trello</li><li>Ensure each ticket has a target resolution date</li><li>Prioritise tickets using the urgent/important matrix</li><li>Integrate Trello to a dedicated Slack channel to receive up to date notifications for each ticket</li><li>Dedicate the first hour of each day tackling support tickets and updating progress reports</li></ol><p>Lastly, I want to make sure that the team I work with is aligned with our goals and targets. This is both financial and aspirational. We set goals and anti-goals with monthly targets for revenue, profit, client profile and sectors. Everything is laid out in the open for all of our teams to view with dashboards with progress reports and tickers to see how we are coming along. Over and above that, we will make sure to get on a monthly team call to debrief the last month’s work and a quarterly call to plan ahead. It’s important to stay nimble and flexible here so that we can adjust and tinker with our marketing, sales and strategy just enough to find little wins here and there.</p><p>Long-term vision and short term actions need to work in tandem so that we understand our north star but also leave enough room for creativity and sudden bursts of brilliance to come from anywhere within the company.</p><p>Until next time.</p><h3>The Back Pocket:</h3><h4><strong>A new perspective:</strong></h4><ul><li><a href="https://www.eisenhower.me/eisenhower-matrix/">The Eisenhower Decision Matrix</a></li></ul><p><em>Originally published at </em><a href="https://www.khanzain.com/irr/pre-empt-your-bottlenecks-and-headaches"><em>https://www.khanzain.com</em></a><em> on May 5, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e3a5109080ca" width="1" height="1" alt=""><hr><p><a href="https://medium.com/i-rr/pre-empt-your-bottlenecks-and-headaches-e3a5109080ca">Pre-Empt Your Bottlenecks And Headaches</a> was originally published in <a href="https://medium.com/i-rr">I,RR</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is It All About The Price?]]></title>
            <link>https://medium.com/i-rr/is-it-all-about-the-price-5210e9b64f66?source=rss-494843a3cdbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/5210e9b64f66</guid>
            <category><![CDATA[pricing-strategy]]></category>
            <category><![CDATA[sales]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[marketing]]></category>
            <dc:creator><![CDATA[Zain Khan]]></dc:creator>
            <pubDate>Thu, 28 Apr 2022 00:00:00 GMT</pubDate>
            <atom:updated>2022-06-14T16:38:20.951Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uRIbifScX2kb8yRu" /><figcaption>Photo by <a href="https://unsplash.com/@m_____me?utm_source=medium&amp;utm_medium=referral">m.</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>That’s right <em>the</em> Fergie. Add this poignant quote to her long list of accolades. Many people argue about the ridiculous prices that Balenciaga charges for their questionable items of clothing. However, nobody argues as much with Walmart or IKEA about how they manage to keep prices so low. Price doesn’t define value but value can define the price.</p><p>If you charge someone £10,000 for a seminar but by the end of it they are completely and sustainably renewed and recharged, you would not hear any protests about the price point given the value generated. You have testimonials and a track record to boot. However, if you charge £50,000 a year for a degree which you can only afford with a high-interest loan then people start to question the true value of your product or service.</p><p>That’s essentially all one needs to know about pricing. Can you justify your price point by demonstrating true value to your customers or clients or are you constantly fighting against it?</p><h3>Different pricing models:</h3><p><strong>Penetration pricing model:</strong></p><p>It’s all in the name. Penetration pricing is used by businesses to attract customers to a new product or service by offering a lower price when first entering a market. ****The lower price helps a new product or service penetrate the market and attract customers away from competitors.</p><p><strong>Cost-based pricing model:</strong></p><p>Cost-based pricing is a pricing method that is based on the cost of producing and distributing the product. This is a common starting point for most founders and business people when deciding on pricing. Assume your new product costs $10 to produce and another $5 to distribute. You add another 20% margin on top for yourself and begin selling. Thus the final price of your product is $18*</p><p>*The math ($10 + $5) * 1.2</p><p><strong>Competitive pricing model:</strong></p><p>If the costs of production for your product are incredibly low and adding a margin doesn’t seem to add up you can offer competitive pricing. Competitive pricing is strategically selecting price points for your goods or services based on competitor pricing in your market.</p><p><strong>Dynamic pricing model:</strong></p><p>Airlines and Uber are great examples of industries and companies that use dynamic pricing. This is a reason your flight prices change based on seasons and timing. The goal of dynamic pricing is to allow a company that sells goods or services over the Internet to adjust prices on the fly in response to supply and demand.</p><p><strong>Fixed pricing model:</strong></p><p>I have never been a fan of fixed pricing as a business person or founder but I think in my young-old age I understand its benefits. A fixed cost pricing model is a model that guarantees a fixed budget for the project, regardless of the time and expense. The main advantage of a fixed price model is that it allows the client to plan and set an exact budget. With <a href="http://themoonshot.digital/">Moonshot Digita</a> l, that’s what I am aiming for. Offer all non-complex tasks to our clients for a fixed monthly fee so they can plan their budgets accordingly without any surprises from their digital team.</p><p>It can be quite easy to over-engineer your pricing model and use price as a leading indicator when most of the time it is the supporting areas that impact your business. Delivering and marketing your value is the tough part, not price setting. There are some cases, such as competitive pricing, where you can’t really go much beyond the fray. Take Netflix for example, their prices kept creeping up but they were able to get away with it because they were producing gems that weren’t available anywhere else. House of Cards, Casa de Papel, Stranger Things and Narcos meant users were getting enough out of the product to let the rising prices slide. However, once they started producing titles like <a href="https://www.theverge.com/2021/10/26/22747488/netflix-suppress-cuties-algorithm-search-results-dave-chappelle">Cuties</a> and other poor reality TV series people started to question it. The pandemic, rising household prices and increased competition added fuel to the fire.</p><p>Focus on what your customers want and they won’t go anywhere, regardless of your pricing. Think Apple and Facebook or Google advertising. You have alternatives, but, for the most part, they aren’t nearly as good.</p><p>Until next time.</p><h3>The Back Pocket</h3><h4><strong>A fascinating read:</strong></h4><p><a href="https://pros.com/learn/blog/airline-dynamic-pricing-why-matters-more-today">PROS: Airline Dynamic Pricing: Why It Matters Even More Today</a></p><p><strong>The headlines:</strong></p><p><a href="https://www.bloomberg.com/news/articles/2022-04-20/netflix-subscriber-woes-hint-at-u-s-consumer-pushback-on-prices#xj4y7vzkg">Bloomberg: Netflix Subscriber Woes Hint at U.S. Consumer Pushback on Prices</a></p><p><em>Originally published at </em><a href="https://www.khanzain.com/irr/is-it-all-about-the-price"><em>https://www.khanzain.com</em></a><em> on April 28, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5210e9b64f66" width="1" height="1" alt=""><hr><p><a href="https://medium.com/i-rr/is-it-all-about-the-price-5210e9b64f66">Is It All About The Price?</a> was originally published in <a href="https://medium.com/i-rr">I,RR</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Credit Card Fraud Detection. An exercise in class imbalance.]]></title>
            <link>https://medium.com/analytics-vidhya/credit-card-fraud-detection-an-exercise-in-class-imbalance-c4b8fd5ba603?source=rss-494843a3cdbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/c4b8fd5ba603</guid>
            <category><![CDATA[fraud-detection]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[imbalanced-data]]></category>
            <dc:creator><![CDATA[Zain Khan]]></dc:creator>
            <pubDate>Wed, 07 Oct 2020 14:43:48 GMT</pubDate>
            <atom:updated>2020-10-13T12:56:34.830Z</atom:updated>
            <content:encoded><![CDATA[<h3>Credit Card Fraud Detection. An Exercise In Class Imbalance.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6g2y1ATrLch1RuCNCoSQWw.jpeg" /></figure><blockquote>‘Criminals thrive on the indulgence of society’s understanding’ — Henri Ducard (Ra’s Al Ghul)</blockquote><p>That’s the energy I channeled for this data science project. To detect fraud, you must first understand why it came to be..</p><p>Well, not really. That was quite excessive. Over the top really..</p><p>(*clears throat*) There are many other ways to detect fraud which I will dive into for this article. The idea to detect fraud was because I wanted a deeper understanding of class imbalances in datasets. Class imbalances are when one certain class in our data (in this case actual fraudulent transactions) are overrepresented.</p><p>This makes intuitive sense. Fraud happens quite often but in the bigger picture it is a small percentage of the overall transactions. So how do we create classification models that can detect fraudulent transactions?</p><h3>Dataset:</h3><p>For this project, I’ll be using an anonymised dataset of transactions made by credit cards in September 2013. We have 30 features in the dataset out of which 28 were all created using Principle Component Analysis to protect personal information. That means that we won’t be able to infer much about the variables themselves. The other two variables are ‘Time’ and ‘Amount.’</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ygdP7CUgO6mwd1iNtlL0bg.png" /><figcaption>A quick peek at the head of our dataframe.</figcaption></figure><p>All in all, we have 284,807 rows of data available to us and no null values. That saves us some time with data cleaning and means we can jump right in.</p><h3>EDA:</h3><p>Since we don’t have much information about what our variables are and how they relate intuitively to credit card transactions, we need to see if data visualisation can help us understand certain relationships and trends.</p><h4><strong>Correlation heat map:</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*g2CDT2a2mGu818JvegEFBQ.png" /><figcaption>No real findings here apart from a few stronger correlations within Time and Amount.</figcaption></figure><h4><strong>Analysing individual features:</strong></h4><p>I wanted to use this section to find if there were any obvious individual columns that could help with future feature engineering.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*Z8AScP_lLR1cR6AU0l2apw.png" /><figcaption>Testing V3, Time and Amount for all fraudulent transactions. I can’t craft any obvious conclusions from this dataframe.</figcaption></figure><p>Let’s look at individual data points.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*Qt7X7pORc0i01JZSlMBtYA.png" /><figcaption>Testing V5, Time and Amount for all fraudulent transactions. It’s interesting that there were 2 transactions under 1 (274,382 and 280,143 respectively).</figcaption></figure><p>Visualising box plots to see the difference in class distributions for various features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0tb5DL5coeh4p0w2UpUqNw.png" /><figcaption>V10 and V14 have some visible distribution spreads and outliers for fraudulent transactions (Class 1) while the rest are not as obvious..</figcaption></figure><h4>Removing Outliers:</h4><p>I had an internal debate to remove outliers or not. After an incredible amount of reading on class imbalance problems, I came to the conclusion that we don’t want information loss which could effect the accuracy of our models going forward. Changing the thresholds of the box plots above from the standard 25th and 75th percentile range can alter the way we analyse ‘extreme’ cases which could be key in discovering fraudulent transactions.</p><h3>Modelling:</h3><p>I set the target variable, Class as y and the rest of the columns as our model inputs, X. Then I split it into train and test sets with 25% in the test set with the remaining data utilised to train our models. Finally, I standardised the features because I believe it’s good practice (research on the true consequences of standardising PCA variables did not yield any glaring issues in constructing machine learning models so I continued with this decision).</p><p>Next, I ran the following models on our data:</p><p>1- LogisticRegression</p><p>2- DecisionTreeClassifier</p><p>3- KNeighborsClassifier</p><p>4- RandomForest</p><p>The baseline score for our data is:</p><pre>0    0.998273 (Normal transactions)<br>1    0.001727 (Fraudulent transactions)</pre><p>It should be clear at this point that we will not be assessing the ability of our models based on the scores, but rather focus on recall and sensitivity.</p><h3>Recall, Sensitivity and Cost-Sensitive Classification:</h3><p>A quick note.</p><p>When working with imbalanced data it is often good procedure to discount accuracy and focus instead on recall (sensitivity).</p><p>Why?</p><p>Well, let’s say we have a fantastic 99.8% accuracy for our model which, as a score, is absolutely fantastic. However, that’s essentially our baseline score. We should be accurately classifying 99.8273% of our target anyways. This is why we focus on recall (sensitivity).</p><p>Recall illustrates the number of correct classifications divided by total number of values in that class. In terms of this project, recall tells us how many fraudulent transactions we correctly identify as fraudulent. That’s the goal.</p><p>Another key area of importance when working with classification models and imbalanced datasets is understanding the cost of correctly or incorrectly classifying a positive class (fraud).</p><p>Would we rather our model err on the side of predicting too many transactions as fraudulent or vice versa? At this point, I can assume that a good percentage of us have experienced that phone call from our bank asking us if a recent transaction is fraudulent or not. That is because banks are much more lenient with False Positives than False Negatives. Intuitively, it makes perfect sense doesn’t it? It would cost the bank a lot more in safeguarding user accounts if their model skewed towards False Negatives rather than False Positives. It is a simple and fairly cheap process for a bank to call up a customer and validate a transaction while it is quite costly to reverse a transaction on a credit card and potentially send the customer a new card altogether.</p><h3>Handling Imbalanced Data:</h3><p>There are many ways to handle imbalanced datasets. Below are a few examples of strategies that I had considered:</p><p>1- SMOTE (Synthetic Minority Oversampling Technique)</p><p>2- Oversampling the minority class</p><p>3- Undersampling the majority class</p><p>There are many advantages and disadvantages to each sampling technique above. One of SMOTE’s biggest issues is that it does not take into consideration neighbouring examples from other classes. Oversampling can lead to overfitting since it is replicating the minority class and undersampling can potentially get rid of some vital information.</p><p>Instead of working through the pros and cons of each, I decided to try something I had previously attempted with the Ames Housing Dataset, altering the ‘class_weight’ parameters for each of my models to ‘balanced.’ This essentially weighs the proportion of data in each class and rebalances the classes accordingly.</p><h3>Results:</h3><p>I created a function I have been using for the longest time:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d8cb8c5a3c0fed1f703ae55a5cc1ab0b/href">https://medium.com/media/d8cb8c5a3c0fed1f703ae55a5cc1ab0b/href</a></iframe><p>Add parameters for GridSearch:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/48404e3d53570e2231c4387f6154a351/href">https://medium.com/media/48404e3d53570e2231c4387f6154a351/href</a></iframe><h4>Logistic Regression:</h4><p>Logistic Regression always amazes me with its flexibility and performance on classification problems. As expected, it also yielded some fairly decent results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U6DNoJijYq3DeI-mVOaLVA.png" /><figcaption>A great result to minimise False Negatives but a large amount of False Positives. Is that you Barclays?</figcaption></figure><h4>Decision Tree Classifier:</h4><p>DecisionTree worked slightly better than my LogisticRegression when dealing with False Positives but also increased my False Negative results by ~300%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TxPOJT2SqZb-pgvoHIZC3g.png" /><figcaption>Lower recall score than Logistic Regression</figcaption></figure><h4>KNeighbors Classifier:</h4><p>A model that seems to only focus on accuracy rather than recall. It predicted 0 fraudulent</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R0j8i5eY0CBWaMrau5rMcA.png" /></figure><h4>Random Forest:</h4><p>Random Forest seems to have the cleanest results with high precision and, more importantly, recall scores. However, the recall score does not compete with the Logistic Regression.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-nQB1GYofSUeP534oAKVvg.png" /><figcaption>23 too many False Negatives. And 12 more than our best model.</figcaption></figure><p>As a result of my first iteration of results, I wanted to explore Logistic Regression further and potentially optimise it to the best of my ability. That led me to expand the parameters for our GridSearch and see if that would make any significant changes to the results.</p><h4>Logistic Regression (after further optimisation):</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*57fNjv0mWwxDoSLNie9OSw.png" /><figcaption>Only 9 False Negatives now in the whole training set. While I have a large amount of False Positives and without the true understanding of the costs for a bank to assess certain fraudulent transactions (correctly or incorrectly), my first impression is that my newly optimised Logistic Regression model is the way forward.</figcaption></figure><h3><strong>Conclusion:</strong></h3><p>Fraud detection is an area of data science that can go as deep as you let it until you reach almost perfect detection. I could have used other models, explored SMOTE, sampling techniques and further feature engineering to improve results and recall scores.</p><p>The aim of this project was to illustrate the ways in which one can tackle imbalanced datasets and keep an eye on ‘real world’ consequences of shifting focus from accuracy to recall.</p><p>To put it quite simply, next time I get a call from any of my banks, I will make sure to ask them more questions about why they called me and assumed a certain transaction was fraud. I invite you to do the same. Just send them this article, I’m sure that won’t piss them off.. right?</p><p>— — —</p><p><a href="http://github.com/xzknx">GitHub</a> | <a href="https://www.linkedin.com/in/xazainkhan/">LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4b8fd5ba603" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/credit-card-fraud-detection-an-exercise-in-class-imbalance-c4b8fd5ba603">Credit Card Fraud Detection. An exercise in class imbalance.</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Can we actually predict market change by analyzing Reddit’s /r/wallstreetbets?]]></title>
            <link>https://medium.com/i-rr/can-we-actually-predict-market-change-by-analyzing-reddits-r-wallstreetbets-9d7716516c8e?source=rss-494843a3cdbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d7716516c8e</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[predictive-analytics]]></category>
            <category><![CDATA[stock-analysis]]></category>
            <category><![CDATA[predictive-modeling]]></category>
            <dc:creator><![CDATA[Zain Khan]]></dc:creator>
            <pubDate>Thu, 27 Aug 2020 16:24:32 GMT</pubDate>
            <atom:updated>2020-09-01T17:35:37.951Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*tdQrKbdUC9z5NzrbARtxGw.jpeg" /></figure><h4>Introduction:</h4><p>Over the last few years, Reddit’s /r/wallstreetbets has been getting an unbelievable amount of attention. The kind of attention previous controversial subreddits like ‘/r/thefappening’, ‘/r/jailbait’ or even ‘/r/The_Donald’ used to attract.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1q2vS_3g6NUdSbXTr_Zepg.png" /></figure><p>It’s not uncommon for redditors (as they are called) to garner recognition as a group nor is it going to be the last time. So why is ‘/r/wallstreetbets’ (we’ll call it WSB from here on out) occupying space in the mind of investors and financial professionals?</p><p>Because they have the ability to move markets.</p><p><em>That’s our hypothesis.</em></p><h4>Thought Process:</h4><p>WSB has over 1,400,000 subscribers (only just in the top 250 subreddits with a rank of 240) but they generate roughly 30,582 comments per day (ranked 7th across all of Reddit, which itself is the 5th most visited website across the entire internet) and has over 2,000,000 comments total (ranked 8th).</p><p>That is a fairly significant amount of comments and a wildly high ranking considering some other subreddits are solely focused on discussions (/r/AskReddit for example, focuses on questions and answers).</p><p>The assumption here is that if WSB selects a long or short position on a stock, then there is a significant chance of generating herd mentality that can create an effect on individual tickers and/or certain ETFs (Exchange Traded Funds).</p><p>WSB users and lurkers (those who visit the subreddit but don’t necessarily post or subscribe) can potentially operate as a single entity working to turn the tides on the market due to their large volume and relative ease of transacting with the public market (largely thanks to platform such as Robinhood).</p><p>I was fortunate enough to grab Robinhood user’s holdings from a site called robintrack.net which was <a href="https://www.bloomberg.com/news/articles/2020-07-23/hedge-funds-approach-robintrack-to-keep-eyes-on-tiny-investors">recently tapped by Hedge Funds</a> to keep an eye on the so-called ‘tiny’ investors operating with the digital broker.</p><p>The goal is to gather the data, clean it up, explore and begin modelling to understand if there is any predictive value within the subreddit.</p><h4>Overview:</h4><p>I’ve set myself the task to do the following to analyse the effects that the subreddit has on the market (if at all):</p><p>1- Gather all comments from a WSB data repository (available on Kaggle)</p><p>2- Clean up the data. Seriously clean it. It’s Reddit.</p><p>3- Sentiment analysis across all comments</p><p>4- Resample the data by day</p><p>5- Gather market data</p><p>6- Merge data along the date index</p><p>7- Run classification models</p><p>8- Analyse results and scores</p><p>9- Scrape Reddit using PRAW (Reddit API) and Pushshift (Reddit Search Application) for up to date data.</p><p>10- Iterate.</p><p>11- Report conclusions.</p><p>A simple 10 step process right?</p><p>Well.. not exactly.</p><h4><strong>Data:</strong></h4><p>The best place to search for Reddit datasets? Well, Reddit! After some digging, I found that a user on /r/algotrading (a subreddit focused on algorithmic trading.. that really didn’t need a description did it?) that had compiled over 2.5 million comments from WSB from 2012 until the end of 2018. Perfect! (<a href="https://www.kaggle.com/theriley106/wallstreetbetscomments">Link</a>)</p><p>After downloading the 1GB data file, I loaded it into Jupyter Lab and began the process of cleaning.</p><p>This is what the first row looked like in its raw form:</p><pre>{&#39;body&#39;: &#39;Lol. Yeah, Welp.&#39;,<br>  &#39;score_hidden&#39;: False,<br>  &#39;archived&#39;: False,<br>  &#39;name&#39;: &#39;t1_cl52foo&#39;,<br>  &#39;author&#39;: &#39;JamesAQuintero&#39;,<br>  &#39;downs&#39;: &#39;0&#39;,<br>  &#39;created_utc&#39;: &#39;1412888184&#39;,<br>  &#39;subreddit_id&#39;: &#39;t5_2th52&#39;,<br>  &#39;link_id&#39;: &#39;t3_2ikdc5&#39;,<br>  &#39;parent_id&#39;: &#39;t1_cl47s92&#39;,<br>  &#39;score&#39;: &#39;1&#39;,<br>  &#39;retrieved_on&#39;: &#39;1426604732&#39;,<br>  &#39;controversiality&#39;: &#39;0&#39;,<br>  &#39;gilded&#39;: &#39;0&#39;,<br>  &#39;id&#39;: &#39;cl52foo&#39;,<br>  &#39;subreddit&#39;: &#39;wallstreetbets&#39;,<br>  &#39;ups&#39;: &#39;1&#39;}</pre><p>As expected, a fair amount of comments were.. colourful and nonsensical without any context and simply useless for our analysis.</p><pre><em>#First thing, let&#39;s load in our data from json</em></pre><pre>file_path = &#39;YOURFILEPATH&#39;</pre><pre>empty = []<br>for line in open(file_path, &#39;r&#39;):<br>    empty.append(json.loads(line))</pre><pre><em>#Cast it to a dataframe</em></pre><pre>df = pd.DataFrame(empty)<br>df.head()</pre><pre><em>#Begin cleaning the data<br>#Drop columns that serve no purpose in our analysis</em></pre><pre>df[&#39;date_created&#39;] = pd.to_datetime(df[&#39;created_utc&#39;].astype(int), unit=&#39;s&#39;)<br>df.drop(columns=[&#39;created_utc&#39;,&#39;archived&#39;, &#39;controversiality&#39;,&#39;retrieved_on&#39;,&#39;downs&#39;,&#39;ups&#39;,&#39;subreddit&#39;], inplace=True)</pre><pre>df[&#39;date&#39;] = df[&#39;date_created&#39;].dt.date<br>df.drop(columns=[&#39;date_created&#39;,&#39;gilded&#39;,&#39;link_id&#39;,&#39;id&#39;, <br>                 &#39;score_hidden&#39;, &#39;name&#39;, &#39;author&#39;, &#39;subreddit_id&#39;, &#39;parent_id&#39;, &#39;author_flair_text&#39;, &#39;author_flair_css_class&#39;,&#39;distinguished&#39;, &#39;score_hidden&#39;, &#39;name&#39;, &#39;author&#39;, &#39;subreddit_id&#39;, &#39;parent_id&#39;, &#39;author_flair_text&#39;, &#39;author_flair_css_class&#39;,&#39;distinguished&#39;], inplace=True)</pre><pre><em><br>#Drop all deleted values</em></pre><pre>df = df.drop(df[df[&#39;body&#39;].map(lambda x: str(x)==&quot;[deleted]&quot;)].index)</pre><p>Now that our dataframe is clean, we have exactly what we need to begin sentiment analysis on each individual comment using <a href="https://pypi.org/project/vaderSentiment/">VADER Sentiment Analysis.</a> Sometimes I wonder if I chose VADER just because of its name.. It’s definitely part of the reason, but the real reason is VADER’s ability to analyse the sentiment across social media comments and posts. The Reddit comments that I have extracted fit that mould perfectly.</p><p><strong><em>VADER Sentiment Analysis: </em></strong><em>VADER (Valence Aware Dictionary and sEntiment Reasoner) is a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed in social media. It is fully open-sourced under the </em><a href="http://choosealicense.com/"><em>[MIT License]</em></a></p><pre>#BRING ME VADER</pre><pre>analyser = SentimentIntensityAnalyzer()</pre><pre>scores=[]<br>for item in tqdm(df[&#39;body&#39;]):<br>    sentiment_score=0<br>    try:<br>        sentiment_score=sentiment_score+analyser.polarity_scores(item)[&#39;compound&#39;]<br>    except TypeError:<br>        sentiment_score=0<br>    <br>    scores.append(sentiment_score)<br>    <br>df[&#39;sentiment_score&#39;] = scores</pre><p>A look at the dataframe:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/1*5Q2j4DBZ_Gf01vSnhtDN4w.png" /><figcaption>Not bad. Nice and clean with the date, score, body and sentiment score.</figcaption></figure><p>At this point, I needed to decide whether or not VADER was sufficient so to be safe and make sure the analysis takes term frequency into consideration, I ran SpaCy (<a href="https://spacy.io/">industrial-strength natural language processing</a>) on the body text to clean it up.</p><p><em>Note: If you are running this code block, keep in mind that it took me a casual and, not at all annoying, 10 hours to run.</em></p><pre>import codecs<br>import unidecode<br>import re<br>import spacy<br>import en_core_web_sm<br>nlp = en_core_web_sm.load()<br>nlp = spacy.load(&#39;en_core_web_sm&#39;)</pre><pre>def spacy_cleaner(text):<br>    try:<br>        decoded = unidecode.unidecode(codecs.decode(text, &#39;unicode_escape&#39;))<br>    except:<br>        decoded = unidecode.unidecode(text)<br>    apostrophe_handled = re.sub(&quot;’&quot;, &quot;&#39;&quot;, decoded)<br>    expanded = &#39; &#39;.join([contraction_mapping[t] if t in contraction_mapping else t for t in apostrophe_handled.split(&quot; &quot;)])<br>    parsed = nlp(expanded)<br>    final_tokens = []<br>    for t in parsed:<br>        if t.is_punct or t.is_space or t.like_num or t.like_url or str(t).startswith(&#39;@&#39;):<br>            pass<br>        else:<br>            if t.lemma_ == &#39;-PRON-&#39;:<br>                final_tokens.append(str(t))<br>            else:<br>                sc_removed = re.sub(&quot;[^a-zA-Z]&quot;, &#39;&#39;, str(t.lemma_))<br>                if len(sc_removed) &gt; 1:<br>                    final_tokens.append(sc_removed)<br>    joined = &#39; &#39;.join(final_tokens)<br>    spell_corrected = re.sub(r&#39;(.)\1+&#39;, r&#39;\1\1&#39;, joined)<br>    return spell_corrected</pre><p>Once the ~2.7 million rows of data were finally cleaned, I decided to run VADER again but this time return the positive and negative sentiment alongside the compound sentiment. The process and code is the same as above with the addition of a positive and negative list:</p><pre>analyser = SentimentIntensityAnalyzer()</pre><pre>compound_scores=[]<br>positive_scores=[]<br>negative_scores=[]</pre><pre>for item in tqdm(df[&#39;body&#39;]):<br>    positive_score=0<br>    negative_score=0<br>    compound_score=0<br>    try:<br>        positive_score=positive_score+analyser.polarity_scores(item)[&#39;pos&#39;]<br>        negative_score=negative_score+analyser.polarity_scores(item)[&#39;neg&#39;]<br>        compound_score=compound_score+analyser.polarity_scores(item)[&#39;compound&#39;]<br>    except TypeError:<br>        sentiment_score=0<br>    <br>    positive_scores.append(positive_score)<br>    negative_scores.append(negative_score)<br>    compound_scores.append(compound_score)<br>    <br>    <br>df[&#39;compound_score&#39;] = compound_scores<br>df[&#39;positive_score&#39;] = positive_scores<br>df[&#39;negative_score&#39;] = negative_scores</pre><p>Dataframe update:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n2WeVqhYRd5zFofbLPkAKw.png" /><figcaption>Ah, how clean.</figcaption></figure><p>In our current dataframe, each date has multiple comments and sentiment scores associated with it so we need to combine each common date together to build our feature set. I tried and tested two options, the first was to resample on the day of the week and take the mean of the sentiment scores which would then neglect the body text that I need to use for NLP later on (you can’t take the mean of a string). Instead I chose to resample and sum each entry for that index position.</p><h4>Feature Engineering:</h4><p><strong>Pulling market data:</strong></p><p>Pulling market data is extremely easy whether you want to use Quandl or Yahoo Finance (as I did). Import the package yfinance and use the download method to get the ticker data (Open, Close, Adj Close, High, and Low).</p><p>Our dates, based on the data are from 2012–04–11 until 2018–10–31. To start the analysis I selected DJI (the Dow Jones Industrial Average, which tracks the stock performance of 30 large US companies). I will also be using SPY (an ETF which tracks the S&amp;P 500) at a later stage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EvKEg29S60qkCmr-NO1gQw.png" /><figcaption>DJI from 2012–04–10 to 2018–10–31. The dates matching our dataframe.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*tCGV-tBMFYHmkg3452ScaQ.png" /><figcaption>Market data for the Dow from 2012 . A quick glance should make you realise that the Dow has increased over 100% since 2012. Crazy.</figcaption></figure><p>Now we merge our dataframe so that everything sits nicely and neatly in one place and create a new column called ‘up’ which is basically the difference between the previous ‘Close’ figure. For example, if yesterday’s Dow closed at 12,000 and today’s at 11,500 then ‘up’ returns 0 and vice versa.</p><p>A<em> quick note: don’t merge your stock data with your dataframe on a datetime index unless you’ve done a great deal of EDA to make sure you aren’t eliminating key values from your data.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LqVr2yRiQhE7g2VLoAutPQ.png" /><figcaption>So beautiful.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M8M_TXBcJMLoFe7ow99J2Q.png" /><figcaption>Graphing the DJI with the compound sentiment score from VADER and the flat green line is the standardised upvotes (score) graph which, looking back, probably should have been removed for the purpose of this exercise. There is a clear albeit only slightly significant relationship but it is too soon to make any conclusions.</figcaption></figure><p><strong>Shifting the ‘Close’ column:</strong></p><p>A way to create feature values for the current time period is to use the .shift() method which.. well.. shifts all the values in our dataframe down one step. This creates a lag in our predictor variables. I’ve shifted the ‘Close’ price 3 times and added them to 3 new columns as additional features for our model.</p><h4>Modelling:</h4><p>Now let’s create our target variable (‘up’), create more features using TF-IDF, stack the sparse matrices and run some models.</p><pre>#Test on the classification first and then regress on the actual close price</pre><pre><em>#Target variable and features set up</em></pre><pre>y = df.up<br>X=df[[score&#39;,&#39;compound_score&#39;,&#39;positive_score&#39;,&#39;negative_score&#39;,&#39;Open&#39;,&#39;High&#39;,&#39;Low&#39;,&#39;Close&#39;,&#39;AdjClose&#39;,&#39;Close_shift_1&#39;,&#39;Close_shift_2&#39;,&#39;Close_shift_3&#39;]]</pre><pre><em><br>#Create a time series split with 1405 rows in our training set and 200 in our test set.</em></pre><pre>n = 1405<br>X_train, y_train = X[:n], y[:n]<br>X_test, y_test = X[n:], y[n:]</pre><pre><br><em>#TF-IDF on the cleaned test column</em></pre><pre>tvec = TfidfVectorizer()<br>tfed_train = tvec.fit_transform(df.cleaned_text[:1405])<br>tfed_test = tvec.transform(df.cleaned_text[1405:])</pre><pre><br><em>#Convert the X_train and X_test to sparse matrices so we can stack it with the tvec variables</em></pre><pre>X_sparse = scipy.sparse.csr_matrix(X_train.values)<br>X_sparse_test = scipy.sparse.csr_matrix(X_test.values)</pre><pre><br><em>#Hstack the sparse matrices</em></pre><pre>X_train = scipy.sparse.hstack((X_sparse, tfed_train))<br>X_test = scipy.sparse.hstack((X_sparse_test, tfed_test))<br></pre><pre><em>#Standardize the stacked X features </em></pre><pre>sc = StandardScaler(with_mean=False)<br>X_train = sc.fit_transform(X_train)<br>X_test = sc.transform(X_test)</pre><pre><br><em>#Create a time series split</em></pre><pre>ts = TimeSeriesSplit(n_splits=7)</pre><pre>splits = [(tr, te) for (tr, te) in ts.split(X_train)]</pre><p><strong>Our baseline:</strong></p><p>We get the value counts across the whole target variable and normalise.</p><pre>y.value_counts(normalize=True)</pre><pre>1: 0.540187</pre><pre>0: 0.459813</pre><h4><strong>The Models:</strong></h4><p>I ran a GridSearch across all 4 models below with a fairly large list of parameters to kickstart the modelling process.</p><p><strong>1- LogisticRegression:</strong></p><p><em>Parameter Grid:</em></p><p><em>{‘C’: np.logspace(-5, 5, 15),<br> ‘penalty’: [‘l1’, ‘l2’],<br> ‘fit_intercept’: [True, False],<br> ‘max_iter’: [100000],<br> ‘verbose’: [1],<br> ‘random_state’: [7]}</em></p><p><strong>2- DecisionTreeClassifier:</strong></p><p><em>Parameter Grid:</em></p><p>{<em>‘n_estimators’: [5, 10, 25, 40],<br> ‘max_depth’: [3, 5, 9]}</em></p><p><strong>3- KNeighborClassifier:</strong></p><p><em>Parameter Gird:</em></p><p><em>{‘n_neighbors’: [1, 3, 5, 10, 15, 20, 25]}</em></p><p><strong>4- RandomForest:</strong></p><p><em>Parameter Grid:<br>{ ‘n_estimators’: [5, 10, 25, 40],<br> ‘max_depth’: [3, 5, 9]}</em></p><h4><strong>The Results:</strong></h4><p>The only model that returned a score above the baseline was: LogisticRegression. The good-ole trusty LogisticRegression.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*2wrSvyEQRHUm85fpakjLBw.png" /><figcaption>Just above the baseline when predicting whether or not the market will close higher or lower than the day before on unseen data.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/371/1*RxFB5hwqvaQUED-D_D1CsA.png" /><figcaption>Our model doesn’t seem to misclassify 0s as much as 1s (decrease in close price versus increase).</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/1*6jpTEOolgWxhSkM0FNEtDQ.png" /><figcaption>A fairly decent AUC. Our model classifies the classes fairly correctly.</figcaption></figure><h4><strong>Conclusions after the first round of modeling:</strong></h4><p>The LR scores were above the baseline while others fell shockingly short. As expected, I suppose. A few key considerations:</p><ul><li>There is an opportunity here to use PCA to remove multidimensionality between correlated variables (unsupervised linear dimensionality reduction algorithm to find a more meaningful basis or coordinate system for our data). PCA is especially useful if we find that some variables are strongly correlated but can’t necessarily tell which ones to remove.</li><li>I question my decision to run TFIDF on each comment and whether or not that was necessary considered we are only interested with the positive or negative sentiment.</li><li>There are more models that I could have tried and implemented to see if the results would be any better or more significant. Support Vector Machines is one I wish I had more time to try. The same goes for Naive Bayes.</li><li>We did not include any Robinhood user data as a feature in these models. The dates preceded Robinhood and thus it did not make sense to include it at all. There was a short period of overlap between my data and Robinhood user holdings, which I will touch upon later in this article.</li><li>Going forward, I would like to change my target variable from the Dow Close to the SPY Close because I believe a full market ETF would give me a better response to the diversity of conversations on WSB.</li><li>What I have not written in the article is the painstaking amount of time I spent trying to extract individual tickers from each comment from 2012 using Regex and a stock list of the most traded Robinhood stocks by iterating through each comment one by one and assigning each a VADER compound sentiment score. Oh well, as I said before, I am also a self-learning machine. I will only get better.</li></ul><p><strong>Quick side analysis of the overlapping dates from our Robinhood user holding dataset and the massive Kaggle dataset (used above):</strong></p><p>The folder from <a href="http://robintrack.net/">robintrack.net</a> contains 8,562 files each corresponding to a stock ticker. I combined all the files together into one dataframe, resampled by day and summed up all the values in the ‘users_holding’ column to get the total number of holdings with Robinhood.</p><p>Our overlapping dates for both datasets are from 2018–05–02 until 2018–10–30. Our final shape ended up at (124, 15), not a large enough dataset to extract any conclusions but a worthy side analysis nonetheless.</p><p>I ran the same exact preprocessing procedures as above and ran a sole GridSearch on a LogisticRegression model with the associated classification report and confusion matrix.</p><p>And.. as expected, the results were phenomenal. The small dataset alongside the addition of aggregated Robinhood holdings over time as an additional feature significantly helped my scores. The model generalised well with great precision and recall scores <em>(precision measures the percentage of our results that were actually relevant to our model (TP/TP+FP) while recall is the percentage of total results that were correctly classified by our model (TP/TP+FN)).</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T3m3Ubvp9gkPGVHaURUAxQ.png" /><figcaption>So far, the best CV training and test score I have seen so far. Almost all models have had perfect training scores so far.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*nxaCrIhr8unMQzb-OdQu3w.png" /><figcaption>The classification report for test predictions.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*yJgR_i3nrIZlXUuD4wKogw.png" /><figcaption>This gives us a very clear picture on the small sample we have for our test set. Great scores, but the sample if far too small to infer anything at all.</figcaption></figure><p>Moving on!</p><h3>Round 2 (Scraping Data From Reddit and Engineering More Features)</h3><h4>Overview:</h4><p>This time we are going to do things a little differently. The goal here isn’t to change everything around in hopes of getting significantly better scores but to illustrate the differences feature engineering, transformations and up-to-date data can make.</p><h4>Thought Process:</h4><p>The existing dataset that we had used was a bulk export without any information about how the data was gathered or which threads they were extracted from. While the data was validated by myself by checking threads in Reddit to match some of the comments on the URL ID and Comment ID, there is an opportunity to gather more relevant data.</p><h4>Data:</h4><p>To scrape Reddit we will be using Pushshift (Reddit search application) and PRAW (Reddit’s API).</p><pre><em>#Function to grab all the information from Pushshift</em></pre><pre>def getPushshiftData(query, after, before, sub):<br>    url = &#39;<a href="https://api.pushshift.io/reddit/search/submission/?title=&#39;+str(query)+&#39;&amp;size=1000&amp;after=&#39;+str(after)+&#39;&amp;before=&#39;+str(before)+&#39;&amp;subreddit=&#39;+str(sub)">https://api.pushshift.io/reddit/search/submission/?title=&#39;+str(query)+&#39;&amp;size=1000&amp;after=&#39;+str(after)+&#39;&amp;before=&#39;+str(before)+&#39;&amp;subreddit=&#39;+str(sub)</a><br>    print(url)<br>    r = requests.get(url)<br>    data = json.loads(r.text)<br>    return data[&#39;data&#39;]</pre><pre><em>#Get the relevant data</em></pre><pre>def collectSubData(subm):<br>    subData = [subm[&#39;id&#39;], subm[&#39;title&#39;], subm[&#39;url&#39;], datetime.datetime.fromtimestamp(subm[&#39;created_utc&#39;]).date()]<br>    try:<br>        flair = subm[&#39;link_flair_text&#39;]<br>    except KeyError:<br>        flair = &quot;NaN&quot;<br>    subData.append(flair)<br>    subStats.append(subData)<br>    <br><em>#Selected subreddit</em></pre><pre>sub=&#39;wallstreetbets&#39;</pre><pre><br><em>#Before and after dates</em></pre><pre>before = &quot;1596240000&quot; #August 1 2020<br>after = &quot;1541030400&quot; #Nov 1 2018</pre><pre><em>#Query</em></pre><pre>query = &quot;Daily Discussion Thread&quot;<br>subCount = 0<br>subStats = []</pre><pre>data = getPushshiftData(query, after, before, sub)</pre><pre>while len(data) &gt; 0:<br>    for submission in data:<br>        collectSubData(submission)<br>        subCount+=1<br>    <br>    <em># Calls getPushshiftData() with the created date of the last submission</em></pre><pre>    print(len(data))<br>    print(str(datetime.datetime.fromtimestamp(data[-1][&#39;created_utc&#39;])))<br>    after = data[-1][&#39;created_utc&#39;]<br>    data = getPushshiftData(query, after, before, sub)</pre><pre><em>#Gather the data into a dataframe</em></pre><pre>data = {}<br>ids = []<br>titles = []<br>urls = []<br>dates = []<br>flairs = []</pre><pre>for stat in subStats:<br>    ids.append(stat[0])<br>    titles.append(stat[1])<br>    urls.append(stat[2])<br>    dates.append(stat[3])<br>    flairs.append(stat[4])</pre><pre>data[&#39;id&#39;] = ids<br>data[&#39;title&#39;] = titles<br>data[&#39;url&#39;] = urls<br>data[&#39;date&#39;] = dates<br>data[&#39;flair&#39;] = flairs</pre><pre>df = pd.DataFrame(data)<br>df = df[df[&#39;flair&#39;] == &#39;Daily Discussion&#39;</pre><p>In my research, I found that the best subreddits for analysis, where the comments are less drunken frat boy conversations (that was me putting it nicely) but more analysis and investment decisions were ‘Daily Discussions’, ‘Technicals’ and ‘Stocks’. There were a few more threads here and there but, to be honest, in the interest of time, I had to pick the most frequent thread with the highest number of good quality investment conversations (which is a tall order, I mean, to the point that I don’t want to paste 80% of the comments I scraped due to the vulgar and obscene content).</p><p>Now we scrape the comments within the threads using PRAW.</p><pre><em>#Connect to PRAW</em></pre><pre>reddit = praw.Reddit(client_id=&quot;YOURIDHERE&quot;, client_secret=&quot;YOURSECRETHERE&quot;, user_agent=&quot;YOURUSERAGENT&quot;)</pre><pre><em>#Collect comments</em></pre><pre>comments_by_day = []</pre><pre>for url in df_1[&#39;url&#39;].tolist():<br>    try:<br>        submission = reddit.submission(url=url)<br>        submission.comments.replace_more(limit=0)<br>        comments=list([(comment.body) for comment in submission.comments])<br>    except:<br>        comments=None<br>    comments_by_day.append(comments)</pre><p>At this point, the dataframe has been populated but before I illustrate its primary form, let’s run VADER and aggregate the sentiment across each comment within a Daily Discussion thread.</p><p>Then we merge it with the market data. This time, as mentioned, we will be pulling the data for the S&amp;P 500 tracker, SPY, from 2018–11–01 (the end date of our last dataset ) to 2020–08–01 (just about a month ago when I started this project).</p><p>Let’s plot the compound sentiment score along with the SPY market data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TJwpUqpSYyrVt7KDRjZncQ.png" /><figcaption>The trends are visible although still quite erratic.</figcaption></figure><p><strong>Adding Bullish And Bearish Sentiment Beyond Vader:</strong></p><p>There’s still a lot of content left on the table so let’s gather post titles using Pushshift and rather than run VADER on the titles, I will create a bullish and bearish list of words and run RegEx to check for the presence of said words in each comment</p><p><strong>Bullish</strong>: <em>call, long, going up, rocket, buy, long term, bulls, green..</em><br><strong>Bearish</strong>: <em>put, short, tits up (sorry about this one), drop, bear, sell, red, leave..</em></p><p>After I complete the RegEx and align the bullish and bearish scores to the dataframe, I’ll standardized the scores across a given day (since there are multiple posts created in a day).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jqlXl15zoeArlmq27_RsLQ.png" /><figcaption>Our bull and bear dataframe with the unique post titles under ‘title’ and the individual bull and bear scores on the rightmost columns pre-standardizaion.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DcYn8448cAMJ3yF85AMk9Q.png" /><figcaption>Our SPY market data combined with the standardised bull and bear scores.</figcaption></figure><p>Now I can plot the bullish and bearish scores with the Close price of SPY over our time period.</p><p><strong>Bullish scores:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FzJr8OZ4SH-VsCsXttFLoQ.png" /><figcaption>A little all over the place but enough to give us an overview of the trends between post titles and market data.</figcaption></figure><p><strong>Bearish scores:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WhyuCb-lrJLYbXnbODCmpw.png" /><figcaption>The bear scores should technically point in the opposite direction and this is most apparent when we look at the massive dip during the pandemic.</figcaption></figure><p>We can tell from the graphs that our VADER compound sentiment score (which I will just call sentiment score for short) and bullish and bearish sentiment scores seesaw quite significantly. It isn’t pleasing to look at (not to say that’s the purpose for these graphs but it does indeed help) and doesn’t give us much room to make inferences.</p><p>That’s when I discovered this amazing article by <a href="https://medium.com/@awrd2019?source=follow_footer--------------------------follow_footer-">Arjun Rohlfing-Das</a> who, by some coincidence is currently studying at UVa, my alma mater. The world is works in wondrous ways. In fact, most of the code I used for this upcoming section was inspired by <a href="https://medium.com/analytics-vidhya/sentiment-analysis-for-trading-with-reddit-text-data-73729c931d01">his fantastic article</a> and tuned and shaped for my own purposes.</p><p>Back to the graph, as Arjun stated, a great way to work with graphs that swing erratically is to use Fourier transformation.</p><p><strong><em>Fourier Transformation</em></strong><em>: In mathematics, a </em><strong><em>Fourier transform</em></strong><em> (FT) is a mathematical </em><strong><em>transform</em></strong><em> that decomposes a function (often a function of time, or a signal) into its constituent frequencies, such as the expression of a musical chord in terms of the volumes and frequencies of its constituent notes.</em></p><p>I transformed the sentiment scores by performing the following:</p><pre>#<em>Fourier transform the sentiment score</em></pre><pre>close_fft = np.fft.fft(np.asarray(merged_df[&#39;compound_score&#39;].tolist()))<br>fft_df = pd.DataFrame({&#39;fft&#39;:close_fft})<br>fft_df[&#39;absolute&#39;] = fft_df[&#39;fft&#39;].apply(lambda x: np.abs(x))<br>fft_df[&#39;angle&#39;] = fft_df[&#39;fft&#39;].apply(lambda x: np.angle(x))<br>fft_list = np.asarray(fft_df[&#39;fft&#39;].tolist())</pre><pre>for num_ in [5, 10, 15, 20]:<br>    fft_list_m10= np.copy(fft_list); fft_list_m10[num_:-num_]=0<br>    merged_df[&#39;fourier &#39;+str(num_)]=np.fft.ifft(fft_list_m10)</pre><p>This gave me a range of transformations components to pick and chose from. Have a look below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jwDmB-b18gKHyN52ZJeDXA.png" /><figcaption>The VADER compound sentiment score with the Fourier transformed curves overlaid.</figcaption></figure><p>This is a much better representation of the swings and changes with the sentiment over time. As always, and at this point I wonder if I should default to this strategy, I will normalise the Fourier transformations I just created.</p><p>They say normalisation is the path to achievement.</p><p>..there’s a high chance that I just say that to myself but we’ll look past it.</p><p>Anyways, here’s more graphs.</p><p><strong>Fourier Bull Scores With Normalised SPY Close Price:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZTsHIbDbklPdJH79GPBErQ.png" /><figcaption>Bull goes down when market goes down. However.. we still don’t know which comes first.</figcaption></figure><h4>Modelling</h4><p>We’re finally at the modelling stage with this dataset. At this point, it looks exciting and I am expecting some overfitting but if I can GridSearch with the right parameters and regularisation techniques, I should be able to reduce that probability.</p><p>Another concern of mine before moving forward is that the data looks slightly sparse. The shape of the dataframe is (255, 24) which means we have at least 51 weeks of data (about a year’s worth) which is quite sufficient considering the volume of online conversations over the time period</p><p>This is what all of our final columns look like at a glance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/113/1*5yerPElbhHSQpKDUe-CEdQ.png" /></figure><p>We shifted the Close price column 3 times for this dataset for this dataset as we did above. There are new additions that weren’t present prior such as the Fourier transformations, and the normalised sentiment as well.</p><p>Our target variable is ‘up’ which is essentially a binary target that returns 0 if the difference between the current day’s Close price and the day prior is less than 0 and vice versa.</p><p>There are a lot of complex time series, regression and neural network models that I could (and should) have modelled for but in all honesty, they are far above my pay grade at the moment nor did I have the computing power or time to run them. That will obviously change as I continue down this fantastic path of predictive analysis and data exploration.</p><p><strong>Baseline To Beat:</strong></p><pre>y.value_counts(normalize=True)</pre><pre>1- 0.6<br>0- 0.4</pre><h4><strong>The Models:</strong></h4><p>All models were GridSearched with a wide range of parameters.</p><p><strong>1- LogisticRegression:</strong></p><p><em>Parameter Grid:</em></p><p><em>{‘C’: np.logspace(-5, 5, 15),<br> ‘penalty’: [‘l1’, ‘l2’],<br> ‘fit_intercept’: [True, False],<br> ‘max_iter’: [100000],<br> ‘verbose’: [1],<br> ‘random_state’: [7]}</em></p><p><strong>2- DecisionTreeClassifier:</strong></p><p><em>Parameter Grid:</em></p><p>{<em>‘n_estimators’: [5, 10, 25, 40],<br> ‘max_depth’: [3, 5, 9]}</em></p><p><strong>3- KNeighborClassifier:</strong></p><p><em>Parameter Gird:</em></p><p><em>{‘n_neighbors’: [1, 3, 5, 10, 15, 20, 25]}</em></p><p><strong>4- RandomForest:</strong></p><p><em>Parameter Grid:<br>{ ‘n_estimators’: [5, 10, 25, 40],<br> ‘max_depth’: [3, 5, 9]}</em></p><h4><strong>The Results:</strong></h4><p>The absolute best results so far. A few model generalised fairly well, returned mean cross validated scores higher than the baseline alongside a great score on unseen data.</p><pre>LogisticRegression<br>Best Parameters:<br>{&#39;C&#39;: 719.6856730011528, &#39;fit_intercept&#39;: False, &#39;max_iter&#39;: 100000, &#39;penalty&#39;: &#39;l2&#39;, &#39;random_state&#39;: 7, &#39;verbose&#39;: 1}<br>Best estimator mean cross validated training score:<br>0.7555555555555555<br>Best estimator score on the full training set:<br>0.9277777777777778<br>Best estimator score on the test set:<br>0.88<br><br><br>DecisionTreeClassifier<br>Best Parameters:<br>{&#39;ccp_alpha&#39;: 0.005, &#39;max_depth&#39;: 16, &#39;max_features&#39;: 3, &#39;min_samples_split&#39;: 20}<br>Best estimator mean cross validated training score:<br>0.6814814814814815<br>Best estimator score on the full training set:<br>0.7944444444444444<br>Best estimator score on the test set:<br>0.5066666666666667<br><br><br>KNN<br>Best Parameters:<br>{&#39;n_neighbors&#39;: 20}<br>Best estimator mean cross validated training score:<br>0.5851851851851851<br>Best estimator score on the full training set:<br>0.6388888888888888<br>Best estimator score on the test set:<br>0.5733333333333334<br><br><br>RandomForest<br>Best Parameters:<br>{&#39;max_depth&#39;: 3, &#39;n_estimators&#39;: 10}<br>Best estimator mean cross validated training score:<br>0.5407407407407407<br>Best estimator score on the full training set:<br>0.8055555555555556<br>Best estimator score on the test set:<br>0.6133333333333333</pre><p>Only DecisionTreeClassifier and KNN failed to beat out the baseline consistently. LogisticRegression is the clear champion while RandomForest had a low CV score but a high score on the test set.</p><p>DecisionTrees are naturally very sensitive to variations in the data set, and our data naturally had a fair amount of variations (think about the key dips in the market, the conversations about lockdown across the internet, etc..) so the model is fitting on all the noise that is present. RandomForest is essentially a collection of decision trees which gives us similar issue.</p><p>The KNeighborsClassifier model operates differently as it assumes similar features exist in close proximity (<a href="https://towardsdatascience.com/machine-learning-basics-with-the-k-nearest-neighbors-algorithm-6a6e71d01761">another fantastic article diving deep into KNN</a>) to the classification in the training set. This generalised fairly well across the models but did not beat the baseline, although it was fairly close.</p><p>The LogisticRegression model works really well for binary classification problems. As explained on this fantastic <a href="https://towardsdatascience.com/introduction-to-logistic-regression-66248243c148">towardsdatascience article</a>: <em>‘the Logistic Regression uses a more complex cost function, this cost function can be defined as the ‘</em><strong><em>Sigmoid function</em></strong><em>’ or also known as the ‘logistic function’ instead of a linear function.’</em></p><p>A fantastic ROC curve on the test set as well:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/1*2tgCqyrNXg9Lq41rhzKuVQ.png" /><figcaption>Almost perfect area under the curve. Fantastic classification results.</figcaption></figure><p><strong>Classification report (training and test set respectively):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*LXpjxr5vqbtAi27jr7K7eA.png" /><figcaption>Report for the training set.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/465/1*yHwNa_zUtmdMw7HUwTpUOA.png" /><figcaption>Report for the test set.</figcaption></figure><p><strong>Confusion matrix (test set):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/366/1*KJTRUC5Yq1mYu4whaqx_pQ.png" /><figcaption>Just beautiful.</figcaption></figure><p><strong>Feature importances:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/212/1*4y67VivhUbg50NeKJUnMBA.png" /><figcaption>As expected the high and low price have serious significant on our data as the day goes on. In terms of our sentiment feature importances, it looks like</figcaption></figure><h4>Conclusion</h4><p>This capstone project opened my eyes to the massive world of data and the power that I now have to craft presentations, analysis and findings to an audience.</p><p>Data science is, indeed, a science but also an art. The science needs to be sound, tested and validated but what is the point without storytelling? One must be able to communicate the preprocessing, hypotheses, mathematical modelling and analysis to every stakeholder involved. That’s the beauty of what I’ve learned in the last 3 months at General Assembly.</p><p>With regards to my project above, the goal was to understand if we can predict a positive or negative movement based on the sentiment obtained from Reddit’s /r/wallstreetbets. What we found is that we can, to a great degree of confidence, predict a binary movement in the market. While there is a lot more analysis to be done with significant feature engineering (potentially removing SPY’s High, Low and Open prices as well as the Volume), I believe there is a story developing here. It isn’t necessarily a finished product (most of data science and storytelling never is) but it illustrates the effects that /r/wallstreetbets can have on the market.</p><p>These traders are active, operate in volumes and, growing day by day.</p><h4>Potential Next Steps</h4><ul><li>After a short break, re-evaluate the dataset, features and analysis.</li><li>Implement non-dynamical and dynamic forecasting.</li><li>Create rolling windows for further analysis.</li><li>Consider further EDA and scrape more up to date data</li><li>Understand the correlation between keeping High, Low and Open prices as features for our modelling.</li><li>Learn more about quantitative analysis in finance.</li><li>Explore further financial algorithms and the way they process inputs.</li><li>Understand if there are new and interesting ratios that I can implement to further facilitate my analysis and story.</li></ul><h4>About Me</h4><p>My name is <a href="https://www.linkedin.com/in/xazainkhan/">Zain Khan,</a> a recent graduate of <a href="https://generalassemb.ly/">General Assembly</a>’s Data Science Immerse taught by<a href="https://scholar.google.co.uk/citations?user=Zp_If8EAAAAJ&amp;hl=en"> Christoph Rahmede</a> and Daniel Chow. A previous 2-time founder and General Partner at Mandeleo Capital.</p><h4>Parting Statement</h4><p>Thank you Christoph and Dan for your undying patience and commitment to all of us. I will be forever grateful.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d7716516c8e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/i-rr/can-we-actually-predict-market-change-by-analyzing-reddits-r-wallstreetbets-9d7716516c8e">Can we actually predict market change by analyzing Reddit’s /r/wallstreetbets?</a> was originally published in <a href="https://medium.com/i-rr">I,RR</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting Data Science Salaries Scraped From Indeed.com]]></title>
            <link>https://medium.com/i-rr/predicting-data-science-salaries-scraped-from-indeed-com-645d699d06b9?source=rss-494843a3cdbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/645d699d06b9</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[regression-analysis]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Zain Khan]]></dc:creator>
            <pubDate>Mon, 20 Jul 2020 20:34:05 GMT</pubDate>
            <atom:updated>2020-09-18T10:42:46.087Z</atom:updated>
            <content:encoded><![CDATA[<p>One of my key projects for the General Assembly Data Science Immersive course was to scrape job listings from any job site we choose and apply machine learning algorithms to predict salaries for Data Science roles in the United States.</p><p><em>Note: This project was completed in July 2020.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GA6LIPVXHMCK-x5lPsITXg.jpeg" /></figure><h3>Introduction and Pre-Processing:</h3><p>The first task was simple and straightforward, find the appropriate tags for each job cards and then extract the respective job title, salary and location to our notebook for pre-processing.</p><h4>Web Scraping:</h4><p>This is the base URL that I used to scrape all of the key job information.</p><blockquote>URL = “<a href="https://www.indeed.com/jobs?q=data+scientist&amp;l=San+Francisco&amp;start=10">https://www.indeed.com/jobs?q=data+scientist&amp;l=San+Francisco&amp;start=10</a>&quot;</blockquote><p>The job titles were represented by <em>q query and the location by l</em>. I used the usual web scraping methods illustrated below and crafted 4 separate functions to extract the relevant information from each job listing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yjb-bnqFGaeDRwJgwZ3xsA.png" /><figcaption>My web scrape in action. Beautiful. Beautiful Soup.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*Sa2-czG9micAdJ23CBHMrA.png" /><figcaption>Each function used to extract the information required.</figcaption></figure><p>From there, I needed to begin scraping with all of the cities I want to include in the scrape as well as job titles. This took a casual 3 hours and 15 minutes. Something I wish I had known at the start.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jnVLVo8Cs1nQ5u1tYxiDSA.png" /><figcaption>Slow and steady wins the race.</figcaption></figure><p>After that long and painful web scrape, I needed to put all of our jobs into a dataframe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kxOL9FpJoDwtkowIbS6NUQ.png" /><figcaption>After a 3 hour web scrape, the hour it took to complete this felt like a minute.</figcaption></figure><p>With all of the data required in a dataframe, I can now run all the amazing models I had learned over the first 6 weeks of the course. Let’s begin by looking at the amount of jobs I actually managed to scrape:</p><pre>Job         17714<br>Company      8756<br>Location     2782<br>Salary        860</pre><p>Since this project is focused on predicting salaries, I need to clean our data to make sure that the job listings I keep for our analysis have an associate salary. This is where I was nervous. The sunk cost fallacy was kicking in. I had waited almost 5 hours to web scrape and add all my data to a dataframe so I needed this to work smoothly.</p><p>After checking for any null values, I found that our lovely ‘Salary’ column had exactly 14,046 missing values. This is getting tight. After removing the rows without an actual salary, I had to clean the column further to remove all of the daily and monthly salary rates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JAL8zyiPLOlB46AiuDPK_Q.png" /><figcaption>I really like the look of this cell.</figcaption></figure><p>A total of 912 values by the end of the data cleaning. Not the biggest sample but sufficient for this project. I probably should not have selected the United States because unlike the United Kingdom most employers or agencies do not list salaries on job postings. You live and learn. I, too, am a self-learning machine.</p><p>Here is what my cleaned dataframe looks like at the end of pre-processing:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kcedWZH0ZDAuMohHo-t85A.png" /><figcaption>I felt like a proud father looking at this.</figcaption></figure><h3>EDA and Setting Up Our Target:</h3><p>The EDA performed on this data set was basic but, as always, necessary. I needed to check if there were any strange and out of the ordinary values in the salary column.</p><pre>count       912.000000<br>mean     120666.084978<br>std       44826.590910<br>min           3.000000<br>25%       91039.000000<br>50%      115555.000000<br>75%      145000.000000<br>max      475000.000000</pre><p>There was a salary value at $3 per year which is definitely a data point that I need to discount so I did just that. Without even thinking twice, I removed it. That’s just how I operate at this point. However, it makes perfect sense because with 912 values, I can afford to move on quickly.</p><p>This made me question whether or not there were other glaring outliers in my dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6GXLEaGVW0wnDPGwjs7Vkg.png" /><figcaption>A histogram on all salaries.</figcaption></figure><p>There are a few values on the higher end which I have an internal debate about removing. Since I am going to be creating dummy variables for my target based on the percentile score, does it really make a difference to my data? Should it? I am going under the assumption that it won’t necessarily effect my model too much, however, after writing and re-reading my analysis, I may learn otherwise.</p><h4>Creating My Target:</h4><p>I decided to test my models on two different target variables.</p><p>1- Multi-class target: 4 different classes each defined by the percentile score of the target.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lh2IaWmOKVa9FpI6yqGrBA.png" /><figcaption>Simple and potentially effective.</figcaption></figure><p><em>Bonus: Baseline Accuracy:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XQTuzNUS2nF7ZniBjT7erQ.png" /></figure><p>2- High/Low target: a simple binary classified target for salaries higher or lower than the median.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BxKXggzfhPggwsqoUJ4H_Q.png" /><figcaption>Extremely simple, and will most likely yield the best results.</figcaption></figure><h3>My Models:</h3><p>The first task was to only use location as a feature for our model. To do that, I had to separate all the locations by city and state and then dummify each city so that we could get a variety of location features for our model.</p><p>This dummification process gave me a total of 196 columns with each city or area as a feature in our model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g8eUm1yyXU5juOuhCAtspQ.png" /><figcaption>Some interesting cities in there even though we had specified a city list when web scraping. So it goes.</figcaption></figure><p>Every X variable (predictor) was split and scaled, it’s just good practice. The target variable here is the multi-class target.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QxCVsfDSjq35uR2jRGXELw.png" /></figure><p>I started by running a grid search on my logistic regression first which yielded a fairly low score:</p><pre>The best parameters for our model: {&#39;C&#39;: 719.6856730011528, &#39;fit_intercept&#39;: True, &#39;max_iter&#39;: 100000, &#39;penalty&#39;: &#39;l2&#39;, &#39;random_state&#39;: 7, &#39;verbose&#39;: 1}</pre><pre>The best LogisticRegression score: 0.31711145996860285</pre><pre>The best score on the test set: 0.28205128205128205</pre><p>This is when I decided to start adding more features like job seniority (analyst, junior and senior) as well as the job titles (data scientist, machine learning, etc..).</p><p>Once again, we split and scale our predictors and begin with the following models:</p><p>1- KNN</p><p>2- DecisionTree</p><p>3- Bagging Classifier with KNN as the base estimator</p><p>4- Bagging Classifier with DecisionTree as the base estimator</p><p>5- Random Forest</p><p>The scores were not flattering at all. In fact, they were barely better than when I just had cities as the predictor. DecisionTree was our best model, but it almost hurts to show how bad of a score it produced.</p><pre>DT CV training score:	 0.36538461538461536<br>DT training score:	 0.7225274725274725<br>DT test score:	 0.34615384615384615</pre><p>I tried and tested many different parameters, went back to clean the data further until I realised maybe I should change my target variable to the simple binary target defining high and low salaries (above or below the median).</p><p>Let’s begin again, this time with a different target, High/Low Salary. Here are the scores and results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/391/1*IQh9PV4mfRuE0pu2wrMJAA.png" /><figcaption>As we can see, an incredible improvement.</figcaption></figure><p>The scores are an obvious improvement and giving our model less work to do when predicting the target seems to have solved our scoring issue. However, we can see that there is still sufficient variance in our model. The cross-validation scores and test scores are still approximately close but our training data seems to vary significantly in comparison.</p><p>This made me want to try and improve the model as much as possible. Others in my class were getting scores slightly (but significantly) better. I figured it may be because they were using data from the United Kingdom which has much better reported and listed salary data.</p><p>I attempted to remove cities that had under 5 job listings because my thinking was that I could feature engineer myself to a better score.</p><p>I was wrong.</p><p>I performed the following on my data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W0Fc-Yfwuj4T3NGEvJsALA.png" /><figcaption>Remove all cities with under 5 jobs reported.</figcaption></figure><p>This lead to scores with much less variance but lower scores overall:</p><pre>logit<br>Best Parameters:<br>{&#39;C&#39;: 0.0071968567300115215, &#39;fit_intercept&#39;: True, &#39;max_iter&#39;: 100000, &#39;penalty&#39;: &#39;l2&#39;, &#39;random_state&#39;: 7, &#39;verbose&#39;: 1}<br>Best estimator mean cross validated training score:<br>0.6497874350495983<br>Best estimator score on the full training set:<br>0.6771978021978022<br>Best estimator score on the test set:<br>0.6538461538461539<br><br><br>cart<br>Best Parameters:<br>{&#39;ccp_alpha&#39;: 0, &#39;max_depth&#39;: None, &#39;max_features&#39;: 3, &#39;min_samples_split&#39;: 40}<br>Best estimator mean cross validated training score:<br>0.6580255077940482<br>Best estimator score on the full training set:<br>0.7129120879120879<br>Best estimator score on the test set:<br>0.6703296703296703<br><br><br>knn<br>Best Parameters:<br>{&#39;n_neighbors&#39;: 1}<br>Best estimator mean cross validated training score:<br>0.5768445914029287<br>Best estimator score on the full training set:<br>0.7074175824175825<br>Best estimator score on the test set:<br>0.6593406593406593<br><br><br>random_forest<br>Best Parameters:<br>{&#39;max_depth&#39;: 3, &#39;n_estimators&#39;: 100}<br>Best estimator mean cross validated training score:<br>0.6401133679735475<br>Best estimator score on the full training set:<br>0.6565934065934066<br>Best estimator score on the test set:<br>0.6428571428571429</pre><p>I was really not impressed at this point.</p><p>I decided to stick with my earlier model and try GradientBoosting to eliminate the effects of poorly chosen variables in my model and analyse the results.</p><p>This is what I ended up with:</p><pre>0.7651098901098901 (training score)<br>0.6222527472527473 (cv score)<br>0.7032967032967034 (test score)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/212/1*fmixuQ6Z9OaQ-5qqi35I5w.png" /><figcaption>Feature importances of my GradientBoostingClassifier</figcaption></figure><p>That’s not too bad. Along with the list of feature importances, I think I have a good model that works better than the baseline. That’s all I could have hoped for at this stage.</p><h4><strong>Here is what I would have done differently:</strong></h4><p>1- I noticed that a few people in class had taken the job summary into account as well. I should have done that had I had more time. I would have used NLP to analyse the different keywords in the job summary and added it as a feature.</p><p>Adding in the summary as a feature would have eliminated some of the guess work that went on once the data was scraped and the EDA began. It would have taken me some more time but it would have been work it to improve my model by 10–15% which, in hindsight, seems doable.</p><p>2- I should have spent more time on EDA to make sure that the right kind of cities and job titles were included and filtered through. This comes back to point 1 above. If I had a summary variable to work with, I could have eliminating a lot of guess work and answered many questions about the validity of my features.</p><p>3- Pick better data. The United States’ job postings were probably not the best selection. The United Kingdom results would have given me more data to play with. My logic was that the US is the largest market for data scientists, hence it will have significantly more data points than the UK (which turned out to be true). However, the quality of the data is equally as important, and while my scraping and analysis was not the best, it would have improved with a significantly better dataset.</p><h3>Classification Reports:</h3><p>I genuinely love classification reports. An insight into how well our models actually are in making any accurate predictions. The final model we selected as the go-to was the last one we discussed, GradientBoostingClassifier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/466/1*uQvDHYoWIDYj23Hp423Kuw.png" /><figcaption>Confusion matrix for our training set.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/471/1*qq-FXVigUKikTG6rLVWoxA.png" /><figcaption>Confusion matrix for our test set.</figcaption></figure><p>Precision measures the percentage of our results that were actually relevant to our model (TP/TP+FP) while recall is the percentage of total results that were correctly classified by our model (TP/TP+FN).</p><p><strong><em>Let’s put it this way. Our model managed to correctly predict the salary 70% of the time (looking at the test report) but only did so 70% of the time.</em></strong></p><p>The ROC-AUC curve is another great illustration of this idea. The area under the curve represents just how often our model is able to predict the classes. The higher the AUC, the better our model is at predicting classes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/408/1*9Nf7OpM43S8WUThQv2Deow.png" /><figcaption>Not too bad considering the flaws that I see in it myself.</figcaption></figure><h3>Conclusion:</h3><p>My model could have been improved in significant ways and the data could have been gathered and cleaned much more effectively. However, I believe I did the best I could with the time given to us.</p><p>Our 70% precision and recall score respectively is better than the baseline, which means we will still predict the classes 20% better than the baseline. That’s a good start. Judging by our ROC curves, we have 3/4 chance that our model will accurately predict the salary based on the models we have selected and tuned.</p><p>Notes for the future: get better at recognising patterns, making decisions, and clearly illustrating the different steps and procedures en route to a model. The modelling part is easy and simple, but the pre-processing, cleaning and EDA are where the magic happens.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=645d699d06b9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/i-rr/predicting-data-science-salaries-scraped-from-indeed-com-645d699d06b9">Predicting Data Science Salaries Scraped From Indeed.com</a> was originally published in <a href="https://medium.com/i-rr">I,RR</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Love In The Time Of Corona]]></title>
            <link>https://medium.com/@xzknz/love-in-the-time-of-corona-6545549a5f2e?source=rss-494843a3cdbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/6545549a5f2e</guid>
            <dc:creator><![CDATA[Zain Khan]]></dc:creator>
            <pubDate>Fri, 13 Mar 2020 14:53:25 GMT</pubDate>
            <atom:updated>2020-03-13T14:53:25.768Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TdHx4ye1fbrdgh9aZURI3Q.jpeg" /></figure><p>What a decade this has been. And it’s only March.</p><p>The last 3 months have felt longer and more intense than all of 2019. Perhaps it is just some recency bias, who knows anymore.</p><p>Speaking of WHO, our global ability to deal with 28 Days Later or World War Z type situations is being put on display. All those hypothetical conversations we had about what we would do during a pandemic are now, unfortunately, slowly being answered.</p><p>Are we doing enough?</p><p>Should we quarantine everyone?</p><p>What should we cancel?</p><p>What should we allow?</p><p>Should I test myself?</p><p>Panic?</p><p>Relax?</p><p>There are too many questions and not enough time to answer them. Those who understand the situation and science behind the virus are busy helping those in need. However, those that don’t necessarily understand the situation thoroughly are starting to share their opinions and, often, agendas.</p><p>I made the mistake of clicking on a Fox News article (<a href="https://www.foxnews.com/opinion/gordon-g-chang-china-falsely-blames-us-for-coronavirus-pandemic">link</a>) which discusses China supposedly blaming the United States for kickstarting the pandemic, and the US, in turn, blaming China for spreading such false information. Somehow, within that conversation they managed to rope in the following: (1) Putin isn’t as big of an enemy as China, especially since they control more supplies, (2) That the Democratic party is being childish by criticising Trump’s response and, my favorite, (3) That isn’t the time to talk about politics and we should just focus on reading the real facts and trust our leaders.</p><p>Crises usually illustrate the true resolve and character of people, cultures and groups. More often than not, we come together to find solutions and answers. Other times we use trajedies to further our own agendas and goals. However, there is something Fox News got right with their final point (albeit with their own ironic agenda). We need to focus our attention on pure facts, trust those in charge and work together.</p><p>There are over 2 billion people on social media networks, with more data shared in a week than the whole calendar year (2009) of the last pandemic (H1N1). You will be inundated with articles, opinions, blog posts (like this one), graphs and images from experts but also from complete novices. Some will be useful and some may not. The only words we should focus on are from our respective country’s health and safety officials. This can and should include the likes of your governor, minister, or whoever else represents you. Leave your issues with them at the door, and realize they are here to help you. We should focus on solutions and provide the least amount of strain on the system that is, unfortunately, designed to serve us during peace time.</p><p>Forget your quabbles, disagreements, anger and resentment to anybody who is around you currently. You voted left but somebody on the right is in charge? So it goes, deal with it and work together for now. You don’t like what Boris said? Understood, and unless you are a doctor or can propose a better solution that can be activated at scale, let’s work with what we have and take precautions. Write down your opinions on a piece of paper if you have and store it for a time when all of this is under control. Until then, save your energy and everyone’s ears for ways in which you can assist.</p><p>There are too many opinion pieces coming out during this time adding fuel to the fire. Those taking advantage of this situation will be remembered after all of this and those who make horrendous decisions will be questioned. I would suggest reading more about the best and worst of social media during this crisis (<a href="https://www.thedrum.com/opinion/2020/03/03/the-pandemic-infodemic-how-social-media-helps-and-hurts-during-the-coronavirus">link</a>).</p><p>But not now.</p><p>For now, let’s take pressure off the system, work together, head to the Winchester and wait for all this to blow over.</p><p>Stay healthy y’all.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6545549a5f2e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>