<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Neil Sinclair on Medium]]></title>
        <description><![CDATA[Stories by Neil Sinclair on Medium]]></description>
        <link>https://medium.com/@neil-ian-sinclair?source=rss-c2f5bfff9f14------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*32FAc9DmhtVfUgsEjuRE0g.jpeg</url>
            <title>Stories by Neil Sinclair on Medium</title>
            <link>https://medium.com/@neil-ian-sinclair?source=rss-c2f5bfff9f14------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 09:01:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@neil-ian-sinclair/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Doing Things The Hard Way]]></title>
            <link>https://neil-ian-sinclair.medium.com/doing-things-the-hard-way-04219e31f904?source=rss-c2f5bfff9f14------2</link>
            <guid isPermaLink="false">https://medium.com/p/04219e31f904</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Neil Sinclair]]></dc:creator>
            <pubDate>Sat, 01 Nov 2025 19:21:01 GMT</pubDate>
            <atom:updated>2025-11-01T19:21:01.278Z</atom:updated>
            <content:encoded><![CDATA[<p>By overusing ChatGPT we’re losing the ability to think — and in the process to make contributions that matter to the world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c6hpiMeYIVivIcwmz4nMsw.png" /><figcaption>Image generated by ChatGPT</figcaption></figure><p>I love using ChatGPT for all sorts of things. When I can explain a code-based task easily and efficiently in English, I often get it to help me write the code: like plotting graphs, processing data, writing API endpoints or much of the frontend of my webiste. Of course, there’s also travel tips, shopping-related research (it helped me massively with the last bicycle I bought) and Github commands I should have learned by now, but still haven’t.</p><p>Of course, there’s also more fun things, like questions about what kind of bird would make the best top-tier management consultant — because I’m sure you’re dying to know, it’s a raven. Sometimes I start to use ChatGPT when I don’t have ideas or, when I’m starting to feel like thinking is an effort. And then I stop…</p><p>It’s been said that social media is eroding our ability to think critically, as we share and consume emotionally charged content presented to us usually with very little context or nuance. Similarly, I think LLMs are eroding our ability to think deeply. Obviously, like almost any tool, ChatGPT/Claude/Gemini etc. are amazing tools if used properly. I have a colleague who uses ChatGPT to help them understand topics in an iterative back and forth manner so they can really grasp the fundamentals of a problem. Another friend uses ChatGPT to help them ask the right kind of questions to take a creative project forward. In both cases they’re using ChatGPT as a kind of mentor.</p><p>I think, however, if you’re asking an LLM to solve something for you as soon as it gets tough, it’s the same as opening a math text book, reading the exerises, skipping to the answers in the back and then saying you’ve grasped the material. Spoiler alert: you haven’t.</p><p>A truth I think we all know about problem solving — and life in general — yet one we often find hard to accept, is that the good, juicy bits always take some trouble to get to. Not only is the reward better (think about that sweet dopamine hit) when you solve a tough problem, but you learn this way. In a deep way. A few days ago, I wrote a little matrix sorting algorithm that used some method from my optimization class in my masters 5 years ago — and it was so much fun turning off Copilot and figuring this out from scratch! I also recently heard a Simon Sinek interview where he pointed out that habitually leaning on ChatGPT to give us the “right” answers in life robs us of some of the hard earned and valuable lessons that we learn when we make mistakes.</p><p>Along with a trend towards developing shallow understandings of topics, I think we’re experiencing a flood of AI-generated content, or “AI slop” as its aptly called. There’s just so much out there and so much of it feels like empty content. It’s essentially information fast food, devoid of anything genuine or authenetic, usually leaving me unsatisfied. When I see one of those posts on LinkedIn with the typical emojis for almost every line (you know the ones, ✅ 🚀) and that snappy short-paragaph style, without even a single mistake, I just scroll past. It’s undoubtedly something unoriginal and empty.</p><p>Not so long ago I used to be such a stickler for perfect grammar, spelling and punctuation, yet nowadays I find the occasional mistake to feel somewhat comforting, as if it signals “This was written by a real human being.”</p><p>The irony of this flood of content is that after a little dip when everyone started to think that creatives were about to all end up unemployed, I think this slop begins to make people who have unique styles — people who worked and developed those for years — more valuable again. The vast majority of content out there just becomes background noise, especially against the few original ideas and art that stand out.</p><p>From my experience as an A&amp;R (talent scout) at Universal Music in the 2010’s I believe that there is a place in the world for generic content. Sometimes we just want to dance and we don’t care what’s playing. But the stuff that really sticks around — and that moves people — is almost always unique and special in some way. This applies to music, to writing, to fine art and to ideas. I don’t think this is going to change in the future.</p><p>What’s the take away here then? In Deep Work, Cal Newport makes an argument (pre-AI content deluge) for why being able to focus in a world constantly driven to distraction by modern technology is a distinguishing trait. I believe that being able to create original content — built upon skills developed through years of deep learning and honing one’s craft — will become something that distinguishes creators amongst the rising tide of AI slop. I lean towards being an AI optimist and believe that AI will help to take over a lot of drudgery, like taking over boring processes or writing throw-away copy that has to be done, but that no one really reads. However, I still think in the coming years that the original art and ideas that really touch and move us will be created by humans who went through the hard work to learn and think through the difficult problems. I’m all for using AI to augment what we do, but in a world that values originality and authenticity, I think we’re still a long, long way from AI replacing us.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=04219e31f904" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Successfully Combining Design Thinking and Data Science]]></title>
            <link>https://medium.com/data-science/successful-combining-design-thinking-and-data-science-8ec6b1b7cc8e?source=rss-c2f5bfff9f14------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ec6b1b7cc8e</guid>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[design-thinking]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[consulting]]></category>
            <dc:creator><![CDATA[Neil Sinclair]]></dc:creator>
            <pubDate>Sun, 13 Feb 2022 12:41:42 GMT</pubDate>
            <atom:updated>2022-02-14T10:43:13.363Z</atom:updated>
            <content:encoded><![CDATA[<h4>Want to know how to use Design Thinking to deliver data science projects that create real business value? Here’s how.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WhijtCPVj03ChXjb" /><figcaption>Photo by <a href="https://www.pexels.com/@picjumbo-com-55570?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels"><strong>picjumbo.com</strong></a> from <a href="https://www.pexels.com/photo/notebook-beside-the-iphone-on-table-196644/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels"><strong>Pexels</strong></a></figcaption></figure><p>Ever since encountering Design Thinking working at Deloitte Consulting back in 2018, I’ve become hooked on it. I love the ethos around redefining problems to get to their core and then quickly iterating on ideas. Mostly, I love the tools and tricks for getting a wide berth of ideas out of a group quickly and creatively in a way that energises everyone.</p><p>I also absolutely love coding, building models and working on analytical problems. So, I left Deloitte in 2020 to head back to school to do a Masters in Data Science. When I re-entered the working world as a data scientist in 2021, I was unsure of whether I’d be able to bring this Design Thinking experience along with me. Luckily for me, I found a natural fit between data science projects and Design Thinking workshops within a few months at my new job. This article outlines how and why Design Thinking can be valuable for your next data science project. It then follows with the ‘what’ of how to implement a Design Thinking workshop in the realm of data science.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gKlUQBpL2VxUTVrp" /><figcaption>The steps of the Design Thinking process; image by author</figcaption></figure><p>When I first became interested in the field of data science it was because I loved the orderly fields of statistics, maths and computer science. I mean, who doesn’t love coming up with predictions where one can confidently state how right (or wrong) the prediction is? However, the truth of working in the field is not so cut and dry. From my perspective, the biggest challenges involve finding the most valuable problems to solve and bringing the people who are impacted by these challenges along for the journey. Now, I know that it’s really easy to look across at another part of a business, put on my data scientist hat, and say something like “oh, what this team needs is a reinforcement learning system to optimise the path of shelf packers”. However, the people <em>in </em>that part of the business have a much better sense of what problems they have and which ones have the highest leverage for them by solving them. Maybe what they really need is a simple dashboard showing which food items have the highest proportion of wastage. It may seem simple, but a tool like that could save millions a week by reducing wastage. I’ve put together a little consulting-esque tool at the end of the <em>Ideating and Feedback</em> section to help you find those leverage points easily when engaging with business.</p><p>A Design Thinking workshop helps a team to go through the first two phases of the Design Thinking process in ‘The Steps of the Design Thinking Process’ image above. We’re then able to ideate, prototype and test based on the insights we collect from the workshop. We empathise by understanding our key stakeholders’ problems and then we go in and further define them so we really understand them. But, why not just do a traditional, requirements defining meeting or workshop? There’s two main reasons: time and energy.</p><p>In my experience, if you need to get executives from one or more areas of the business into a meeting room together, you’d better hurry up and wait. Naturally, they’re incredibly busy and everyone wants a piece of their time and attention. And when you do find a slot in their diaries, you’re incredibly lucky if you can get more than an hour. One of my favourite things about the Design Thinking process is the ability to get a group of different people coming up with ideas really quickly. Although I find that an hour and a half long workshop is cutting it a bit fine, it’s sufficient to get enough of an understanding to start designing solutions. The energy in the room is a second factor that makes these workshops powerful for me. The participants need to come up with ideas quickly and everyone gets to share their ideas. I find that, for most people, this helps to inspire creativity and, in the process, gets peoples’ energy levels up. By using a process that allows us to get the most ideas out of the least time whilst getting people excited about what we’re working on, we’re able to create a real sense of buy-in from our stakeholders. Let’s dive into the process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UUZCsJ7zf3hZymea" /><figcaption>Photo by <a href="https://www.pexels.com/@brunogobofoto?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels"><strong>Bruno Bueno</strong></a> from <a href="https://www.pexels.com/photo/person-writing-on-pink-sticky-notes-3854816/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels"><strong>Pexels</strong></a></figcaption></figure><h4>What you’ll need</h4><ul><li>Different coloured sticky notes — enough so that everyone has at least 15–20</li><li>Whiteboard markers for the sticky notes. Permanent markers will likely cause some unintended permanent damage and thin, ballpoint pens are difficult to read unless you’re right up close to them</li><li>1 white board per group, or alternatively, 2 x large A2 pieces of paper</li></ul><p><strong>Who you’ll need</strong></p><ul><li>Key business stakeholders involved in the area you’re working on — you <strong>need</strong> management buy-in if anything is going to happen</li><li>Two or three people who will actually use the tools or insights you’ll be delivering</li><li>A facilitator (probably you) and a few members of your data science or analytics team</li></ul><h3><strong>Structure of the Workshop</strong></h3><p>From my experience, two approaches have worked for me, although I’d encourage you to mix and match your own ideas.</p><p><strong>Challenge Identification Workshop</strong></p><p>This type of workshop is more broad than the second type and is suited for a situation when the business stakeholders know what their challenges are, but don’t know what you can do for them. The purpose of this workshop is to try to understand the customer pain points as well as you can. You <strong>don’t</strong> want to walk away from this session with any ideas about how to solve these problems. You just really want to understand them.</p><p>Start with a brief introduction of your team and mention that the focus of the day is on understanding customer pain points. I like to do a brief 5-minute ‘Best In Class’ slide or two where I look at companies who are currently doing amazing things, preferably in the data science domain, in the area that our stakeholders work. Although we’re just focusing on pain points and avoiding solution mode, these ‘Best in Class’ examples help to set an inspiring and creative tone for the workshop.</p><p>I then lay down the rules for the session, which go something like: the idea is to get as many ideas out as we can today. We’ll split into two or three groups (more on this below) and everybody will have a set of stickies and a pen. You’re welcome to chat about ideas, but if you get a good idea, you can just write it down and stick it on your whiteboard/sheet. You don’t need agreement from the group. We can get rid of bad ideas later. If you see themes popping up, go ahead and group similar sticky-notes together.</p><p>We then break the workshop up into two or three groups with between four to five people per group. Ideally, put senior managers or executives in separate groups. Get a good mix of your team of analysts or data scientists in the groups, as well as the stakeholders who will be using the tool(s) you build. It’s important for your team to get a real first-hand understanding of the problem. It’ll also be useful later when you’re doing a debrief and trying to remember what was meant by what was scribbled on the stickies. Putting people in groups also helps to speed up the ideation process by getting more ideas out in the room at once — I really found this aspect of the workshops to be one of the key success factors. Putting people in smaller groups also gives participants more space to talk and share ideas, adding more diversity to the mix.</p><p>A key consideration for me here is how much structure we give to the ideation process. Do we go as far as laying out a customer journey map or some other kind of operational map? Or do we just pose a few questions to guide the process? Given that you’re most likely going to be constrained for time, I’d suggest getting <em>some</em> kind of operational map in front of the participants. Here’s an example of one for a fictitious delivery service company.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WXkhIYHqEMYSJ1RG" /><figcaption>Customer journey with operational steps of a fictitious online retailer; image by author</figcaption></figure><p>The advantage of a map like this is that it allows the stakeholders to focus on specific areas of the business or customer journey, which invariably makes it easier to home in on pain points right away. Make sure, of course, that you work with someone in the business stakeholder team beforehand to get these customer journeys sketched out. You won’t have time to do this in the workshop. Obviously, if the participants share a pain point that doesn’t live firmly on the journey map, or if they just want to focus on one or two areas of it, that’s fine. Having this there just helps to streamline the process.</p><p>Keep going around the groups to see how they’re doing. Remind them of the rules (get down as many ideas as possible) and prompt them with questions to get more ideation happening within the groups. Make sure you’re keeping time, and after giving the groups 25–30 minutes, stop the session. Ask one member from each of the groups to present their key ideas back to the rest of the room. Give them three to five minutes per group to do this. Now is also be a good time to place similar stickies into groups or themes. This makes the next part of the process simpler.</p><p>Now for the best part. Hopefully all the groups have come up with a ton of ideas. Now you’ll vote on the best ones. There’s an excellent method called “Dot Voting” that was introduced to me at Deloitte and comes from the excellent book <a href="https://www.thesprintbook.com/">Sprint</a>. Basically, everyone gets three <a href="https://www.carlyandkin.com/home/2018/11/16/4-simple-dot-sticker-activities">dot stickers</a> and is allowed to vote for any ideas — represented here by sticky notes — that they like. They can put all their stickies on one idea or divide them up however they like. No discussions though. Give everyone around three minutes to go vote. They should have a decent understanding of the ideas on the different team’s boards based on the prior feedback sessions. Obviously some stickies will be very close to each other in terms of the ideas on them and hopefully these will be grouped together. If it’s the case that one sticky within a theme of stickies gets all the votes, or even if the votes are all shared equally, I would consider all of those as votes towards the theme.</p><p><strong>Predefined Problem Sculpting Workshop</strong></p><p>The difference between this approach and the first one is that here the business stakeholders already have an idea about what they need. Perhaps it’s a metric of some sort, some kind of customer segmentation or some kind of tool. We follow the same steps as above, except that it’s not really necessary to do ‘best in class’ here. We can also replace the journey map with two simple questions:</p><ul><li>What is it that we’re trying to do?</li><li>Why do we want to do this?</li></ul><p>Together, these questions enable everyone in the room to understand what we’re building and what we’re going to use it for. To give a very retail-led example, imagine you’re working at a bank and you’re developing an ‘affluency score’ for each customer. It’s vital that everyone in the room understands what is meant by affluence. For example, is it how much money someone currently has? It is a measure of their potential spend? Does it refer to their lifestyle and how much debt they may take on and can realistically pay? Although these are all similar, they aren’t quite the same and will have significant implications on how you gather features and build the metric. Defining what it is you’re trying to do would help to define what is meant by the metric/segment/measure you’re developing. Understanding ‘why’ you want this metric is also vital. Is it something we use to set strategic goals? Do we use it to identify cross or upsell opportunities? These, too, will have design implications.</p><p>In a setting like this give each team 15 minutes to answer the ‘what’, followed by a feedback session to the wider group for around three minutes. This is then followed by another 15–20 minute session to answer the ‘why’, again followed by a feedback session to the group. It’s not necessary to vote here on the ideas though. If your institution currently has a metric similar to the one you’re discussing in the session, it’s valuable to get a sense of how it measures up against the ‘what’ and ‘why’ you’ve discussed during the session. This also helps to understand what gaps may or may not need to be filled and can be done in five minutes at the end of the session.</p><h3><strong>Ideating and Feedback</strong></h3><p>It’s not going to be possible to understand the key issues your stakeholders have <strong>and</strong> come up with innovative solutions for them in the short workshop that you have. Because of this, I like to do the following:</p><ul><li>After the workshop (make sure it’s on the same day) get your team together and do a debrief. You should have a few whiteboards or A2 sheets filled with stickies. Write down the key idea that goes with each sticky and/or group of stickies. You could, for example, write out the stickies in an Excel sheet and then put the key messages next to them. Later, you can move this to a powerpoint. An example for the e-tailer whose fictitious customer journey was shown above might look like this:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yXvrBbNpxtO3TzybKy9_bQ.jpeg" /><figcaption>An example of stickies from a fictitious workshop along with the key ideas that would accompany these; image by author</figcaption></figure><ul><li>You might need some clarification from the key stakeholders about what you think the key points are from the workshop. As you’re starting to think of solutions to the pain points you discussed, reach out to stakeholders to get their opinions on your thoughts and to understand where you might be able to get the data from. For example, if a key pain point for your stakeholders is not knowing what customers’ biggest issue are, but they don’t collect any customer feedback data, it won’t be possible in the near term to provide a solution here</li></ul><p>I would encourage the data science team to ideate about how you think you’ll be able to solve the customer pain points. I’m a firm believer in coming back to stakeholders with some ideas that you can refine instead of coming back to them solely with a set of refined problem statements. It also allows your team to use their data science expertise, but with clear business goals in mind.</p><p>When sketching out solutions, I’d encourage you to think about the solutions as products. That is, think about the final state of what your team could co-create with the stakeholders to meet their customer pain points in one or more areas. Let’s say a key pain point for customers is long waiting times at the call centre, often for the same, simple-to-fix problems. This could be, for example, to get help on submitting a claim if you’re an insurer. The overall project here might be ‘automated assistance for customers’ which will, of course, be broken up into multiple steps. The first step might be a simple rule-based chatbot MVP, whereas the final step might be a conversational AI.</p><p>You’re almost certain to come up with more ‘products’ than your team will be able to complete in any kind of reasonable time. So, here comes one of my favourite devices from management consulting: the 2 x 2 matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/490/0*a6czeX__sRxDwnsA" /><figcaption>Priortisation Matrix for projects; image by author</figcaption></figure><p>Here we plot our projects based on <em>impact</em> and <em>difficulty</em>. In the top left we have ‘low hanging fruit’ — these are easy to achieve, impactful projects that we should start on right now. In the lower left we have ‘quick wins’ — these are easy to achieve, but less impactful. In the top right, we have ‘long term projects’ — these are very valuable, but also difficult to achieve; it only makes sense to embark on these after grabbing the low hanging fruit. Finally, in the bottom right, we have ‘distractions’ — these are difficult to achieve and don’t have much business value.</p><p>How you decide to measure <em>impact</em> and <em>difficulty</em> is obviously up to you, but it’s best to have some kind of moderately objective way of scoring these metrics. For example, you could put a back of the envelope monetary value to the business impact. You could also include additional metrics, although more difficult to measure, such as increase in customer experience score / NPS. Similarly, difficulty can be gauged by a mixture of how long you believe the project will take to build, whilst taking resource constraints into consideration, as well as how difficult the data may be to acquire and clean. You will also need to break each project down into a set of smaller projects. For example, the first would be an MVP to test if the idea is feasible. The second may be a more functional version of the MVP, whilst the third may be the final, productionized version of the product. Each of these milestones are potential ‘go/no-go’ areas to decide whether to progress further on a project.</p><p>The process here is a little simpler if you’re working on a predefined problem. It’s still necessary to do the same-day team debrief and to ideate with the team about how to solve the problem. And, as always, checking in with stakeholders is essential. However, the prioritisation step isn’t necessary. Although you might develop a road map for the piece you’re working on, you don’t really need to weigh up different paths of action.</p><h3><strong>Prototyping and Testing</strong></h3><p>If you’ve gone down the prioritisation matrix route, set up a feedback session to go over the prioritisation matrix, as well as the scores you’ve given to projects, with the key stakeholders. From this, you should be able to finalise the matrix and decide which of the projects to add to your team’s roadmap for the short to medium term. Begin with the low hanging fruit and decide with your stakeholders which of the long-term projects or easy wins to dive into.</p><p>When it comes to thinking about iteratively building projects, I absolutely love <a href="https://locallyoptimistic.com/post/lean-data-science/">this article</a> on Lean Data Science and can’t really express the idea better than the author. However, essentially, it’s about building things that deliver value quickly. That involves testing, failing and iteratively building. As you build your prioritised projects (or the predefined project), work towards ‘good enough’ versions of the tool or project. Give these to stakeholders to use and then take the feedback and iterate on them.</p><h3><strong>Keeping Score</strong></h3><p>One last point which is much easier said than done: if you can track the impact of each of the sub-projects that comprise the larger projects you’re building, it will make the ‘go/no-go’ decisions a lot easier to make. For example, if building the rule-based chatbot manages to successfully help 10,000 customers a month, it’s clear that moving forward with the next phase of this project is likely to bear fruit. You could even put a monetary value on the MVP, with a simple calculation. For example, 10,000 customers X 3min average call = 30,000 mins = 500 hours. If an hour of a call-centre agent’s time costs R200 in terms of salary and infrastructure, then you’re saving R100K a month with this project. You could likely save even more with a more sophisticated chatbot. Measuring the impact of a project is obviously incredibly challenging for a number of reasons — like the additional time overhead required to measure the impact — but it’s essential for understanding which projects are worthwhile pursuing</p><h3><strong>Wrap Up</strong></h3><p>This has been a lightning overview of how to mix Design Thinking and data science together to get your data science team working to co-create solutions with business stakeholders. I was over the moon when I discovered that I can bring together two things that I love — “creative” Design Thinking and “analytical” data science — together in one field. Let me know how this works for you if you try it. And, as with everything, this is also an iterative process. Find what works best in your organisation and what doesn’t. Keep trying new ideas to make these kinds of workshops more successful as you help to bridge the gap between business and the data science community.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ec6b1b7cc8e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/successful-combining-design-thinking-and-data-science-8ec6b1b7cc8e">Successfully Combining Design Thinking and Data Science</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Mathematical Thinking to Find the Perfect Apartment]]></title>
            <link>https://medium.com/data-science/using-mathematical-thinking-to-find-the-perfect-apartment-e421c6e621fb?source=rss-c2f5bfff9f14------2</link>
            <guid isPermaLink="false">https://medium.com/p/e421c6e621fb</guid>
            <category><![CDATA[decision-making]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Neil Sinclair]]></dc:creator>
            <pubDate>Mon, 29 Mar 2021 06:52:23 GMT</pubDate>
            <atom:updated>2021-03-29T16:26:24.488Z</atom:updated>
            <content:encoded><![CDATA[<h4>Overwhelmed with options, I used mathematical thinking to enhance my decision-making and find the perfect apartment in a week.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i9D0RlUfJ8tVGEPrUqVLuw.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@chaitaastic?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Chait Goli</a> from <a href="https://www.pexels.com/photo/green-2-seat-sofa-1918291/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Pexels</a></figcaption></figure><p>After an intensive couple of months of interviewing, I landed an excellent data science role at an exciting company. I had gone back to school to do a Masters in Data Science in 2020 and, getting ready to settle in, I was now on the lookout for an awesome apartment. To help me find the best place in the shortest amount of time, I created a concise measuring tool for the apartments I’d be looking at and then planned to use that to find the best one. Also, being a data nerd, I did a little extra analysis just for fun. A lot of the inspiration around applying this method came from reading “Thinking, Fast and Slow” and attempting to avoid shortcuts in my thinking. All of the data used in this article was generated from a short survey I conducted when I viewed each apartment.</p><p>This short article presents a straightforward way for bringing data into one’s decision-making process and hopefully the ideas here can be more useful to you than just drawing up a pros and cons list next time you have to make a big decision. It doesn’t describe a sophisticated method for finding the perfect place, but does highlight a way for adding quantitative thinking to what could otherwise be a predominantly emotive decision. It also shows how to use two tools from the data science toolkit to visualize data one collects oneself to solve their own, personal problem.</p><h3>How to compare properties?</h3><p>I started off by wondering what the key features of properties are for me. After thinking about this and allowing for a slight adjustment after the first properties that I saw, I decided on a list of the nine most important factors for me. I then took ratings for each feature on a scale from 1 to 10 using a survey. It’s worth noting that not all features are created equal. I needed to divide them between ‘hygiene’ factors and ‘motivators’. This way of thinking is based on a theory from industrial psychology called <a href="https://en.wikipedia.org/wiki/Two-factor_theory">‘Two-Factor Theory’</a>. The gist is that very low scores on hygiene items are likely to have more of a negative impact on my day-to-day living, whilst higher scores on the motivators will have less of noticeable impact. I also tended to have an idea from the apartment adverts of what an apartment’s motivators already were, so they varied less than the hygiene factors, which I could only observe when viewing the apartments. I’ve indicated hygiene factors with an (H) and motivators with an (M)</p><ul><li>Affordability (M)</li><li>Parking (H)</li><li>Safety (H)</li><li>Quiet (H)</li><li>Size (M)</li><li>Block (H)</li><li>Area (H)</li><li>Aesthetics (M)</li><li>Views (M)</li></ul><p>Now, there are a few things to consider here. A lot of these are subjective measures, for example, ‘aesthetics’ or ‘view’, however these things are very important to me and I needed some way to compare the properties. Secondly, some of these bits of data could have been collected by more sophisticated means. For example, ‘safety’ could have comprised crime statistics for the area coupled with some categorical variables for having a security guard and the level of security, such as fencing. I didn’t think that this level of detail was necessary for helping me to make a better decision — the ROI of adding it to the model would have been very low. Finally, when it came to tallying up the scores for each apartment, I weighted the hygiene factors by two and the motivators by one because of their unbalanced effect on the level of happiness I assumed they would impart.</p><p>Using trusty old Google Sheets, I put together a simple set of the weighted scores and took the average. What was interesting for me here is that I was really stuck between the apartments H. Park (G) and HL (GP), but the results clearly leaned towards H. Park. I managed to get an extra viewing in at the last minute though and FH (G) came out a clear winner and was the one I ended up taking.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aTlSuhMwtfOeddSh" /><figcaption>Scores for each of the features for each apartment where each feature was rated from 1 to 10 and then multiplied by its relevant hygiene or motivator weighting; image by author</figcaption></figure><h3>Getting a little more technical</h3><p>For my own nerdy, data scientist mind I decided to take a look at two things in the data. Firstly, I was interested in the correlation between the different factors. There’s a really basic notebook <a href="https://colab.research.google.com/drive/1wSZmebxa3qi80JBMae-R5N_9muQguhfh?usp=sharing">here</a> that you can check out if you want to see or use the code yourself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/408/0*Wk_MollTN9qhEBRU" /><figcaption>Feature correlation matrix; image by author</figcaption></figure><p>Unsurprisingly, affordability is negatively correlated with everything else; spending more leads to better apartments. The best apartment blocks had the worst parking. Nicer areas also tended to be noisier and have smaller apartments as well as being safer — although definitely more expensive. Parking also appears quite negatively correlated with good views.</p><p>I also wanted to run a bit of PCA on the data to see if the apartments naturally clustered together. I didn’t quite get the results that I’d expected to see, however it’s interesting that generally the low (average) scoring apartments are on the bottom of the graph. Given what I remember about the apartments, I didn’t think they could make meaningful clusters in this two-dimensional space. It could, however, be interesting to see what a non-linear dimensionality reduction method would produce.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mBFRvgn4d_MrYrlv" /><figcaption>Principal components of the apartments; image by author</figcaption></figure><p>I investigated the PCA reduction further by taking a look at the factor weightings. It’s clear to see that safety and area are highly correlated. In terms of how apartments are positioned, the blocks in the lower half of the graph were more affordable, but had lower weightings on the other features; this is unsurprising given the correlation plot above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-0sZUnfLMGIz1rzU" /><figcaption>Principal components and factor weightings; image by author</figcaption></figure><p>It’s worthwhile noting here that I deliberately didn’t scale the data, although I did centre it. I didn’t scale the data because the difference in the scales of the hygiene and motivator variables is significant and I didn’t want to remove this difference from the data.</p><h3>Conclusion</h3><p>The most valuable aspect of this research was that it helped to show, in numbers, the difference between the apartments that I was particularly interested in and helped with the decision-making process. It was also fun being able to apply a quantitative approach to a problem that was largely intuition-driven for me. It’s interesting how some of the apartments that I wouldn’t have really considered after viewing actually didn’t score too badly, however still not in the top half of scores. I think more refinement could be applied to the process, but I saw it as being iterative and will definitely apply a similar approach to decisions that I make in the future.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e421c6e621fb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/using-mathematical-thinking-to-find-the-perfect-apartment-e421c6e621fb">Using Mathematical Thinking to Find the Perfect Apartment</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building A Graph-Based Grocery Recommender in Dash on AWS]]></title>
            <link>https://medium.com/data-science/building-a-graph-based-grocery-recommender-in-dash-on-aws-3d96d262e365?source=rss-c2f5bfff9f14------2</link>
            <guid isPermaLink="false">https://medium.com/p/3d96d262e365</guid>
            <category><![CDATA[recommender-systems]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[dashboard]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Neil Sinclair]]></dc:creator>
            <pubDate>Tue, 09 Feb 2021 20:05:55 GMT</pubDate>
            <atom:updated>2021-05-18T18:45:31.524Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5hyjhEf07AWXgA_eba7ebA.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@wdnet?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">PhotoMIX Company</a> from <a href="https://www.pexels.com/photo/vegetables-stall-868110/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Pexels</a></figcaption></figure><p>In preparation for a recent job interview at a large retailer, I wanted to get my hands dirty with some retail data whilst focusing on answering a real-life business question — how can we build more profitable grocery baskets. Having recently become familiar with the NetworkX package in Python, I saw an excellent opportunity to build something visually enticing and interactive to help answer this question. This article gives an overview of:</p><p>1) Setting out the business case</p><p>2) Building the graph in NetworkX</p><p>3) Making graph-based recommendations</p><p>4) Creating an App in Dash and deploying it to AWS</p><p>All the code is available here on my <a href="https://github.com/NeilSinclair/GroceryNetwork">GitHub</a>.</p><h3>Setting Out the Business Case</h3><p>How can we get shoppers to spend more at our store and less at other stores? I started off with this question and thought that by visualizing the relationships between products in the baskets in a dataset, one might easily be able to find a few products that essentially “glue” other products together. We could then surmise that these products that glue baskets together — these <strong>anchor products</strong> — get customers into stores and that if we lowered the prices of these items, we could get customers to come to our store over other stores. But to be sustainable, we need to make the shortfall up somewhere else. So, looking at the baskets, we could then increase the price/margin of less central items which are not getting customers into the door, but which nonetheless land up in their baskets. Having recently discovered the NetworkX package in python, I thought that displaying grocery items as nodes in a network would be an excellent way to visually represent the relationships between items in baskets.</p><p>I set out to find a dataset and found <a href="https://www.kaggle.com/c/instacart-market-basket-analysis/data">this excellent one on Kaggle</a> (subsequently, I also found <a href="https://www.dunnhumby.com/source-files/">an excellent one</a> on Dunnhumby — called “The Complete Journey” — which I’m still hoping to sink my teeth into).</p><p>I thought it would also be interesting to see the difference between the graph structure for customers who bought small baskets (bottom quartile, less than 5 items) and customers who bought medium-sized baskets (second quartile, 5 to 9 items).</p><h3><strong>Building the Graph in NetworkX</strong></h3><p>It would be remiss of me to not first mention that displaying this data as a graph is not necessarily the best way of finding anchor products. There are other approaches, such as the <a href="https://medium.com/@sawravador/overview-about-apriori-algorithm-a5f20f0a4d39">apriori algorithm</a> used in market-basket analysis, however, I really wanted to experiment with initially building insights from the graph visualisation, as well as using the structure of the graph for building recommendations later, so went with this method. Another thing to note is that while it seems logically sensible that one could also just look at which are the 20 most bought products and consider those to be the anchor items, it’s also possible that there might be items that are say, in the top 1000, but not the top 20, which are still items that glue a number of other items together. If you’re a South African, perhaps an anchor item for a basket might be boerewors (a type of sausage) for your braai (a barbeque), however this item is unlikely to be one of your store’s top-selling items, yet it’s still valuable to be aware of how this grocery item may bind other items together.</p><p>The use of NetworkX is incredibly straightforward. With a little pre-processing done on the data, basket items are treated as nodes with edges created between nodes that are in the same basket. The edges are then weighted by the number of times two items appear in a basket together. To give a concrete example, the below image shows what a sub-graph of items would look like, with these representing all of the items that were found in a basket with Almonds in the dataset, as well as the connections between each of those items.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/334/1*ayYafOYVnSW4T0JVfHUSIQ.png" /><figcaption>Sub-graph of groceries occurring with Almonds; image by author</figcaption></figure><p>The code below shows how to build the <em>node — edge — node</em> pairs and then pass them to NetworkX to create the network:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d7b30928e40247f9517c7fbd1b9707b2/href">https://medium.com/media/d7b30928e40247f9517c7fbd1b9707b2/href</a></iframe><p>And voila, we have a graph network. The best part about this is that we can now visualise the connections between nodes in the network. I omit the code here as it’s a little bit long, but take a look at the Jupyter Notebook on the <a href="https://github.com/NeilSinclair/GroceryNetwork">GitHub page</a> to get an idea of how to put this together. There was so much data here that I thought this was an ideal opportunity to build an interactive plot using Plotly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zlFN6mm9khtnmKIYeaBh4A.png" /><figcaption>Image of the Grocery Graph for small baskets; image by author</figcaption></figure><p>Exploring the graph one can see a few things. Firstly, there are a few very central items here (the ones darker in colour) that seem to be connected to most of the other items. These are <em>bananas </em>and <em>strawberries</em>. Now, going back to the original business question, “what can we do with this information?”, this suggests that we could look at lowering the price of these items to bring more customers into the store. Now, if we look at items on the outside of our graph “cloud”, such as l<em>arge eggs</em>, these are items that we could look to raise the price (and increase our margin) on, as these are less likely to be items that customers come specifically into our store to buy, yet still land up in a number of baskets.</p><p>It’s worthwhile noting here that I expected to see different patterns of clustering in the network graph. I imagined that there would be multiple clusters, or areas of high density in the network, where certain kinds of products were connected with each other; however, this was not the case. Whilst this does somewhat erode the value of visually exploring grocery items in a graph network format, building the graph still allows one to use it to create a recommender system.</p><h3><strong>Building A Graph-Based Recommender System</strong></h3><p>For those unfamiliar with recommender systems, check out the <a href="https://en.wikipedia.org/wiki/Recommender_system">Wikipedia page</a>. Whilst methods such as collaborative filtering are responsible for a host of the recommendations that influence a lot of our choices, such as Netflix recommendations, graph-based recommender systems have also been used in a host of applications. In this article, the graph-based recommender focuses simply on recommending similar products, without taking into consideration what the customer has purchased before. For an excellent article on recommender systems with Bipartite Graphs (i.e. with users and products), check out <a href="https://medium.com/dunnhumby-data-science-engineering/network-models-for-recommender-systems-7f0d6d210ccf">this article</a>.</p><p>I took two approaches to building the recommender system. The first approach is the most obvious, whereby the recommender system recommends the top K neighbours of an item, sorted by the edge weight of the neighbours of the current item. The second approach is more interesting, and also looks at recommendation in a slightly different way.</p><p>The second approach generates a graph-embedding for each node in the graph, where the embedding for each node is similar to the embedding created when one creates word vectors (if you’re unfamiliar with word vectors, check out <a href="http://jalammar.github.io/illustrated-word2vec/">this brilliant article</a>). In fact, one of the methods used to create these embeddings uses exactly the same process for creating word embeddings to create the node embeddings. The method covered in this article is called Deep Walk. I built it after reading <a href="https://towardsdatascience.com/deepwalk-its-behavior-and-how-to-implement-it-b5aac0290a15">this awesome overview of the topic</a>, which covers it in greater detail than I will here. Basically, node embeddings are created by selecting a node in the network and then taking a k-step random walk from that node and recording every other node visited in the network. This selection of k+1 nodes is then stored as a string (k+1 because we include the first node in the string). In the shopping list example, say you started on the node for “Almonds”, then you might make a string of nodes that looks like this by taking a random walk from that node:</p><p>Almonds -&gt; Tortilla Chips -&gt; Virgin Olive Oil -&gt; Banana -&gt; Apples -&gt; Clementines</p><p>You would do this <strong><em>n</em></strong> times for each node in the network, taking <strong><em>k</em></strong> steps and storing all of these “strings” of length k+1 comprising the nodes visited in the process. Once you have a collection of size <strong><em>k x n</em></strong> of these strings of nodes, you can use a nifty package like Gensim to learn Word2Vec embeddings for these nodes, and viola, you’ve created embeddings for your graph. Below is the code for this, but keep in mind that this is very liberally borrowed from <a href="https://towardsdatascience.com/deepwalk-its-behavior-and-how-to-implement-it-b5aac0290a15">this great article</a> which you should check out.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6dde6bf5203646b49e459ae491cd2166/href">https://medium.com/media/6dde6bf5203646b49e459ae491cd2166/href</a></iframe><p>Now, as one can see in the code above, the way to find similar items is simple, just use the Word2Vec method .most_similar().</p><p>However, there is something here that needs to be taken into consideration when comparing the two recommendation approaches given here. The “most similar” embeddings method is ingenious in the manner in which the representations for each of the nodes captures contextual information about the nodes in the same way that Word2Vec captures contextual information about words. However, when we buy, say, chicken broth, we might be more interested in the other items that people by <em>with</em> chicken broth than items that are contextually similar. When using the two methods, we end up with recommendations like:</p><p><strong>Most Similar (Embeddings) to Almonds:</strong> Cola Cans, Pistachios, Raisin Bran, Dried Mangos, Classic Roast Coffee, Variety Diet Tea, Salt &amp; Pepper Pistachios, Rice Krispies Treats</p><p><strong>Nearest Neighbours of Almonds:</strong> Soda, Banana, Trail Mix, Pistachios, Raspberries, Clementines, Blueberries, Strawberries, Seedless Cucumbers, Granola Bars</p><p>So, which is better? It’s difficult to say. The nearest neighbours are more generic. Banana, Raspberries and Strawberries are in a lot of baskets so, in this case, some of the recommendations may be not much more valuable than recommending common products that the customer was going to buy anyway. The most similar embeddings method suggests that the recommended items play a similar role (through their contextual embeddings) to the main item and thus may be less complementary and perhaps more interchangeable.</p><p>Although this is interesting, and fun to play with, this suggests that to really add value it may be worthwhile to rather suggest items to customers based on similarity with other customers. The dataset I looked at didn’t have customer level data, but if you check out <a href="https://medium.com/dunnhumby-data-science-engineering/network-models-for-recommender-systems-7f0d6d210ccf">this article</a> (mentioned above), it provides an excellent example of using both customer and item data together to make recommendations.</p><h3><strong>Creating a Dash App for Interactivity</strong></h3><p>Although Dash is very similar in many ways to R-Shiny, the differences led to quite a sharp learning curve. Having developed one or two small things with Flask, JS and HTML, I found some of the ways that Dash utilises CSS to be rather challenging to understand — for example having a CSS class “six columns” instead of “col-6” and taking a while to figure that out. Although I won’t go into too much detail here about the Dash App — you can view the code in my <a href="https://github.com/NeilSinclair/GroceryNetwork">GitHub account</a> — I will give a brief overview.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oLlHTBNuWYSsCjS6nR_R0Q.jpeg" /><figcaption>Shopping List Builder section of the Dash app; image by author</figcaption></figure><p>As Plotly and Dash work so well together, I decided to take this project and turn it into an interactive app and deploy to AWS. I found <a href="https://austinlasseter.medium.com/deploying-a-dash-app-with-elastic-beanstalk-console-27a834ebe91d">this fantastic article</a> on Medium that took me through the basic steps of uploading an app to Elastic Beanstalk on AWS and was able to get my app up and running quite painlessly. I won’t recap what was written there as it’s done so well, there’s no need to, but follow that article and you’ll be good to go in a couple minutes.</p><p>A couple things to consider about the process of building the app. Dash has a system of callbacks that allow your functions to alter Dash objects on the page. Trying to do something with a lot of dynamically-created moving parts can be quite a challenge, although once you get the hang of it, there’s a ton of stuff you can do. The learning curve is steep, but I think it’s worth it for future dashboards. To prevent the app from having to regenerate the graph every time a tab is loaded or a dropdown changes, I saved the graph data as pickle files, and in the case of the recommender tab, load this up as the page loads. This might not be best practice, but it got the job done. I’m also unlikely to win any UX/UI awards for this little web app, but the idea was more to build something and explore some data, rather than make something that looks perfect.</p><h3>Wrap Up</h3><p>This project really helped me to get started with trying to answer a business problem first, which was “how can we get customers to buy more goods at our stores”. By finding a novel way to explore the data visually, I was able to disprove an initial hypothesis I had around items being clustered together — it appears more that one ends up with a cloud-like network of grocery items. However, this then led me to explore using the graph to make recommendations for products and ultimately to build a basic Dash app to explore the data online. Overall, it was an excellent learning experience and has left me with a number of thoughts on improvements:</p><ul><li>One could rather build a graph with both customers and orders to recommend products to customers based on similarity with other customers</li><li>There are <a href="https://sigmod2020.org/keynote.pdf">other methods</a> for creating graph embeddings which could be explored and compared</li><li>Adding financial (price and cost) data into analysis would be valuable in understanding which products one could increase prices on and the effect on overall basket value</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d96d262e365" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/building-a-graph-based-grocery-recommender-in-dash-on-aws-3d96d262e365">Building A Graph-Based Grocery Recommender in Dash on AWS</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Gentle Introduction to Transfer Learning in NLP]]></title>
            <link>https://medium.com/data-science/a-gentle-introduction-to-transfer-learning-in-nlp-b71e87241d66?source=rss-c2f5bfff9f14------2</link>
            <guid isPermaLink="false">https://medium.com/p/b71e87241d66</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Neil Sinclair]]></dc:creator>
            <pubDate>Wed, 27 Jan 2021 07:50:33 GMT</pubDate>
            <atom:updated>2021-01-27T14:00:11.736Z</atom:updated>
            <content:encoded><![CDATA[<h4>Transfer Learning is one of the hottest topics in NLP — learn what it is and how you can apply it to your own projects today</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*bTck-8uw1MZLuYg3H3iREA.jpeg" /><figcaption>Photo by Lisa Fotios from Pexels</figcaption></figure><p>This article gives a brief overview of what Transfer Learning is in Natural Language Processing (NLP) and why it’s the greatest thing since sliced bread. If you want to experience what it’s like to play around with these awesome pretrained models, check out the<a href="https://colab.research.google.com/drive/1hBwM6QITP8elGEI05L9SwAGqozgI1vRT?usp=sharing"> link to the code</a> here and at the end of the article; you can do some really advanced in NLP with almost zero programming experience.</p><p>As a brief primer for this article, Natural Language Processing or NLP, refers to using machine learning to process natural text, with ‘natural’ referring to the kind of text we find in books and newspapers, as opposed to, for example computer programming code (okay, some of the models are <a href="https://twitter.com/sharifshameem/status/1283322990625607681">learning to code</a>, but we’ll stick to talking more generally about ‘natural’ language here). This technology is driving amazing things from automatic article summarisation, to responsive chatbots and even creative writing generation.</p><p>Training a computer vision or natural language model can be expensive. It requires lots of data, and in the case of computer vision, that data needs to be labelled by humans, and it can take lot of time to train on expensive hardware. GPT-2, a benchmark-setting language model released in 2019 by Open AI, is <a href="https://arxiv.org/pdf/2004.08900.pdf">estimated to have cost around $1.6m to train</a>. With costs like that, how is anybody going to compete with Open AI or any of the other organisations doing research in the NLP space?</p><p>Here’s the good news though, you don’t need to compete with them. You can download these models for free on the internet, pre-trained on enormous datasets and ready to go. What’s even better is that you can fine-tune these models really quickly to work on the nuances of your specific dataset. To give you a sense of the difference between fine-tuning and training a model, fine-tuning is like taking your car to the mechanic and getting new spark plugs whereas training/pre-training is like getting a whole new engine. As a concrete example, let’s say you run a large food delivery company and you want to get the sentiment (positive or negative intention) of tweets people make about your company so you can quickly attend to the negative ones. Considering you have enough tweets to feed into the model, you could fine-tune it in a really short time to predict whether the tweets are positive or negative. For example, I recently fine-tuned a classification model on food reviews for my master’s research that achieved 98% classification accuracy (predicting whether a review was positive or negative) with only about 40 minutes of training. If you were to train a model from scratch it would likely take a few hours or longer, need an enormous amount of data that you’d need to collect and pre-process yourself and would be less accurate than the fine-tuned model. What a win!</p><p>This process of taking a model that’s been trained to do one task — as these pre-trained models have been trained to do — and then fine-tuning it to work on a related but different task is at the essence of what’s called Transfer Learning. It’s been used in computer vision for just over half a decade and has been making waves in NLP for about three years. To understand better how it works, we need to take a short diversion into a high-level understanding of how a language model works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/0*3zHRMHK7BOZNKLOQ.jpeg" /><figcaption>Image by Author</figcaption></figure><p>After converting words into a numerical form machine learning models can understand, these are fed into the main part of the model which is (most often) a deep, multi-layered <a href="https://www.forbes.com/sites/bernardmarr/2018/09/24/what-are-artificial-neural-networks-a-simple-explanation-for-absolutely-anyone/#7f276ec51245">neural network</a>. The most popular language models at the moment, the Transformer, have a structure where they build a very deep set of relationships between every word in the sentence, creating an incredibly rich (numerical) representation of a sentence. This rich representation is then fed into the last layer of the model which, given a part of the sentence, is trained to predict the next word. It does this by giving its confidence of predicting what the next word is over all of the words in its vocabulary. In the example in the image above, the model is most confident that the next word is ‘oceans’.</p><p>One of the fascinating things with Transfer Learning in NLP was that researchers discovered that when you train a model to predict the next word, you can take the trained model, chop off the layer that predicts the next word and put on a new layer and train just that last layer — very quickly — to predict the sentiment of a sentence. Now, keep in mind that this is <strong>not</strong> what the model was trained to do; it was trained to predict the next word in the sentence; however, it appears to be capturing a lot of pertinent information in a sentence when it processes and transforms into the rich representations which are fed into the last layer to predict the next word. But you can do more than just fine-tune the model to predict the sentiment of a sentence. You can fine-tune the model to be a conversational AI, really quickly (<a href="https://arxiv.org/abs/1901.08149">these guys</a> did it in two hours), or you can fine-tune the model to take in questions and pull the answers out of a piece of text relatively quickly.</p><p>So why should you care? Previously, language models were generally trained for very specific tasks and often trained from scratch and believe me, training a language model from scratch can be a pain! As a researcher, this means you can test out new ideas really quickly by downloading these pretrained models and tweaking them based on your use case. For example, I’m doing something in unsupervised style transfer for my master’s thesis and being able to use pretrained models for these tasks means I can iterate ideas incredibly quickly. As a practitioner, this means that you can start incorporating NLP into your products or services relatively easily; you need less data to fine-tune the model, you can start to prototype new ideas that utilise cutting-edge technology and you can start to get really creative with it. One caveat though: getting a model that’s between 250MB to 1.5GB to run quickly enough for customers’ needs is surely a mean feat of cloud engineering.</p><p>The team over at <a href="https://huggingface.co/">HuggingFace</a> have done an incredible job of making these models incredibly easy to use. You can basically load up these pre-trained models with just a couple lines of code and start experimenting with the possibilities. If you want to check it out, I’ve put together <a href="https://colab.research.google.com/drive/1hBwM6QITP8elGEI05L9SwAGqozgI1vRT?usp=sharing"><strong>this online notebook</strong></a> that you can play with. You hardly need any programming experience to play around — just read the instructions — and start to imagine how you could incorporate this amazing technology in your products or services. For reference if you’re not familiar with python, a notebook is basically a simple way of running python scripts.</p><p>For something quite light hearted, <a href="https://transformer.huggingface.co/doc/gpt2-large">head over here</a> to play around with the GPT-2 model referenced in this article. Clear the text, type in a sentence or two of your own, press tab and see what the AI comes up with. Playing around with this makes me think about what the future of human-AI artistic interaction will look like.</p><p>If you want a more in-depth, technical overview of transfer learning, check out <a href="https://ruder.io/transfer-learning/">this incredible article</a> and if you want to know how transfer learning applies to other domains like computer vision, check out <a href="https://medium.com/analytics-vidhya/a-gentle-non-technical-introduction-to-transfer-learning-in-artificial-intelligence-2fd178bd7e54">this article</a> I wrote previously.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b71e87241d66" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/a-gentle-introduction-to-transfer-learning-in-nlp-b71e87241d66">A Gentle Introduction to Transfer Learning in NLP</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Teaching BART to Rap: Fine-tuning Hugging Face’s BART Model]]></title>
            <link>https://medium.com/data-science/teaching-bart-to-rap-fine-tuning-hugging-faces-bart-model-41749d38f3ef?source=rss-c2f5bfff9f14------2</link>
            <guid isPermaLink="false">https://medium.com/p/41749d38f3ef</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[hugging-face]]></category>
            <dc:creator><![CDATA[Neil Sinclair]]></dc:creator>
            <pubDate>Sun, 18 Oct 2020 18:12:52 GMT</pubDate>
            <atom:updated>2020-10-18T20:56:32.235Z</atom:updated>
            <content:encoded><![CDATA[<h4>I taught BART to rap as part of the process of learning how to tweak the incredibly powerful Hugging Face Transformers models.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jYwMLcsRryHL4RrZ_9CZgg.jpeg" /><figcaption>Image by Krystyna Kaleniewicz from Pixabay</figcaption></figure><p>Transfer learning has provided an unimaginable boon to artificial intelligence over the past few years, making waves in the computer vision space and more recently in the NLP space with researchers discovering that a model trained on a language modelling task can easily (quickly and cheaply) be adapted for other tasks. From a practitioner’s perspective, aside from the deluge of new discoveries — easily accessible on <a href="https://arxiv.org/">Arvix </a>— <a href="https://huggingface.co/transformers/">Hugging Face</a> have developed unbelievably easy-to-use APIs that allow anyone to access these latest developments with a few lines of code.</p><p>In spite of the ease with which one can use the Hugging Face APIs both for on-the-fly inference and for fine tuning through command line style arguments, I got a little stuck trying to fine-tune the <a href="https://arxiv.org/abs/1910.13461">BART model</a>. I’m aiming to use it for my Masters thesis and it took me an inordinate amount of time to write the code to fine tune the model because I got stuck in the process. However, once I’d managed to get past this, I’ve been amazed at the power of this model.</p><p><strong>TL; DR: Check out the fine tuning code </strong><a href="https://colab.research.google.com/drive/1Cy27V-7qqYatqMA7fEqG2kgMySZXw9I4?usp=sharing&amp;pli=1"><strong><em>here</em></strong></a><strong> and the noising code </strong><a href="https://colab.research.google.com/drive/1n45bHMFw5dSTEUxcGYrQxhApPaltqGsS"><strong><em>here</em></strong></a><strong>.</strong></p><p>This article will give a brief overview of how to fine-tune the BART model, with code rather liberally borrowed from Hugging Face’s finetuning.py script. However, this will allow a bit more control over how one can experiment with the model. I’ve used <a href="https://pytorch-lightning.readthedocs.io/en/latest/">PyTorch Lightning</a> to handle the training, and if you’re new to it, I encourage you to get familiar with it. The implementation is incredibly straightforward and may be able to streamline some of your projects going forward. Although I’ve taught BART to rap here, it’s really just a convenient (and fun!) seq2seq example as to how one can fine-tune the model.</p><p>Just a quick overview of where I got stuck in the training process. The loss on my model was declining at a rapid pace over each batch, however the model was learning to generate blank sentences. For a long time, I couldn’t figure out why this was happening. It turns out that you need to manually shift the tokens to the right before you feed them to the decoder, but that you must pass the unshifted tokens to the loss function.</p><p>So, without further ado, this is how to teach BART to rap.</p><h3>The training set</h3><p>I found a great set of lyrics from <a href="http://www.github.com/fpaupier/RapLyrics-Scraper">this GitHub repo</a>. The author explains how they scraped the lyrics using the Genius python API, but I just downloaded the lyrics that had already been scraped. I then spun up a quick notebook (<a href="https://colab.research.google.com/drive/1n45bHMFw5dSTEUxcGYrQxhApPaltqGsS">here</a>) where I created a set of line, next-line pairs of lyrics, such that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/93/1*WCg8ARKw1Q1Sx1fBgZCpTQ.jpeg" /></figure><p>Here L(n) represents line “n”, L(n+1) represents the following line and -&gt; indicates the lines are paired in the training data. I also did a small amount of additional processing to ensure that songs wouldn’t bleed into each other and that a verse line wouldn’t be followed by a chorus line in the training pairs and vice versa. I also removed a lot of duplicate lines as failing to do this led to a model that often just generated the same lines, repeated, over and over again (due to the significant portion of line -&gt; repeated-line pairs in the data).</p><p>From here, I noised the data. Because BART is trained as a denoising autoencoder I thought it best to pass noised data into the model for training. I’m not sure if this is necessary though. I replaced 25% of the data with the &lt;mask&gt; token, however, I excluded the final word of the lyric line from being added to the replacement pool as this word plays a crucial role in supporting a rhyming scheme.</p><p>I also tried to set up the training set in such a way that a line could be predicted by more than just the previous line. This was done in the hope that during generation the model would be able to have greater coherence across a four-line verse. Concretely, as above if L(n) represents the nth line in the training set, it was set up such that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/1*axK_5P7YcMio8pJiwr9GyQ.jpeg" /></figure><p>However, when doing this I found that although the number of training examples grew significantly (obviously), the model was learning to copy output lines from the dataset. I thus abandoned this version of the training data.</p><h3>The training</h3><p>The training was relatively straight forward (after I solved the plummeting loss issue). I used PyTorch Lightning to simplify the process of training, loading and saving the model. I also used ‘bart-base’ as the pre-trained model because I had previously had some GPU memory issues on Google Colab using ‘bart-large’. I trained the model for around 10 epochs. The code is available <a href="https://colab.research.google.com/drive/1Cy27V-7qqYatqMA7fEqG2kgMySZXw9I4?usp=sharing&amp;pli=1">here</a>.</p><h3>Generating the text</h3><p>When generating the text, I did two things. First, I fed a seed line into the generate_text() method (which used the BartForConditionalGeneration generate() method) and auto-regressively generated k new lines; secondly, I noised each line with 25% — 35% noised tokens. I found that noising the tokens like this generally gave more variation in the outputs of the model. In the end, I was pretty amazed with the results that I was able to get. It’s quite addictive playing with it. Here’s one 4 line piece I thought could be in a song (it’s no Bukowski, but hey) — first line is mine:</p><blockquote>You and me forever cruising city lights</blockquote><blockquote>I hope you and your homies don’t ever see these flashing lights</blockquote><blockquote>And me forever shining in the city lights</blockquote><blockquote>But you can’t ever change the way you shine</blockquote><h3>Thoughts on improvements</h3><p>Adding a longer range of lyrics for generation — although my experiment with creating more training data with longer lead-in lyrics (two and three lines leading to the target line) weren’t successful, there might be a way to improve this, by for example, adding &lt;sep&gt; tokens to the sentences.</p><p>I think it would also be interesting to see how BART works with a different genre of music, such as Country or Punk Rock. I also think it would be interesting to see what happens if one doesn’t noise the source lines when training the model. Although, as I recall, from the <a href="https://arxiv.org/abs/1910.13461">original BART paper</a> all of the data was noised when training the model, I’m not sure if not noising the data would work.</p><p>Finally, every now and then when a really great line would pop up, I’d manually go search through the training data to see if the model was just copying it or if it was generating the text. Most of the time it was generating the text, however adding in a BLEU-like metric to get a sense of whether the model is copying or being “original” would be helpful.</p><h3>Final Thoughts</h3><p>There are a lot of options that one has available now when utilising pre-tuned models, especially with the great work that Hugging Face is doing with democratising access to the latest and greatest ones. BART shows a lot of promise for a wide-range of seq2seq tasks and having spent some time getting to know the model better, I’m very keen to see what else is possible.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41749d38f3ef" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/teaching-bart-to-rap-fine-tuning-hugging-faces-bart-model-41749d38f3ef">Teaching BART to Rap: Fine-tuning Hugging Face’s BART Model</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Gentle, Non-Technical Introduction to Transfer Learning in Artificial Intelligence]]></title>
            <link>https://medium.com/analytics-vidhya/a-gentle-non-technical-introduction-to-transfer-learning-in-artificial-intelligence-2fd178bd7e54?source=rss-c2f5bfff9f14------2</link>
            <guid isPermaLink="false">https://medium.com/p/2fd178bd7e54</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[transfer-learning]]></category>
            <dc:creator><![CDATA[Neil Sinclair]]></dc:creator>
            <pubDate>Wed, 16 Sep 2020 18:47:01 GMT</pubDate>
            <atom:updated>2020-10-04T17:13:26.736Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*kjKORCTQNEGWg-AkdF3r3Q.jpeg" /><figcaption>Credit: Markus Spiske — Unsplash</figcaption></figure><p>Transfer learning in the realm of artificial intelligence — specifically in the case of neural networks — is somewhat akin to that scene in the Matrix when Neo downloads his martial arts ability almost instantaneously. It allows one to transfer the knowledge learned by a neural network in one setting to another related but different setting, essentially enabling your neural network to learn how to do a new task (relatively) well, almost instantaneously. It’s been responsible for some of the most exciting advances in artificial intelligence over the past few years and is one of the key reasons why I’m so excited about the <a href="https://www.linkedin.com/pulse/democratisation-artificial-intelligence-why-im-excited-neil-sinclair/">democratisation of AI</a>.</p><p>This article is intended for people with minimal knowledge of how neural networks or machine learning works, but who want to know a bit more about the basics of one of the most powerful trends in the space, Transfer Learning, and how it can be applied (if you want a more technical/applied approach, <a href="https://ruder.io/transfer-learning/">this</a> is an excellent article). I’ll keep it as short and sweet as I can. The article starts with a very high-level overview of how neural networks are built and then dives into how Transfer Learning works in the domain of computer vision. I’ll share a follow-up article on Transfer Learning in natural language processing (NLP) in the coming weeks.</p><p>Let’s dive right in by defining a neural network, which has been one the biggest drivers of progress in AI over the past decade. At a very high level, it’s essentially a collection of small processing units, called neurons, put together into networks that are loosely (very loosely) modelled on how neurons in the brain might work together to process information (here’s a slightly more in-depth <a href="https://www.forbes.com/sites/bernardmarr/2018/09/24/what-are-artificial-neural-networks-a-simple-explanation-for-absolutely-anyone/#7f276ec51245">explanation</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S_XxdIIh6IS3k6Ww12xsNA.jpeg" /><figcaption>A basic, 2 hidden layer neural network</figcaption></figure><p>In the fields of computer vision and natural language processing, neural networks are essentially built up in blocks called layers. As data moves through the network, the layers have been designed so that they build richer and richer representations of the data that was fed into them — essentially processing the data into information-dense mathematical abstractions. In the case of computer vision, lower layers in a neural network essentially extract high-level features, like lines and edges, whilst layers deeper in the network extract more complex features — for instance eyes or mouths in the case of a face detection network.</p><p>Ultimately, you want your neural network to <em>do</em> something with this rich representation of your data. If it’s image data, perhaps you want to build a system to recognise your employees’ faces when they get to work so that the front door unlocks automatically for them. Based on this understanding, we can conceptually split a neural network into two parts: the part that processes the data and builds a rich representation of the image of an employee’s face; and the part that decides what to do with this processed data by classifying it, which in this case might be categorising it as someone who works at your company, thus triggering the door to open.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x8WIxrxVGFyL2-ufORHLjg.jpeg" /><figcaption>An overview of the layered structure of neural networks and the increasingly rich representations created by Convolutional Neural Networks for image classification</figcaption></figure><p>One of the key drivers of the neural network revolution has been that, generally, as you feed more data into a neural network its accuracy continues to improve continually (this isn’t the case for all machine learning methods). However, neural networks have in the past generally been trained to be very good at narrow tasks that used data very similar to the data they were trained on. When you needed to train a neural network on a new task, you’d find that collecting and curating (e.g. labelling and processing) data is expensive (check <a href="https://enlabeler.com/">these guys</a> out for a great solution to that problem). Often it can also be challenging to get enough data in whatever domain you’re focusing on to train your model adequately. Even if you’re able to collect enough data, training the network from scratch can start to get prohibitively expensive and take a long time, from days to weeks.</p><p>What if we could take a neural network trained in a domain or on a dataset where there is a lot of data and then use it in our domain? It turns out we can.</p><p>Although Transfer Learning has been discussed for over 20 years, tangible results have been shown more recently with image processing neural networks. It was seen that if you trained a computer vision network on one task, like classifying 1,000 different types of images (e.g. classifying images as containing a cat, a dog or a car etc.) you could actually take the part that was trained to process the images (and create those rich representations) and use it in another setting. Often, all you have to do is retrain just the “classifier” on the top of the network. It turns out that the part that processes the images takes a very long time to train, potentially at considerable cost, whereas the part that does the classification is quite cheap to train. This fundamental discovery showed that training neural networks on some more general tasks and then tweaking how they classified new data — by retraining the classifier part — could allow one to transfer their learning into another domain.</p><p>To give a concrete example, let’s say that I ran a clothing company and wanted to build a system to classify the most popular item of clothing worn in photos posted on social media at the next Ultra or Rocking the Daisies so I could stock these types of items in my store. What we could do is take a relatively general model that we’re able to download online that’s been trained to classify a wide array of different things and use it to classify items of clothing. We would start by replacing the <em>classifying layer(s)</em> on top and then train those for a short time on a labelled dataset of people wearing different items of clothing (i.e. each photo in our dataset also comes with a label stored in a file saying what item of clothing is being worn in the photo). If we still weren’t getting great results, we could retrain some of the later layers in the network — the ones that have built more complex representations from the simpler ones — thus training the model to identify some of the more domain-specific (clothing-specific) features of our dataset. Even though we have to tweak part of the model, it will still take only a fraction of the time to train compared with a completely new model and most importantly, we only require a fraction of the data to get decent results.</p><p>So, what does this mean for you? First of all, what it doesn’t mean is that you can achieve incredible results with any AI task without having any data. You still need some data, but degrees of magnitudes less than you needed before. And sure, you’ll likely need to do some fine-tuning with your data, but it’s just a drop in the ocean compared to training a model from scratch — and believe me, if you’ve spent <em>hours</em> trying to train an image classification model from scratch to predict something, only to get out garbage because it needs more training and lots more data, you’ll be grateful to just have to do a little fine-tuning. It means that you can start experimenting with new ideas more quickly and that you can imbue more products and services with computational intelligence to deliver better products and be more in tune with your customers’ needs. It means that often you can be in touch with the cutting edge of what’s happening in machine learning — and there’s a lot happening and changing all the time — by leveraging off pre-trained, opensource models.</p><p>Although this certainly doesn’t mean that it’s <em>easy</em> to make awesome AI-infused products, it does mean that it’s <em>easier</em>. Outside of training your model, there’s a myriad of other things one needs to consider, not least of all how to scale the product and make it run quickly and efficiently enough to keep your customers happy. However, when it comes to the kinds of products you can deliver, increasingly your imagination is the only barrier to what can be done.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2fd178bd7e54" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/a-gentle-non-technical-introduction-to-transfer-learning-in-artificial-intelligence-2fd178bd7e54">A Gentle, Non-Technical Introduction to Transfer Learning in Artificial Intelligence</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Democratisation of Artificial Intelligence, or Why I’m excited about the next few years in AI]]></title>
            <link>https://neil-ian-sinclair.medium.com/the-democratisation-of-artificial-intelligence-or-why-im-excited-about-the-next-few-years-in-ai-746d29e7cc74?source=rss-c2f5bfff9f14------2</link>
            <guid isPermaLink="false">https://medium.com/p/746d29e7cc74</guid>
            <dc:creator><![CDATA[Neil Sinclair]]></dc:creator>
            <pubDate>Thu, 09 Jul 2020 07:59:11 GMT</pubDate>
            <atom:updated>2020-07-09T08:06:25.926Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KH0GjfDFcGBcbF8hn0_CFw.jpeg" /><figcaption>Photo by Alex Knight from Pexels</figcaption></figure><p>When the internet initially began expanding into the world in the early 90’s it was the purview of hackers and computer science majors and an act of creation required a large degree of technical expertise and know-how. As time moved on, the barriers to entry for the layperson began to diminish, however I think even a garish platform like Geocities in the late 90’s required some coding skills.</p><p>Fast forward less than 20 years and think of the seamless way in which the internet has become an enabling and democratising force in our lives. 15 to 20 years ago, if I wanted to sell things online, I had to build an ecommerce site from scratch! Now, I can build an online shop almost effortlessly through Shopify or sell my wares through Takealot here in South Africa. I can advertise my business on Facebook, Instagram and Google with minimal expertise and build a community of customers by creating promotional content on a blog or a platform like Quora.</p><p>The internet has opened doors and built bridges to customers for small, medium and large businesses that were unimaginable 25 years ago. It’s helped to foster connections, from the rise to ubiquity of social networks, to enabling platforms like MeetUp that connect people with like-minded souls, to the “long-tail” of commerce where people can access an endless array of niche products.</p><p>The internet has allowed new companies to flourish that answer the question, “how can we connect people?”. Technical expertise, whilst worthwhile and lucrative, is no longer the main resource required for success through the internet; now with barriers to entry relatively low, creativity in how you connect people distinguishes you in the internet age.</p><p>It’s exciting to see that the same thing is starting to happen with AI.</p><p>There are two reasons for this. First off, there is an avalanche of free learning material relating to artificial intelligence and machine learning online. The biggest challenge is not where to find it, it’s where to start (<a href="https://www.coursera.org/learn/ai-for-everyone">this is a good one</a> if you want a very high-level view; <a href="https://www.coursera.org/learn/machine-learning">this is the one</a> that made me fall in love with Machine Learning). In my case, even though I’m doing a “formal” MSc in Data Science at UCT this year, I’m really grateful to be able to explore the material I find particularly interesting more deeply with content I find online. Also, when you’re ready to move on from the foundations, almost all of the cutting-edge research papers are <a href="https://arxiv.org/">available for free</a>… and they’re surprisingly accessible. Secondly, the advent of transfer learning has made previously out-of-reach models available to everyone. At a very high-level, transfer learning means I can take a model trained by, say some researchers at Stanford to identify 100 different types of images (e.g. cat, dog, airplane, moose), and tweak it very slightly to identify, say different species of sea-birds. Whilst the original model may have taken perhaps weeks to train at a high cost (you pay per hour for that kind of processing), I can now download the model and tweak it in a fraction of the time — for free — on Google Colab (a service on Google Cloud). On top of this, the tools to do this are becoming increasingly easy to use. For example, I built a question and answering system where you type in a question in plain English and it extracts the answer from a piece of text in about 15 minutes this past weekend. I literally laughed with astonished joy at the results.</p><p>So what does this mean? If we’re able to leverage so much, with a small amount of code, perhaps we need to start thinking about having coders on all the teams we create? Also, I believe that the accessibility of these tools means that some really exciting things are about to start happening. If almost anyone with a modicum of programming experience (and again, with learning to program, the toughest choice is <em>which </em>free site to use — check out this <a href="https://www.freecodecamp.org/news/best-python-tutorial/">one</a> and this <a href="https://campus.datacamp.com/courses/intro-to-python-for-data-science/chapter-1-python-basics?ex=1">one</a> if you want to start) can start to put together services that solve problems that would have seemed like science fiction a few years ago, well… we’re going to start to see some really cool things happening. We’re going to start to see people solving more niche problems that unlock value for themselves and others. And I think we’re going to start to see people using AI to solve even some very individual problems, whether it’s using text generation to help with creative projects (<a href="https://transformer.huggingface.co/doc/gpt2-large">this</a> blew my mind), a small company using simple machine learning to improve customer communication by segmenting their customer base or someone trying to understand themselves better (on this note, I created a “digital journal” to help understand the drivers of good and bad days during the depths of lockdown — but more on that and the lessons learned another time).</p><p>Thinking of this, I can’t wait to see what happens as more and more people are able to unlock the power of AI. If the internet has essentially been about building better connections between people, I think the next few years of AI will be about enhancing those connections and building greater knowledge — of ourselves, our customers and the world. And soon, you’ll see, the biggest differentiator is not going to be expertise or resources, but creativity around what we can do with this incredible technology.</p><p>I want to end with a quote from Steve Jobs that I find incredibly insightful.</p><p><em>“And that’s what a computer is to me. What a computer is to me is it’s the most remarkable tool that we’ve ever come up with, and it’s the equivalent of a bicycle for our minds.” ~ Steve Jobs</em></p><p>I think that with the democratisation of AI, we’re ready to start seeing the training wheels come off.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=746d29e7cc74" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Finding under-valued suburbs in Cape Town]]></title>
            <link>https://neil-ian-sinclair.medium.com/finding-under-valued-suburbs-in-cape-town-e5c4537c364b?source=rss-c2f5bfff9f14------2</link>
            <guid isPermaLink="false">https://medium.com/p/e5c4537c364b</guid>
            <category><![CDATA[maps]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Neil Sinclair]]></dc:creator>
            <pubDate>Tue, 07 Apr 2020 08:10:56 GMT</pubDate>
            <atom:updated>2020-04-07T08:10:56.325Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e3ZWHeyyJ6apMp0nw3mQLw.png" /><figcaption>An overview of changes in house prices by suburb in Cape Town from 2017–2019; view the interactive map <a href="https://neilsinclair.shinyapps.io/maps_-_final/">here</a></figcaption></figure><p>We’ve seen tremendous property price increases in South Africa over the past two decades, with many people in my generation feeling like it’s increasingly difficult to get onto the property ladder. However, with the continuing energy crisis, sluggish economy and political uncertainty in the country, I was intrigued to see how this might be impacting house prices.</p><p>I also wanted to find a way to create a visually impactful data science project that uses publicly available data that could have economic value to someone. So, I sat down and created a map of average house prices, coupled with the average <em>change </em>in house prices for most of the suburbs in Cape Town over the time period 2011–2019. The idea behind looking at the change in house prices is that one could potentially find an area that traditionally has seen strong growth in prices, but is currently undervalued; this would be seen through a historic growth in prices followed by a decline in prices over a recent period. I chose to focus on Cape Town because I wanted to use something from the <a href="http://odp.capetown.gov.za/">City of Cape Town Open Data Portal</a> and also because I recently moved back to Cape Town from Joburg to pursue a Masters in Data Science at UCT.</p><p>You can see the result of this project <a href="https://neilsinclair.shinyapps.io/maps_-_final/"><strong>here</strong></a>.</p><h3>The Process</h3><p>When recently looking for a place to stay in Cape Town, I noticed that Property 24 had a graph on the main page for each suburb indicating suburb price trends. I created a web-scraper using BeautifulSoup in Python to scrape this data (see <a href="https://github.com/NeilSinclair/CTHousePrices">Github</a> for the code of this web-scraper). It took a few attempts to ensure that I got the majority of suburbs in Cape Town and it took a bit of searching to find the JavaScript-generated data points I was after, but the data was relatively easy to get.</p><p>I also downloaded the shapefiles for the boundaries of suburbs in Cape Town as well as the boundaries of larger sub-councils from the City of Cape Town (CoCT) Open Data Portal which I used to group the suburbs and display the data.</p><p>From here, I moved over to R to take advantage of the ease of plotting spatial data with the SF and Leaflet packages and to make the data easy to navigate with a Shiny App. The scraped data was processed to remove NA values followed by calculating the annualized property price growth (or decline) per suburb for each pair of years (e.g. 2011 &amp; 2012, 2011 &amp; 2013… 2018 &amp; 2019), similarly, the average house price was also calculated for these periods. The scraped data and CoCT data sets were then joined to be displayed in the Shiny App, with differences in the names of some of the suburbs adjusted on the CoCT data set to enable the joining of the datasets. The shapefile for the CoCT sub-councils was also used to group suburbs together; this was necessary to add an option to the data display that would load more quickly than the “all suburbs” option.</p><p>In terms of functionality, the Shiny app allows users to view the change in house prices (as an annualised %) or the average house prices over a time period specified by a slider. One is also able to select specific areas (groups of suburbs) or individual suburbs to compare.</p><h3>Extracting insights from the data</h3><p>A useful exercise to perform using the app is to look for the suburbs that have shown the greatest historic (2011–2017) price increase; as there are many suburbs in Cape Town that have shown an annual price increase of between 10% to 25%, we’ll pick a subset of these here for this analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kO03v9wpNLngwVlwUk_qNQ.png" /><figcaption>Annualised % price changes across suburbs in Cape Town</figcaption></figure><p>Based on this, we can isolate suburbs which grew by more than 10% per year between 2011 and 2017 and take a look at their yearly price change from 2017 to 2019. For this example, I picked suburbs in the Southern Suburbs and Atlantic, CBD and Surrounds areas to start the search and then picked a number of these suburbs where prices had grown by more than 10% a year (which was most of them). I then adjusted the time slider to see which of these properties had experienced the greatest <em>declines </em>in prices over the past two years (2017 to 2019); Vredehoek, Gardens, Woodstock, Bo Kaap, Camps Bay, Cape Town City Centre, Three Anchor Bay and Bishopscourt had all experienced annual price declines over these two years.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iXdslX89JRNUgkvgA2ynzw.png" /><figcaption>Price increase of selected suburbs from 2017–2019 showing the potential for undervalued sites</figcaption></figure><p>One can see that Three Anchor Bay has seen an average 12% annualised decline in prices between 2017 and 2019, with Camps Bay showing an even steeper price reduction. Depending on your viewpoint, this either suggests that properties are currently undervalued in these suburbs and could potentially be good purchases, or that prices were historically over inflated.</p><p>If you’re looking for places that meet your budget, a similar exercise could also be carried out by looking at the average house price for the past two years on the map and seeing where you can afford to live. All else being equal, one could then look at the change in house prices to get a sense of which areas are overpriced and which areas might be experiencing a short-term dip in prices before looking for a property in that area.</p><h3>Challenges with the data</h3><p>The way Property24 organises suburbs and the way CoCT organises suburbs (i.e. their respective suburb hierarchies) isn’t always the same. This has led to some gaps on the map as well as some potential errors in average suburb prices. For example, Property24’s data for “Noordhoek” comprises the average of the average house price for a number of suburbs within that area, which when joined with the City of Cape Town data gives an inaccurate picture of house prices in the area; this is exacerbated if one of the areas on Property24 comprises numerous socioeconomic levels within the area.</p><p>Secondly, in a minority of cases the Property24 average house prices swing wildly from year to year. This may be due in some cases to actual wild fluctuations of house prices in an area, or perhaps more likely, to an outlier price (such as an apartment block going on sale) which skews the mean.</p><h3>Wrap up</h3><p>This project showed how an exploratory data analysis approach can help to uncover high level insights in a data set that can point one towards insights in the data (for example, where have prices gone down recently). It also allowed me to use some of the great libraries across Python and R and to use publicly available data to answer a question I had for myself, namely, “Does it appear that the downturn in the economy is affecting house prices”.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5c4537c364b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>