<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aqib Mehmood on Medium]]></title>
        <description><![CDATA[Stories by Aqib Mehmood on Medium]]></description>
        <link>https://medium.com/@aqibme141?source=rss-3ab78c814aa6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*zheQDV09TxfoQ93NCxk-BA.jpeg</url>
            <title>Stories by Aqib Mehmood on Medium</title>
            <link>https://medium.com/@aqibme141?source=rss-3ab78c814aa6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 19:11:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aqibme141/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Differences between learning programming on your own and on the job programming]]></title>
            <link>https://medium.com/@aqibme141/differences-between-learning-programming-on-your-own-and-on-the-job-programming-fc5c0f0ee018?source=rss-3ab78c814aa6------2</link>
            <guid isPermaLink="false">https://medium.com/p/fc5c0f0ee018</guid>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Aqib Mehmood]]></dc:creator>
            <pubDate>Sun, 28 Aug 2022 10:08:49 GMT</pubDate>
            <atom:updated>2022-08-28T10:08:49.825Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*71MqGQc-9APWb2bM" /><figcaption>Photo by <a href="https://unsplash.com/es/@flipboo?utm_source=medium&amp;utm_medium=referral">Philippe Bout</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote>It can be very exciting and also very terrifying when learning a new skill to enhance your career or to pick up a side hustle. I believe learning is something that stays with the person the entirety of their lifetime. So, if learning is such an important part of a person’s life, it is imperative that we should learn how to do it right.</blockquote><p>In this article, I’ll be specifically talking about learning a programming language or pretty much anything else in the IT field. With the recent increase in jobs in the field, it has also created a large group of people wanting to learn programming and getting into the field themselves. However, in my experience, there are subtle differences and nuances that you should keep in mind when learning programming on your own vs when actually doing it on the job.</p><p>So I thought I should list the differences here and my thoughts regarding them and how can one keep them in mind when going through their own journey. So let’s start.</p><p><strong>In summary, here are the differences. We’ll be going through each of these points in detail below.</strong></p><ul><li>When learning something, people are just following a boiler plate method off of some tutorial or YouTube video to solve a <strong>dummy problem</strong>. While when on the job, there is no boiler plate method and people are solving <strong>actual real problems</strong>.</li><li>When learning something, people are often doing something <strong>from scratch</strong> and doing it end to end while on the job, most of the time you’ll be making changes to an <strong>already present architecture</strong> to incorporate a new business use case. So it’s also very important to learn how to build off of someone else’s work as well.</li><li>While learning, most of us make use of <strong>dummy data </strong>or <strong>already processed data</strong> off of some website like Kaggle and incorporate that into our code. While when you’re on the job, data is really <strong>messy and crude</strong> in some cases and we have to figure out a way to clean or process it.</li><li>Localhost vs actual host. When you’re learning something, we just want to see it work so we end up locally hosting our work. <strong>No, http://localhost:8080 is not accessible to the internet :)</strong> While on the job, you’ll never be locally hosting anything.</li><li>When learning something, we don’t take the <strong>cost</strong> of the infrastructure into account. We just want to build that one thing, using that new special tool everyone’s been talking about. While in actual on the job, cost consideration is a massive factor in all of the major infrastructure decisions as things can scale very quickly and we don’t want to end up paying $100K in the end.</li><li>While learning something, we don’t care if a certain infrastructure is <strong>reliable or not</strong>. We just want to see it work. Doesn’t matter if it comes crumbling down in two hours (Guilty :)). While on the job, we want to make sure things stay in tact for many years to come.</li><li><strong>Git</strong> is also a tool that hardly comes up when you’re learning programming. We just CTRL + S on our computer and consider our work done for the day :) While in actual on the job scenario, all of our code will be version controlled using Git.</li><li><strong>Time pressure</strong> is also something that is very different when you’re learning something vs when you’re on the job. While learning, you feel like you got all the time in the world. While when you’re on the job and you’re team is working on a project that needs to be delivered in two days to solve for a business use case, things can get pretty intense.</li></ul><p>So now we’ve listed down the differences. Now let’s talk about these differences in a little more detail and how you can <strong>bridge the gap</strong> in all of these cases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/297/1*QIF1r_9Mb9HODtFhjnk9IQ.jpeg" /></figure><h3>1</h3><blockquote>When learning something, people are just following a boiler plate method off of some tutorial or YouTube video to solve a dummy problem. While when on the job, there is no boiler plate method and people are solving actual real problems.</blockquote><p>When you’re learning something and following tutorials, you’re not really solving any real problem. You’re copying whatever the person in the tutorial is saying. This might help you in getting to know the tools you’re using a little bit better but this can be a problem in an interview.</p><p>The interviewer will not be impressed by you saying that you have built a machine learning model using the famous <strong>Titanic data set. </strong>No matter what complicated models you used. Mainly because of two reasons.</p><ul><li>Titanic data set is a very famous data set readily available online and takes no effort to acquire.</li><li>Building a machine learning model on top of that solves no business problem.</li></ul><p>What will be more impressive to the interviewer is that you figured out a problem that is specific to you and solved it using tech tools you’re interviewing for. You collected the data yourself and did the cleaning as per your requirement and built on top of that.</p><p>And I cannot stress this enough but, <strong>complexity of your project does not matter for the most part. </strong>Your project does not have to be rocket science and hard to understand. It just has to be solving a real problem. A real problem of extremely average complexity is a lot more attractive to the interviewer as compared to a very complex project that doesn’t solve any problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/321/1*P3NH_pItuJNxQYelTVVbAw.jpeg" /></figure><h3>2</h3><blockquote>When learning something, people are often doing something from scratch and doing it end to end while on the job, most of the time you’ll be making changes to an already present architecture to incorporate a new business use case. So it’s also very important to learn how to build off of someone else’s work as well.</blockquote><p>When you’re alone and learning and doing something yourself, you don’t care about what others have done. You just want to see yourself doing it from scratch. But that’s usually not how things are done in a real business setting. In whichever company you might end up working, its very much likely that they already have a built in infrastructure that they are using and that is solving business use cases for them.</p><p>Most of the time you’ll not be asked to do something from scratch but to rather make slight modifications to an already present system. So I think it’s important to learn to build off of someone else’s work and to understand what the other person has done.</p><p>On the flip side, it is also important to do things yourself in such a way that other people can understand what you have done.</p><p>Here are a few tips that will make your work easy to understand.</p><ul><li>Leave <strong>comments</strong> when you code.</li><li>Incorporate <strong>logging</strong> into your code.</li><li>Have <strong>descriptive variable names</strong>. Don’t name variables as a,b,c,x,i ,er. You know who you are. We’re all guilty of this.</li><li>Incorporate <strong>Docstrings</strong> in your methods whenever you use them.</li><li>Try to follow the <strong>object oriented approach</strong> whenever coding something.</li><li>Follow <strong>abstraction</strong> and <strong>encapsulation</strong> principles. These are also very important from an interview stand point.</li><li>Incorporate a <strong>linter</strong> in your code like Pylint if you’re coding in python.</li></ul><p>When you do these things, it will not only help others in understanding your work but it will also help you in remembering what you have done if you try to revisit your code after a while :D</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/337/1*EV6xYZ2NRCiLwXII_-xygw.jpeg" /></figure><h3>3</h3><blockquote>While learning, most of us make use of dummy data or already processed data off of some website like <a href="https://www.kaggle.com/"><strong>Kaggle</strong></a> and incorporate that into our code. While when you’re on the job, data is really messy and crude in some cases and we have to figure out a way to clean or process it.</blockquote><p>When you’re learning data science or data engineering for that matter, we mostly make use of already available data sets that take no effort to acquire. We just download them off of some website and use them in our code.</p><p>However, this will not impress the interviewer irrespective of what you did with that data as this is certainly not how things are done in a real business setting. On the job, acquiring useful data is a mission in it of itself and should not be taken lightly when learning.</p><p>Here is what I recommend.</p><ul><li>Think about the problem that you want to solve and what data is required for that problem.</li><li>Then think about where you can get this data.</li><li>Then think about how you will get that data i.e. scraping or REST API</li><li>Then write up the code to acquire that data directly from the internet.</li><li>Pre-process that data and then use it in your project</li></ul><p>The interviewer will be a lot more impressed if they know that you have acquired the data yourself rather than downloaded pre-processed data off of some website.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/259/1*gkwS7eFyB4WY7QqOZCUENA.png" /></figure><h3>4</h3><blockquote>Localhost vs actual host. When you’re learning something, we just want to see it work so we end up locally hosting our work. No, <a href="http://localhost:8080/">http://localhost:8080</a> is not accessible to the internet :) While on the job, you’ll never be locally hosting anything.</blockquote><p>While it certainly feels like an achievement whenever you write <a href="http://localhost:8080/">http://localhost:8080</a> in your browser and you see your work in the shape of a web page and you want to tell the world about it. Things work a little differently in a real business environment.</p><p>On the job, we want our work to be accessible to the entire internet, or just our colleagues depending upon what it is. So locally hosting your work will not do. You actually have to hire a server and put your work there so that other people can see it too.</p><p>Here is what I recommend</p><ul><li>Go on to AWS console and launch up a free tier EC2 instance.</li><li>SSH into the instance and figure out how to host your work there.</li><li>Host it and then access it through a specific IP address specific to that instance.</li><li>Configure the security groups appropriately so that you can show it to everyone else as well.</li><li>Maybe purchase a domain name and set it on top of the IP address. Access the site through the domain name.</li></ul><p>Voila! now you have an actual website that you can link to in your CV and that everyone can see.</p><p>If you want to take a step further than this, I’d recommend you set up a database at the back-end as well. But this is a separate topic for another time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/259/1*QOu3ugFaa1iYQ3EclomxEw.jpeg" /></figure><h3>5</h3><blockquote>When learning something, we don’t take the cost of the infrastructure into account. We just want to build that one thing using that new special tool everyone’s been talking about. While in actual on the job, cost consideration is a massive factor in all of the major infrastructure decisions as things can scale very quickly and we don’t want to end up paying $100K in the end.</blockquote><p>This is something that rarely comes up when learning something new. Usually when we’re learning something, the scale of project tend to be small and not enough to make us think about the cost impact of our infrastructure.</p><p>But in a real business setting, things can scale pretty rapidly, especially when its a startup.</p><p>Here is what I recommend we should do in this case.</p><ul><li>Think about what if your data was a thousand or even a million times larger than it is now. What would you do?</li><li>Think about what if the traffic on your website suddenly grows a thousand times. What would you do?</li></ul><p>Try to factor in the cost of your approaches in these scenarios and how would you go about making a decision.</p><p>Usually decision making consists of the following factors</p><ul><li>Is it cost efficient?</li><li>Is it scaleable?</li><li>Is it reliable?</li><li>Is it easy to integrate with our other infrastructure?</li><li>Is it easy to understand and pick up for a new hire?</li></ul><p>Although, you might not have to deal with these things while learning but going through this thought process and explaining it in an interview will 100% put you ahead of the other candidates :)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/318/1*-jQHCeWGAhOddyoBCSLM-A.png" /></figure><h3>6</h3><blockquote>While learning something, we don’t care if a certain infrastructure is reliable or not. We just want to see it work. Doesn’t matter if it comes crumbling down in two hours (Guilty :)). While on the job, we want to make sure thing stay in tact for many years to come.</blockquote><p>This is also a very technical detail that we mostly overlook while learning something new. We usually don’t consider the impact of what if this particular thing stops working and what steps can we take in that situation.</p><p>This step is a major factor in making infrastructure decisions in all companies. In a real business, we want to make sure that things stay in tact and plan for a time when they don’t.</p><p>Here is what I recommend we should do to bridge the gap</p><ul><li>Think about the different variables that will be interacting with your system. They can be anything from company people to foreign factors.</li><li>Think about the likelihood of the system crashing whenever one of these variables go rogue.</li><li>Think about what kind of failures can happen e.g. system outages, gaps in processing, down times etc.</li><li>Then plan a strategy to execute whenever these specific failures happen.</li></ul><p>This is again one of those points that you might not have to think about while learning. But knowing these things in an interview and explaining them clearly will certainly put you ahead of the other candidates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/301/1*SOPkRD3GvDwr7KsW-_CZQQ.png" /></figure><h3>7</h3><blockquote>Git is also a tool that hardly comes up when you’re learning programming. We just CTRL + S on our computer and consider our work done for the day :) While in actual on the job scenario, all of our code will be version controlled using Git.</blockquote><p>Git is something I learned on the job as I hadn’t come across it while my learning phase.</p><p>While it is also okay to save your code in your hard drive and retrieve it whenever you require. But ask yourself this.</p><ul><li>What if instead of just 1 file there were 100s of files.</li><li>What if I was not the only person working on this project.</li><li>What if I wanted to see what changes have I made since I last saved my code.</li><li>What if I wanted to see what changes other people have made.</li></ul><p>You see the problem of not using Git?</p><p>I know that these things you will only need to worry about in an actual company and most probably in your own learning project, you <strong>will</strong> be the only person working on it and there will most likely be very few files you’re working with.</p><p>But I would still suggest to go ahead and push your code to an actual Git repository i.e. GitHub or GitLab anyway. This will not only give you all of the above benefits, but at the end, you will also have a very much rich Git profile to showcase in your CV and then you can talk about it with your interviewer.</p><p>You just can’t go wrong with using Git :)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/299/1*Fxy_PLMNVfJfiVht3pZSsA.jpeg" /></figure><h3>8</h3><blockquote>Time pressure is also something that is very different when you’re learning something vs when you’re on the job. While learning, you feel like you got all the time in the world. While when you’re on the job and you’re team is working on a project that needs to be delivered in two days to solve for a business use case, things can get pretty intense.</blockquote><p>While it is okay to take things slow while learning, it is also important to note that in a real business, you will not always have to luxury of having ample amount of time. There will most likely be multiple projects that you’re working on, all of which have their own deadlines that you have to manage.</p><p>I do think it is okay to start your learning journey thinking that you have ample amount of time but I also think that after a while it is important for you to start giving yourself realistic deadlines to meet. And then you have to work extra hard to keep up with those deadlines.</p><p>This I think will condition you for how an actual business environment is like and will make you more likely to adapt quickly wherever you choose to work.</p><p>So there you have it, things that everyone should keep in mind and lookout for while on their learning journey. Do let me know if you can think of some other differences. I’ll be happy to add them here :)</p><p>Hope you enjoyed the read!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc5c0f0ee018" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pivoting towards a data engineering career from a non-tech background.]]></title>
            <link>https://medium.com/@aqibme141/pivoting-towards-a-data-engineering-career-from-a-non-tech-background-9cc613654404?source=rss-3ab78c814aa6------2</link>
            <guid isPermaLink="false">https://medium.com/p/9cc613654404</guid>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Aqib Mehmood]]></dc:creator>
            <pubDate>Sat, 21 May 2022 14:21:03 GMT</pubDate>
            <atom:updated>2022-08-14T11:50:44.097Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sHQIBFbEQ7rNjCNh" /><figcaption>Photo by <a href="https://unsplash.com/@foxxmd?utm_source=medium&amp;utm_medium=referral">Matt Duncan</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote><strong>Well now let’s assume that I have convinced you to look a bit more into data engineering and how to get into it. Let’s also assume that right now you are doing a job in a career that has nothing to do with IT. What do you do?</strong></blockquote><p><strong>Data Engineering</strong> is one of the fastest growing areas of tech today. This title might have been previously held by its not so distant cousin <strong>Data Science,</strong> whose ascend began after the famous HBR article published in October 2012.</p><p><a href="https://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century">Data Scientist: The Sexiest Job of the 21st Century</a></p><p>Suffice to say it has been almost 10 years since the article was published and by the looks of it, things seem a little different.</p><p>Well what could be the reason of it? Why is the growth of data science coming to a halt? Is there no need for Artificial Intelligence or Machine Learning any more?</p><p>Well no, none of that. I am of the opinion that requirement for business insights obtained through machine learning is more than ever and is growing. With that being said, the field itself has gone through tremendous changes over the years so much so that most of the tasks required from a data scientist have been automated and are requiring less and less people to perform them.</p><p>Well since the requirement for business insights are growing, that means that the need for useful data is growing and that is where <strong>data engineering</strong> comes in. <strong>Data Engineering, simply put, is the collection and management of data.</strong></p><p>Well now let’s assume that I have convinced you to look a bit more into data engineering and how to get into it. Let’s also assume that right now you are doing a job in a career that has nothing to do with IT and since you are already doing a job, you can only devote limited time towards learning data engineering. What do you do?</p><p>Well this was exactly my situation when I started and I will try to answer this question with my experience. Let’s begin.</p><p>Before I tell you what to learn, I feel like I must tell you <strong>how to learn.</strong> It can be very tempting to press the play button on any video course on YouTube, Coursera, Udemy or Code Academy and sit back and watch and get the impression that you are learning. Suffice to say, this is the first pitfall that people fall into and the first that I want you to avoid.</p><p>With that being said, let me share my personal <strong>LASR </strong>technique that I found very effective.</p><ul><li><strong>Learn</strong></li><li><strong>Apply</strong></li><li><strong>Share</strong></li><li><strong>Repeat</strong></li></ul><p>Let me explain. Anything that you are learning (be it through videos or through literature), <strong>read or watch as little as you can</strong> and as minimum as it is required for you to start doing something with it. e.g. if you’re learning python, just learn the basic syntax of it and start making your own program, if you’re learning data-processing through pandas, learn the basic functions of the library and start meddling with your own data set. The point is to spend as little time learning and as much time doing as you can.</p><p>Next step would be to share what you have done. Let’s say that you have made an extremely basic program that takes in dirty and cluttered data and turns it into well structured and processed data, write about it. Share with the public on any platform that suits you be it Facebook, LinkedIn and an article on Medium like this one. This will do 2 things.</p><ul><li>Let the people know that you are actually learning something by doing it instead of just saying that you are which gives your journey so much credibility.</li><li>It will solidify your own concepts and will help you learn faster.</li></ul><p>I cannot stress this enough. <strong>DO NOT LEARN IN ISOLAION!</strong></p><p>It’s super beneficial to have a mentor that you can talk to and who can guide you through the process whenever things get tough. In my case, it was the all talented <a href="https://medium.com/u/b7a71d4661e7">Shafiqa Iqbal</a>.</p><p>Now coming to the fun stuff. The reason that you’re probably reading this article. <strong>What to learn?</strong></p><p>By looking at the very basic stuff that a data engineer does, you should learn the following.</p><ul><li>A programing language to write your data pipelines in(Python or R). I prefer Python.</li><li>A querying language. (SQL obviously)</li><li>A container management system like Docker.</li><li>A workflow management system (Airflow or Argo). Let’s go with Airflow.</li><li>A data lake solution like Amazon S3.</li><li>A data warehouse to store the data in. (Redshift or BigQuery)</li></ul><p>I will go through each of the these and maybe list down the resources that I found helpful and also mention the estimated time they will require.</p><h3>1. Python</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/119/1*-rnTX9F8I5w16IUKzdTRIw.jpeg" /></figure><p>Python is relatively a user friendly and an easy to learn language. Lucky for us because it also the most used language in the data space.</p><p>I particularly found this specialization on Coursera below very helpful in clearing some of the basic concepts of the language in a very interactive way.</p><p><a href="https://www.coursera.org/specializations/python-3-programming">Python 3 Programming</a></p><p>This should take you anywhere from 6–9 weeks. I recommend that you follow along and write your own code as you’re going through it either in an editor IDE or in a Jupyter Notebook. Jupyter is very beginner friendly so I recommend that for now.</p><p>When you are going through this course. Try pulling up your own data set and see if you can make sense of it using code and also see if you can visualize it and make graphs with it using python.</p><p>I would also recommend getting any publicly available API on the internet and getting data from that. Twitter has a very beginner friendly API and WhatsApp has recently also launched their API as well. This will also help you in finding out how APIs work.</p><p>Now you’re probably 9 weeks in your journey. It’s time to move to the next step.</p><h3>2. SQL</h3><p>If you work or plan to work in a field that has anything to do with data, SQL is absolutely non-negotiable.</p><p>I found this course on data camp particularly useful in getting familiar with SQL.</p><p><a href="https://www.datacamp.com/courses/introduction-to-sql">https://www.datacamp.com/courses/introduction-to-sql</a></p><p>They have an interactive interface which I found particularly helpful. You can apply a SQL query immediately after learning it.</p><p>After the introductory course, I recommend that you take their intermediate level course on SQL as well. This one will be a little more advanced and will let you see the actual power and use of SQL in a real business setting.</p><p>This might take you anywhere from 4–6 weeks.</p><p>Okay now we’re almost 15 weeks in our journey and we’re getting good. It’s time to take things up a notch.</p><h3><strong>3. Docker</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/1*1Xxy8RKXBscQ7ugCSSjCMA.png" /></figure><p>If you ever want to code in a development environment, Docker is a must know tool. It basically allows you to bundle up you code dependencies and environment in a container and push the container on some sort of an online repo like <strong>Docker Hub or AWS ECR</strong>.</p><p>Regarding Docker, I would recommend checking out their own website and documentation. They provide an easy to walk through tutorial which can bring you up to speed with the basic concepts.</p><p>I personally have also written an article on Docker which you can check out in which I walk you through a step by step process of making your first containerized application. Although, I make use of a simple pandas library use case, I believe it will be a good starter to familiarize yourself with the basic concepts.</p><p><a href="https://medium.com/@aqibme141/using-a-docker-container-to-run-a-python-script-8a6e0b618a38">Using a Docker container to run a python application</a></p><p>This should take you anywhere around 2–3 weeks. The goal is not to be an expert but just to get the hang of the basic concepts.</p><p>Now we’re around 18 weeks deep in our journey. It’s time to work on some actual workflows. Let’s move on.</p><h3><strong>4. Airflow</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g5LTfLtkg3uhLnDdACCpmw.png" /></figure><p>Airflow is a tool that let’s you set up workflows through the concept of DAGs(Directed Acyclic Graphs).</p><p>Let’s say that you have a large process that consists of lots of small processes. Let’s also say that some of those processes are dependent on one another in a sense that one process can only start after completion of another process. Airflow let’s you do this kind of orchestration.</p><p>Airflow is also one of those tools that I recommend learning from their own website and documentation. Their tutorial also makes use of Docker containers that we learned in our earlier step so this tutorial can also serve as a Docker use case.</p><p>Fortunately, I also have written an article on Airflow that you can go through. In the article I go through a step by step process of getting a fairly basic one step DAG up and running using Airflow’s documentation.</p><p><a href="https://medium.com/@aqibme141/data-pipeline-automation-using-airflow-dags-in-docker-containers-8ac8d68cef1b">Data pipeline automation using Airflow DAGs in Docker containers</a></p><p>This should also take you anywhere between 2–3 weeks. The goal in this one is also not the become a master at the tool but to get the hang of the basic concepts.</p><p>Now, we’re almost 21 weeks deep. It’s time to get the data to some point.</p><h3><strong>5. AWS S3</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*B9CIOrxdROHvtdmouQA1_A.png" /></figure><p>You can think of S3 as a file system similar to the one on your computer but on the Amazon cloud.</p><p>It provides you a place to store your structured, semi-structured or unstructured data in the form of files. The benefit of storing data on a blob store like S3 is that it provides scalability, retrievability and version control. It is fairly easy to learn and can be learned from their own documentation below.</p><p><a href="https://aws.amazon.com/s3/">Amazon S3 - Cloud Object Storage - AWS</a></p><p>At this point, I would encourage you to make use of everything we have learned up until this point and make a complete workflow that takes data from an API, processes it and loads onto S3 in the form of CSV or JSON. Amazon provides a very handy Python SDK Boto3 that you use to interact with S3.</p><p>This should not take you more than 1–2 weeks.</p><p>Now we’re almost 23 weeks deep and it’s time to get to our final step. Storing data in you data warehouse. Let’s go.</p><h3>6. Redshift</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mRQgByd5nR-d5qkp-8bV-A.png" /></figure><p>A data warehouse is simply an extremely scalable and efficient structured data storage. When I say scalable, I mean millions and billions of rows of data and up to petabytes of data. It makes use of the concept of clusters and nodes that will be very helpful to learn as it is one of those concepts that are key in <strong>Big Data </strong>handling.</p><p>It is also a place where you can make use of your SQL skills. I would recommend also learning this from AWS’s own documentation here.</p><p><a href="https://aws.amazon.com/redshift/">Cloud Data Warehouse - Amazon Redshift - AWS</a></p><p>At this point in our journey, I would recommend you to make use of everything we have learned up until this point and set an automated workflow that runs after a fixed interval.</p><p>In the last step, S3 was our end point. You can just add to that workflow and use the <strong>COPY </strong>command by Redshift to load data from S3 to Redshift. You can make use of the <strong>Pysycopg2 </strong>library in this.</p><p>This also shouldn&#39;t take you more than 1–2 weeks.</p><p>Now let’s take a deep breath and see what we have done.</p><p>We have set up a workflow using Python that takes data from a data source like an API and processes it using Docker and Airflow and loads it to S3 and finally uses the COPY command and loads the data to Redshift.</p><p>Congratulations! We have done it. We have set up an end to end batch data pipeline(the bread and butter of data engineers).</p><p>Now we’re ready to apply for jobs.</p><p>Remember in the beginning I asked you to document your journey? This is where that will help us. Sharing your journey with your audience will open opportunities for networking for you with the people who are already in the data space. Networking is, as we all know, hands down the most effective way of landing a job.</p><p>Now this journey has taken us around 25 weeks and it should also look like that based on the journey that you have shared with us.</p><p><strong>Now, beware that being interview ready is completely different from doing projects</strong>. You will need to study up on <strong>Data Structures and Algorithms</strong> and some other famous interview questions around Python and SQL which you can learn about from a thoughtful search on YouTube.</p><p>Now it’s time to start applying for jobs. Polish up your resume and start applying. Good Luck.</p><p>Hope this article helped :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9cc613654404" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data pipeline automation using Airflow DAGs in Docker containers]]></title>
            <link>https://medium.com/@aqibme141/data-pipeline-automation-using-airflow-dags-in-docker-containers-8ac8d68cef1b?source=rss-3ab78c814aa6------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ac8d68cef1b</guid>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[docker-compose]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Aqib Mehmood]]></dc:creator>
            <pubDate>Sun, 02 Jan 2022 22:19:33 GMT</pubDate>
            <atom:updated>2022-01-02T22:19:33.509Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jc0vI_CscDWTik98" /><figcaption>Photo by <a href="https://unsplash.com/@hharritt?utm_source=medium&amp;utm_medium=referral">Hunter Harritt</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this article we will walk through the step by step process of how to automate your data pipelines using Airflow. Airflow is an open source tool very popular in Data Engineering as it is used to author, monitor and automate your pipelines which can be very handy in some routine tasks.</p><p>In this article, I assume that you have some familiarity with Docker &amp; Docker Compose along with some basic python and the pandas library.</p><p>Following are the tools we will cover in this article</p><ul><li>Airflow</li><li>Docker</li><li>Docker-Compose</li><li>Windows Command Prompt</li><li>Python(Pandas)</li></ul><p>Following are the steps we will follow to reach our target</p><p>1- Configuring the Docker-Compose file and launching Airflow<br>2- Coding our first DAG<br>3- Launching and testing our DAG</p><p>So let’s get started.</p><h3><strong>1- Configuring the Docker-Compose file and launching Airflow</strong></h3><p>I would say that the hardest thing about Airflow is its configuration so best to get it out of the way at the beginning.</p><p>1- Make your project directory and in it make a new file and name it <strong>docker-compose.yaml. </strong>This is the configuration file that <strong>Docker </strong>will use to create our <strong>Airflow </strong>image and launch our <strong>container.<br></strong>2- Copy and paste the code below. Don’t worry. We’ll go through what it says in a minute.</p><pre>version: &#39;3&#39;</pre><pre>x-airflow-common:<br>  &amp;airflow-common<br>  image: apache/airflow:2.2.3-python3.8<br>  environment:<br>    &amp;airflow-common-env<br>    AIRFLOW__CORE__EXECUTOR: CeleryExecutor<br>    AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow<br>    AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow<br>    AIRFLOW__CELERY__BROKER_URL: redis://:<a href="http://twitter.com/redis">@redis</a>:6379/0<br>    AIRFLOW__CORE__LOAD_EXAMPLES: &#39;False&#39;<br>    AIRFLOW__CORE__FERNET_KEY: &#39;&#39;<br>    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: &#39;true&#39;<br>    AIRFLOW__API__AUTH_BACKEND: &#39;airflow.api.auth.backend.basic_auth&#39;<br>    _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}<br>  volumes:<br>    - ./dags:/opt/airflow/dags<br>    - ./logs:/opt/airflow/logs<br>    - ./plugins:/opt/airflow/plugins<br>  user: &quot;${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-50000}&quot;<br>  depends_on:<br>    &amp;airflow-common-depends-on<br>    postgres:<br>      condition: service_healthy</pre><pre>services:<br>  postgres:<br>    image: postgres:13<br>    environment:<br>      POSTGRES_USER: airflow<br>      POSTGRES_PASSWORD: airflow<br>      POSTGRES_DB: airflow<br>    volumes:<br>      - postgres-db-volume:/var/lib/postgresql/data<br>    healthcheck:<br>      test: [&quot;CMD&quot;, &quot;pg_isready&quot;, &quot;-U&quot;, &quot;airflow&quot;]<br>      interval: 5s<br>      retries: 5<br>    restart: always</pre><pre>redis:<br>    image: redis:latest<br>    expose:<br>      - 6379<br>    healthcheck:<br>      test: [&quot;CMD&quot;, &quot;redis-cli&quot;, &quot;ping&quot;]<br>      interval: 5s<br>      timeout: 30s<br>      retries: 50<br>    restart: always<br>  airflow-init:<br>    &lt;&lt;: *airflow-common<br>    entrypoint: /bin/bash -c &quot;/bin/bash -c \&quot;$${@}\&quot;&quot;<br>    command: |<br>      /bin/bash -c &quot;<br>        airflow db init<br>        airflow db upgrade<br>        airflow users create -r Admin -u admin -e <a href="/service/mailto:airflow@airflow.com">airflow@airflow.com</a> -f admin -l user -p airflow<br>      &quot;<br>    environment:<br>      &lt;&lt;: *airflow-common-env</pre><pre>airflow-scheduler:<br>    &lt;&lt;: *airflow-common<br>    command: scheduler<br>    environment:<br>      &lt;&lt;: *airflow-common-env<br>    restart: always</pre><pre>airflow-worker:<br>    &lt;&lt;: *airflow-common<br>    build:<br>      context: .<br>      dockerfile: Dockerfile<br>    command: celery worker<br>    healthcheck:<br>      test:<br>        - &quot;CMD-SHELL&quot;<br>        - &#39;celery --app airflow.executors.celery_executor.app inspect ping -d &quot;celery@$${HOSTNAME}&quot;&#39;<br>      interval: 10s<br>      timeout: 10s<br>      retries: 5<br>    environment:<br>      &lt;&lt;: *airflow-common-env<br>      # Required to handle warm shutdown of the celery workers properly<br>      # See <a href="https://airflow.apache.org/docs/docker-stack/entrypoint.html#signal-propagation">https://airflow.apache.org/docs/docker-stack/entrypoint.html#signal-propagation</a><br>      DUMB_INIT_SETSID: &quot;0&quot;<br>    restart: always<br>    depends_on:<br>      &lt;&lt;: *airflow-common-depends-on<br>      airflow-init:<br>        condition: service_completed_successfully</pre><pre>airflow-triggerer:<br>    &lt;&lt;: *airflow-common<br>    command: triggerer<br>    healthcheck:<br>      test: [&quot;CMD-SHELL&quot;, &#39;airflow jobs check --job-type TriggererJob --hostname &quot;$${HOSTNAME}&quot;&#39;]<br>      interval: 10s<br>      timeout: 10s<br>      retries: 5<br>    restart: always<br>    depends_on:<br>      &lt;&lt;: *airflow-common-depends-on<br>      airflow-init:<br>        condition: service_completed_successfully</pre><pre>airflow-webserver:<br>    &lt;&lt;: *airflow-common<br>    command: webserver<br>    ports:<br>      - 8081:8080<br>    healthcheck:<br>      test: [&quot;CMD&quot;, &quot;curl&quot;, &quot;--fail&quot;, &quot;<a href="http://localhost:8080/health">http://localhost:8080/health</a>&quot;]<br>      interval: 10s<br>      timeout: 10s<br>      retries: 5<br>    restart: always<br>    environment:<br>      &lt;&lt;: *airflow-common-env</pre><pre># Names our volume<br>volumes:<br>  postgres-db-volume:</pre><ul><li>This entire file is written in <strong>yaml </strong>format guidance of which can be taken from <strong>Docker’s </strong>official <a href="https://docs.docker.com/compose/compose-file/compose-file-v3/">website</a>.</li><li>At the top, we have specified the <strong>Airflow Image</strong> that we would like to build upon “<strong>image: apache/airflow:2.2.3-python3.8”</strong></li><li>Next we have specified the executor type as <strong>CeleryExecutor </strong>which is recommended for production as it is much faster.</li><li>Next we have specified the <strong>Postgres</strong> database for our <strong>Executor </strong>to work with.</li><li>Next in the <strong>Volumes</strong> heading we have specified our directories that we will be needing for our <strong>DAGs</strong> most important of which at the moment is our <strong>dags </strong>directory because we will code our <strong>DAGs</strong> here.</li><li>Finally in the <strong>Services </strong>section, we have specified our <strong>6 containers </strong>that we require for Airflow to launch. Please see the following snippet below from Airflow’s official <a href="https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html">website</a>.</li></ul><h3><strong>“</strong></h3><ul><li><strong><em>airflow-scheduler - The </em></strong><a href="https://airflow.apache.org/docs/apache-airflow/stable/concepts/scheduler.html"><strong><em>scheduler</em></strong></a><strong><em> monitors all tasks and DAGs, then triggers the task instances once their dependencies are complete.</em></strong></li><li><strong><em>airflow-webserver - The webserver is available at</em></strong><a href="http://localhost:8080."><strong><em>http://localhost:8080.</em></strong></a></li><li><strong><em>airflow-worker - The worker that executes the tasks given by the scheduler.</em></strong></li><li><strong><em>airflow-init - The initialization service.</em></strong></li><li><strong><em>postgres - The database.</em></strong></li><li><strong><em>redis - </em></strong><a href="https://redis.io/"><strong><em>The redis</em></strong></a><strong><em> - broker that forwards messages from scheduler to worker.</em></strong></li></ul><h3><strong>“</strong></h3><p>3- Now open up your terminal and go to the project directory and run the following command.</p><pre>mkdir -p ./dags ./logs ./plugins</pre><p>4- Make another file called <strong>Dockerfile </strong>in the project directory and paste the following code in it.</p><pre># syntax=docker/dockerfile:1</pre><pre>COPY . .</pre><p>This file copies all your files and folders in the project directory into the <strong>airflow-worker</strong> container. If you want, you can also add additional <strong>libraries</strong> which you will require for your DAGs using the <strong>RUN pip install &lt;name&gt; </strong>format.<br>5- Make another file call <strong>.env </strong>and paste the following code in it. This is for permissions so that <strong>Airflow </strong>can access the directories on your pc.</p><pre>AIRFLOW_UID=50000<br>AIRFLOW_GID=50000</pre><p>6- Now when everything is set up, go to your windows command line and type <strong>docker-compose up airflow-init.</strong> This will initialize the database and may take some time depending upon your internet connection. So be patient.<br>7- Now once the above command has finished executing, type again <strong>docker-compose up</strong>. This will launch all your containers.</p><p>Now your <strong>Airflow </strong>is up and running. As proof, go to your web browser and type <a href="http://localhost:8081/"><strong>http://localhost:8081/</strong></a><strong>. </strong>You will see the Airflow interface.</p><h3><strong>2- Coding our first DAG</strong></h3><p>Ok first things first. What is a <strong>DAG?</strong></p><p>DAG stands for <strong>Directed Acyclic Graph.</strong> You can think of it as a flow chart of processes that are acyclic. DAGs are mostly used in data pipelines.</p><p>1- Go to your <strong>dags</strong> folder in your project directory and make a new file. Call is <strong>demo dag.py.<br></strong>2- Copy and paste the below code.</p><pre>from airflow import DAG<br>from airflow.operators.python import PythonOperator, BranchPythonOperator<br>from airflow.operators.bash import BashOperator<br>from datetime import datetime<br>from random import randint<br>import pandas as pd<br>import os<br>import json</pre><pre>#dag_path=os.getcwd()<br>AIRFLOW_HOME = os.getenv(&#39;AIRFLOW_HOME&#39;)</pre><pre>def adding_date_column():<br>    df1=pd.DataFrame({&#39;a&#39;:range(0,10),&#39;b&#39;:range(50,60),&#39;c&#39;:range(100,110)})<br>    #df1=pd.read_csv(f&quot;{AIRFLOW_HOME}/Before/SampleCSVFile.csv&quot;,encoding=&#39;latin1&#39;)<br>    #df1.columns=[f&quot;Column{x}&quot; for x in range(3)]<br>    df1.insert(3,&quot;d&quot;,[datetime.now().time() for x in range(len(df1[&quot;a&quot;]))],True)<br>    #df1.to_csv(f&quot;{AIRFLOW_HOME}/After/SampleCSVFile.csv&quot;)<br>    return df1.to_json()</pre><pre>with DAG(&quot;excel_transform&quot;,<br>    start_date=datetime(2022, 1 ,1),<br>    schedule_interval=&#39;* * * * *&#39;,<br>    catchup=False) as dag:</pre><pre>training_model_tasks = PythonOperator(<br>        task_id=&quot;excel_transform&quot;,<br>        python_callable=adding_date_column)</pre><pre>adding_date_column</pre><p>This is a fairly basic <strong>DAG</strong> consisting of only one process. It makes a very basic <strong>DataFrame </strong>and adds a column to it filling it with <strong>current time values.</strong></p><p>In order to define a DAG we must create an instance of a <strong>DAG Class </strong>and specify the parameters which <strong>Airflow</strong> uses to know when to trigger this DAG and what are the steps in it and what <strong>function </strong>they will each call.</p><p>We specify the frequency of <strong>DAG </strong>execution using <strong>CRON </strong>expressions. In the above DAG, I have set the frequency of <strong>1 Minute. </strong>Meaning that out DAG will be executed every minute.</p><p>In the end you specify the order of tasks by using the <strong>&gt;&gt; notation. </strong>I have not used this notation here as our DAG only consists of 1 step.</p><p>Okay. Now we have created our DAG. Now lets trigger it.</p><h3><strong>3- Launching and testing our DAG</strong></h3><p>1- Now to to <strong>localhost:8081 </strong>and enter credentials. User name will be <strong>admin </strong>and password will be <strong>airflow<br></strong>2- On the interface, you will see the DAG that we just created. Airflow will pick it up automatically.<br>3- Click on it and see the <strong>Tree View. </strong>Wait a minute and see that our DAG will execute automatically. Wait another minute and it will execute again.</p><p>Now we have successfully automated our <strong>Data Pipeline</strong>. But where is that data going? Well now we haven’t connected our DAG to any output so only way to see it is by going to the <strong>Logs </strong>and seeing the <strong>Xcoms.</strong></p><p>That’s it for this article. I hope it was informative and easy to follow. Until next time :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ac8d68cef1b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using a Docker container to run a python application]]></title>
            <link>https://medium.com/@aqibme141/using-a-docker-container-to-run-a-python-script-8a6e0b618a38?source=rss-3ab78c814aa6------2</link>
            <guid isPermaLink="false">https://medium.com/p/8a6e0b618a38</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[docker-image]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pandas]]></category>
            <dc:creator><![CDATA[Aqib Mehmood]]></dc:creator>
            <pubDate>Fri, 31 Dec 2021 18:46:10 GMT</pubDate>
            <atom:updated>2021-12-31T18:48:12.612Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KJphpMjf7trStdSj" /><figcaption>Photo by <a href="https://unsplash.com/@carrier_lost?utm_source=medium&amp;utm_medium=referral">Ian Taylor</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this article we are going to go through the step by step procedure of how to make a docker container to run a basic python application using pandas library.</p><p>We will use the following in this tutorial</p><p>1- Docker<br>2- Python<br>3- Pandas</p><p>Often in the development stage of your applications you need a consistent environment across all your devices and all the devices of your fellow developers. It can be cumbersome to install all the dependencies from scratch whenever you try to run your code on a new device or in a new environment. Docker is here to solve exactly this problem.</p><p>Docker provides a way to package everything you need to run an app or a simple script in a unit called a <strong>Container </strong>which you can download or pull into all the devices you use and just start working right away.</p><p>Following are the steps that we will follow to make and run a Docker container.</p><p>1- Installing python and pandas onto our system and making a python script.<br>2- Creating a Docker <strong>Image </strong>file which will be used to make a <strong>Container.<br></strong>3- Testing the container.</p><p>So let’s get started.</p><h3><strong>1- Installing python and pandas onto our system and making a python script</strong></h3><p>1- I’m assuming that you already have python installed on your computer. If not, you can visit their website and download the installer file and complete the installation process.<br>2- Open your windows <strong>CMD </strong>and install <strong>Pandas </strong>using <strong>Pip install pandas </strong>command.<br>3- Create a project directory and give it a name. For this tutorial we’ll call it <strong>python-docker.<br></strong>4- In this folder location make a new python file and give a name <strong>demo app.py.<br></strong>5- Next we will write a very basic program that reads and prints a <strong>CSV</strong> file using <strong>pandas DataFrame</strong>. First click on this <a href="https://sample-videos.com/csv/Sample-Spreadsheet-1000-rows.csv">link</a> and download the CSV file that we will be using for this and save it in your project folder.<br>6- Next just copy and paste the script below.</p><pre>import pandas as pd</pre><pre>df = pd.read_csv(&quot;SampleCSVFile.csv&quot;,encoding=&#39;latin1&#39;)<br>print(df.head(1000))</pre><p>7- Run the script by typing <strong>python ‘demo app.py’</strong> in your windows <strong>CMD</strong> and see if it works. It should.</p><h3><strong>2- Creating a Docker Image file which will be used to make a Container</strong></h3><p>1- Install <strong>Docker</strong> by going to their official <a href="https://docs.docker.com/engine/install/">website</a> and following the installation steps.<br>2- Make a new file in your project directory and give it a name <strong>Dockerfile.</strong> Be careful to not give it an extension.<br>3- Copy and paste the below code. We will go through every line next.</p><pre># syntax=docker/dockerfile:1</pre><pre>FROM python:3.8-slim-buster</pre><pre>RUN /usr/local/bin/python -m pip install --upgrade pip</pre><pre>RUN pip install pandas</pre><pre>COPY . .</pre><pre>CMD [ &quot;python&quot;, &quot;demo app.py&quot;]</pre><ul><li>First line tells the docker engine on how to read/parse the lines coming after it. This line always has to be the first line before any spaces or characters.</li><li>Second line downloads a base python image from <strong>Dockerhub. </strong>It is an environment that already has python installed on it.</li><li>Third line upgrades the <strong>Pip </strong>as the above image is running an old version of it.</li><li>Fourth line installs <strong>Pandas </strong>library in the image. This is a crucial step that we will also use to test our Docker image next</li><li>Fifth line copies all the files in the current directory and stores them onto the file system attached with the docker image.</li><li>Finally the sixth line tells Docker image on what to do when the <strong>Run </strong>command is given.</li></ul><p>3- Now go to your windows <strong>CMD </strong>and make a docker image by typing</p><pre>docker build --tag python-docker .</pre><p><strong>python-docker </strong>will be the repository that the image will be in.</p><h3><strong>3- Testing the container</strong></h3><p>Now it’s finally time to test the container and to do that we will have to uninstall the pandas library from our system.</p><p>1- Type <strong>Pip uninstall pandas </strong>in your windows <strong>CMD </strong>and uninstall the pandas library.<br>2- Run the docker image by typing <strong>Docker run python-docker</strong> in your windows <strong>CMD </strong>and see the results.</p><p>Now you will see a pandas dataframe printed on your screen despite not having pandas library installed on your system. Where is the pandas library coming from? This is the power of Docker.</p><p>Hope you found this informative and fun.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a6e0b618a38" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Querying data from S3 using AWS Athena]]></title>
            <link>https://awstip.com/querying-data-from-s3-using-aws-athena-18a41d061d94?source=rss-3ab78c814aa6------2</link>
            <guid isPermaLink="false">https://medium.com/p/18a41d061d94</guid>
            <category><![CDATA[glue]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[athena]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Aqib Mehmood]]></dc:creator>
            <pubDate>Tue, 21 Dec 2021 19:09:56 GMT</pubDate>
            <atom:updated>2021-12-23T06:03:29.030Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QgpMBjFXd5DKwj2v" /><figcaption>Photo by <a href="https://unsplash.com/@urielsc26?utm_source=medium&amp;utm_medium=referral">Uriel SC</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>S3 is a data lake provided by AWS for storage of both structured and unstructured data. It is easy to query the data if the data is well structured in the form of rows and columns. But what if we had rows and columns but not in a database but rather in an excel file. Can you query the excel data without opening the file directly from S3?</p><p>That’s what we will do in this article.</p><p>Following are the tools we will use in this article</p><p>1- Amazon S3<br>2- AWS Glue<br>3- Amazon Athena<br>4- SQL (to query the data)</p><p>We will follow the following steps in order to achieve our goal.</p><p>1- Create an S3 bucket and upload an excel file.<br>2- Create a crawler in AWS Glue.<br>3- Use the Glue Catalogue to query the S3 excel file data using SQL.</p><p>Let’s get started</p><h3><strong>1- Create and S3 bucket and upload an excel file.</strong></h3><p>1- Go to your AWS console and type S3 in the search bar and press enter.<br>2- Click on <strong>Create Bucket </strong>button. Type in a bucket name. Remember that your bucket name has to be unique across all AWS accounts.<br>3- You can leave everything as is and click on <strong>Create Bucket </strong>at the bottom.<br>4- Click on your created bucket on the next page and click on <strong>Upload</strong>.<br>5- Download this dummy CSV file <a href="https://sample-videos.com/csv/Sample-Spreadsheet-1000-rows.csv">here</a> and upload it.<br>6- Now create another S3 bucket and leave it empty. This will be the bucket that the results of our queries get stored in.</p><p>Now moving on to the next step.</p><h3><strong>2- Create a crawler in AWS Glue</strong></h3><p>1- Type <strong>Athena </strong>in the search bar and hit enter.<br>2- Click on <strong>Explore the query editor.<br></strong>3- At the top click on <strong>Settings </strong>and then click on <strong>Manage.<br></strong>4- Click on <strong>Browse </strong>and select the empty bucket that we just created.<br>5- At the <strong>Editor </strong>tab click on <strong>Create </strong>and then select <strong>AWS Glue Crawler.<br></strong>6- A new window will open up to create the crawler. Give it a name. Click on <strong>Next.</strong><br>7- Again leave everything as it and click <strong>Next.<br></strong>8- In the new window, click on the File icon in the <strong>Include Path </strong>field and select the S3 bucket which has the excel file in it. Click on <strong>Next.</strong><br>9- Again click on <strong>Next. </strong>Now select <strong>Create an IAM role </strong>and give it a name below. Click on <strong>Next.<br></strong>11- Select <strong>Run on demand </strong>and click on <strong>Next.<br></strong>12- Click on <strong>Add database </strong>and give it any name at the top.<br>13- Click on <strong>Next.</strong> Now review everything we just did and click on <strong>Finish.<br></strong>14- Now select the crawler that we just created and click on <strong>Run crawler.</strong></p><p>This will take a couple of minutes to please be patient.</p><p>Now move on to the next step.</p><h3><strong>3- Use the Glue Catalogue to query the S3 excel file data using SQL.</strong></h3><p>Now if you go back to the Athena page you will see that a new database has been created. Click on the three dots in front of it and select <strong>Preview. </strong>This will automatically generate a SQL query and run it and then below you will be able to see your table.</p><p>Now you can run some awesome SQL queries on the table to show you specific rows or columns.</p><p>That’s it. Until next time.</p><p>Keep learning.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18a41d061d94" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/querying-data-from-s3-using-aws-athena-18a41d061d94">Querying data from S3 using AWS Athena</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Boto3 to get data into S3 through Kinesis]]></title>
            <link>https://medium.com/@aqibme141/using-boto3-to-get-data-into-s3-through-kinesis-1f6e6171ba0?source=rss-3ab78c814aa6------2</link>
            <guid isPermaLink="false">https://medium.com/p/1f6e6171ba0</guid>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[kinesis]]></category>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[kinesis-firehose]]></category>
            <category><![CDATA[boto3]]></category>
            <dc:creator><![CDATA[Aqib Mehmood]]></dc:creator>
            <pubDate>Sun, 19 Dec 2021 15:01:09 GMT</pubDate>
            <atom:updated>2021-12-19T17:14:37.106Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UwoKv5BIlkg1_YJc" /><figcaption>Photo by <a href="https://unsplash.com/@markusspiske?utm_source=medium&amp;utm_medium=referral">Markus Spiske</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>We will cover the following tools in this article</p><p>1- Exchange Rate API for Python<br>2- Boto3<br>3- Kinesis Data Stream<br>4- Kinesis Data Firehose<br>5- S3<br>6- Lambda<br>7- SNS</p><p>Following will be the steps that we will follow</p><p>1- Use the Exchange Rate Python API to get the currencies data.<br>2- Create an S3 bucket.<br>3- Create Kinesis Data Steam and set Kinesis Data Firehose as consumer.<br>4- Use the Boto3 Amazon SDK for Python to push data into the data stream.<br>5- Create a Lambda function that gets triggered every time an object is loaded onto an S3 bucket and send data to SNS.</p><p>So lets get started.</p><h3><strong>1- Use the Exchange Rate Python API to get the currencies data.</strong></h3><p>First you need to go to <a href="https://app.exchangerate-api.com/">https://app.exchangerate-api.com/</a> and create an account which will give you an API Key which will be shown on your dashboard.</p><p>Be careful to not show this key to anyone as it is meant to be kept private. I will show you a dummy key in the code.</p><h3><strong>2- Create an S3 bucket.</strong></h3><p>Creating an S3 bucket is fairly easy. Just type S3 in your AWS Console and then click on <strong>Create Bucket. </strong>Give it a name and choose your region.</p><p>You can leave everything else as is because we are not concerned with anything else at the moment.</p><h3><strong>3- Create Kinesis Data Steam and set Kinesis Data Firehose as consumer.</strong></h3><p>1- Go to Amazon Kinesis and select <strong>Kinesis Data Streams </strong>in the top menu and click on <strong>Create Data Stream.<br></strong>2- Give it a name and select <strong>On Demand. </strong>Then click on <strong>Create Data Stream </strong>below.<br>3- Then on the new page you will see <strong>Kinesis Data Streams </strong>under <strong>Consumers</strong>. Click on <strong>Process with delivery stream</strong>.<br>4- Then on the next page, select <strong>Amazon S3 </strong>from the dropdown <strong>Destination </strong>menu.<br>5- Then if you scroll down you will see <strong>Destinations Settings. </strong>Click on <strong>Browse</strong> and choose the S3 bucket that we created earlier.<br>6- Then click on <strong>Create delivery stream.</strong></p><h3><strong>4- Use the Boto3 Amazon SDK for Python to push data into the data stream.</strong></h3><p>Now its time to write some code. See below</p><pre>import boto3<br>import requests<br>import pandas as pd<br>import json</pre><pre>api_key=’9f7aXXXXXXXXXXX69589401&#39;<br>currency=’PKR’<br>get_url=f’<a href="https://v6.exchangerate-api.com/v6/%7Bapi_key%7D/latest/%7Bcurrency%7D&#39;">https://v6.exchangerate-api.com/v6/{api_key}/latest/{currency}&#39;</a></pre><pre>rates = requests.get(get_url)</pre><pre>rates_json=rates.json()<br>rates_dict=rates_json[“conversion_rates”]</pre><pre>for cur in rates_dict.keys():<br> rates_dict[cur]=1/rates_dict[cur]</pre><pre>y=json.dumps(rates_dict)</pre><pre>client = boto3.client(‘kinesis’)</pre><pre># Send message to Kinesis DataStream<br>response = client.put_record(<br> StreamName = “Rates”,<br> Data = y,<br> PartitionKey = “demo-shard”<br>)</pre><pre>print(‘Message sent’)</pre><pre>if response[‘ResponseMetadata’][‘HTTPStatusCode’] != 200:<br> print(‘Error!’)<br> print(response)</pre><p>Remember to use your own API key in above code.</p><p>Write this code in an IDE of your choice and save it in your project directory.</p><h3><strong>5- Create a Lambda function that gets triggered every time and object is loaded onto an S3 bucket and send data to SNS.</strong></h3><p>1- Go to <strong>Lambda </strong>in your console and click on <strong>Create Function.<br></strong>2- Give it a name. Choose the latest Python version as <strong>Runtime.<br></strong>3- Leave everything as is and click on <strong>Create Function</strong> below.<br>4- On the next page, click on <strong>Add Trigger </strong>and choose <strong>S3 </strong>from the dropdown menu on the next page.<br>5- Select the bucket that we created in the second step and then click on <strong>Add below.<br></strong>6- Now before we create destinations, we must create an <strong>SNS topic.<br></strong>7- Go to SNS in your console and click on <strong>Create topic. </strong>Give it a name and click on <strong>Create topic.</strong><br>8- Now click on <strong>Add subscriptions </strong>and then under <strong>Protocol </strong>select <strong>Email. </strong>Type in your email address and click on <strong>Create subscription.<br></strong>9- Now go back to adding <strong>Destinations </strong>in Lambda. Select <strong>Asynchronous invocation</strong> and then select <strong>SNS. </strong>Select your topic that you just created and then click on <strong>Save.</strong></p><p>Now all the infrastructure has been setup and now it is time to run our code.</p><p>Open up your <strong>Command Prompt </strong>and go to the location where you saved your code. Final thing before we run our code, make sure that you are logged in on the command line through your IAM role with Administrator access.</p><p>Now run your Python code. It should show ‘Message sent’. Which means that our experiment has been a success.</p><p>Voila!!. Now you should receive an email in a couple of minutes.</p><p>Keep learning :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f6e6171ba0" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>