<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Abhijeet Kamble on Medium]]></title>
        <description><![CDATA[Stories by Abhijeet Kamble on Medium]]></description>
        <link>https://medium.com/@abhijeet5793?source=rss-71cf2226f9f5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*IjZbw6tZSypTcTn_.jpg</url>
            <title>Stories by Abhijeet Kamble on Medium</title>
            <link>https://medium.com/@abhijeet5793?source=rss-71cf2226f9f5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 14:01:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@abhijeet5793/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[SQL problems]]></title>
            <link>https://abhijeet5793.medium.com/sql-problems-279f977a8c92?source=rss-71cf2226f9f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/279f977a8c92</guid>
            <dc:creator><![CDATA[Abhijeet Kamble]]></dc:creator>
            <pubDate>Thu, 05 Dec 2019 03:41:20 GMT</pubDate>
            <atom:updated>2019-12-05T03:41:20.736Z</atom:updated>
            <content:encoded><![CDATA[<p>Two common tasks in data science work are filtering data so we can work with just the data we’re interested in, and the other is grouping data so we can calculate values for an entire group such as a product category or a department. Let’s start with filtering data using numeric values. We’ll use the staff table for this example. Let’s assume we want to list everything with a salary greater than $100,000. So I’ll use the select command, and for this example, we’ll include the last name, department, and of course, salary. And we will select this from the staff table. Now, we only want to include a certain subset of the employees, so we use a WHERE clause. We’re going to say WHERE salary is greater than 100,000, and if we execute that statement, we’ll get a list of employees who work in various departments, but all of whom earn salaries over $100,000. Filtering based on character strings works in similar ways. Now, let’s generate a list of employees who work in the tools department, and we’ll keep the columns the same. So I’ll just change the WHERE clause to say WHERE department = tools, and I’ll execute that statement. And now all of the employees listed are in the tools department, and if we were to browse a little bit, we’d notice that yes, in fact, they are all in the tools department. Now let’s combine the two filters. It’s pretty common to have more than one condition we want to filter on. In fact, sometimes WHERE clauses can be the longest part of a select statement, because there are so many conditions that need to be satisfied. To filter for rows that satisfy both conditions, we use the AND keyword. So I’ll include that in my WHERE clause, and I’ll say WHERE department = tools AND salary &gt; 100,000. And we’ll execute this statement, and we’ll notice that sure enough, everyone listed here is in the tools department and has a salary greater than $100,000. Now, if we want at least one of the conditions to be met but maybe not both, then we can use the OR keyword. And I’ll change the AND to OR, execute that statement, an you’ll notice, now we have some people who work in departments other than tools, but all of the people who aren’t in the tools department have salaries over $100,000. And if we browse a little bit, we’ll notice that some of the people in the tools department are listed, but they have salaries less than $100,000. For example, in this case, Watson is in the tools department and earns a salary of $81,870. Now, in SQL, the LIKE operator is used to match patterns. There are many options for working with the LIKE operator. For example, if we wanted to look at all of the departments that begin with the letter B, we could change this WHERE clause to say WHERE department LIKE ‘B%.’ And this means select all staff from the departments where the department name begins with a B and has zero or more characters after it. And let’s, to make this a little more interesting, let’s add a GROUP BY statement here, and let’s GROUP BY department, and we’ll change this. Instead of doing less than the department, we’ll remove the last_name, we’ll GROUP BY department, and we will sum the salary. And this will allow us to see how much we’re spending on salaries in each department that begins with a B. We’ll execute that statement, and you’ll notice, there are three departments listed, and then the sum of the salaries is listed as well. Now, we can make the LIKE operator actually be more specific. For example, if we just wanted to see departments that began with the letter B followed by the letter O, we could specify that, and if we execute the statement, as we expected, we see just one department, and that’s the one that begins with the letters B-O. The special symbol percent can be used in different parts of the string. It does not have to be at the end of the string. For example, if you want information on the departments that begin with B and end with Y, you could use the following query: like that, we’ll have a percent in the middle, and we’ll end with Y. With this query, any department that begins with B and ends with Y will match. So if we execute this statement, as we expect, we have two departments. Both begin with B and both end with Y. One note, be careful when using percent at the beginning of a string. When this happens, the database cannot use indexes that may be on the column. This will cause the database to scan every row in the table looking for matches. That’s okay, it just may take a lot longer than you expect.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=279f977a8c92" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Statistics with SQL]]></title>
            <link>https://abhijeet5793.medium.com/statistics-with-sql-6c718f8ba75b?source=rss-71cf2226f9f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c718f8ba75b</guid>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Abhijeet Kamble]]></dc:creator>
            <pubDate>Thu, 05 Dec 2019 03:23:41 GMT</pubDate>
            <atom:updated>2019-12-05T03:23:41.727Z</atom:updated>
            <content:encoded><![CDATA[<p>Data scientists often use statistics to better understand data sets. SQL includes functions for describing data sets, so let’s see how to use some of the most commonly-used statistic functions, to better understand our data. I’m going to start by selecting from the staff column, I will select star, from staff, and I’ll limit this to 10, cause I’m just trying to get a feel for the data. And we’ll notice that this table includes information about employees, which includes some string characters and some dates, and some integer values to work with. Now, one of the things I might like to know is, how much does the company pay in salary to all of its staff, across a given year? Well, the way to do that is to first of all, we’ll remove the limit clause, cause we want to look at everyone, and we want to use the sum function, which is one of those statistic functions, and it’s applied to a column, like salary, and I’ll execute the query, and notice that the result is approximately $97.3 million is paid per year, across all of the staff. Now next, I might like to understand how much does each department pay in salary? Well, to do that, I’d want to use the group by clause again. So I will group by department, and I’ll be sure to include the department name in the results set. I’ll execute that query, and I’ll notice that I have 22 rows returned, one for each department, and also, a value of the total amount spent in salary each year for that department, okay, that’s really useful information. Different departments may have different number of employees so it’s also helpful to understand what’s the average salary paid per employee in department. To find that, I can use the AVG or average function, and ask for the average salary, and since I’m grouping by department, the average function will calculate the average salary per department, and I’ll simply execute the statement, and now we’ll notice that I have a third column added. And I’m just going to spread it out here, to make it easier to read. And this is the average salary per department. There are a couple of other statistics that are sometimes used by statisticians and others who are familiar with statistics. These are called the variance, and standard deviation. We won’t go into details about what these mean, but I’ll roughly describe them as both good ways of measuring how spread out data is around an average. So, let’s look first at the variance, and how to calculate that. SQL has a function called var pop, short for variance of population, and that’s applied to a numeric column, like salary. And when we execute this statement, we’ll get a fourth column added to our results set, and this is known as the variance. I won’t go into details about it, but it’s a good way of measuring how spread out the data is. A more commonly used statistic for understanding the distribution of data is something called the standard deviation. In SQL, that function is stddev, underscore pop, which is short for standard deviation of population. And we want to know the standard deviation over the salary, so we’ll use that function, and execute, and again, another column is added to our result set, and as we can see, this number indicates the standard deviation over the salary, by department. Now you’ll notice that when we use functions like average, and standard deviation, we’re getting a lot of decimal places after the decimal point. Since we’re talking about salaries, we’re talking about monetary values, it’s probably helpful to round these to two decimal points. To do that, we simply use the round function, and apply it to the average, the variance, and the standard deviation. So I’ll do that right now, I’ll add round, and one of the things I want to do is tell the round function to round the average salary, and keep just two decimal points, and I’ll do the same thing with variance, I’ll ask to round it and keep just two decimal points, and finally I’ll do the same for standard deviation. And when I execute, you’ll notice that each column that has the round function applied to it, has only two decimal places, this makes the results a little easier to read. Now I will say, if you’re not familiar with statistics, and terms like variance and standard deviations are not things you’ve come across before, that’s nothing to worry about, it’s just important to know that if you do need them in the future, the functions are available to you in SQL.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c718f8ba75b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Window Functions]]></title>
            <link>https://abhijeet5793.medium.com/window-functions-e9e428a89803?source=rss-71cf2226f9f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/e9e428a89803</guid>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[Abhijeet Kamble]]></dc:creator>
            <pubDate>Thu, 21 Nov 2019 04:35:30 GMT</pubDate>
            <atom:updated>2019-11-21T04:35:30.227Z</atom:updated>
            <content:encoded><![CDATA[<p>Sometimes when working with data, you’re going to need to perform calculations that change across the span of time or other discrete values. A good example is a rank. If you wanted to calculate the rank of something. And these are known as window functions in SQL terms. Again, not many or not all data platforms support these more advanced functions, but Presto does regardless of whether or not the underlying database does. So let’s dig in here and see some of these functions in action. First I want to find the highest ranked parts from my tiny database, the <strong>tpch</strong> benchmarking database. So I’ll start by saying presto — server localhost logging in. And I want to run a function against this, so let’s do select, and we’ll do part key, extended price, which is the price listed, rank, this is our function, this is our window function. And when we do this we have to say over. What are we ranking over? Now we could partition this let’s say by category or by year, but in this case I just want to find the rank over all of them. So I’ll just order by the extended price in a descending order, so the highest to lowest. We’ll call it as rnk for rank. We’ll call that from tpch.tiny.lineitem and we’ll order by rank. So there you have the part key, the extended price, and then the rank. And notice that we have a tie for our third one here, so we don’t have a fourth place. We have two that are tied at three, and then fifth. And in fact we have a tie there, a tie at seven. A lot of these make sense because they’re all similarly priced items. So this is how you can calculate a rank using the over function in a window function built into Presto. Another thing you might want to do where you’re calculating something which changes over time is a moving average. And this is all going to lead up to something called Bollinger Bands, which I’ll show you how we can graph that and what that looks like at the end. So first we need to create an aggregation table. Now this performs the rolling average, and then we’ll reference that additionally after we create it. So the way we do that is using something called with. So we start out with sales weekly, that’s the name of our table. We’re going to say as select date_trunc, which is a date function we reviewed already, week, and orderdate. We’ll call that wk for week. So that’s going to give me the rolled up date to the week level. Then we’ll do the sum of totalprice as total. This is coming from tpch.tiny.orders, and we’re grouping by the date_trunk week of orderdate. So, this is our with table, this is our temporary table, essentially, that we’re creating. Now we’re going to use a function and join to that. So, we’re going to say select wk avg(total) over, this is our rolling function, order by wk, so we’re ordering sequentially here by time, and we’re going to say rows between, so we’re getting it a range, so before, we were just ranking over all, now we’re going to create an average with just a specific period in mind. So it’s going to be rows between five preceding and current row. So what that’s going to do, that’s going to give me, essentially, a six-week rolling average that’s going to calculate this average over, and it’s ordering by week, chronologically there, the five preceding rows and the current row. So, the first five are just going to be leading up to where it actually becomes moving or rolling. So, here I say, from sales_weekly and we order by wk. So, it starts out with our first week and our second week and all that, where it’s just kind of starting to build the aggregate, and then after six weeks, we’re actually into the rolling average. So this is really key in something that you’re probably going to get asked to do a lot if you’re working with sales data. Hit “q” to exit there, and another question you may come into is a running total. So I want to find the total for every customer over time. And this is something that is really common, so to do that, let’s start out by saying select, and we’ll give it the key that we want to use for the customer, so custkey is the field, and we’ll do it by order date. Now, on a new line, I’m going to say sum of totalprice, this is going to be the value that I’m actually doing a running total of, and again, I use the over clause, and I need to partition this by customer key. What that does is that means that every time I have a new customer key, I’m going to start my running total, my sum of totalprice, over again. That way, the running total doesn’t span multiple customers, which isn’t what I want in this case. I also need to tell it what to order by, so order by orderdate as running_total from tpch.tiny.orders, and order by custkey and orderdate. So now, what you have is the customer key, the first order that they ever placed with us, in fact, every order, the date there, and a rolling total. So these numbers will only get bigger. And then, as you go down, you’re going to find the next customer key, and when that happens, you’re just going to have that new totalprice, the new order amount, instead of the rolling total. So you can see every time we have a new customer key, it sort of resets back to the first amount that that new customer had with our first order.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e9e428a89803" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Presto]]></title>
            <link>https://abhijeet5793.medium.com/presto-e86110253904?source=rss-71cf2226f9f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/e86110253904</guid>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Abhijeet Kamble]]></dc:creator>
            <pubDate>Tue, 12 Nov 2019 14:52:24 GMT</pubDate>
            <atom:updated>2019-11-12T14:52:24.371Z</atom:updated>
            <content:encoded><![CDATA[<p>Let’s start by talking about where Presto comes from. And I think it’s always important to understand the nature of something, because this will help us understand its purpose and how we can use this platform to solve the challenges that we face when we’re trying to deliver insights to our customers. It all started in 2006, when a group of developers from a company called Nutch were hired by Yahoo. This project they were working on became Hadoop and they open-sourced it at this time, and the point of Hadoop was to help Yahoo index all of the data on the internet that they were ingesting and then trying to provide search for. Now this worked exceedingly well because it was able to spread the data out across multiple different machines, different nodes in the cluster, and create what we now call a distributed computing platform. The problem with Hadoop was that in order to access and work with the data, you had to understand some very low-level APIs and know Java. This is when Google added MapReduce. MapReduce is a Java-based interface for working with data in the Hadoop distributed file system. Now, MapReduce helped developers start to use Hadoop for all the data storage needs that they had. So that was great, so far, so good. But when companies really began to realize that their data analysts and data scientists needed access to this new data store, they faced a big problem. Most data people simply weren’t interested or weren’t going to learn Java and these low-level APIs in order to access this data. They wanted to use the tools that they were familiar with, such as SQL. Now this need for SQL access to Hadoop data is what led to, in 2008, Facebook developing a SQL abstraction on top of MapReduce. This was called Hive. And this is when Hadoop really took off, in my opinion, because now data scientists and data analysts had the ability to access and process and analyze these big data volumes that were stored in Hadoop. So over the years, Hadoop has grown tremendously in popularity. It, in fact, is synonymous with big data. So in my opinion, Hive really helped Hadoop grow. And the reason is because now the analysts could actually access, process, and analyze huge amounts of data stored in the platform. Now, the big problem with Hive was that it was still batch-oriented and really just a translator. So you would write your code in Hive, and then the engine would translate that into MapReduce, which would then process the data. This was, again, great and very useful. However, it was really, really slow. So over the years, Hadoop has grown tremendously in popularity and today is the most prevalent big data engine out there. The challenge still remains for most companies that processing and analyzing the data takes a lot of time, however. Now, this is where the nature of Presto comes from. And many of the analytical database vendors out there use an architecture known as MPP, or Massively Parallel Processing. What this does is to distribute parts of the query across different nodes in a cluster, and each little tiny query runs very quickly, then pulls all that data back together in memory and returns it to the user, much faster than if it were a single database or a single node or a batch operation such as how MapReduce works. The problem with many of these MPP database vendors is that in order to use and gain the efficiency in speed, you have to move your data from where it lives, what’s in Hadoop, and put it into their database system. So not only do you have to maintain the data flows and pipelines capturing the data and putting it in Hadoop from the start, you also have another set of data pipelines that has to maintain and synchronize the data between Hadoop and this third-party system. So at Facebook, they saw this problem, and they developed a solution that would allow you to run these MPP-style really fast queries wherever the data lives. And this is what Presto gives you. Now Presto is licensed under the Apache license. However, it’s not a top-level project. But it may become one as popularity seems to be growing around this idea of running fast queries on Hadoop data without having to migrate it to another system first.</p><p>here are four main reasons I think why you would want to use Presto. The first one is speed. And I think this is where Presto really shines because when you’re working with large data volumes speed can really slow the analysis process down. Many times you just generally accept that it isn’t going to be possible with such large volumes of data. However this can cost your company a lot of money. If I’m a highly-paid data scientist making over six figures per year, and if you add up all the time I spent waiting for a query to finish and for me to get the answer I’m looking for, it will cost the company a ton of money over the years that I work there. So speed is incredibly important and it is often worth whatever the cost is to speed up those queries. Especially if you have a data-centered company like a high tech organization from Silicon Valley. Another important factor when thinking about why you would use Presto is the open source nature of the platform itself. Big data really works best when you have an open platform that a large community can get behind and support. And as well as advance the platform without worrying about licensing restrictions or the proprietary nature of a vendor software. Open source has won this battle. And Presto is obviously one of those systems that has a large community just like Hadoop and Hive and many of the other big data platforms out there. One of the features that is really interesting with Presto, is the pluggable nature of the platform. That nature is one that will allow you to extend Presto to query your data no matter where it lives. So if your data is in a relational database, such as MySQL or Oracle, you can still use Presto to query it. And in fact you can combine those data sources in a single query in Presto. So with this pluggable architecture, Presto becomes a query abstraction layer. Which let’s you access all of your data sources regardless of the underlying data platform. Now this is incredibly powerful and it will save your analyst tons of times and eliminate the need to move the data between these systems to run your queries. The last reason I’ll mention to use Presto is the scalable nature of it. Because it’s a distributed engine, it allows you to scale as large as you need to. This is a very . brief introduction aboout Presto and we will talk about it more in the next blog.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e86110253904" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simple Arithematics]]></title>
            <link>https://abhijeet5793.medium.com/simple-arithematics-1a139f3c20eb?source=rss-71cf2226f9f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a139f3c20eb</guid>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Abhijeet Kamble]]></dc:creator>
            <pubDate>Thu, 07 Nov 2019 04:01:44 GMT</pubDate>
            <atom:updated>2019-11-07T04:01:44.200Z</atom:updated>
            <content:encoded><![CDATA[<p>In this blog, we’re going to be talking about how to do simple arithmetic in Python. The benefit of the NumPy library is that it makes it really easy to do math on data that’s stored in either arrays or matrices. Now, an array is just a one-dimensional container for elements that are all of the same data type. And a matrix is just a two-dimensional container for elements that are stored in an array. The basic arithmetic operators in Python are addition, subtraction, multiplication, and division. Let me give you an example of where NumPy can come in handy. Have you ever tried to use a spreadsheet application to perform mathematical operations on a data set that has more than 300,000 rows? What happened? If the application didn’t crash, then it took a lot of time and effort to get the program to make the computation. With NumPy on the other hand, you can quickly and easily do mathematical and statistical operations on data sets, with even millions of records. Simply put, NumPy makes it easy to do math on large data sets. With this blog here, I just wanted to familiarize you with the operators that you will use in Python to achieve these arithmetic operations. So, we’re starting out the coding demonstration with having imported NumPy and then also imported the random number generator from NumPy. It never looks good to see more than two digits after a decimal point, so let’s ahead and limit the number of decimal places returned in this demonstration. So to do that, you just say: “np.set_printoptions,” and then you pass in the parameter: “(precision=2).” And then we just only get two places after the decimal point. Okay, so you run that. Now, the first thing we’re going to look at is how to do math with arrays. So, we of course need to create some arrays to do that math, and our first array will be array “a,” and we’ll set it equal to an array of six values, from one to six. Now to do that, we need to call the array function, which is: “np.array.” And then we just pass in the list with the values that we want the array to contain, which would be: “1,2,3,4,5,6.” And we can just print that out so we can see it. All right, great. And then, let’s create the second object we will create will be a matrix. And we’ll call it “b.” And then we just call the “np.array” function. And this time, we’re going to pass in two lists. So, the first one is going to contain the numbers “10,20,30.” And then the next one will contain the numbers “40,50,60.” Okay, and then we print that out. Very good. Okay. Now before going into actually doing the math, let’s just create an array via assignment. I wanted to show you how to do that. So, let’s this time use the random number generator in NumPy, and what we’re going to do is we’re going to create six random values. So, to set the seed for our random number generator, we say: “np.random.seed.” And this is so you get the same numbers on your machine as that are showing randomly on mine. And then we just assign a value to the variable “c.” This is going to create an array of six values. And, let’s put a multiplier on here, we’ll say 36 times the random numbers that are generated by the random number generator in NumPy. So we’ll say: “np.random.randn,” this is our function for generating random numbers, and then we’ll pass in the number “6” here. And “6” just says, “Okay we want six values.” And then we print this out. Okay, cool. So one thing I would like to point out here is that when we use the randn function, which is the random number generator in Python, what that actually is doing is it’s generating both positive and negative random numbers. And let’s create a fourth array, which is going to be called “d.” So what we’re going to do here is we are going to generate a series of sequential numbers between 1 and 34. And so, since it’s a sequential series of numbers we can use the arrange function which would be “np.arrange.” And then we just pass in the start and end number we want in our series, and then we can print this out. All right, so, now that you kind of understand how to create arrays and matrices, let’s just start performing some math. First things first, let’s just multiply the array “a” by the number 10. So, let’s just look back at “a” here. So it’s just 1–6, and we want to multiply each element of this array by the number 10. To do that, it’s very easy, you would just say: “a*10,” and run it. And then as you can see, each element here, each number in the array, has been multiplied by the number 10, so this is very straight-forward. Now, let’s try an addition operation. So we’ll say: “c+a.” And, okay so what we have here, we have our “c” array and then we have our “a” array, so we would expect that we add these two it’s going to be 9.22, right? And then you can see the first element here is “9.22.” So it’s very simple to do straight out arithmetic with arrays and matrices in Python. Now let’s just do “c-a.” It’s the same exact functions so in this case we’re going to have one less than 8.22. So, okay, “7.22,” all of this makes perfect sense. Let’s try “c*a.” And if we go back up and just make sure it makes sense. “a” is a “1,” “c” here is “8.22,” so 8.22*1 is 8.22, so yeah. Okay, cool. So it’s really, really simple as you can see. And if you wanted to do division here, you would just go “c/a.” Okay, cool. And 8.22/1 is 8.22, this also makes sense.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a139f3c20eb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[APIs]]></title>
            <link>https://abhijeet5793.medium.com/apis-795f9b661dc5?source=rss-71cf2226f9f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/795f9b661dc5</guid>
            <category><![CDATA[api]]></category>
            <dc:creator><![CDATA[Abhijeet Kamble]]></dc:creator>
            <pubDate>Thu, 31 Oct 2019 03:00:07 GMT</pubDate>
            <atom:updated>2019-10-31T03:00:07.540Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/318/1*UQkPxiAHjdZlPJv2_yxI5Q.png" /></figure><p>Luckily through the internet of things we are having an abundance of this data and this data can be accessed through the use of APIs. Let’s say you want to add some finance data to your website. There’s an API for that. You want to add funny GIFs, there’s an API for that. Or if you want to add data about the best reviewed pubs in your neighborhood, there’s an API for that. APIs are a fundamental piece of how most web applications work today. But what exactly is an API and how do you use one? In this blog, I’ll explain the fundamentals of APIs and how they work. I’ll break down APIs in simple terms and I’ll demonstrate how you can get started working with APIs to incorporate data into a website quickly and easily.</p><p>API stands for Application Programming Interface. While it may not be completely clear what that means at first, it’s actually a fairly straight forward concept. One way that may help to understand APIs is to think of them like a restaurant. Let’s say you’re really in the mood for a pastry. You could make it from scratch, but you’d have to go to the store, buy all the ingredients and bake it yourself. It can be expensive. It could take a lot of time. And you may not have the expertise to bake it properly. Or you could go to a restaurant. At a good restaurant the food is excellent and all the work has already been done for you. You order from a menu that lists all the available dishes and organizes them into different categories. Like appetizers, salads, entrees, and desserts. The menu typically lists information about the dishes, including ingredients and price. Once you decide you simply tell the server what you’d like. I’ll get the muffin, please. — All right. — They take your order to the kitchen and the food is brought to you ready to eat. — Here you go. — You just had to know how to order it and now, you get to enjoy delicious food without having to do any of the work. While this isn’t a perfect analogy, it is a helpful way to think about how APIs work. With an API you also want to get something. Although it’s not food, it’s data. There are all kinds of companies out there that provide data through the use of their APIs. Data on sports, weather, finance, outer space, gifs, you name it. Like a menu in a restaurant, companies provide documentation that explains what data’s available and how to ask for and get that data. Then like a server at a restaurant, you request that data from the API, it goes and gets it and brings it back for you to use as you like.</p><p>Documentation on APIs, The order is important too, like in a restaurant you order soup first then the appetizers then the main course and desert. Similarly, for an API you need to Make a request to the API, then filter Responses to the API, access data in the response , use the response data in HTML(Hyper text markup language) and finally loop through the response data. The scope of this blog is just to familiarize with APIs, I will have another blog that goes through each and every step, one at a time.</p><p>Finally, one should look out for error handling when dealing with an API. If there is an error with your request the <strong>xmlhttp</strong> <strong>request object</strong> has an event handler for handling these errors. We’ll use the on error event handler to deal with any errors that might come up. So in our code, so we’ll type <strong>request.onerror</strong> and we’ll create a function to handle any errors. Inside this function we’ll just type console.log and into our console we’ll just print out a very simple error message.</p><p>Certainly you could be much more involved with the way you handle errors and you probably should, but as I mentioned we’re not really concerned with errors at this point. We just wanted to get started working with APIs. But I wanted to briefly introduce you to this because you will likely deal with it in the future and there’s a very convenient event handler to deal with this.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=795f9b661dc5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Natural language processing]]></title>
            <link>https://abhijeet5793.medium.com/natural-language-processing-1b71e15101b8?source=rss-71cf2226f9f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b71e15101b8</guid>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Abhijeet Kamble]]></dc:creator>
            <pubDate>Thu, 24 Oct 2019 05:04:03 GMT</pubDate>
            <atom:updated>2019-10-24T05:04:03.682Z</atom:updated>
            <content:encoded><![CDATA[<p>Natural language processing</p><p>Being familiarized with the basics of what NLP and NLTK, we’re going to continue the process of assembling some tools and knowledge that we’ll put to use as we move forward in this blog. In this blog, we’re going to learn how to read in semi-structured data. This probably isn’t something you really think about if you’re used to reading in well-formatted CSV or PRK files in the past, which would be represented by this nice column or row-based depiction on the left, but reading in text files can be a little bit of a battle sometimes, represented by this figure on the right. Text data will typically be in a semi-structured or unstructured format. So, I’ve mentioned unstructured a couple times. What does that really mean? Well, it could mean that it’s binary data, it could mean no delimiters, or it could mean no indications of any rows. A few examples might be an email, PDF file, social media post, these may just get dumped into a file with no indication of where, maybe, a subject of an email ends and the body of the email begins, or even where one email ends and the next begins. It could also get cluttered by things like HTML tags, and we won’t really get into it in this blog, but the idea is just that it can get really messy. It’s important to note that Python is pretty smart, but ultimately, unless it’s told otherwise, it basically sees everything as a string of characters. It needs to be told what those characters mean. So, to illustrate how to read in semi-structured data, we’re going to be using a dataset from the UCI Machine Learning Repository. This dataset is a collection of text messages, each with a label of either spam or ham. We’ll be using the same dataset for the duration of the blog, and it’s all contained in your exercise files, so there’s no need to download it. Now, before we dive into this, I want to show you what the raw text file looks like, so we know what we’re working with. It’s not a clean CSV file, but it’s not terribly unstructured, either. Each row has a distinct text message and a distinct label as either spam or ham. So, in the context of text datasets, this is actually pretty well structured, so this shouldn’t be too difficult. Now, let’s jump back to our Notebook. So, we’re going to start by opening this file and just reading it, which basically just means that you’re pulling in the text without any semblance of structure at all. You may do this if you don’t know what your data looks like, and you’re just reading it in, this gives you a way to get a first look at the data, and then you can determine your next step. So, we’re opening it, and then we’re reading it. I should note that this lesson is as much about text manipulation as it is reading in a file. The way we’ll read this in certainly isn’t the easiest, but many times, you don’t know what the easiest way is up front, and you have to explore the data first. Additionally, I don’t want to limit you from taking on more complicated datasets moving forward. So, we’re going to take a roundabout, inefficient way to reading this, with the purpose of arming you with some tools that you’ll need to take more difficult datasets later on. So, we’ve read in our dataset using this open .read. Now, let’s print it out and take a look at what’s contained in here. So, because we’ve read it in without any indication of format, it doesn’t know if there are rows, columns, words or anything. All Python sees is a very long string. So, to tell it how much to print out, we basically just tell it how many characters. So, we’re telling it, “Print out the first 500 characters.” So now, you could see that it’s just basically a block of text, and you’ll see that you have these \t and these \n separators. The \t’s are between the labels and the text message bodies, and the \n’s are typically at the end of those lines. There are many ways to tackle this. The way that I’m going to do it is replace the \n’s with \t’s, and that’ll allow us to split this into a list, and then we can split it up from there. So, what we’re going to do is, we’re going to assign this to a list called parsedData, and we’re going to pass in, say, we want to use this rawData string, and what we’re going to do is we’re going to use this replace function. What that’s expecting to see, as the first parameter, is the thing that it’s looking for. In this case, that’s \t. And then, the second parameter is what it’s going to replace that with. In our case, we’re going to do \n. So, every time it sees \t, it’ll replace it with \n, and what this allows us to do is then do .split, and what this will do is it’ll take our string and split it on a certain character and return a list. So, we’ll tell it, split on \n, so now every time it sees an N, it’ll chop it off and add that component to a list. So, if we run this, and then we’ll print out parsedData. Instead of printing out the first 500, now it’s an actual list, so we’ll just print out the first five. Now, you can see that this is basically alternating where your labels exist and your zeroth position, the second position, the fourth position, so it’s every other, and the same thing with the actual labels. So, this gives us some sort of structure. So, what we’re going to do is we’re going to create a new list called labelList, and what we’re going to do is we’re going to pull from that parsedData list, every other item. So, what we tell it is, parsedData, and we want to start in position zero, so that’ll be the very first one, and then we tell it, we want to go to the end, so we don’t enter anything in the second place. And then, we tell it, two, so what that’s telling it is, start in the zeroth position, go to the very end and take every other, so it’ll grab ham, spam, ham and so on. Now, we want to do the same thing for text lists, so that we can get the text bodies in a different list. So, again, we’re taking parsedData, but this time, it’s going to be starting in the position one, which is the second element in the list. So, we’ll start with this one, and it’ll grab every other one. So, we’ll run that, and then, let’s take a look at the first five entries of labelList, and then we’ll take a look at the first five entries of textList, just to make sure that it did what we expected it to do. And we’ll go ahead and print both of these, because if you don’t include the print statement, Jupyter Notebooks will only print the most recent statement, so it would only print textList, and it wouldn’t print labelList, so we’ll add the print statement on both, print those out, and we can see, the lists are separated in the way that we expected them to be. So, now we have the information that we need separated into two lists, so now we can start thinking about combining these in a way that we can actually use for analysis. The first thing we have to do is import Pandas, so just do import pandas, and we’re going to save that as PD, so we don’t have to call pandas every time that we need to use one of its functions. And we’re going to create a data frame that we’ll call fullCorpus, and the way that we create this data frame is pd., call the data frame, method, and within this data frame, parenthesis, we’re going to need to pass it a dictionary, where, in the dictionary, the keys are going to be the name of the columns, and then the values will be the lists that we’ve stored the actual values in. For the first column, we’ll call it Label, and we’ll pass in labelList, because that contains the actual values, and then, in the second column, we’ll call bodyList, and we’ll pass in textList. So, what this should create is, it’ll take labelList, textList, create a data frame out of them, and then label the column names label and bodyList. So, let’s go ahead and print this out. fullCorpus.head, that’ll print out the first five rows. And, we have an error. So, it throws an error, saying the arrays must have the same length. So, let’s dig into that. So, we’re going to check the length of each of these lists to see where the issue lies. So, we’ll do print, LEN, labelList, to print the length of that list, and then we’ll do the same thing for the textList. Print those, and you see that labelList has one extra entry that textList does not have. My guess is, it picked up on something at the very end that is creating the mismatched length. So, let’s print out the last five items of labelList. So, the way we print out the last five items is we do negative five til the end, so what that’ll say is, go to the very end, count five backwards, and print out those last five. And you’ll see that the very last entry here is empty. So, it is picking up on one extra entry that we don’t need it to. So, if we just drop that, then it’ll have the same length as textList, and they’ll match up. We can just copy this code from here, because we’re going to be using the same exact thing to create our dataframe. The only change we’re going to make is we’re going to tell labelList, instead of grabbing all of them, just don’t grab the last one, so this says, start at the beginning, and capture all of them except for the very last one. So, now when we do fullCorpus and print out the first five, it should print out the way that we expect. And so, now, you have a nice, clean version, compared to what we had over here, where it was rather messy, and you could see some structure, but now we have it all cleaned up, where it’s all in the data frame and you can start to see how you could actually do some analysis with this. Now, I’d like to revisit the fact that I mentioned we would take a bit of a roundabout way, for the purpose of arming you with the tools to tackle more difficult datasets. To give you an idea of where we could have taken a shortcut to reading this in, as soon as we noticed the \t up here, that tells us immediately that this is tab delimited. Pandas allows you to read in tab-separated files very easily. You’ll learn that different details like this \t are giveaways to allow you to take shortcuts, essentially, to reading in your data. The easiest way to actually read this in will be to use the Read CSV function from Pandas. So, we’ll assign this to dataset, and we’ll call pandas.read_csv, and we just pass in our spam collection dataset, and the thing that tells it that it’s spam separated is there’s a separator parameter here, and we tell it \t, so that tells it that it is tab separated. And the only other thing that we need to add in here is, header = None. The reason we need to do that is because the raw dataset does not have column names, and if we don’t say header = None, it’ll take that first column and assume those are column names. So, let’s go ahead and print this out. And, as you can see, the column order is switched, but it’s the same content, and there’s just no column labels here. So, in this lesson, we took a dataset where we didn’t have any idea what the structure was, and we explored it to uncover the structure of the data set and organized it in a clean way, so now it’s prepared for the next steps in our NLP pipeline.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b71e15101b8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NPL and NLTK]]></title>
            <link>https://abhijeet5793.medium.com/npl-and-nltk-85430f0f4ce0?source=rss-71cf2226f9f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/85430f0f4ce0</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Abhijeet Kamble]]></dc:creator>
            <pubDate>Tue, 22 Oct 2019 06:37:26 GMT</pubDate>
            <atom:updated>2019-10-22T06:37:26.078Z</atom:updated>
            <content:encoded><![CDATA[<p>NPL and NLTK</p><h3>What are NLP and NLTK?</h3><p>Let’s jump straight into what Natural Language Processing actually means. There are various definitions out there, but one that I like is this: Natural language processing is a field concerned with the ability of a computer to understand, analyze, manipulate, and potentially generate human language. By human language, we’re simply referring to any language used for everyday communication. This can be English, Spanish, French, anything like that. Now it’s worth noting that Python doesn’t naturally know what any given word means. All it will see is a string of characters. For instance, it has no idea what natural actually means. It sees that it’s seven characters long, but the individual characters don’t mean anything to Python and certainly the collection of those characters together don’t mean anything, either. So we know that, what an N is, what an A is, and we know that together, those seven characters makes up the word natural, and we know what that means. So NLP is the field of getting the computer to understand what naturally actually signifies, and from there we can get into the manipulation or potentially even generation of that human language. You probably experience natural language processing on a daily basis. They may not really even know it. So here are a few examples that you may see on a day to day basis. The first would be a spam filter, so this is just where your email server is determining whether an incoming email is spam or not, based on the content of the body, the subject, and maybe the email domain. The second is auto-complete, where Google is basically predicting what you’re interested in searching for based on what you’ve already entered and what others commonly search for with those same phrases. So if I search for natural language processing, it knows that many other people are interested in learning NLP with Python, or learning it through a course(that’s how I learned), or looking for jobs related to natural language processing. So it can auto-complete your search for you. The last is auto-correct, where say a smart . keyboard is trying to help you correct a misspelling. I like this example because it shows how auto-correct has actually evolved over time and continues to evolve and learn. So with iOS6, if you’re trying to say, “I’ll be ill tomorrow,” It wouldn’t necessarily correct I’ll be I’ll tomorrow until iOS7, where it actually corrects, it auto-completes tomorrow and corrects I’ll into ill. So it’ll correctly send as I’ll be ill tomorrow. So that just kind of shows how NLP is has evolved in it’s early . stages and how even super powerful systems like iOS(in Steve Jobs’ time) were learning what natural language even means. Now NLP is a very broad umbrella that encompasses many topics. A few of those might be sentiment analysis, topic modeling, text classification, and sentence segmentation or part-or-speech tagging. The core component of natural language processing is extracting all the information from a block of text that is relevant to a computer understanding the language. This is task specific, as well. Different information is relevant for a sentiment analysis task than is relevant for a topic modeling task. So that’s a very quick introduction into what natural language processing is. Now let’s start thinking about the tools that we actually need to build these processes. The natural language toolkit is the most utilized package for handling natural language processing tasks in Python. Usually called NLTK for short, it is a suite of open-source tools originally created in 2001 at the University of Pennsylvania for the purpose of making building NLP processes in Python easier. This package has been expanded through the extensive contributions of open-source users in the years since its original development. NLTK is great because it basically provides a jumpstart to building any NLP process by giving you the basic tools that you can then chain together to accomplish your goal rather than having to build all those tools from scratch. A lot of tools are packaged into NLTK, and in the next blog we’ll dive into the package and explore some of those tools.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85430f0f4ce0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Time-series analysis with AUTOREGRESSIVE MOVING AVERAGES (ARMA)]]></title>
            <link>https://abhijeet5793.medium.com/time-series-analysis-with-autoregressive-moving-averages-arma-c278c8a0a473?source=rss-71cf2226f9f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/c278c8a0a473</guid>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Abhijeet Kamble]]></dc:creator>
            <pubDate>Thu, 10 Oct 2019 03:46:12 GMT</pubDate>
            <atom:updated>2019-10-10T03:46:12.059Z</atom:updated>
            <content:encoded><![CDATA[<p>Random experiments and random variables</p><p>Random experiments are opportunities to observe the outcome of a chance event. If we were rolling dice, the random experiment is observing and recording the outcome, which brings us to a random variable. A random variable is the numerical outcome of a random experiment. If we rolled a two and a three, our random variable would be five.</p><p>This would be an example of a discrete random variable since when we roll the die, the possible outcomes are one, two, three, four, five, or six. These are discreet numbers, so we cannot get an outcome of 2.4 or 5.99. On the other hand, if we are measuring the time it took runners to run 100 meters, now the outcomes are continuous.</p><p>A collection of random variables is called a stochastic process. A stochastic process can be stationary, stationarity means, its values do not change over a factor of time.</p><blockquote>We all know how a line is represented,<strong> <em>Y = mx + b</em></strong></blockquote><p>Y is equal to mx plus b, where y is the variable that we’re trying to predict, x is some set of data that we’ve got, and then a is what we call m is the slope. It’s the relationship between x and y, and b is some sort of base value, otherwise called the y-intercept. We’re trying to fit a line to our data. In general, a line gives us a starting point for predicting the Y value based on the x value.</p><blockquote><strong><em>basic linear regression model is something like Y = xB + e</em></strong></blockquote><p>As we can see both of them are very similar, here, we see the Y value is the dependent variable and the x is the dependent variable and we are trying to get the best fit or a line for the predicted value of ‘Y’ based on ‘x’. A more complex version of this is what’s known as multiple regression, and that’s going to use multiple, different sets of x’s to try to predict the value ‘y’.</p><p><strong>What is a time series?</strong></p><p>A time series is a series of data points indexed (or listed or graphed) in time order. Most commonly, a time series is a sequence taken at successive equally spaced points in time. Thus it is a sequence of discrete-time data. I feel a time series is a collection of random variables indexed in time.</p><p>We can use this sequence in order to extract meaningful statistics view patterns in data and other characteristics of the data. Time series provide the opportunity to forecast future values based on previous values, mostly used to forecast trends in economics, weather, capacity planning, Risk management and stocks for this post.</p><p><strong>Data</strong></p><p>I have used data from a kaggle data set and time series API.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*0S6OVo66PjZbjpgKa5dGsQ.png" /></figure><p>If we plotted this data, we would get something like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AaxNPvVpI7S1KTTASDSKAw.png" /></figure><p>Time series analysis usually consists of trend, seasonal and errors, I have decomposed them as follows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2er_HrT_9CffBdBsndOvZg.png" /></figure><p>We can see, there is an obvious upward trend and we can confirm that with data, Jeff Bezoz being the richest man alive with amazon growing a lot. This upward trend was gradual earlier, but it increased exponentially because of AWS(amazon web services)</p><p>We are using this model to predict the stock prices of amazon, a best way to predict stocks is through moving averages. We might want to go through and develop a moving average that sort of smooths out some of the inter-day fluctuations in the stocks, and this gives us a better idea of where the trend is over time, and potentially allows us to develop some sort of a trading strategy around that. To do this, we’ll need to put together a moving average for the stock in question.</p><p>There are different combinations of moving averages, we are trying to look for something that is going to help us to smooth out the data, but at the same time, capture what’s really going on with the stock. Reasonable minds could differ on what type of moving average that might be. In our case, let’s just say that we’re going to use a moving average over the last two years. So, what we need to do then is go out two years, and then go back and average the prices from the previous two years.</p><p>There is another way that is ingrined in a SARIMAX model, which is p,d,q values, which optimize the model in my interest because I do not have other variables that I am considering which could potentially drive my stocks. We can do a grid search for these values in the range (0,2), because they are three values. A sample example would look like this</p><pre>ARIMA(0, 0, 0)x(0, 0, 0, 12)12 - AIC:44929.81118772419<br>ARIMA(0, 0, 0)x(0, 0, 1, 12)12 - AIC:40862.1409133712<br>ARIMA(0, 0, 0)x(0, 1, 0, 12)12 - AIC:27223.68012203509<br>ARIMA(0, 0, 0)x(0, 1, 1, 12)12 - AIC:27105.889277035287<br>ARIMA(0, 0, 0)x(1, 0, 0, 12)12 - AIC:26990.948653346153<br>ARIMA(0, 0, 0)x(1, 0, 1, 12)12 - AIC:26984.93513938784<br>ARIMA(0, 0, 0)x(1, 1, 0, 12)12 - AIC:27111.03801408202<br>ARIMA(0, 0, 0)x(1, 1, 1, 12)12 - AIC:27094.050620986014<br>ARIMA(0, 0, 1)x(0, 0, 0, 12)12 - AIC:40784.52085785904<br>ARIMA(0, 0, 1)x(0, 0, 1, 12)12 - AIC:36813.25909358517<br>ARIMA(0, 0, 1)x(0, 1, 0, 12)12 - AIC:24428.461308469847<br>ARIMA(0, 0, 1)x(0, 1, 1, 12)12 - AIC:24338.5925225698<br>ARIMA(0, 0, 1)x(1, 0, 0, 12)12 - AIC:24274.613349213498<br>ARIMA(0, 0, 1)x(1, 0, 1, 12)12 - AIC:27442.79295300477<br>ARIMA(0, 0, 1)x(1, 1, 0, 12)12 - AIC:24353.530388003874<br>ARIMA(0, 0, 1)x(1, 1, 1, 12)12 - AIC:24339.28166775049<br>ARIMA(0, 1, 0)x(0, 0, 0, 12)12 - AIC:19709.113840489765<br>ARIMA(0, 1, 0)x(0, 0, 1, 12)12 - AIC:19638.246455470813<br>ARIMA(0, 1, 0)x(0, 1, 0, 12)12 - AIC:21857.121695046477<br>ARIMA(0, 1, 0)x(0, 1, 1, 12)12 - AIC:19610.07412416587<br>ARIMA(0, 1, 0)x(1, 0, 0, 12)12 - AIC:19643.43467324836<br>ARIMA(0, 1, 0)x(1, 0, 1, 12)12 - AIC:19639.44649165436<br>ARIMA(0, 1, 0)x(1, 1, 0, 12)12 - AIC:20776.627923808832<br>ARIMA(0, 1, 0)x(1, 1, 1, 12)12 - AIC:19608.62913989576<br>ARIMA(0, 1, 1)x(0, 0, 0, 12)12 - AIC:19685.555910564344<br>ARIMA(0, 1, 1)x(0, 0, 1, 12)12 - AIC:19614.408562646815<br>ARIMA(0, 1, 1)x(0, 1, 0, 12)12 - AIC:21836.602191405334<br>ARIMA(0, 1, 1)x(0, 1, 1, 12)12 - AIC:19588.066501426496<br>ARIMA(0, 1, 1)x(1, 0, 0, 12)12 - AIC:19625.123662900285<br>ARIMA(0, 1, 1)x(1, 0, 1, 12)12 - AIC:19615.633999147754<br>ARIMA(0, 1, 1)x(1, 1, 0, 12)12 - AIC:20762.893365248525<br>ARIMA(0, 1, 1)x(1, 1, 1, 12)12 - AIC:19585.89949980239<br>ARIMA(1, 0, 0)x(0, 0, 0, 12)12 - AIC:19696.823820843863<br>ARIMA(1, 0, 0)x(0, 0, 1, 12)12 - AIC:19644.424994667563<br>ARIMA(1, 0, 0)x(0, 1, 0, 12)12 - AIC:21736.430358589456<br>ARIMA(1, 0, 0)x(0, 1, 1, 12)12 - AIC:19611.450034828522<br>ARIMA(1, 0, 0)x(1, 0, 0, 12)12 - AIC:19623.141781259677<br>ARIMA(1, 0, 0)x(1, 0, 1, 12)12 - AIC:19624.919842116753<br>ARIMA(1, 0, 0)x(1, 1, 0, 12)12 - AIC:20713.039534778174<br>ARIMA(1, 0, 0)x(1, 1, 1, 12)12 - AIC:19610.432288304906<br>ARIMA(1, 0, 1)x(0, 0, 0, 12)12 - AIC:19676.612363886677<br>ARIMA(1, 0, 1)x(0, 0, 1, 12)12 - AIC:19628.59134629185<br>ARIMA(1, 0, 1)x(0, 1, 0, 12)12 - AIC:21691.024674182925<br>ARIMA(1, 0, 1)x(0, 1, 1, 12)12 - AIC:19590.695753054977<br>ARIMA(1, 0, 1)x(1, 0, 0, 12)12 - AIC:19608.471281482973<br>ARIMA(1, 0, 1)x(1, 0, 1, 12)12 - AIC:19606.13257975408<br>ARIMA(1, 0, 1)x(1, 1, 0, 12)12 - AIC:20688.153786629882<br>ARIMA(1, 0, 1)x(1, 1, 1, 12)12 - AIC:19588.87655406686<br>ARIMA(1, 1, 0)x(0, 0, 0, 12)12 - AIC:19692.55669396984<br>ARIMA(1, 1, 0)x(0, 0, 1, 12)12 - AIC:19621.375847990577<br>ARIMA(1, 1, 0)x(0, 1, 0, 12)12 - AIC:21844.543744424107<br>ARIMA(1, 1, 0)x(0, 1, 1, 12)12 - AIC:19595.44655959739<br>ARIMA(1, 1, 0)x(1, 0, 0, 12)12 - AIC:19621.01793830233<br>ARIMA(1, 1, 0)x(1, 0, 1, 12)12 - AIC:19622.594983639956<br>ARIMA(1, 1, 0)x(1, 1, 0, 12)12 - AIC:20757.94605550419<br>ARIMA(1, 1, 0)x(1, 1, 1, 12)12 - AIC:19593.48476707819<br>ARIMA(1, 1, 1)x(0, 0, 0, 12)12 - AIC:19684.948044797922<br>ARIMA(1, 1, 1)x(0, 0, 1, 12)12 - AIC:19614.03965430033<br>ARIMA(1, 1, 1)x(0, 1, 0, 12)12 - AIC:21691.94553706001<br>ARIMA(1, 1, 1)x(0, 1, 1, 12)12 - AIC:19587.178750448988<br>ARIMA(1, 1, 1)x(1, 0, 0, 12)12 - AIC:19619.337245966366<br>ARIMA(1, 1, 1)x(1, 0, 1, 12)12 - AIC:19615.306015134967<br>ARIMA(1, 1, 1)x(1, 1, 0, 12)12 - AIC:20756.528009620055<br>ARIMA(1, 1, 1)x(1, 1, 1, 12)12 - AIC:19585.252712355243</pre><p>As we can see this is returning an AIC score ie (Alkaike information criterion), this is a ranked score which is conveniently returned with ARIMA models fitted using statsmodels. The AIC measures how well a model fits the data while taking into account the overall complexity of the model. A model that fits the data very well while using lots of features will be assigned a larger AIC score than a model that uses fewer features to achieve the same goodness-of-fit. Therefore, we are interested in finding the model that yields the lowest AIC value.</p><p>Once, we fit our data in the model we have something that looks like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_jfKijALovReeGkUFwmtTg.png" /></figure><p>This results column describes how my models performed, the coeff column shows the importance of each feature ie AR1 and AR2 and the P&gt;|z| shows us how powerful of an impact is being created on my variables. In this case for constants = 0.994, which implies that my data needs to alter before I do any further calculations on it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rayv0jGvEWQlIKzeqI_JwA.png" /><figcaption>time series prediction</figcaption></figure><p>Just for fun, I also tried to compare the closing values of Amazon and apple stocks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-eaV5YL_9X5OtXYxzHRr3A.png" /><figcaption>Whereas R-squared is a relative measure of fit, <strong>RMSE</strong> is an absolute measure of fit. As the square root of a variance, <strong>RMSE</strong> can be interpreted as the standard deviation of the unexplained variance, and has the useful property of being in the same units as the response variable. Lower values of <strong>RMSE</strong> indicate better fit.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vBFrWnIdn0l5RBV9ul741A.png" /><figcaption>model diagnostics test</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c278c8a0a473" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data_Science_Road_Blocks_!_!_!]]></title>
            <link>https://abhijeet5793.medium.com/data-science-road-blocks-d0a521e306ac?source=rss-71cf2226f9f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0a521e306ac</guid>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Abhijeet Kamble]]></dc:creator>
            <pubDate>Tue, 04 Jun 2019 14:53:36 GMT</pubDate>
            <atom:updated>2021-01-19T20:41:57.880Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Data science road blocks</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MuxcSKNlkUei6Y6JLgxoZA.png" /></figure><p>Hello Internet! This is my third blog! First good one, sadly, because like many data scientists, I’m much better with numbers than words.</p><p>I am a student at the Flatiron School and I’ve been tinkering around with Big Data for the last couple of months. The results of it have been fascinating but I can’t help but feel numbingly dull with my analysis because of certain aspects in the field. I found a few ways around this and thought that I would start a blog to help fellow data scientists experiencing similar mundane issues. So, in the hopes to make all of our lives as a data scientist a little less tedious, take what you will from this blog.</p><p>I did a Data-science project last weekend where I attempted to solve a <strong>classic classification problem</strong>. However, like many data scientist’s experiences, some of the largest roadblocks I faced were Exploratory Data Analysis (EDA), Feature Engineering, and the time it took to see results for the computational complexity of the advanced machine learning models (more on that later).</p><p>I feel a lot of budding data scientists face the same issues while dealing with their projects, so, I want to address these particular issues here.</p><h3><strong>EDA (Exploratory Data Analysis)</strong></h3><p>A lot of the EDA is done manually — bleh!- such as looking for missing values, checking for correlation between two columns, whether the data is normally distributed or not, unique value counts, the averages trio mean, median, mode, the maxima-minima, kurtosis, etc. (oh, my eyes!)</p><p>Comparing all these values is time expensive and cumbersome! Bleh!</p><p>I found a library in python called pandas_profiling, here is an example of how this works with one of the most common datasets on kaggle, the titanic data-set. Just a one-liner “ pandas_profiling.ProfileReport(df) ”</p><p>image pandas</p><p>This is what the pandas_profile report does to your initial EDA, you can, of course, dive deeper with this.</p><p>In my opinion, it is better than the inbuilt .describe() function of pandas.</p><h3><strong>FEATURE ENGINEERING</strong></h3><p>In the context of data-science, a <strong>feature</strong> can be described as a <strong>characteristic</strong> or a set of characteristics, that explains the occurrence of a phenomenon. When these characteristics are converted into some numerical measurable form, they are called <strong>features</strong>.</p><p>For example, the different columns, in the picture down below, can be different features in reference to the titanic data. Eg AGE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mp_cGeJUKXBjjRpLeCETew.png" /></figure><p><strong>Feature Engineering</strong> can be simply defined as the <strong>process of creating new features from the existing features in a dataset.</strong></p><p>Example: we could combine Fare with Class and make it into a new feature. This process of creating new relevant features for a calculated estimation is called Feature Engineering.</p><p>The performance of a predictive model is heavily dependent on the quality of the features in the model; if it can convey more information about the target variable in the most efficient way, its performance will definitely rise.</p><p>This is a juggernaut of a task for a data-scientist (again bleh!), as it involves a lot of manual labour and brainstorming in the form of feature tuning, creating new features, examining their relevance and coming to a collective decision. Trust me, it’s easier said than done! And I cringed through it all.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XbtdvCi3XXqqO5p41Rbk1Q.png" /></figure><p>I spent the vast majority of my time struggling to determine which features to use and which to eliminate. The rest of my peers in the program are in the same struggle boat. One stated that “Feature Engineering is truly an art!” and I cannot help but agree with them.</p><p>In the quest to surpass these tedious tasks, I’ve come across this dope python library called <a href="https://docs.featuretools.com/">featuretools</a>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yfSYfNckNK2suG-pqDMm3w.png" /><figcaption>python library</figcaption></figure><p>Featuretools could dramatically automate this arduous task of FE. But here’s the kicker…</p><p>Despite being a valuable time saver, the tool will only get you through <strong>half</strong> of the process; i.e. it can only automate the feature engineering process to a certain degree, freeing up time so that we can focus on the important parts for model building. This freed up time is great, but not enough.</p><p>Before taking <strong>Featuretools</strong> for a spin, there are<strong> three major components of the package</strong> that we should be aware of:</p><ul><li>Entities</li><li>Deep Feature Synthesis (DFS)</li><li>Feature primitives</li></ul><p>a) An <strong>Entity</strong> can be considered as a representative of a Pandas DataFrame. A collection of multiple entities is called an <strong>Entityset</strong>.</p><p>b) <strong>Deep Feature Synthesis (DFS)</strong> has got nothing to do with deep learning. Don’t worry. DFS is actually a Feature Engineering method and is the backbone of Featuretools. It enables the creation of new features from single, as well as multiple data frames.</p><p>c) DFS create features by applying <strong>Feature primitives</strong> to the Entity-relationships in an EntitySet. These primitives are the often-used methods to generate features manually. For example, the primitive “mean” would find the mean of a variable at an aggregated level.</p><p>Installation<strong> “!pip install featuretools”</strong> in jupyter notebook</p><p>And<strong> import featuretools as ft</strong></p><p><strong>Step I Create an Entity set</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3xfnyzj9bqYEIMNUm57lVw.png" /><figcaption>Entity code snippet</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iTRZb5o-WW7RpKsOXJqe8Q.png" /><figcaption>output for Entity</figcaption></figure><p><strong>Step II Create a DFS</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IxmOXSe5QkvA9g8ajc19Ow.png" /><figcaption>code &amp; output for DFS</figcaption></figure><p>Step III Calling the DFS Matrice</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O_jY-Kay1VvicQp8RMrCcA.png" /><figcaption>Various Features</figcaption></figure><p>The featuretools package is truly a game-changer in data-science. While its applications are understandably still limited in cases of industry use, the amount of time it saves, and the usefulness of feature it generates, has won me over.</p><h3><strong>Computational Complexity</strong></h3><p>Before entering this data-driven Bootcamp, the staff suggested that I use a MacBook. At first, I was like “why do I need one? I have a top of the line windows device with killer specs.”</p><p>The first week into the boot camp and I realized how wrong I was. Not only is a MacBook so much more reliable (Rest in Power Steve Jobs), but it was computationally faster than my killer state of the art windows device as well! (Points to 🍎 once again!)</p><p>So, I considered buying one, and I was really happy with the new machine- which had 4 cores of a processor and 16 gigs of ram and what not! They are not groundbreaking specs, of course, but they are still better than most laptops out there.</p><p>However, as soon as I encountered <strong>complex machine learning algorithms</strong> like SVM, Random Forests, etc., my laptop took <strong><em>2–3 hours</em></strong> to run a <strong>grid search with these models for parameter tuning</strong>. During this time, as my model run its course, I could do nothing but sit and stare at it.</p><p>I had little to no clue that an extensive grid search heavily relies on a <strong>GPU</strong>, something my Mac barely has.</p><p>Google solved this issue for all of us by providing Google Colab. Thank you, Google! That real VIP!</p><p>Google Colab is a free cloud service and now it supports <strong><em>free</em></strong> GPU! You can:</p><ul><li>improve your <strong>Python </strong>programming language coding skills.</li><li>develop deep learning applications using popular libraries such as <strong>Keras</strong>,<br><strong>TensorFlow</strong>, <strong>PyTorch</strong>, and <strong>OpenCV</strong>.</li><li>You can create notebooks in Colab, upload notebooks, store notebooks, share notebooks.</li><li>Mount your Google Drive and use whatever you’ve got stored in there, import most of your favourite directories, upload your personal Jupyter Notebooks.</li><li>Upload notebooks directly from GitHub, upload Kaggle files, download your notebooks.</li><li>You can use it to tweak/run your code in python. It is quite user-friendly with jupyter notebooks- if you are familiar with jupyter you will adapt easily. But it’s not that bad for anyone who doesn’t know jupyter to begin with google colab, etc.</li></ul><p>The most important feature that distinguishes Colab from other free cloud services is: <strong>Colab </strong>provides GPU and is totally free.</p><p>Again. It is free, free, free, free, free, free, free, free, and even freer.</p><p>It has its limits like it is not supported for Scala and R yet, it does limit your session size, but there are legit loopholes to get around that, eg re-upload your files.</p><p>Here’s how you get started with it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pbj_0Yzp2h1e1TEbvRLnYg.png" /><figcaption>create a folder on google drive</figcaption></figure><p>Next, create a google colab notebook with google drive</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tdMRsUQUQYnuWSPVedWhfA.png" /><figcaption>create a new Colaboratory notebook</figcaption></figure><p>Finally, set the GPU</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aOqCfNCzIiTAyNhddTJhaw.png" /><figcaption>GPU setup</figcaption></figure><p>Just kidding, that more power is just an April fool’s prank by Google.</p><p>To set up the GPU which is not on by default go to Runtime &gt;&gt; change Runtime type and then select GPU from there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*21eqQFrWtvN7IPSgZJflsg.png" /></figure><p>And you are all set, use it just like your jupyter notebook and your complex ml models run in a matter of seconds, worse minutes.</p><p>With great power comes great responsibility!!</p><p>So, that’s it for now.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0a521e306ac" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>