<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[hacking-datascience - Medium]]></title>
        <description><![CDATA[My journey into data science with python. - Medium]]></description>
        <link>https://medium.com/hacking-datascience?source=rss----8f6eff91310d---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>hacking-datascience - Medium</title>
            <link>https://medium.com/hacking-datascience?source=rss----8f6eff91310d---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 21:42:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/hacking-datascience" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[SQLAlchemy — Python Tutorial]]></title>
            <link>https://medium.com/hacking-datascience/sqlalchemy-python-tutorial-abcc2ec77b57?source=rss----8f6eff91310d---4</link>
            <guid isPermaLink="false">https://medium.com/p/abcc2ec77b57</guid>
            <category><![CDATA[sqlalchemy]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[Vinay Kudari]]></dc:creator>
            <pubDate>Thu, 23 Aug 2018 02:13:26 GMT</pubDate>
            <atom:updated>2018-08-23T02:59:32.659Z</atom:updated>
            <content:encoded><![CDATA[<p>We often encounter data as Relational Databases. To work with them we generally would need to write raw SQL queries, pass them to the database engine and parse the returned results as a normal array of records.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/1*D1TpfUof-8amlhN1E2A6JA.png" /></figure><p>SQLAlchemy provides a nice “Pythonic” way of interacting with databases. So rather than dealing with the differences between specific dialects of traditional SQL such as MySQL or PostgreSQL or Oracle, you can leverage the Pythonic framework of SQLAlchemy to streamline your workflow and more efficiently query your data.</p><h4>Installing The Package</h4><pre>pip install sqlalchemy</pre><h4><strong>Connecting to a database</strong></h4><p>To start interacting with the database we first we need to establish a connection.</p><pre>import sqlalchemy as db<br>engine = db.create_engine(&#39;dialect+driver://user:pass<strong>@host</strong>:port/db&#39;)</pre><p>Some examples of connecting to various databases can be found <a href="http://docs.sqlalchemy.org/en/latest/core/engines.html#postgresql"><strong>here</strong></a></p><h4>Viewing Table Details</h4><p>SQLAlchemy can be used to automatically load tables from a database using something called reflection. Reflection is the process of reading the database and building the metadata based on that information.</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d1596c2dcabfd778f53716b57049399d/href">https://medium.com/media/d1596c2dcabfd778f53716b57049399d/href</a></iframe><h4>Querying</h4><p>Table and MetaData have already been imported. The metadata is available as metadata.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a6aeec7df852e29063adf0d33f23d3a9/href">https://medium.com/media/a6aeec7df852e29063adf0d33f23d3a9/href</a></iframe><p>ResultProxy:<em> The object returned by the </em><em>.execute() method. It can be used in a variety of ways to get the data returned by the query.</em></p><p>ResultSet: <em>The actual data asked for in the query when using a fetch method such as </em><em>.fetchall() on a ResultProxy.</em></p><p><strong>Dealing with Large ResultSet</strong></p><p>We use .fetchmany() to load optimal no of rows and overcome memory issues in case of large datasets</p><pre>while flag:<br>    partial_results = ResultProxy.fetchmany(50)<br>    if(partial_results == []): <br>	flag = False<br>    //<br>	code<br>   //<br>ResultProxy.close()</pre><p><strong>Convert to dataframe</strong></p><pre>df = pd.DataFrame(ResultSet)<br>df.columns = ResultSet[0].keys()</pre><p><strong>Filtering data</strong></p><p>Lets see some examples of raw SQLite Queries and queries using SQLAlchemy.</p><p><strong>where</strong></p><pre><strong>SQL :</strong><br>SELECT * FROM census <br>WHERE sex = F</pre><pre><strong>SQLAlchemy :<br></strong>db.select([census]).<em>where</em>(census.columns.sex == &#39;F&#39;)</pre><p><strong>in</strong></p><pre><strong>SQL :</strong><br>SELECT state, sex<br>FROM census<br>WHERE state IN (Texas, New York)</pre><pre><strong>SQLAlchemy :</strong><br>db.select([census.columns.state, census.columns.sex]).where(census.columns.state.<em>in_</em>([&#39;Texas&#39;, &#39;New York&#39;]))</pre><p><strong>and, or, not</strong></p><pre><strong>SQL :</strong><br>SELECT * FROM census<br>WHERE state = &#39;California&#39; AND NOT sex = &#39;M&#39;</pre><pre><strong>SQLAlchemy :</strong><br>db.select([census]).where(db.<em>and_</em>(census.columns.state == &#39;California&#39;, census.columns.sex != &#39;M&#39;))</pre><p><strong>order by</strong></p><pre><strong>SQL :</strong><br>SELECT * FROM census<br>ORDER BY State DESC, pop2000</pre><pre><strong>SQLAlchemy :</strong><br>db.select([census]).<em>order_by</em>(db.desc(census.columns.state), census.columns.pop2000)</pre><p><strong>functions</strong></p><pre><strong>SQL :</strong><br>SELECT SUM(pop2008)<br>FROM census</pre><pre><strong>SQLAlchemy :</strong><br>db.select([db.<em>func.sum</em>(census.columns.pop2008)])</pre><p>other functions include <em>avg, count, min, max</em>…</p><p><strong>group by</strong></p><pre><strong>SQL :</strong><br>SELECT SUM(pop2008) as pop2008, sex<br>FROM census</pre><pre><strong>SQLAlchemy :</strong><br>db.select([db.func.sum(census.columns.pop2008).label(&#39;pop2008&#39;), census.columns.sex]).<em>group_by</em>(census.columns.sex)</pre><p><strong>distinct</strong></p><pre><strong>SQL :</strong><br>SELECT DISTINCT state<br>FROM census</pre><pre><strong>SQLAlchemy :</strong><br>db.select([census.columns.state.<em>distinct</em>()])</pre><p><strong>case &amp; cast</strong></p><p>The case() expression accepts a list of conditions to match and the column to return if the condition matches, followed by an else_ if none of the conditions match.</p><p>cast() function to convert an expression to a particular type</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1f38ad030d8ac051b01d25bc7748ec61/href">https://medium.com/media/1f38ad030d8ac051b01d25bc7748ec61/href</a></iframe><p>We use .scalar to the result when the result contains only single value</p><p><strong>joins</strong></p><p>If you have two tables that already have an established relationship, you can automatically use that relationship by just adding the columns we want from each table to the select statement.</p><pre>select([census.columns.pop2008, state_fact.columns.abbreviation])</pre><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9544fd229798f505a9a389d0f3c5b2ff/href">https://medium.com/media/9544fd229798f505a9a389d0f3c5b2ff/href</a></iframe><h4>Creating and Inserting Data into Tables</h4><p>By passing the database which is not present, to the engine then sqlalchemy automatically creates a new database.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4fba55fcaa5cb24716ce7adfb8bd44ff/href">https://medium.com/media/4fba55fcaa5cb24716ce7adfb8bd44ff/href</a></iframe><h4>Updating data in Databases</h4><pre>db.update(table_name).values(attribute = new_value).where(condition)</pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5475d8dded9400c47473ead446276a9c/href">https://medium.com/media/5475d8dded9400c47473ead446276a9c/href</a></iframe><h4>Delete Table</h4><pre>db.delete(table_name).where(condition)</pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0d73cbe89f9ebaf7b686318db8a49b8d/href">https://medium.com/media/0d73cbe89f9ebaf7b686318db8a49b8d/href</a></iframe><h4>Dropping a Table</h4><pre>table_name.drop(engine) #drops a single table</pre><pre>metadata.drop_all(engine) #drops all the tables in the database</pre><p>IPython Notebook’s and other assets of this story can be found <a href="https://github.com/vinaykudari/hacking-datascience/tree/master/notebooks/sqlalchemy"><strong>here</strong></a></p><p>See ya again 😃</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=abcc2ec77b57" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hacking-datascience/sqlalchemy-python-tutorial-abcc2ec77b57">SQLAlchemy — Python Tutorial</a> was originally published in <a href="https://medium.com/hacking-datascience">hacking-datascience</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Probability and Statistics]]></title>
            <link>https://medium.com/hacking-datascience/introduction-to-probability-and-statistics-9c8ed6f6a6bc?source=rss----8f6eff91310d---4</link>
            <guid isPermaLink="false">https://medium.com/p/9c8ed6f6a6bc</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Vinay Kudari]]></dc:creator>
            <pubDate>Mon, 06 Aug 2018 07:56:15 GMT</pubDate>
            <atom:updated>2018-08-06T09:39:58.765Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*Uu7Bc_2baQ6WojJkRreukw.png" /></figure><p>Why Probability and Statistics?</p><p>We live in a very uncertain world, we are unsure about our next minute. In this kind of world probability and statistics provide a rational way to deal with uncertainty.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*Rsws0afjeqJ5e7fzmMJaTg.png" /><figcaption>The fastest route may not be the shortest.</figcaption></figure><p>It is certain to find the fastest route between point a and point b, it can be found my measuring the distance between them, while it is uncertain to find the fastest route as it depends on various factors such as traffic at that moment, time of the day and also the day.</p><p><strong>Probability Theory</strong></p><p>We all know that the probability of getting head and tail are equal to 1/2, So what does this actually mean? If suppose we toss a coin 1000 times we I’ll have the heads approximately <strong>about</strong> 500 times. Let me simulate this using python <a href="https://github.com/vinaykudari/hacking-datascience/blob/master/notebooks/statistics-for-ds/Intro%20to%20Prob%20and%20Stats.ipynb"><strong>code</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/1*h7AnOu1CNhVhEQcRO1U6_w.png" /><figcaption>histogram of generate_counts()</figcaption></figure><p>Head is counted as +1 and tail as -1 so if three times head and 2 times tail showed up then the sum is 1, if the no of tail and heads are same, the sum is o, from the histogram we can say that sum is ~0 for maximum no of times.</p><pre>                  sum = x1 + x2 + x3 + ...... xk</pre><p>After doing some stimulations of summing k numbers, <strong>x</strong>i = 1 with probability 1/2 and <strong>x</strong>i = -1 with probability 1/2 we see that the sum is almost always in the range <strong>[-4√k, +4√k] </strong>as k tends to a large number sum/k tends to 0.</p><p><strong>What is Statistics?</strong></p><p>Statistics is about analyzing the real-world data and drawing conclusions. For example if you would like to predict which party is going to win elections one way is to ask all potential voters about their opinion, but it would be extremely expensive instead we could select random unbiased group and take their opinions, now this exactly equivalent to coin problem.</p><p><strong>Frequentist Statistics</strong> draw conclusions from repeating the event multiple number of times under same conditions and comparing the frequency of each outcome, just like flipping a coin. The result entirely depends on the number of times the event is repeated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/307/1*ViNoy99m6Y0q3xtYbXho9w.png" /></figure><p><strong>Bayesian Statistics</strong> comes into the play where we cannot perform multiple experiments under same set of conditions, like in diagnosing the patient. He might have unique set of medical conditions which the doctor might not have encountered in the past.</p><p>Homework Notebook can be found <a href="https://github.com/vinaykudari/hacking-datascience/blob/master/notebooks/statistics-for-ds/HW/HW_Topic1.ipynb"><strong>here</strong></a>. 😄</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c8ed6f6a6bc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hacking-datascience/introduction-to-probability-and-statistics-9c8ed6f6a6bc">Introduction to Probability and Statistics</a> was originally published in <a href="https://medium.com/hacking-datascience">hacking-datascience</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[hacking-pandas]]></title>
            <link>https://medium.com/hacking-datascience/hacking-pandas-4af7f10e3a31?source=rss----8f6eff91310d---4</link>
            <guid isPermaLink="false">https://medium.com/p/4af7f10e3a31</guid>
            <category><![CDATA[group-by]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[melt]]></category>
            <category><![CDATA[pivot]]></category>
            <category><![CDATA[timeseries]]></category>
            <dc:creator><![CDATA[Vinay Kudari]]></dc:creator>
            <pubDate>Fri, 27 Jul 2018 22:06:18 GMT</pubDate>
            <atom:updated>2018-08-07T21:12:33.332Z</atom:updated>
            <content:encoded><![CDATA[<p>Pandas is a powerful python package which makes importing, cleaning, analyzing, and exporting the data easier. In this story I’d like to write down some of the key functions which I found them useful.</p><p>Will keep on appending new stuff to the story.</p><p><strong>#Update 08/08/18 : Categorical Data, Grouping.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*6WvQ6gUxIRNMv65AobkB-Q.png" /></figure><h4>Importing the data</h4><p>The first part of any data science problem is to import data, now lets see how few hacks can make importing much easier and cleaner.</p><p><strong>syntax</strong></p><pre>pd.read_csv(file_name,argument1, argument2,...)</pre><p><strong>key arguments</strong></p><p>delimiter = character : <strong>delimiter</strong> is a sequence of one or more characters used to specify the boundary between separate, independent regions. The <strong>character</strong> can be &#39; &#39; or &#39;,&#39; or a tab delimiter &#39;\t&#39; .</p><p>header = n : header implies the start of the data, If header argument is not passed it is assumed to take the 0th row, therefore column names are inferred from the first line of the file.</p><p>index_col = n : columns to use as row labels, If a sequence is given, a <em>MultiIndex</em> will be used.</p><p>use_cols = [ ]: list can have column numbers or name, only those mentioned in the list will be selected.</p><p>nrows = n : no of rows that are to be read.</p><p>comment = character : character indicates the beginning of the line which should not be parsed, it can be &#39;#&#39; or &#39;\\&#39;</p><p>na_values = list : additional string list to recognise as NaN value.</p><p>parse_dates = True : parses dates into <strong>yyyy-mm-dd hh:mm:ss</strong> format.</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/623c1f4d102358cd528fa83f53b36dfe/href">https://medium.com/media/623c1f4d102358cd528fa83f53b36dfe/href</a></iframe><h4>Cleaning and analyzing the data</h4><p>Data almost never comes in the form which can be used for analyzing or training, The most common problems in raw data include missing values, inconsistent data, duplicate records.</p><p>Tabular data can be organized in many ways some are suitable for reporting and some are suitable for analysis, The process of turning an untidy data-set into a form which is suitable for data analysis is know as <strong>data</strong> <strong>tidying</strong>.</p><p><strong>The principles of tidy data are</strong></p><ol><li>columns represent unique variables.</li><li>rows represent individual observations.</li></ol><p>Lets explore some functions which make data cleaning easier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/1*x66pKHSARDDODBMGnIrNOw.png" /></figure><p><strong>Melting</strong></p><p>Melting is a way of tidying the data, the columns are turned into rows making data good for analysis.</p><p><strong>syntax</strong></p><pre>pd.melt(df, argument1, argument2,....)</pre><p><strong>key arguments</strong></p><p>id_vars = [ ] : represents columns you do not want to melt</p><p>value_vars = [ ] : represents columns you do wish to melt into rows if this argument is not provided all the columns not set in id_vars will be melted.</p><p><strong>Pivoting</strong></p><p>Exactly opposite to melting, turning every unique value into columns is called pivoting. Making the data report friendly.</p><p><strong>syntax</strong></p><pre>df.pivot(argument1, argument2, ...)</pre><p><strong>key arguments</strong></p><p>index = [ ] : represents columns you do not want to pivot</p><p>columns = [ ] : represents columns you want to pivot</p><p>values = [ ] : values to be used when the column is pivoted</p><p>aggfunc = np.ufunc : in case of duplicate rows the values are computed as per the ufunc mean, max, min..</p><p><strong>why?</strong></p><p>makes data cleaning easier for better data analysis.</p><h4>Date Time Series</h4><p>It is one of most common series data, I have found the pandas package to be extremely useful for time series manipulation and analysis.</p><p><strong>functions</strong></p><h4>pd.to_datetime(series, format=time_format)</h4><p>Passing list of strings and specifying the format in which the strings are present will parse the strings into datetime objects.</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/37f3d1e175333ee7af8a087df9087ee0/href">https://medium.com/media/37f3d1e175333ee7af8a087df9087ee0/href</a></iframe><p><strong>why?</strong></p><p>datetime object is powerful and we can get the individual elements of the date <strong>Eg. datetime_object.dt.hour( )</strong></p><h4><strong>df.reindex(list, method=func)</strong></h4><p>reindexing a dataframe df according to the indices given in the list, if the indices doesn&#39;t match then the values are filled according to optional logic func, it can be ffill or bfill they represent forward fill and backward fill.</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ace6b2dc68bb3bd7ac75af785425c330/href">https://medium.com/media/ace6b2dc68bb3bd7ac75af785425c330/href</a></iframe><p><strong>why?</strong></p><p>when we need to compare two dataframes it would be better if we have the same indexing order.</p><h4><strong>pd.resample(rule).func( )</strong></h4><p>Resampling involves changing the frequency of your observations, the records are up-sampled or down-sampled according to the <strong><em>rule</em></strong>, Most common rules in the table below. This can be chained with any statistical functions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*MD1HrXj73EWIMYm9vXeZyQ.png" /></figure><p>downsampling is reducing the frequency of records example turning daily data to weekly data in contrast up sampling is increasing the frequency, while we compute the later we often come up with many missing values we can overcome this by interpolating the values.</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7126616c34daf9319ff16c9abb614fca/href">https://medium.com/media/7126616c34daf9319ff16c9abb614fca/href</a></iframe><p><strong>why?</strong></p><p>to provide additional structure or insight into the learning problem for supervised learning models</p><p><strong>#Update 08–08–18</strong></p><h4>Categorical Data</h4><p>Often in real world data each column consists of repetitive values, like gender, country. We can convert the series data type to category type using</p><p><strong>syntax</strong></p><p>df[column_name].astype(&#39;category&#39;)</p><p><strong>why?</strong></p><p>Save memory especially in case of large datasets also speed up operations like groupby.</p><h4>Grouping</h4><p>It involves <strong>splitting</strong> the dataframe by the column name into groups and <strong>applying</strong> a function on each group in the column and <strong>merging</strong> all the results.</p><p><strong>syntax</strong></p><pre>df.groupby(column_name) <strong><em>or</em></strong> df.groupby(condition)</pre><p>groupby return an object on which we apply a function and finally combine the results</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d751e14e8854ec8c6775e0709767a743/href">https://medium.com/media/d751e14e8854ec8c6775e0709767a743/href</a></iframe><h4><strong>functions</strong></h4><p><strong>aggregation</strong> function returns a single aggregated value for each group</p><p><strong>syntax</strong></p><pre>df.groupby(column_name).agg([func1, func2, ...])</pre><p>If we want to apply different functions to different columns we could pass a dictionary to agg( ) instead of a list.</p><p>df.groupby(column_name).agg({column1:func1, column2:func2})</p><p><strong>transformation </strong>function applies the function to each group and returns the same size data frame where each group has same aggregated value.</p><p><strong>syntax</strong></p><pre>df.groupby(column_name).transform(function)</pre><p><strong>filter</strong> function filters the dataframe on some Boolean condition</p><p><strong>syntax</strong></p><pre>df.groupby(column_name).filter(function)</pre><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4cbe5480486a78d5b73382b4d62fd754/href">https://medium.com/media/4cbe5480486a78d5b73382b4d62fd754/href</a></iframe><p><strong>visualization</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*fcGsJmbYsUpyFQHL98LVvg.png" /><figcaption>Thanks to pbpython :)</figcaption></figure><p><strong>why?</strong></p><p>helps to properly analyse the data also fill missing values appropriately using transform.</p><p>See ya soon 😄</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4af7f10e3a31" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hacking-datascience/hacking-pandas-4af7f10e3a31">hacking-pandas</a> was originally published in <a href="https://medium.com/hacking-datascience">hacking-datascience</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[hacking-numpy]]></title>
            <link>https://medium.com/hacking-datascience/hacking-numpy-29ed0a6ec8ef?source=rss----8f6eff91310d---4</link>
            <guid isPermaLink="false">https://medium.com/p/29ed0a6ec8ef</guid>
            <category><![CDATA[numpy]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Vinay Kudari]]></dc:creator>
            <pubDate>Tue, 24 Jul 2018 11:44:51 GMT</pubDate>
            <atom:updated>2018-08-23T02:17:54.031Z</atom:updated>
            <content:encoded><![CDATA[<p>The core python packages like pandas, scikit-learn are built on top of numpy, So in this story I’m going to list some key functions of numpy package which made my computations much easier/efficient.</p><p>Will keep on appending new stuff to the story.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*sESyI6dq-_QYutdNlg9zpg.png" /></figure><p>Get the ipython notebook of this story from <a href="https://github.com/vinaykudari/hacking-datascience/blob/master/notebooks/hacking-numpy.ipynb"><strong>here</strong></a>.</p><pre>import numpy as np #importing the package</pre><h4><strong>np.random.seed( )</strong></h4><p>generating random number’s is one of the major task of any data science problem, numpy does a very good job in this regard. There will be times when we want numpy to generate same set of random values on every invocation, In this scenario we could set a seed <strong><em>instead of copying</em></strong> the random array.</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b89de97ef0b01da362f6306d247e79e5/href">https://medium.com/media/b89de97ef0b01da362f6306d247e79e5/href</a></iframe><p><strong>why?</strong></p><p>when we want to check the performance of our model we often need to test on same data to get the accurate results, using seed will help save memory.</p><h4>np.reshape(array, shape) / array.reshape(shape)</h4><p>as the name says it resizes the given ndarray where <em>shape = (rows, columns).</em></p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/362ebfa4dd6205c86e3ba115b3bcc3ae/href">https://medium.com/media/362ebfa4dd6205c86e3ba115b3bcc3ae/href</a></iframe><p><strong>why?</strong></p><p>reshape is especially useful when we want to reshape matrices.</p><h4>np.nditer( )</h4><p>it is an efficient multi dimension iterator object, it does iterate over every element in a ndarray.</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7b769e1bc63585871e34ef9f81605e61/href">https://medium.com/media/7b769e1bc63585871e34ef9f81605e61/href</a></iframe><p><strong>why?</strong></p><p>efficiently iterate over multi dimensional arrays.</p><h4>np.flatnonzero( )</h4><p>returns the indices which are non zero in the input array.</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e4473e7a4cb18c5c5bc2b131338224cb/href">https://medium.com/media/e4473e7a4cb18c5c5bc2b131338224cb/href</a></iframe><p><strong>why?</strong></p><p>selecting specific sequence in an array obeying required condition.</p><h4><strong>np.argsort( )</strong></h4><p>returns the index of each element of the input array after sorting.</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c558d9c7e1b77d40ee59ab1f8f4c050e/href">https://medium.com/media/c558d9c7e1b77d40ee59ab1f8f4c050e/href</a></iframe><p><strong>why?</strong></p><p>useful when we need to sort an array according to another array.</p><h4>np.ufunc.at(input_array, indices, second_operand_ufunc)</h4><p>applies ufunc(Universal Functions) at particular indices, the list of ufunc are available <a href="https://goo.gl/7V2ZTo">here</a></p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/10f0e129914dd4fb3c5360afd4910569/href">https://medium.com/media/10f0e129914dd4fb3c5360afd4910569/href</a></iframe><p><strong>why?</strong></p><p>it can used to apply mathematical operations to a part of array satisfying certain condition.</p><p>See you again 😃</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=29ed0a6ec8ef" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hacking-datascience/hacking-numpy-29ed0a6ec8ef">hacking-numpy</a> was originally published in <a href="https://medium.com/hacking-datascience">hacking-datascience</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python hacks for data-science.]]></title>
            <link>https://medium.com/hacking-datascience/python-hacks-for-data-science-9009a3d7e6e5?source=rss----8f6eff91310d---4</link>
            <guid isPermaLink="false">https://medium.com/p/9009a3d7e6e5</guid>
            <category><![CDATA[reduce]]></category>
            <category><![CDATA[generator]]></category>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[maps]]></category>
            <dc:creator><![CDATA[Vinay Kudari]]></dc:creator>
            <pubDate>Sat, 21 Jul 2018 22:53:31 GMT</pubDate>
            <atom:updated>2018-08-15T19:59:50.659Z</atom:updated>
            <content:encoded><![CDATA[<p>In this story I’ll show how some tasks can be made easier/efficient using few built-in python tools, I’m considering Python 3.6.5 in all my stories henceforth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*OrwbmUm3yD7VJQu4ebJqSA.png" /></figure><p>Get the ipython notebook of this story from <a href="https://github.com/vinaykudari/hacking-datascience/blob/master/notebooks/python_hacks_for_datascience.ipynb"><strong>here</strong></a></p><h4>Generators</h4><p>Before getting to know about generators let’s first understand iterators, So what is an iterator?</p><blockquote><em>Iterator in python is simply an object that can be iterated upon.</em></blockquote><p>next(iterator) will return one value at a time, when we reach the end there is no data to return then an exception will be raised. If an object can return an iterator then that object is know an <strong>iterable</strong>, few examples are <em>lists, dictionaries, tuples, </em>and<em> strings</em>.</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/634705524f02ad70ec542280da229260/href">https://medium.com/media/634705524f02ad70ec542280da229260/href</a></iframe><p>So now what is a generator?</p><blockquote>Generators are iterator objects, they get executed only when we need them.</blockquote><p>When a generator function is called it returns an generator object without even executing the function, when we apply<strong> next( )</strong> on that object the function starts executing until it reaches <strong><em>yield</em></strong> statement, and the yielded value is returned.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8394cdf3525ef1029075e93475fbf7f4/href">https://medium.com/media/8394cdf3525ef1029075e93475fbf7f4/href</a></iframe><p><strong>why?</strong></p><p><strong>To save memory space </strong>: When we are dealing with very large datasets which cannot be loaded into memory all at ones, we can use generators.</p><h4>zip( )</h4><blockquote>zip( ) takes iterables as arguments and returns a zip object which contain list of tuples.</blockquote><p><strong>syntax</strong></p><pre>zip(iterable1, iterable2, …)</pre><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5db7ff014789ae44264b3c392f511752/href">https://medium.com/media/5db7ff014789ae44264b3c392f511752/href</a></iframe><p>to unzip the zipped object we can use <strong>zip(*zipped_object)</strong></p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/46d8a5c0ceb98c3bf21fefa2b7a23d6d/href">https://medium.com/media/46d8a5c0ceb98c3bf21fefa2b7a23d6d/href</a></iframe><h4>List Comprehension</h4><blockquote>They are used to create new lists quickly from other iterables.</blockquote><p><strong>syntax</strong></p><pre>[expression <em>condition</em> loop <em>condition</em>]</pre><p><strong>note</strong> : condition’s are optional</p><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/18b32567dab380cb1ae2d9bec7632d47/href">https://medium.com/media/18b32567dab380cb1ae2d9bec7632d47/href</a></iframe><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6c0035bea9a911dde4ae7758bf122c76/href">https://medium.com/media/6c0035bea9a911dde4ae7758bf122c76/href</a></iframe><p><strong>why?</strong></p><p><strong>compact and faster : </strong>title says it so<strong> 😛</strong></p><h4><strong>Lambda function</strong></h4><blockquote>Lambda function doesn&#39;t have any name and sometimes called as an <em>anonymous</em> function, they can take arguments similar to a normal function but can only have a single expression, when it is called the <strong><em>expression</em></strong> is evaluated and the result is returned which can be assigned to any variable.</blockquote><p><strong>syntax</strong></p><pre>lambda input-arguments : expression</pre><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/21842b6122606027f9b365a367743947/href">https://medium.com/media/21842b6122606027f9b365a367743947/href</a></iframe><p><strong>why lambda function?</strong></p><p><strong>ease of use</strong> : when the function isn&#39;t complex we could use lambda function instead of a normal function.</p><h4>map( )</h4><blockquote>map( ) takes a function and a sequence as arguments, It returns a map object which contain the function call results.</blockquote><p>map object can be converted to list using <strong>list(map-object)</strong></p><p><strong>syntax</strong></p><pre>map(function, iterable1, iterable2, …) </pre><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3b619a6f7359a327a565317922f9745c/href">https://medium.com/media/3b619a6f7359a327a565317922f9745c/href</a></iframe><h4><strong>filter( )</strong></h4><blockquote>filter() takes a function and a sequence similar to a map function and filters the given sequence with the help of the input function that tests each element in the sequence to be true or not and returns a filter object.</blockquote><p><strong>syntax</strong></p><pre>filter(function, iterable)</pre><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1815488d738fb51267adf129a3c99215/href">https://medium.com/media/1815488d738fb51267adf129a3c99215/href</a></iframe><p><strong>why?</strong></p><p><strong>super fast execution</strong> : given multiple sequence arguments, it sends items taken form sequences in parallel as distinct arguments to the function.</p><h4>reduce( )</h4><blockquote>reduce( ) takes a function and a sequence as arguments, when the reduce function is called the first two items of the sequence are passed on to the input function then the computed result along with the next item in sequence is again passed on to the function, this will repeat until the end of the sequence and the final result is returned.</blockquote><p><strong>note : </strong><em>reduce( ) has to be imported from functools</em></p><p><strong>syntax</strong></p><pre>functools.reduce(function, iterable)</pre><p><strong>example</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/15296d601cd0e19eb25ef30ada17d82f/href">https://medium.com/media/15296d601cd0e19eb25ef30ada17d82f/href</a></iframe><p><strong>why?</strong></p><p><strong>fast computation</strong> : way faster than using loops.</p><h4>glob( )</h4><blockquote>Returns an iterable of names matching the given pattern.</blockquote><p><strong>syntax</strong></p><pre>files = glob(&#39;sales*.csv&#39;)</pre><p>returns a list which contains all the filenames like sales_1.csv, sales_2.csv….</p><p><strong>why?</strong></p><p>This is very useful when we have data in many files, we could read the list of files using a for loop.</p><p>See you again. 😃</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9009a3d7e6e5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hacking-datascience/python-hacks-for-data-science-9009a3d7e6e5">Python hacks for data-science.</a> was originally published in <a href="https://medium.com/hacking-datascience">hacking-datascience</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting started…]]></title>
            <link>https://medium.com/hacking-datascience/getting-started-c159a5e8ec1d?source=rss----8f6eff91310d---4</link>
            <guid isPermaLink="false">https://medium.com/p/c159a5e8ec1d</guid>
            <category><![CDATA[about]]></category>
            <category><![CDATA[course]]></category>
            <category><![CDATA[education]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[summary]]></category>
            <dc:creator><![CDATA[Vinay Kudari]]></dc:creator>
            <pubDate>Sat, 21 Jul 2018 22:50:02 GMT</pubDate>
            <atom:updated>2018-08-06T07:30:13.609Z</atom:updated>
            <content:encoded><![CDATA[<p>Hey 😃 Bored of watching video lectures? Tired of doing courses which have almost the similar content, <em>introduction to …..</em> stuff? or need a bit revision of the key concepts? Welcome, you are at the right place. I have been in your shoes sometime back, so I had thought to pen down the summary of hours of video lectures, days of discussions, and months of practice. So here it is <strong>hacking-datascience </strong>for you<strong>.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8ukaiQ5iA-tg1SafPVSUag.jpeg" /></figure><p>I was introduced to machine learning by my aunt during the 7th semester of my undergrad college, since then <em>(8 months until today)</em> I have been taking online courses, reading books, also taking advice from experts. I’ll try to summarise my learning and highlight the key concepts which are often missed in majority of the intro courses. I’ll keep on adding stuff to the stories as I encounter.</p><p>Feel free to <a href="https://www.linkedin.com/in/vinaykudari/"><strong>contact</strong></a> me for any queries, suggestions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c159a5e8ec1d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hacking-datascience/getting-started-c159a5e8ec1d">Getting started…</a> was originally published in <a href="https://medium.com/hacking-datascience">hacking-datascience</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>