<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dr. Shamiul Hauqe on Medium]]></title>
        <description><![CDATA[Stories by Dr. Shamiul Hauqe on Medium]]></description>
        <link>https://medium.com/@dr-sami?source=rss-dc8bf5718d9d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*NenvQpfyhQbdT2ha.png</url>
            <title>Stories by Dr. Shamiul Hauqe on Medium</title>
            <link>https://medium.com/@dr-sami?source=rss-dc8bf5718d9d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 05:19:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dr-sami/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Scripting for big data: Part 1: Parsing csv]]></title>
            <link>https://dr-sami.medium.com/scripting-for-big-data-part-1-parsing-csv-cbd9f51e18e9?source=rss-dc8bf5718d9d------2</link>
            <guid isPermaLink="false">https://medium.com/p/cbd9f51e18e9</guid>
            <dc:creator><![CDATA[Dr. Shamiul Hauqe]]></dc:creator>
            <pubDate>Sat, 19 Dec 2020 19:26:06 GMT</pubDate>
            <atom:updated>2020-12-19T19:26:06.627Z</atom:updated>
            <content:encoded><![CDATA[<h3>A. Raw python</h3><h4>Openig a file and checking the size</h4><p>In [2]:</p><pre><em># Due to limitation of uploadable file size in Github, we are using a small file in this example.</em><br><em># However, these codes are scalable for large files (files in GBS).</em></pre><pre><em># Opening a file named &#39;access_subset.log&#39;by read() method and checking its size:</em></pre><pre><strong>try</strong>:<br>    f = open(&#39;access_subset.log&#39;,&#39;r&#39;)<br><strong>except</strong> <strong>Exception</strong> <strong>as</strong> e:<br>    print(str(e))<br>    exit<br>s = f.read()<br>print(len(s))<br>f.close()</pre><pre><strong>Output:</strong></pre><pre>647478</pre><h4>Parsing and printing selected contents</h4><pre>lines = s.split(&quot;<strong>\n</strong>&quot;) <em># Splitting the &#39;lines&#39; by new line (&#39;\n&#39;)</em></pre><pre>print(len(lines)) <em># Checking the number of elements</em></pre><pre>n = 0<br><strong>for</strong> line <strong>in</strong> lines:<br>    row = line.split() <em># Further splitting (default is splitting by white space)</em><br>    ip = row[0] <em># Since &#39;row&#39; is a list, it could be indexed to parse desired information ... here, row[0] contains ip address</em><br>    <strong>if</strong> len(row) &gt; 5:<br>        dts = row[3].replace(&quot;[&quot;,&quot;&quot;) <em># row[3] contains full datetime like 04/Mar/2018:21:32:18, we are splitting it in smaller chunks</em><br>        dtl = dts.split(&quot;:&quot;)<br>        ds = dtl[0]<br>        ts = &#39;:&#39;.join(dtl[1:])</pre><pre>        print(ip,dts,ds,ts)</pre><pre>    <strong>if</strong> n &gt; 5:<br>        <strong>break</strong><br>    n += 1</pre><pre>3350<br>104.229.180.74 04/Mar/2018:21:32:18 04/Mar/2018 21:32:18<br>104.229.180.74 04/Mar/2018:21:32:18 04/Mar/2018 21:32:18<br>104.229.180.74 04/Mar/2018:21:32:24 04/Mar/2018 21:32:24<br>104.229.180.74 05/Mar/2018:02:12:32 05/Mar/2018 02:12:32<br>104.229.180.74 05/Mar/2018:02:12:32 05/Mar/2018 02:12:32<br>104.229.180.74 05/Mar/2018:02:12:32 05/Mar/2018 02:12:32<br>104.229.180.74 05/Mar/2018:02:12:32 05/Mar/2018 02:12:32</pre><h3>B. CSV module</h3><p>CSV module provides two different classes to read the contents of a csv file: csv.reader and csv.DictReader</p><p>With csv module’s reader class object we can iterate over the lines of a csv file as a list of values.</p><h4>Using the csv module’s reader class object</h4><pre><strong>import</strong> <strong>csv</strong><br><em>&#39;&#39;&#39;</em><br><em>For large CSV files, it is better to use the reader object in a for loop.</em><br><em>This avoids loading the entire file into memory at once.</em><br><em>&#39;&#39;&#39;</em><br><em>#counter to limit print content</em><br>n=0</pre><pre><em>#Importing reader function of csv module</em><br><strong>from</strong> <strong>csv</strong> <strong>import</strong> reader</pre><pre><em># open file in read mode</em><br><strong>with</strong> open(&#39;access_subset.log&#39;, &#39;r&#39;) <strong>as</strong> f:<br>    <br>    <em># pass the file object to reader() to get the reader object</em><br>    rows = csv.reader(f)<br>      <br>    <em># Iterate over each of the rows in the csv using reader object</em><br>    <strong>for</strong> row <strong>in</strong> rows:<br>        <em># row variable is a list that represents a row in csv</em><br>        print(type(row))<br>        print(len(row))<br>        print(row)<br>        <strong>if</strong> n==3:<br>            <strong>break</strong><br>        n+=1</pre><pre>&lt;class &#39;list&#39;&gt;<br>2<br>[&#39;104.229.180.74 - - [04/Mar/2018:21:32:18 +0000] &quot;GET / HTTP/1.1&quot; 404 571 &quot;-&quot; &quot;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML&#39;, &#39; like Gecko) Chrome/64.0.3282.140 Safari/537.36&quot; &quot;-&quot;&#39;]<br>&lt;class &#39;list&#39;&gt;<br>2<br>[&#39;104.229.180.74 - - [04/Mar/2018:21:32:18 +0000] &quot;GET /favicon.ico HTTP/1.1&quot; 404 571 &quot;https://hilite.clarksonmsda.org/&quot; &quot;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML&#39;, &#39; like Gecko) Chrome/64.0.3282.140 Safari/537.36&quot; &quot;-&quot;&#39;]<br>&lt;class &#39;list&#39;&gt;<br>2<br>[&#39;104.229.180.74 - - [04/Mar/2018:21:32:24 +0000] &quot;GET / HTTP/1.1&quot; 404 571 &quot;-&quot; &quot;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML&#39;, &#39; like Gecko) Chrome/64.0.3282.140 Safari/537.36&quot; &quot;-&quot;&#39;]<br>&lt;class &#39;list&#39;&gt;<br>2<br>[&#39;104.229.180.74 - - [05/Mar/2018:02:12:32 +0000] &quot;GET / HTTP/1.1&quot; 200 2518 &quot;-&quot; &quot;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML&#39;, &#39; like Gecko) Chrome/64.0.3282.140 Safari/537.36&quot; &quot;-&quot;&#39;]</pre><p>It iterates over all the rows of access_subset.log file. For each row it fetched the contents of that row as a list and printed that list.</p><p>How did it work ?</p><p>It performed the following steps:-</p><p>Open the file in read mode and create a file object. [with open(‘access_subset.log’, ‘r’) as f:]</p><p>Create a reader object (iterator) by passing file object in csv.reader() function. [rows = reader(f)]</p><p>Now, once we have an iterator (returned by the reader object) we used this iterator with for loop to read individual row of the csv as list of values (where each value in the list represents an individual cell).</p><p>In this way, only one line will be in the memory at a time while iterating through csv file.</p><h4>Using csv module’s DictReader object</h4><p>With csv module’s DictReader class object we can iterate over the lines of a csv file as a dictionary i.e.: for each row a dictionary is returned, which contains the pair of column names and cell values for that row. For this example we are using another csv that has column heading. Theese headers will act as keys.</p><p>In [17]:</p><pre>n=0<br><strong>from</strong> <strong>csv</strong> <strong>import</strong> DictReader</pre><pre><strong>with</strong> open(&#39;wifi_list.csv&#39;, &#39;r&#39;) <strong>as</strong> f:<br>    <br>    <em># pass the file object to DictReader() to get the DictReader object</em><br>    rows = DictReader(f)<br>    <br>    <em># iterate over each line as a ordered dictionary</em><br>    <strong>for</strong> row <strong>in</strong> rows:<br>        <br>        <em># row variable is a dictionary that represents a row in csv</em><br>        print(row[&#39;MAC&#39;])<br>        <strong>if</strong> n==3:<br>            <strong>break</strong><br>        n+=1</pre><pre><strong>Output:</strong></pre><pre>e2:91:f5:de:fa:51<br>e0:91:f5:de:fa:50<br>e0:91:f5:5e:34:60<br>90:84:0d:d6:8b:69</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cbd9f51e18e9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>