<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Priyansh Jain on Medium]]></title>
        <description><![CDATA[Stories by Priyansh Jain on Medium]]></description>
        <link>https://medium.com/@priyansh.jain0246?source=rss-ed0f446b72ec------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*-MTB8qVsVU9-H2nmHwY72Q.png</url>
            <title>Stories by Priyansh Jain on Medium</title>
            <link>https://medium.com/@priyansh.jain0246?source=rss-ed0f446b72ec------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 23:40:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@priyansh.jain0246/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[On the Internalization of concepts from books]]></title>
            <link>https://medium.com/@priyansh.jain0246/on-the-internalization-of-concepts-from-books-fdde6c95fcf?source=rss-ed0f446b72ec------2</link>
            <guid isPermaLink="false">https://medium.com/p/fdde6c95fcf</guid>
            <category><![CDATA[reading]]></category>
            <category><![CDATA[books]]></category>
            <category><![CDATA[nonfiction]]></category>
            <dc:creator><![CDATA[Priyansh Jain]]></dc:creator>
            <pubDate>Sat, 26 Mar 2022 12:32:41 GMT</pubDate>
            <atom:updated>2022-03-28T16:52:14.622Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/754/0*k0h3Wi-8cjoXfo_3" /></figure><p>I recently picked up the habit of reading nonfiction books, and I had a significant realization that I will state below.</p><p>Most books can easily be summarized in a few pages, as there are essentially a small number of major ideas that the author lays down. It seems only logical to read through atomic points the author tries to make, and gather the most valuable and important information they’re giving as your brain can hold only a limited amount of info. Just google the summary of the book, find the blog post and read through it. While I cannot speak for how others would make sense of this thought, I disagree with its entire premise.</p><p>I use text-heavy social media like Reddit and Twitter a lot. For Twitter users, I’m sure you must be aware of a new finance/life/incident thread posted every second by an influencer. The authors very conveniently break down information into bite-sized content (the consequence of the character limit per tweet). A fascinating example is Naval’s thread on <strong><em>“Getting rich without getting lucky”</em></strong> -</p><h3>Naval on Twitter: &quot;How to Get Rich (without getting lucky): / Twitter&quot;</h3><p>How to Get Rich (without getting lucky):</p><p>You can now try out an experiment (or just read through my experience). Read through the entire thread, and try to remember the tweets about an hour later. Chances are you’ll retain a good amount of information. Now try to remember it the next day. Chances are you’ll retain less than half of it. And poof! Just like that half of the information is not available in your head. Now, remember — this could be extremely valuable information, and half of this doesn’t exist in your brain anymore.</p><p>Why does this happen? You know for a fact that the information presented in the tweetstorm is valuable, it is completely bite-sized, broken down into good components. Logically, your brain should be able to remember these sets of tweets.</p><p>Now when I first encountered these tweets, I forgot most of them within a month of reading them. When I actually picked up books, there was a very clear difference in how I was able to remember content. Why is it that key points from lengthy prose are much easier to remember rather than having read the summarized version? That’s when it hit me. I <strong><em>internalized </em></strong>these concepts.</p><p><strong>So what is internalization?</strong></p><p>Let us consider an example of musicians. Have you ever seen a musician listen to a song by ear and are able to immediately play it on their instrument? If not, see this happening here ⬇</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fyq2u9kWJ2rk%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dyq2u9kWJ2rk&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fyq2u9kWJ2rk%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/aa15195b4624d021730a8e28dbfd7c1f/href">https://medium.com/media/aa15195b4624d021730a8e28dbfd7c1f/href</a></iframe><p>There is not just memorization, there is internalization. This <em>SuperDad </em>has internalized the position of the notes on the piano, and the mapping of external sound heard to the note — he has trained the ear, his brain and has developed muscle memory for playing the piano.</p><p>This right here is the internalization of a concept. You are able to access the concept whenever needed and have it right on cue at any given time.</p><p><strong>So how do books internalize concepts?</strong></p><p>Authors structure their ideas in chapters. Each chapter usually holds a central idea, and the chapter encompasses an example or an idea strengthening point — it could be anything like a research study, a personal experience, an excerpt from a memoir of another person. This example provides a fundamental value to the point being conveyed. It plays out as a story — we humans are accustomed to hearing stories and often are able to remember stories, and from those stories are able to derive the concepts that were intended. It’s the same thing with these examples — these are like the <a href="https://www.dictionary.com/e/slang/eli5/">ELI5s</a> that help you understand any concept at the foundational level. And when you spend time with a book (it could be anything from a few days to weeks), your brain is able to store this information in the long term memory.</p><p>Let me illustrate the same with an example:</p><p><strong>Core-Idea: </strong>Re-routing</p><p><strong>Summarized concept:</strong> Skip dwelling on what happened in the past, use it in retrospective to plan for the future</p><p><strong>Internalizer: </strong>I’m sure you might have used Google Maps. Assume that you are going from Point A to Point B. The route that you have passes through 3 to 4 different roads. What happens when instead of taking a left at the required point, you keep going forward? How does Google Maps react to this? <br>Does the voice shout at you for having taken the wrong route? Does it stop talking to you? NO. It simply charts the next possible path to get to your destination. This is what the re-routing function does. If it would have reprimanded you for the same, or stopped working, why would you even use the application? It would not make sense.</p><p>While I cannot speak for others, the internalise section really helped me understand the fundamental meaning of the core-idea, and why it should be used. Whenever I run into such a roadblock, I am able to instantly remember the re-routing example, and that helps me plan ahead.</p><p>What books do, is cover these core ideas with a lot of supporting evidence and examples that strengthen the idea within your psyche. So what I concluded, is that books are a fascinating way of internalize concepts. Since books are carefully written mind-nuggets of all these amazing authors, they are 100% worth making a daily habit out of.</p><p>Hope you enjoyed reading this blog and derived something useful from it 😁</p><p><strong><em>Priyansh Jain</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fdde6c95fcf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hands-on with distributed file systems and storage virtualization!]]></title>
            <link>https://codeburst.io/hands-on-with-distributed-file-systems-and-storage-virtualization-1cc41b93481f?source=rss-ed0f446b72ec------2</link>
            <guid isPermaLink="false">https://medium.com/p/1cc41b93481f</guid>
            <category><![CDATA[virtualization]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[storage]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Priyansh Jain]]></dc:creator>
            <pubDate>Tue, 30 Apr 2019 19:34:10 GMT</pubDate>
            <atom:updated>2019-04-30T19:34:10.695Z</atom:updated>
            <content:encoded><![CDATA[<p>A short article on builiding a cool storage virtualization project from scratch</p><p>We had a course this spring, Virtualization, which had a project component.<br> You must’ve heard about virtualization, how things look unified from one end but are an orchestration of multiple services at the other end. We’ve multiple types of virtualization — ones that range from virtualizing entire Operating Systems (dual boots / Oracle VirtualBox) to virtualizing the desktops. I had to select a topic for the project and on some research found storage virtualization interesting.</p><p><a href="https://github.com/Presto412/StoreV">Here’s</a> the source code for the project.</p><p>Storage Virtualization is practically providing a unified view on a cluster of storage devices, managed in such a manner that we have data safety and size/performance optimization.</p><p>A decent issue with having a storage solution is data de-duplication, which means trying to remove unnecessary copies of files that take up your resources. And I intended to solve the issue with my project(a distributed file system) while getting hands-on experience with managing multiple servers and implementing a management system within it.</p><h3>The stack</h3><p>The web framework I am most comfortable working with is Express(Node.js), and hence chose the same coupled with EJS templating for the frontend views.<br> I also decided to make use of docker for containerized services and using docker swarm to orchestrate them.</p><h3>Development Methodology</h3><p>I wrote some general upload code in express using multer for managing them, then packaged it into a docker image. The base image was the LTS Node.js one, and I used docker-compose to do it in an organized manner.<br> Docker swarm is used to manage the cluster and provides fault tolerance by having replicas and whatnot. What I wanted here was easy to perform hostname mappings for my storage servers.<br> The dockerfile I used for building the image</p><pre>FROM node:10<br><br># Create app directory<br>WORKDIR /usr/src/app<br><br># Install app dependencies<br># A wildcard is used to ensure both package.json AND package-lock.json are copied<br># where available (npm@5+)<br>COPY package*.json ./<br><br>RUN npm install<br># If you are building your code for production<br># RUN npm install --only=production<br><br># Bundle app source<br>COPY . .</pre><p>We see that I set the working directory as /usr/src/app. During development, I mounted the current folder as a volume, so nodemon would automatically detect all the changes and reflect them immediately. The development docker-compose file -</p><pre>version: &quot;3.2&quot;<br><br>networks:<br>  testnet:<br>    external:<br>      name: testnet<br><br>services:<br>  server1_server:<br>    deploy:<br>      replicas: 1<br>      restart_policy:<br>        condition: on-failure<br>        delay: 5s<br>        max_attempts: 3<br>      # placement:<br>      #   constraints:<br>      #     - node.hostname == your-hostname-2-here<br>    image: presto412/storev1<br>    hostname: server1.example.com<br>    command: ./node_modules/.bin/nodemon<br>    volumes:<br>      - &quot;/tmp/uploads:/tmp/uploads&quot;<br>      - &quot;.:/usr/src/app/&quot;<br>    environment:<br>      - SELF_HOSTNAME=server1.example.com<br>    ports:<br>      - &quot;3000:3000&quot;<br>    networks:<br>      testnet:<br>        aliases:<br>          - server1.example.com</pre><h3>First attempt</h3><p>Since I had developed on blockchain platforms during the past year, my initial go-to logic was to make use of not block-chaining itself, but the tech that lies underneath it, Distributed Ledger Technology.</p><p>The general process flow -</p><ul><li>The file is uploaded via a form</li><li>On upload, the backend hashes the file contents by making use of the <a href="https://dev.to/presto412/hands-on-with-distributed-file-systems-and-storage-virtualization-322g">sha1</a> library. I also considered using <a href="https://dev.to/presto412/hands-on-with-distributed-file-systems-and-storage-virtualization-322g">xxHash</a> but didn’t implement it because speed wasn’t a concern for a small indie project.</li><li>It maintains a mapping JSON of the following structure</li></ul><pre>{<br>  &quot;0347f8b35b22104339b5f9d9d1c8d3f0251b6bdc&quot;: {<br>    &quot;details&quot;: {<br>      &quot;name&quot;: &quot;JKSUCI_AuthorAgreement (2).pdf&quot;,<br>      &quot;mimetype&quot;: &quot;application/pdf&quot;,<br>      &quot;date&quot;: &quot;Sun, 07 Apr 2019 23:55:04 GMT&quot;,<br>      &quot;size&quot;: 27087<br>    },<br>    &quot;backups&quot;: [<br>      {<br>        &quot;hostname&quot;: &quot;bangalore.storage.com&quot;,<br>        &quot;path&quot;: &quot;/tmp/uploads/528518c7e306e2f04e34117a5b245e6f&quot;<br>      },<br>      {<br>        &quot;hostname&quot;: &quot;amsterdam.storage.com&quot;,<br>        &quot;path&quot;: &quot;/tmp/uploads/600405f52831535e54aaf590f2b14052&quot;<br>      }<br>    ]<br>  }<br>}</pre><ul><li>So here you can see that the key is the hash of the file, and the value is where it is stored and some metadata.</li><li>This new file’s hash is now checked with the above JSON, and if it exists, the system denies the upload and doesn’t save the file.</li><li>Otherwise, it randomly selects two servers to store it in and updates the mapping as well. This mapping is then updated for every other server, thus making it a distributed ledger implementation.</li></ul><p>At this point, the project was ready and showed the implementation for the first review. Professor suggested to include some meaningful changes, and the one I liked most was delivering files by geographical proximity to the server.</p><h3>Improvements</h3><p>I moved to the cloud this time. Created 5 droplets on <a href="https://dev.to/presto412/digitalocean.com">digitalocean</a>, with each one having a separate data centre — New York, Toronto, Singapore, Bangalore, Amsterdam. <br> I changed the architecture a bit this time — had a central server that served the frontend and maintained metadata of each file stored(I know that it is a bottleneck, but was enough for a class project)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/0*ODsS1rJ06aU_1Tey.jpg" /></figure><p>What I also did, was hash the files in the client side, by making use of the FileReader API coupled with the Forge library. Here’s some code for it.</p><pre>&lt;script type=&quot;text/javascript&quot;&gt;<br>    function submitFile() {<br>      var form = document.getElementById(&quot;uploadFileForm&quot;);<br>      var reader = new FileReader();<br>      reader.onload = function() {<br>        var fileContent = reader.result;<br>        var md = forge.md.sha1.create();<br>        md.update(fileContent);<br>        var fileContentHash = md.digest().toHex();<br>        document.getElementById(&quot;fileHash&quot;).value = fileContentHash;<br>        $.ajax({<br>          type: &quot;POST&quot;,<br>          url: &quot;/checkHash&quot;,<br>          contentType: &quot;application/json;charset=utf-8&quot;,<br>          data: JSON.stringify({<br>            fileHash: fileContentHash<br>          }),<br>          success: function(response) {<br>            console.log(response);<br>            console.log(&quot;sresponse&quot;, response.success);<br>            if (!response.success) {<br>              form.submit();<br>            } else {<br>              console.log(&quot;not submtting form&quot;);<br>              alert(&quot;File already exists&quot;);<br>            }<br>          },<br>          error: function(response) {<br>            console.log(&quot;fresponse&quot;, response);<br>          }<br>        });<br>      };<br>      const file = document.getElementById(&quot;fileItem&quot;).files[0];<br>      reader.readAsText(file);<br>    }<br>  &lt;/script&gt;</pre><p>So now, only the file hash was being sent to the server as an ajax, and if the hash wasn’t present, the file was freshly uploaded.</p><p>When the central server received the file, it selected two storage servers to store it in and then updated its metadata file.</p><p>So this time, when a new request for a file comes in, I used APIs like <a href="http://api.ipstack.com/">ipstack</a> and <a href="https://www.distance24.org/">distance24</a> to determine what location(city, preferably) the IP is from, and how far the city was from each of the storage servers. The servers are then sorted by distance, and the URL is generated for the closest server and returned. When the user clicks on the link, the file is downloaded.</p><p>Docker and Docker Swarm have been extremely helpful in this scenario since I can simply specify the deployment constraints, pull and distribute images and so on.</p><p>What was your first project with distributed systems? Let me know in the comments below. <a href="https://github.com/Presto412/StoreV">Here’s</a> the source for the project.</p><p>Thanks for reading!</p><p><em>Originally published at </em><a href="https://dev.to/presto412/hands-on-with-distributed-file-systems-and-storage-virtualization-322g"><em>https://dev.to</em></a><em>.</em></p><p><strong>Priyansh Jain</strong><br><a href="http://github.com/Presto412">Github</a>| <a href="https://www.linkedin.com/in/priyanshjain412/">LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1cc41b93481f" width="1" height="1" alt=""><hr><p><a href="https://codeburst.io/hands-on-with-distributed-file-systems-and-storage-virtualization-1cc41b93481f">Hands-on with distributed file systems and storage virtualization!</a> was originally published in <a href="https://codeburst.io/">codeburst</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The assumption/validation approach I developed towards software development]]></title>
            <link>https://codeburst.io/the-assumption-validation-approach-i-developed-towards-software-development-e987d1b914d8?source=rss-ed0f446b72ec------2</link>
            <guid isPermaLink="false">https://medium.com/p/e987d1b914d8</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[developer-stories]]></category>
            <dc:creator><![CDATA[Priyansh Jain]]></dc:creator>
            <pubDate>Fri, 04 Jan 2019 07:37:34 GMT</pubDate>
            <atom:updated>2019-04-09T02:59:34.585Z</atom:updated>
            <content:encoded><![CDATA[<p><em>My method of gaining quick hands-on experience in software development</em></p><p>I got into general programming around 4 years back, when I was in my high school years. Our Computer Science syllabus comprised C++, and that was the language I started with. When I got into uni, we had Python to start with</p><p>So we had general projects for almost every course we took ( still have to ), and it depended on the course content.</p><p>We had one such course, web data mining, for which I had to assemble a project. So I had this random idea where one could grab a bunch of ratings from different movie reviewing sites, and aggregate them to show a unified thing.</p><p>I just googled, how to extract data from sites, and got introduced to web scraping, which turned out to be pretty daunting at first, for a person who never was introduced to HTML and JavaScript. Managed to find a GitHub repository that scrapes off ratings from the IMDb website, and it was a general Python script. I installed the dependencies and got it to work.</p><p>Now there were multiple, sections in the code that dealt with different parts of the task, like getting the response, parsing it, filtering the rating etc. I read those lines and registered the fact that yes, if I try this approach on another, simple website, with the same tree style grammar parsing technique, it should work. I did not look at the documentation of the library they used (BeautifulSoup), did not understand what GET or POST meant, but I still cooked up a script to scrape some numbers from the other website. Some parts worked, some didn&#39;t. Those that didn&#39;t, I searched why and replaced my solution. Got the script to work finally, and in the process, I managed to learn the web request-response model, how HTML grammar is parsed ( tree-like structure ), and in the base of all this, how a browser works in the first place.</p><p>It seemed quite unconventional to me, as most peers were undertaking proper development courses, and I did the trial and error experiment.</p><p>What followed this was that after a few months, some of my seniors who liked my work with Python, took me into their team, and asked me to learn Node.js and contribute to their app. Well, here again, I started by referring to what the JavaScript syntax was and quickly understood that it isn’t as strictly types as C++ was, and managed to write up some very simple algorithms, that did not utilize the asynchronous nature, much like how you would write a Python program. Next “textbook” task was to understand callbacks and asynchronous nature, but I was so excited I jumped right into their code base.</p><p>Their code was entirely Promisified — and I was able to understand the flow of the code, without actually knowing what a promise is, or why we use it. I <em>assumed</em> that if I use this as boilerplate, I can modify it according to however I want.</p><p>The result? I learned Promises, and how to use them before I was even introduced to callbacks. What went on internally was of least concern to me at the time, and I just wanted to make a new module. I used parts of their code and managed to make a module. Now again, history repeated itself, I got bugs, googled how to solve them, and only then understood the logic of asynchronous code flow in JavaScript. I actually wrote a complete route, without even understanding how the Express framework works. Eventually, with a little practice, the concepts became very clear to me.</p><p>This continued again when I was introduced to blockchain technology, and Hyperledger Fabric in particular, during my first internship. I was asked to write some business logic in Golang, and I simply asked for existing code that they had on the topic. Without knowing what interfaces/pointers or anything “special” in the language, all I did was try to understand the flow and replicate it. Made assumptions based on my observations, and mostly, everything turned out to be right. Eventually, when I worked with the language more, I understood the core concepts.</p><p>In a sense, I developed this ability to start working with anything that is given to me and learned by either mine or others’ experience, rather than going the conventional way by following tutorials and video lectures. Give me some existing work, I will base my work on it, and validate if my assumption was right. If there’s a new requirement, I learn how to implement it, and if it required core concepts, I learned that too, until my knowledge was sufficient.</p><p>Now I know, that this approach isn’t a great way, as usually knowing something completely will allow you to assess and analyze all possible approaches to solving a problem, and pick out the best one. You can learn in a variety of methods. For example, if you prefer guided, well-assisted learning online, <a href="https://www.microverse.org/">Microverse</a> is a great resource for learning software development, with plus points like pair-programming.</p><p>This method, allowed me to get hands-on experience, and learn way faster than a person who would go the usual way. Because eventually, if you really are learning something new, you will at some time have to understand everything that lies underneath.</p><p>What do you think about this approach? Let me know in the comments below.</p><p>Priyansh Jain</p><p><a href="https://www.linkedin.com/in/priyansh-jain-925133142/">LinkedIn</a> | <a href="https://www.github.com/Presto412">GitHub</a></p><figure><a href="https://www.microverse.org/"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CFDIw64cKLqCzqsV53sqUQ.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e987d1b914d8" width="1" height="1" alt=""><hr><p><a href="https://codeburst.io/the-assumption-validation-approach-i-developed-towards-software-development-e987d1b914d8">The assumption/validation approach I developed towards software development</a> was originally published in <a href="https://codeburst.io/">codeburst</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I developed a captcha cracker for my University’s website]]></title>
            <link>https://codeburst.io/how-i-developed-a-captcha-cracker-for-my-universitys-website-fa4cacdc4dfa?source=rss-ed0f446b72ec------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa4cacdc4dfa</guid>
            <category><![CDATA[captcha]]></category>
            <category><![CDATA[image-processing]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Priyansh Jain]]></dc:creator>
            <pubDate>Fri, 23 Nov 2018 19:37:14 GMT</pubDate>
            <atom:updated>2018-11-25T20:21:14.661Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Hello!</strong></p><p>Consider this a spinoff of my original <a href="https://dev.to/presto412/how-i-automated-the-boring-university-stuff-with-python-5gib">article</a>. I had some requests from the readers to explain how I developed the captcha cracker, and hence I decided to share the story of my first (significant?) project with you guys. <br> <a href="https://github.com/Presto412/VtopBetaCaptchaParser/tree/master/Blog%20Post">Repository Link</a></p><p><strong>Let’s start!</strong></p><p>When I developed these set of scripts, I had zero knowledge of Image Processing or the algorithms used in it. It was in my fresher year that I worked on this.</p><p>The basic ideas I had in mind when I started:</p><ul><li>An image is basically a <strong>matrix</strong>, with pixels as individual cells.</li><li>A color image has a tuple (Red, Green, Blue) values for every pixel, and a grayscale image has a single value and each pixel value ranges from (0, 255) in a general image</li></ul><p>So the student login portal in my college looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/0*Jjkzl88eXr58H-lb.png" /></figure><p>To begin with, I had made some very useful observations about the image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/180/0*Wc2a5ggkaJNjQAWz.png" /></figure><ul><li>The number of characters in the captcha is always 6, and it is a grayscale image.</li><li>The spacing between the characters looked very constant</li><li>Each character is completely defined.</li><li>The image has many stray dark pixels, and lines passing through the image.</li></ul><p>So I ended up downloading one such image and using this <a href="https://www.dcode.fr/binary-image">tool</a> visualized the image in binary(0 for black and 1 for white pixel).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/0*yRx3FD7sGQHxHFbV.png" /><figcaption>Binary visualization of the image</figcaption></figure><p>My observation was right — the image is 45x180 in dimension, and each character is allotted a space of 30 pixels to fit, thus making them evenly spaced.<br> Thus I got my step 1, that was</p><ul><li>Crop any image you get into 6 different parts, each having a width of 30 pixels.</li></ul><p>I chose Python as my prototyping language, as its libraries are easiest to use and implement. <br> On some simple searching, found the <a href="https://pillow.readthedocs.io/en/5.1.x/">PIL</a> library. I decided to use the <a href="https://pillow.readthedocs.io/en/5.1.x/reference/Image.html">Image</a> module, as my operation was limited to only cropping and loading the image as a matrix.<br> So, according to the documentation, the syntax for cropping an image is</p><pre>from PIL import Image<br>image = Image.open(&quot;filename.xyz&quot;)<br>cropped_image = image.crop((left, upper, right, lower))</pre><p>In my case, if you want to crop just the first character,</p><pre>from PIL import Image<br>image = Image.open(&quot;captcha.png&quot;).convert(&quot;L&quot;) # Grayscale conversion<br>cropped_image = image.crop((0, 0, 30, 45))<br>cropped_image.save(&quot;cropped_image.png&quot;)</pre><p>The image that got saved:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/30/0*292tTvIWHJuq8UC2.png" /></figure><p>I wrapped this in a loop, wrote a simple script that fetches 500 captcha images from the site, and saved all the cropped characters into a folder.</p><p>Going to the third observation — Every character is well defined.<br> In order to “clean” a cropped character from the image (remove the unnecessary lines and dots), I used the following method.</p><ul><li>All the pixels in a character are pure black(0). I used a simple logic — If it’s not completely black, it’s white. Hence for every pixel that has a value greater than 0, reassign it to 255. The image is converted into a 45x180 matrix using the load() function, and then it is processed.</li></ul><pre>pixel_matrix = cropped_image.load()<br>for col in range(0, cropped_image.height):<br>    for row in range(0, cropped_image.width):<br>        if pixel_matrix[row, col] != 0:<br>            pixel_matrix[row, col] = 255<br>image.save(&quot;thresholded_image.png&quot;)</pre><p>For clarity sakes, I applied the code to the original image.</p><p>Original:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/180/0*q8Hf4Cg8bmU6Oful.png" /></figure><p>Modified:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/180/0*-1emxd3TXARWwVqg.png" /></figure><p>So you can see, that all the pixels that weren’t completely dark have been removed. This includes the line that passed through the image.</p><p>It was only later after the project was completed that I learned that the above method is called thresholding in Image Processing.</p><p>Moving on the fourth observation — There are many stray pixels in the image.<br> Looped through the image matrix, and if an adjacent pixel is white, and pixel opposite to the adjacent pixel is also white, and the central pixel is dark, make the central pixel white.</p><pre>for column in range(1, image.height - 1):<br>    for row in range(1, image.width - 1):<br>        if pixel_matrix[row, column] == 0 \<br>            and pixel_matrix[row, column - 1] == 255 and pixel_matrix[row, column + 1] == 255 :<br>            pixel_matrix[row, column] = 255<br>        if pixel_matrix[row, column] == 0 \<br>            and pixel_matrix[row - 1, column] == 255 and pixel_matrix[row + 1, column] == 255:<br>            pixel_matrix[row, column] = 255</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/180/0*3h1wk01xsFoipmzD.png" /></figure><p>So you see, the image has been reduced to the individual characters themselves! Even though it may look like some characters have lost their base pixels, they serve as very good skeletons for other images to compare with. After all, the main reason we’re doing so many changes is to generate a proper image for every possible character.</p><p>I applied the above algorithm to all the cropped characters and stored them in a new folder. The next task was to name at least one sample for each character belonging to “ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789”. This step was like the “training” step, where I manually selected a character image for each one and renamed them.</p><p>With this step complete, I had a skeleton image for every character!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/0*zU6wLbA76APEgrMM.png" /></figure><p>I ran couple other scripts to get the best image among all images of a character — for example, if there were 20 ‘A’ character images, the image with the least number of dark pixels was obviously the one with the least amount of noise and was hence best fit for the skeleton images. So there were two scripts:</p><ul><li>One to group similar images sorted by character (constraints: no. of dark pixels, and similarity &gt;= 90–95 %)</li><li>One to get the best images from every grouped character</li></ul><p>Thus by now, the library images were generated. Converted them to pixel matrices, and stored the “bitmaps” as JSON file.</p><p>Finally, here is the algorithm that solves any new captcha image</p><ul><li>Reduced the unnecessary noise in the new image using the same algorithms</li><li>For every character in the new captcha image, brute forced through the JSON bitmaps I generated. The similarity is calculated on the basis of corresponding dark pixels matching.</li><li>This means that, if a pixel is dark and at position (4, 8) in the image to crack the captcha for, and if the pixel is dark at the same position in our skeleton image/ bitmap, then the count is incremented by 1.</li><li>This count, when compared with the number of dark pixels in the skeleton image is used to calculate the percentage match. The percentage and the character for which the percentage is calculated is pushed to a dictionary.</li><li>The character that has the highest percentage of match is selected.</li></ul><pre>    import json<br>    characters = &quot;123456789abcdefghijklmnpqrstuvwxyz&quot;<br>    captcha = &quot;&quot;<br>    with open(&quot;bitmaps.json&quot;, &quot;r&quot;) as f:<br>        bitmap = json.load(f)<br><br>    for j in range(image.width/6, image.width + 1, image.width/6):<br>        character_image = image.crop((j - 30, 12, j, 44))<br>        character_matrix = character_image.load()<br>        matches = {}<br>        for char in characters:<br>            match = 0<br>            black = 0<br>            bitmap_matrix = bitmap[char]<br>            for y in range(0, 32):<br>                for x in range(0, 30):<br>                    if character_matrix[x, y] == bitmap_matrix[y][x] and bitmap_matrix[y][x] == 0:<br>                        match += 1<br>                    if bitmap_matrix[y][x] == 0:<br>                        black += 1<br>            perc = float(match) / float(black)<br>            matches.update({perc: char[0].upper()})<br>        try:<br>            captcha += matches[max(matches.keys())]<br>        except ValueError:<br>            print(&quot;failed captcha&quot;)<br>            captcha += &quot;0&quot;<br>    print captcha</pre><p>And the final result we get is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/0*jZfXgUnO7b7kzSna.png" /></figure><p>which is <strong>Z5M3MQ</strong> — The captcha has been solved successfully!</p><p>So that’s pretty much how I did it. It was a great learning experience, and I developed a <a href="https://chrome.google.com/webstore/detail/enhancer-for-vit-vellore/hafeeaangmkbibcaahfjdmmmeappjbbp">chrome extension</a> that uses the algorithm, and it has 3000+ users!</p><p>Would love to see your opinions and suggestions!<br> The above code is hosted <a href="https://github.com/Presto412/VtopBetaCaptchaParser/tree/master/Blog%20Post">here</a>.</p><p><em>Originally published at </em><a href="https://dev.to/presto412/how-i-cracked-the-captcha-on-my-universitys-website-237j"><em>dev.to</em></a><em>.</em></p><p>Priyansh Jain</p><p><a href="https://www.linkedin.com/in/priyansh-jain-925133142/">LinkedIn</a> | <a href="https://www.github.com/Presto412">GitHub</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa4cacdc4dfa" width="1" height="1" alt=""><hr><p><a href="https://codeburst.io/how-i-developed-a-captcha-cracker-for-my-universitys-website-fa4cacdc4dfa">How I developed a captcha cracker for my University’s website</a> was originally published in <a href="https://codeburst.io/">codeburst</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I automated the boring University stuff with Python]]></title>
            <link>https://codeburst.io/how-i-automated-the-boring-university-stuff-with-python-f2935b10d2ce?source=rss-ed0f446b72ec------2</link>
            <guid isPermaLink="false">https://medium.com/p/f2935b10d2ce</guid>
            <category><![CDATA[google-calendar]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Priyansh Jain]]></dc:creator>
            <pubDate>Fri, 23 Nov 2018 19:29:19 GMT</pubDate>
            <atom:updated>2018-11-25T20:21:19.799Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello!</p><p>My college has a general student — login, where students can view their profile, upload assignments, get due dates, download course materials, and stuff.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/0*ZrvIJZ-DYe7BOJBI.png" /></figure><p>But the site is kind of tedious to navigate through, and hence I decided to use python to automate the boring stuff. One of them is the Assignment Reminder Service.</p><p>As for this article, you need to know a bit about how the web request — response model works and simple python knowledge. It will be split into two parts.<br> Here we go!</p><h3>The Reminder Service</h3><p>Being a generally lazy person, there were certain disappointing instances of me missing assignment uploads. To counter this, I learned Web Scraping using Python.</p><p>This is a resource I found useful:<br> <a href="https://www.dataquest.io/blog/web-scraping-tutorial-python/">https://www.dataquest.io/blog/web-scraping-tutorial-python/</a></p><p>Once I learned how to use the BeautifulSoup and Requests libraries, it became quite clear that if we sequentially emulate what the browser does, we can work with the HTML/JSON responses that we get.</p><p>I would also have you know, as a project in my first year, I developed a cracker for the Captcha that is present in the student login. The captcha looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/180/0*F31P5xUciwJVMw3M.png" /></figure><p>I have written up a simple article explaining how I did that <a href="https://dev.to/presto412/how-i-cracked-the-captcha-on-my-universitys-website-237j">here</a> on Dev.to.<br> You can find the repository on my GitHub profile. Here’s the <a href="https://github.com/Presto412/VtopBetaCaptchaParser">link</a>.</p><p>The chrome developer tools are pretty handy, and I used the network monitor in it.<br> The logic I used was this:</p><ul><li>Opened the Website, and in developer tools opened the network monitor tab.</li><li>Observed the requests being sent and their details(headers, form — data, URL, method)</li><li>For example, if a request looks like this:</li></ul><pre>method = POST, <br>form-data = {&#39;uname&#39;: &quot;username&quot;, &#39;passwd&#39; &quot;password&quot;}, <br>url = &quot;https://example.com/login&quot;, <br>headers = {&quot;User-Agent&quot;:&quot;blah blah&quot;}</pre><ul><li>What I did was with python, emulate the request. So the request in python syntax would be:</li></ul><pre>import requests from bs4 <br>import BeautifulSoup <br>response = requests.post( &quot;https://example.com/login&quot;, # URL {&#39;username&#39;: &quot;username&quot;, &#39;password&#39; &quot;password&quot;}, # form-data <br>headers = {&quot;User-Agent&quot;:&quot;blah blah&quot;}, # headers <br>verify=False # this simply disables SSL security check ) <br>root = BeautifulSoup(response.text, &quot;html.parser&quot;) print(root.prettify()) # pretty prints the response HTML</pre><p>The assignment page looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/735/0*GXuKSOHn5fscdJyV.png" /></figure><ul><li>This way, I was able to acquire the HTML content and then wrote a few lines to navigate to the assignments page. Once that was done, I used <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/">BeautifulSoup</a>, a parsing library in python.</li><li>What BeautifulSoup does is, parse the HTML response into a tree-like structure, and if I want to navigate to a lower branch I can simply go to the next child and so on. The documentation link is mentioned above.</li><li>A list of all the pending assignments is returned, in a dictionary-list format after parsing each link.</li><li>Next thing to do was sync it with the <a href="https://developers.google.com/calendar/">Google Calendar API</a>. The documentation is on — point and pretty easy to understand and implement. The <a href="https://developers.google.com/calendar/quickstart/python">quickstart guide</a> is more than enough to get started.</li><li>Wrote some lines of code that do sanity checks, and finally, request the google calendar API by creating an event, and the code looked like this.</li></ul><pre># Refer to the Python quickstart on how to setup the environment:<br># https://developers.google.com/calendar/quickstart/python<br># Change the scope to &#39;https://www.googleapis.com/auth/calendar&#39; and delete any<br># stored credentials.<br><br>event = {<br>  &#39;summary&#39;: &#39;Data Structures Assignment - 1&#39;,<br>  &#39;description&#39;: &#39;Theory Assessment&#39;,<br>  &#39;start&#39;: {<br>    &#39;dateTime&#39;: &#39;2018-03-28T09:17:30+05:30&#39;, <br>    &#39;timeZone&#39;: &#39;Asia/Kolkata&#39;, # my timezone<br>  },<br>  &#39;end&#39;: {<br>    &#39;dateTime&#39;: &#39;2018-03-28T17:18:30+05:30&#39;, # 12 midnight is the deadline<br>    &#39;timeZone&#39;: &#39;Asia/Kolkata&#39;,<br>  },<br>  &#39;recurrence&#39;: [<br>    &#39;RRULE:FREQ=DAILY;COUNT=1&#39;<br>  ],<br>  &#39;reminders&#39;: {<br>    &#39;useDefault&#39;: False,<br>    &#39;overrides&#39;: [<br>      {&#39;method&#39;: &#39;email&#39;, &#39;minutes&#39;: 24 * 60}, # a day before start<br>      {&#39;method&#39;: &#39;popup&#39;, &#39;minutes&#39;: 100}, # 100 minutes before start<br>    ],<br>  },<br>}<br><br>event = service.events().insert(calendarId=&#39;primary&#39;, body=event).execute()<br>print &#39;Event created: %s&#39; % (event.get(&#39;htmlLink&#39;))</pre><ul><li>I wrapped this snippet into a function, and for every pending assignment called the function which added the event to my calendar.</li><li>And voila! All my assignment due dates showed up on my calendar, and I could use them on my phone. This is how my home-screen widget looks:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*oweRAdlYRSM1sq94.jpeg" /></figure><ul><li>In addition to this, I developed a chrome extension that automatically cracks the captcha and fills the textbox. It also managed downloads from the university website, categorizing them on the basis of course code and faculty name, and organizing into the respective folders. How the course resources page looks:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/0*stCh3cFQDDEEIVNw.png" /></figure><ul><li>How the downloads are saved without the extension:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/0*gI5Mp9nEfcwVAdum.png" /><figcaption>Stupidly named files</figcaption></figure><ul><li>How the downloads are saved when I download them via the extension:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/0*v4wmsJeED9g97kAG.png" /><figcaption>Smart renamed files</figcaption></figure><ul><li>You can view the <a href="https://chrome.google.com/webstore/detail/enhancer-for-vit-vellore/hafeeaangmkbibcaahfjdmmmeappjbbp">extension page</a>, it has over 3000 users, who are students here at my college. It was great fun learning to build using JavaScript!</li></ul><h3>Bonus!</h3><p>This one is partial automation, but you can still have a read! Highly Practical.</p><p>In order to be more productive, I planned to use <a href="http://sheets.google.com/">Google Sheets’</a> Daily Schedule Template. This is basically a week-long activity chart that has days for columns, and times for rows. My current week looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/0*6UJkOYqOZwvl44-t.png" /></figure><p>The red colored regions imply that I won’t be available at the time.</p><p>The beauty of this was that since my timetable always remained the same for an entire semester, all I had to do was sit for like 15 minutes on every Sunday night, and plan the entire week.<br> Now what I wanted was, to sync it to the Google Calendar, so I could see what I’ve planned on my phone. The method I used is as follows:</p><ul><li>Fill in the schedule sheet. While you’re at it, adjust the times accordingly. On the top bar, navigate to Tools -&gt; Script Editor.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/295/0*C8naftU9evyRQmYY.png" /></figure><ul><li>The Script Editor is for Google Scripts, that lets you create JavaScript functions to perform while using Google apps APIs. Here is a sample code snippet you can use. Note that we’ll be using the indexes for the same template</li></ul><pre>var cal = CalendarApp.getDefaultCalendar();<br>var dayOffset = 0; // replace this with the loop index<br>var currSheet = SpreadsheetApp.getActiveSheet(); // gets the currently active sheet<br>var startTimeString = currSheet.getRange(row, column).getValue() // replace row and column with desired values<br>var startTime = new Date(startTimeString);<br>startTime.setDate(startTime.getDate() + dayOffset);<br>var endTime = new Date(startTimeString);<br>endTime.setDate(startTime.getHour() + 1);<br>var title = currSheet.getRange(row, column).getValue(); // specify the row and column of the cell<br>cal.createEvent(title, startTime, endTime); // creates the event in calendar and syncs it</pre><ul><li>This is just a sample snippet, but you should have an idea. Adjust it according to your will, use it in a loop to sync all times. I kind of hard-coded it as I was too lazy to look at some of the APIs.</li><li>Here’s my code that should work for a template with a small change — The time-rows weren’t in a macro and were strings, so I converted them to a macro using this simple formula:</li></ul><pre>=C2 + 0.0416666 * &lt;hour_number&gt;</pre><p>Here, C2 is the current week cell. To adjust a cell’s property, click on a cell, and head over to the macro section right above the index cells.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/304/0*iyvkIOz2C94FiYSF.png" /></figure><p>The hour number if it is 8, then it will equalize to 8:00 AM. I made this change to all the other times, as I wanted an hour wise split up. Here is the script I use now. “0.041666” is basically 1/24th of the day = an hour.</p><pre>function myFunction() {<br>  var cal = CalendarApp.getDefaultCalendar();<br>  var currSheet = SpreadsheetApp.getActiveSheet();<br>  var times = currSheet.getRange(&quot;B6:B20&quot;).getValues()<br>  var weekDate = currSheet.getRange(&quot;C2&quot;).getValue()<br>  var columns = [&quot;C&quot;,&quot;D&quot;,&quot;E&quot;,&quot;F&quot;,&quot;G&quot;,&quot;H&quot;,&quot;I&quot;]<br>  var offSet = 6;<br>  var totalDays = 7;<br>  var totalRows = 15;<br>  for(var i = 0; i &lt; totalDays ; i++) {<br>    var column_no = columns[i] + &quot;5:&quot; + columns[i] + &quot;20&quot;;<br>    var schedule = currSheet.getRange(column_no).getValues();<br>    for(var j = 0; j &lt; totalRows; j++) {      <br>      var startTime = new Date(times[j]);<br>      startTime.setDate(startTime.getDate() + i);<br>      var title = currSheet.getRange(columns[i] +  (offSet + j).toString()).getValue();<br>      if(title === &quot;&quot;){<br>        continue;<br>      }<br>      var startTZoffSet = 240; // adjust hour in timezone<br>      var endTZoffSet = 180; // adjust hour in timezone<br>      var correctedStart = new Date(startTime.getTime() + (60000*(startTime.getTimezoneOffset() - startTZoffSet)));<br>      var correctedEnd = new Date(startTime.getTime() + (60000*(startTime.getTimezoneOffset() - endTZoffSet)));<br>      cal.createEvent(title, correctedStart, correctedEnd);<br>    }<br>  }<br>}</pre><p>I ran this script, and all the events got synced to my google calendar.</p><p>So this is how I’ve automated some stuff in my college. Let me know if there is anything I can help you with, or if you have any suggestions, in the comments!</p><p><em>Originally published at </em><a href="https://dev.to/presto412/how-i-automated-the-boring-university-stuff-with-python-5gib"><em>dev.to</em></a><em>.</em></p><p>Priyansh Jain</p><p><a href="https://www.linkedin.com/in/priyansh-jain-925133142/">LinkedIn</a> | <a href="https://www.github.com/Presto412">GitHub</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f2935b10d2ce" width="1" height="1" alt=""><hr><p><a href="https://codeburst.io/how-i-automated-the-boring-university-stuff-with-python-f2935b10d2ce">How I automated the boring University stuff with Python</a> was originally published in <a href="https://codeburst.io/">codeburst</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The ABCs of Kafka in Hyperledger Fabric]]></title>
            <link>https://codeburst.io/the-abcs-of-kafka-in-hyperledger-fabric-81e6dc18da56?source=rss-ed0f446b72ec------2</link>
            <guid isPermaLink="false">https://medium.com/p/81e6dc18da56</guid>
            <category><![CDATA[fault-tolerance]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[hyperledger-fabric]]></category>
            <category><![CDATA[consensus]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Priyansh Jain]]></dc:creator>
            <pubDate>Wed, 06 Jun 2018 08:49:26 GMT</pubDate>
            <atom:updated>2018-06-07T04:44:20.250Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Kafka is primarily a distributed, horizontally-scalable, fault-tolerant, commit log. A commit log is basically a data structure that only appends. No modification or deletion is possible, which leads to no read/write locks, and the worst case complexity O(1). There can be multiple Kafka nodes in the blockchain network, with their corresponding Zookeeper ensemble.</p><p>If you want to skip directly to the implementation, you can find the repository <a href="https://github.com/skcript/Kafka-Fabric-Network">here</a>.</p><h3>How Kafka Works</h3><p>Let’s understand how Kafka works in general. You can skip this section to know how Hyperledger Fabric implements this.</p><p>Kafka is, in essence, a message handling system, that uses the popular <strong>Publish-Subscribe</strong> model. <strong>Consumers</strong> subscribe to a <strong>Topic</strong> to receive new messages, that are published by the <strong>Producer</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*brLQW_K4WrLhFKCTdmsr-w.png" /></figure><p>These topics, when they get bigger, are split into partitions, and Kafka guarantees that all messages inside a partition are sequentially ordered.</p><p>Kafka does not keep track of what records are read by the consumer and delete them but rather stores them a set amount of time (e.g one day) or until some size threshold is met. Consumers themselves poll Kafka for new messages and say what records they want to read. This allows them to increment/decrement the offset they are at as they wish, thus being able to replay and reprocess events. Consumers are in reality consumer groups, that run one or more consumer processes. Each partition is assigned to a single consumer process so that the same message is not read twice.</p><h3>Crash Tolerance</h3><p>The crash-tolerance mechanism comes into play by replication of the partitions among the multiple Kafka brokers. Thus if one broker dies, due to a software or a hardware fault, data is preserved. What follows is, of course, a leader-follower system, wherein the leader owns a partition, and the follower has a replication of the same. When the leader dies, the follower becomes the new leader.</p><p>If a consumer is the one subscribing to a topic, then how does he know which partition leader to grab the subscribed messages from?</p><p>The answer is the Zookeeper service.</p><h3>The Zookeeper Service</h3><p>Zookeeper is a distributed key-value store, most commonly used to store metadata and handle the mechanics of clustering. It allows clients of the service (the Kafka brokers) to subscribe and have changes sent to them once they happen. This is how brokers know when to switch partition leaders. Zookeeper is also extremely fault-tolerant as it ought to be, since Kafka heavily depends on it.</p><p>Metadata stored here comprises,</p><ul><li>The consumer group‘s offset per partition (although modern clients store offsets in a separate Kafka topic)</li><li>ACL (Access Control Lists) — used for limiting access/authorization</li><li>Producer &amp; Consumer Quotas — maximum message/sec boundaries</li><li>Partition Leaders and their health</li></ul><h3>Kafka in Hyperledger Fabric</h3><p>To understand the working, it is important to know certain terminologies.</p><ul><li><strong>Chain</strong> — The log that a group of clients (a “<strong>channel</strong>”) has access to.</li><li><strong>Channel — </strong>A channel is like a topic that authorized peers may subscribe to and become members of the channel. Only members of a channel may transact on that channel, and transactions on a channel are not visible on other channels.</li><li><strong>OSN</strong> — It is the Ordering Service Node, or in fabric terms the Orderer node. These are connected to the endorsers and peers, which are our clients. These ordering service nodes</li><li>Do client authentication,</li><li>Allow clients to write to a chain or read from it using a simple interface, and</li><li>Do transaction filtering and validation for configuration transactions that either reconfigure an existing chain or create a new one.</li><li><strong>RPC</strong> — Remote Procedure Call is a protocol that one program can use to request a service from a program located in another computer on a network — without having to understand the network’s details. In general, a procedure call is also sometimes known as a function call or a subroutine call.</li><li><strong>Broadcast RPC</strong> — The transaction submission call, from a node to the orderers.</li><li><strong>Deliver RPC</strong> — The Deliver request, what happens after the transaction is processed via the Kafka brokers and is served to the clients on the receiving side.</li></ul><p>Note that even though Kafka is the “consensus” in Fabric, stripped down to its core, it is an ordering service for transactions, and has the added benefit of crash tolerance.</p><h3>What Actually Happens</h3><ul><li>For every chain, we have a separate partition.</li><li>Each chain maps to the single partition topic.</li><li>The OSNs after confirming the permissions in the chain relay the incoming client transactions (received via the Broadcast RPC) belonging to a certain chain to the chain’s corresponding partition.</li><li>The OSNs can then consume that partition and get back an ordered list of transactions that is common across all ordering service nodes.</li><li>The transactions in a chain are batched, with a timer service. That is, whenever the first transaction for a new batch comes in, a timer is set.</li><li>The block (batch) is cut either when the maximum number of transactions are reached (defined by the <strong>batchSize</strong>) or when the timer expires (defined by <strong>batchTimeout</strong>), whichever comes first.</li><li>The timer transaction is just another transaction, generated by the timer described above.</li><li>Each OSN maintains a local log for every chain, and the resulting blocks are stored in a local ledger.</li><li>The transaction blocks are then served to the receiving clients via the Deliver RPC.</li><li>In the case of a crash, the relays can be sent through a different OSN since all the OSNs maintain a local log. This has to be explicitly defined, however.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/0*RvhKCEATyCWmaKrt" /></figure><h3>An Example</h3><p>Consider the figure below. We consider OSNs 0 and 2 to be the nodes connected to the Broadcasting Client, and OSN 1 to be the node connected to the Delivery Client.</p><ul><li>OSN 0 already has transaction foo relayed to the Kafka Cluster.</li><li>At this point, OSN 2 broadcasts baz to the cluster.</li><li>Finally, bar is sent to the cluster by the OSN 0.</li><li>The cluster has these three transactions, at three offsets as seen in the diagram.</li><li>The Client sends a Delivery request, and in its local log, the OSN1 has the above three transactions in Block 4.</li><li>Thus it serves the client with the fourth block, and the process is complete.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/0*MXH0EAdq-D3RyCyw" /></figure><p>Kafka is probably a great thing to happen to the Hyperledger Fabric, due to its high performance. Multiple orderers use Kafka for being in sync, Kafka isn’t an orderer in itself. It just connects two orderers with a stream. Even though it supports crash tolerance, it does not offer protection against malicious nodes in the network. An SBFT based solution can help avoid this problem but is yet to be implemented in the Fabric framework. The issue can be tracked <a href="https://jira.hyperledger.org/browse/FAB-378">here</a>.</p><p>All in all, it is suited for production, as compared to the Solo mechanism also implemented in Fabric. As mentioned earlier, you can play with the network by following the steps in the repository hosted <a href="https://github.com/skcript/Kafka-Fabric-Network">here</a>.</p><p>If you have any queries or doubts, do let me know by commenting below.</p><figure><a href="https://www.skcript.com/blockchain?utm_source=codeburst&amp;utm_medium=blog&amp;utm_campaign=cb2bc"><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VnVj6YoFgv_u9wrh.png" /></a></figure><figure><a href="http://bit.ly/codeburst"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i3hPOj27LTt0ZPn5TQuhZg.png" /></a></figure><blockquote>✉️ <em>Subscribe to </em>CodeBurst’s<em> once-weekly </em><a href="http://bit.ly/codeburst-email"><strong><em>Email Blast</em></strong></a><strong><em>, </em></strong>🐦 <em>Follow </em>CodeBurst<em> on </em><a href="http://bit.ly/codeburst-twitter"><strong><em>Twitter</em></strong></a><em>, view </em>🗺️ <a href="http://bit.ly/2018-web-dev-roadmap"><strong><em>The 2018 Web Developer Roadmap</em></strong></a><em>, and </em>🕸️ <a href="http://bit.ly/learn-web-dev-codeburst"><strong><em>Learn Full Stack Web Development</em></strong></a><em>.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=81e6dc18da56" width="1" height="1" alt=""><hr><p><a href="https://codeburst.io/the-abcs-of-kafka-in-hyperledger-fabric-81e6dc18da56">The ABCs of Kafka in Hyperledger Fabric</a> was originally published in <a href="https://codeburst.io/">codeburst</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Explaining the logic behind JavaScript event loop to a five year old using a Pizza restaurant…]]></title>
            <link>https://codeburst.io/explaining-the-javascript-event-loop-to-a-five-year-old-b4d9e81c1920?source=rss-ed0f446b72ec------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4d9e81c1920</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Priyansh Jain]]></dc:creator>
            <pubDate>Mon, 09 Apr 2018 16:49:52 GMT</pubDate>
            <atom:updated>2018-04-12T12:42:03.784Z</atom:updated>
            <content:encoded><![CDATA[<h3>Explaining the logic behind JavaScript event loop to a five year old using a Pizza restaurant analogy!</h3><p>Originally the answer to a question series labelled <a href="https://dev.to/t/explainlikeimfive"><strong>#explainlikeimfive</strong></a></p><p>Consider a pizza restaurant.<br>There are two types of <strong>orders</strong> that we currently have from a single customer one is an elaborate order, that requires a pizza with an olive topping(1), a cheese filling(2), and a large base(3). The other one is just a simple one, mayonnaise(a) with garlic bread(b).</p><p>The <strong>chef</strong> on receiving the order starts making the first pizza, by taking a large base(3), adding the filling inside it(2), and then adding the olive toppings(1). The same chef also has to make garlic bread. The <strong>manager</strong> suddenly realizes that the restaurant is completely <strong>out of mayonnaise</strong>. The manager adds the ‘get mayonnaise’ task to a <strong>chart</strong> and sends the only available <strong>errand boy</strong> to go fetch some.</p><p>Technically, if orders were to be taken together and delivered together, the customer would have to wait until an errand boy goes to a supermarket five blocks away, get the mayonnaise, and give it to the cook for completing the order. But this is a <strong>restaurant</strong>, and customers don’t need to have the entire order delivered to them all at once. <br>The chef decides to continue making the pizza, bakes it and sends it to the customer. <br>When that is done, the errand boy arrives with the mayonnaise, and the cook takes some garlic bread(b) and adds the mayonnaise(a) on top, and <strong>delivers the order</strong> to the customer.</p><p>What did we learn about Javascript here?</p><ol><li>The customer’s orders(<strong>make Pizza + make GarlicBread</strong>) are the functions in JavaScript code.</li><li>The <strong>order details</strong> are simply about how to customize the pizza and the bread, and they can be treated as functions called inside of make Pizza — and the order is taken from top to bottom — the toppings, the filling, and the size. These details are basically a <strong>representation of the call-stack</strong>, that executes all of these events in reverse order.</li><li>The <strong>absence of mayonnaise</strong> from the restaurant — that was an <strong>Event</strong> that got triggered and called an <strong>asynchronous function</strong>, that was to fetch the mayonnaise from a supermarket.</li><li>Since the restaurant doesn’t need to send the entire order together, they complete the tasks in the <strong>call stack </strong>in a reverse order, as implied above in the story.</li><li>The manager here is the <strong>event table</strong> — his job is to keep track of all the mishaps(<strong>Events</strong>) that happen, in a chronological order.</li><li>The <strong>errand boy</strong> is the <strong>event queue</strong>, that is, if he is already asked to fetch something and a new item is requested, the item has to wait until the errand boy fetches the old item.</li><li>The <strong>chef</strong> is the <strong>event loop</strong>, that is continuously making the orders(executing all the functions).</li><li>The <strong>restaurant</strong> is a <strong>browser</strong>, that doesn’t need to freeze until all the content is loaded and doesn’t need to wait for other things to complete before one is completed.(No need of serving the entire order together)</li></ol><p>So in essence, the event loop looks if the call stack is empty, and if so, looks into the event queue. If there is something in there, it adds it to the call stack, and executes it. The event loop constantly runs until their shift is over(browser content is loaded). The event table keeps track of all the events that have been triggered and sends them to the event queue to be executed.</p><p>This was my attempt, let me know if you found this analogy interesting!</p><figure><a href="http://bit.ly/codeburst"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i3hPOj27LTt0ZPn5TQuhZg.png" /></a></figure><blockquote>✉️ <em>Subscribe to </em>CodeBurst’s<em> once-weekly </em><a href="http://bit.ly/codeburst-email"><strong><em>Email Blast</em></strong></a><strong><em>, </em></strong>🐦 <em>Follow </em>CodeBurst<em> on </em><a href="http://bit.ly/codeburst-twitter"><strong><em>Twitter</em></strong></a><em>, view </em>🗺️ <a href="http://bit.ly/2018-web-dev-roadmap"><strong><em>The 2018 Web Developer Roadmap</em></strong></a><em>, and </em>🕸️ <a href="http://bit.ly/learn-web-dev-codeburst"><strong><em>Learn Full Stack Web Development</em></strong></a><em>.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4d9e81c1920" width="1" height="1" alt=""><hr><p><a href="https://codeburst.io/explaining-the-javascript-event-loop-to-a-five-year-old-b4d9e81c1920">Explaining the logic behind JavaScript event loop to a five year old using a Pizza restaurant…</a> was originally published in <a href="https://codeburst.io/">codeburst</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>