<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Manoj G T on Medium]]></title>
        <description><![CDATA[Stories by Manoj G T on Medium]]></description>
        <link>https://medium.com/@gtmanoj235?source=rss-8f6a2bdfb745------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*vY1naQgIvNM5kTJ4.</url>
            <title>Stories by Manoj G T on Medium</title>
            <link>https://medium.com/@gtmanoj235?source=rss-8f6a2bdfb745------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 23:26:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gtmanoj235/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[NGINX as a Web Server]]></title>
            <link>https://medium.com/@gtmanoj235/nginx-as-a-web-server-754eae209219?source=rss-8f6a2bdfb745------2</link>
            <guid isPermaLink="false">https://medium.com/p/754eae209219</guid>
            <category><![CDATA[nginx]]></category>
            <category><![CDATA[web-server]]></category>
            <dc:creator><![CDATA[Manoj G T]]></dc:creator>
            <pubDate>Tue, 07 Jan 2020 03:50:42 GMT</pubDate>
            <atom:updated>2020-01-07T03:50:42.969Z</atom:updated>
            <content:encoded><![CDATA[<h4>C10k Problem</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*jb13d7vDI0Aa84gWN1zKyQ.png" /></figure><p>NGINX was created to solve the well-known C10k problem.</p><p>C10k is a term devised in 1999 to depict the trouble that existing in web servers at that time experienced in taking care of substantial numbers of concurrent connections. The C10k problem refers to the inability of a server to scale a large number (the 10K) of clients/connections (the C) due to resource exhaustion and addressing the issue by optimizing network sockets.</p><h4>NGINX Overview</h4><p>NGINX was created in 2004 by a Russian Software Engineer Igor Sysoev, as he was frustrated with Apache and wanted to build a replacement of handing 10000 concurrent connections. NGINX solved the C10k problem with its event-driven, asynchronous architecture.</p><p>NGINX is an open-source web server which can also act as a reverse proxy server, load balancer, HTTP cache and a mail proxy server.</p><p>In this blog post, we will examine and centre NGINX as a Web Server. NGINX is a Reverse Proxy Server at its core, and due to this design, it performs so well.</p><p>NGINX mainly focus on the below:</p><ol><li>High Performance — Fast</li><li>Low Resource Usage — Memory Efficient</li><li>High Concurrency</li></ol><h4>NGINX vs. Apache</h4><p>NGINX is Faster than Apache!! This is the most common perception. But we will see how and why.</p><p>NGINX’s core design as a Reverse Proxy Server makes it novel and aids is to outpace Apache in many ways.</p><p>As a matter of course, Apache is configured in a prefork mode, which implies it generates a set number of processes, every one of it can process a single request at any given moment. Nevertheless, that request is for dynamic content (PHP Script) or static content (Image).</p><p>NGINX on the other hand deals requests asynchronously, which implies a single NGINX process can serve multiple requests concurrently. This number mostly depends on the system resources available to NGINX process. However, this also prevents NGINX to embed server-side programming languages like PHP into its process. This implies all request for dynamic content must be managed with a separate process like PHP-FPM (FastCGI Process Manager) and reverse proxied back to the client via NGINX. This makes NGINX significantly less resource hungry.</p><p>A well-configured NGINX server, serving both static and dynamic content should always be more efficient and less demanding on system resources compared to a similarly configured Apache server.</p><p>Now we know NGINX is Faster than Apache due to the below:</p><ol><li>It serves static resources much faster than Apache</li><li>It handles a much more significant number of concurrent requests than Apache</li></ol><p>Another critical difference is NGINX translates all incoming requests as URI locations whereas Apache prefers to read incoming requests as filesystem locations. This design allows NGINX to function not only as a web server but also as a load balancer or a mail server or an HTTP cache.</p><h4>Using Both NGINX and Apache</h4><p>Apache is known for its power and NGINX for its speed. This implies NGINX can serve static contents quicker but it needs help to serve dynamic contents, on the other hand, Apache includes powerful modules required to work with back-end application servers and run scripting languages faster but its memory hungry. Now it’s evident that both NGINX and Apache have their advantages and drawbacks, a better design is to use NGINX in the front as a proxy server for serving static files and forwarding requests and Apache as the backend to serve dynamic content.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=754eae209219" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[In-Memory Database Systems]]></title>
            <link>https://medium.com/@gtmanoj235/in-memory-database-systems-176c705e0c31?source=rss-8f6a2bdfb745------2</link>
            <guid isPermaLink="false">https://medium.com/p/176c705e0c31</guid>
            <category><![CDATA[imdb]]></category>
            <category><![CDATA[in-memory]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Manoj G T]]></dc:creator>
            <pubDate>Tue, 07 Jan 2020 03:43:20 GMT</pubDate>
            <atom:updated>2020-01-07T03:44:11.154Z</atom:updated>
            <content:encoded><![CDATA[<h4>In-Memory Database Systems (IMDS) are faster than disk-based databases.</h4><h4>Introduction</h4><p>An In-Memory Database Systems (IMDS) is a database management system that primarily relies on the main memory of the computer (RAM) for data storage compared to a traditional database management system which employs a disk storage mechanism. They are a growing subset of a database management system (DBMS) emerged in response to new application goals, system requirements, and operating environments.</p><p>In-Memory Database Systems (IMDS) are faster than disk-based databases since the internal optimization algorithms are more straightforward and execute fewer CPU instructions. Accessing data from the RAM (main memory) eliminates seek-time when querying the data, which provides faster and more predictable performance than disk. In theory, IMDS scales well beyond the terabyte size range.</p><p>The key reasons behind concerning ‘‘data storage mechanism” as the hearth of enterprise software systems is it is the most significant part of the software that determines how quick an application responds to a request, and the loss of data is mostly unacceptable since the critical business operations.</p><h4>Key Categories/Type of In-Memory Database Systems</h4><p>We can categorize in-memory databases systems into four critical classes according to different optimizations they have adapted.</p><p><strong>Key-Value Store:</strong> In a Key-Value Store we store the data as key-value pairs like the data structure ‘‘hash table” where keys are used to retrieve the data. Example: Redis &amp; Memcached</p><p><strong>Document Store:</strong> In a Document Store we store the data in collections that contain key-value pairs which encapsulate key-value pairs in JSON (JavaScript Object Notation) or JSON like documents. In a document store since values are not opaque to the system, we can query the data by values as well as keys. Example: MongoDB</p><p><strong>Column Family Store: </strong>In a Column Family Store the data is stored as a set of rows and columns where we group columns according to the relationship between the data. Column family shares the characteristic of being schema-free with document stores. However, the implementation is very different. Example: Cassandra</p><p><strong>Graph Database:</strong> In a Graph Database or Graph DBMS we represent data in the form of graph-structured as nodes and edges. It is easy and flexible to process data in this form. Example: Neo4j</p><h4>Storage Optimization for In-Memory Database Systems</h4><p>Recent evolution in RAM sizes and rapidly dropping market prices of RAM in the past years has inspired considerable increases in the prevalence of database systems that keep the entire database in memory. Due to this in-memory computing became economic among a wide variety of applications. Though it’s economically feasible to use the main memory as the primary data store still RAM is a limited resource and expensive compared to a disk.</p><p>A primary goal of recent research works around IMDB is to improve in-memory storage optimization. The freer memory, the larger systems to be stored in the database, which enhances the performance and the cost-efficiency. In the In-Memory Database Systems research, recent works focus on approaches to identify hot and cold data so that we can store the hot data in the RAM and move the raw(cold) data to a cheaper disk store.</p><p>A traditional RDBMS relies on ACID (Atomicity, Consistency, Isolation, Durability) consistency model that ensures all the transactions are correctly committed and do not the corrupt database, and the data are consistent. Whereas an IMDB relies on BASE (Basically Available, Soft-state, Eventually Consistent) consistency model to achieve scalability, high availability, and high performance.</p><h4>Advantages of In-Memory Database Systems</h4><p>Some of the critical benefits of IMDS are</p><p>1) IMDS is used to cache countless amounts of data continuously. Using this cached data, we can ensure speedy response times for searches.</p><p>2) The ability of IMDS to store session data, allowing for the customization of live sessions and guaranteeing optimum website performance.</p><p>3) IMDS is used to process events for improved complex event processing.</p><h4>Applications of In-Memory Database Systems</h4><p>Some of the critical demands of IMDS are</p><p>1) IMDS is used widely in algorithmic trading and some other applications used in financial markets to provide instant manipulation of data and to identify and leverage the market opportunity.</p><p>2) Various B2C web applications like e-commerce and social networking sites also use IMDS to cache a portion of their on-disk database.</p><p>3) IMDS is also used widely in real-time embedded systems or real-time operating systems (RTOSs) like IP network routing, telecom switching, and industrial control.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=176c705e0c31" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Overview of Big Data for Beginners]]></title>
            <link>https://medium.com/@gtmanoj235/overview-of-big-data-for-beginners-a8cea682d416?source=rss-8f6a2bdfb745------2</link>
            <guid isPermaLink="false">https://medium.com/p/a8cea682d416</guid>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Manoj G T]]></dc:creator>
            <pubDate>Tue, 07 Jan 2020 03:35:13 GMT</pubDate>
            <atom:updated>2020-01-07T03:47:23.700Z</atom:updated>
            <content:encoded><![CDATA[<h3>Five Vs of Big Data</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x4PU0GxaD50tcVXQGSdTDQ.jpeg" /></figure><p>You may be wondering why I’m starting with the Five V’s of Big Data before even explaining What Big Data is. But, trust me on this one, it will make more sense to explain what Big Data is after learning the Five V’s of Big Data.</p><p>When we deal with data, especially Big Data, there are <strong><em>“Five Vs”</em></strong> you should know.</p><blockquote><em>The first </em><strong>“V” is Volume</strong><em> — this refers to the vast amount of data gets generated every second in the current information age.</em></blockquote><p>The quantity of data which gets generated every year is exponentially higher compared to the previous years. By the year 2025, it’s expected that the total volume of data which got created will be ten times more than the currently accumulated data volume.</p><p>In a traditional database system, the data is measured in Gigabytes (GB) and Terabytes (TB), but Big Data is measured in Petabytes (PB) and Exabytes. The data is so huge, and it’s impossible to manage it using a traditional database system.</p><p>The primary reason for the increase in the volume of data gets generated is due to the rise in the number of devices like mobile phones, surveillance camera, sensors or any other IoT device. Another important source of data is the web access logs. Every search we do on Google and every website we visit in the world wide web generates data.</p><blockquote><em>The second </em><strong>“V” is Velocity</strong><em> — this refers to the speed at which the data moves and the rate at which new data gets generated.</em></blockquote><p>Not only has the volume of generated data that increased, but the speed at which this data arrives and moves has also improved drastically.</p><p>The primary reason for this is the significant increase in the speed of communication of data through the internet. Also, more and more people are getting access to high-speed internet every day, this automatically increases the rate at which the new data gets generated and moves.</p><blockquote><em>The third </em><strong>“V” is Variety</strong><em> — this refers to the different types of data which gets generated and used.</em></blockquote><p>Since the number of types of devices which generates data increased, the type of data that each of these devices produces also increased drastically. It’s no longer the traditional structured data, or text message gets created, it’s also various unstructured data or images or videos and much more.</p><blockquote><em>The fourth </em><strong>“V” is Veracity</strong><em> — this refers to the messiness or integrity of the data.</em></blockquote><p>The data which we obtain from its source does not always need to be accurate. The quality of the data received will sometimes be of low standards due to any technical issues or human error or intentional malicious manipulation.</p><blockquote><em>The fifth and the final </em><strong>“V” is Value</strong><em> — this refers to the value we get by analyzing the data.</em></blockquote><p>There is no point in handling Big Data if we are not going to get any business value out of it. So, it’s essential to analyze and get value from the data which gets generated by various sources.</p><h3>What Is Big Data?</h3><p>It’s always ambiguous when it comes to the definition of Big Data. To be more precise, it’s difficult to say when data becomes big data. You may think if the size of the data is enormous, then it’s Big Data, but this is not entirely true. But, now that we know the Five Vs of Big Data, let’s go ahead and define what Big Data is.</p><blockquote><strong>“Big Data”</strong><em> is a term used to refer an extremely large, high-speed, vastly diverse and complex data, which can be analyzed to derive business value.</em></blockquote><p>The data can be both structured and unstructured which can’t be managed with our traditional database management systems, and it should be handled by developing a highly scalable, maintainable, and fault-tolerant data systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8cea682d416" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache NiFi Overview]]></title>
            <link>https://medium.com/@gtmanoj235/apache-nifi-overview-6f48ba731995?source=rss-8f6a2bdfb745------2</link>
            <guid isPermaLink="false">https://medium.com/p/6f48ba731995</guid>
            <category><![CDATA[apache-nifi]]></category>
            <category><![CDATA[nifi]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[etl]]></category>
            <dc:creator><![CDATA[Manoj G T]]></dc:creator>
            <pubDate>Tue, 07 Jan 2020 03:31:32 GMT</pubDate>
            <atom:updated>2020-01-07T03:31:32.338Z</atom:updated>
            <content:encoded><![CDATA[<h3>What Is Apache NiFI?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*TYAAkAy6OXG_K-paQXVaag.jpeg" /></figure><p>Apache NiFi is a robust open-source Data Ingestion and Distribution framework and more. It can propagate any data content from any source to any destination.</p><p>NiFi is based on a different programming paradigm called Flow-Based Programming (FBP). I’m not going to explain the definition of Flow-Based Programming. Instead, I will tell how NiFi works, and then you can connect it with the definition of Flow-Based Programming.</p><h3>How NiFi Works</h3><p>NiFi consists of atomic elements which can be combined into groups to build simple or complex dataflows.</p><p>NiFi has processors and process groups.</p><h3>What Is a Processor in NiFi?</h3><p>A processor is an atomic element in NiFi which can do some specific task.</p><p>The latest version of NiFi has around 280+ processors, and each has its own responsibilities.</p><p>For example, the GetFile processor can read a file from a specific location, whereas the PutFile processor can write a file to a particular location. Like this, we have many other processors, that each address a unique aspect of the data pipeline.</p><p>We have processors to get data from various data sources and processors to write data to various data sources.</p><p>The data source can be almost anything. It can be any SQL database server like Postgres, Oracle, or MySQL, or it can be NoSQL databases like MongoDB or Couchbase. It can also be your search engines like Solr or Elasticsearch, or it can be your cache servers like Redis or HBase. It can even connect to Kafka Messaging Queue.</p><p>NiFi also has a rich set of processors to connect with Amazon AWS entities likes S3 Buckets and DynamoDB.</p><p>NiFi have a processor for almost everything you need when you’re working with data. We will go deep into various types of processors available in NiFi in later posts. Even if you don’t find the right processor for your requirements, NiFi gives a simple way to write your custom processors.</p><p>Now let’s move on to the next term, FlowFile.</p><h3>What Is a FlowFile in NiFi?</h3><p>The actual data in NiFi propagates in the form of a FlowFile. The FlowFile can contain any data, say CSV, JSON, XML, or plain text, and it can even be SQL queries or binary data.</p><p>The FlowFile abstraction is the reason that NiFi can propagate any data from any source to any destination. A processor can process a FlowFile to generate a new FlowFile.</p><p>The next important term is connections.</p><h3>What Is a Connection in NiFi?</h3><p>In NiFi, all processors can be connected to create a data flow. This link between processors is called connections. Each connection between processors can act as a queue for FlowFiles as well.</p><p>The next one is the process group and input/output ports.</p><h3>What Are Process Groups, Input Ports, and Output Ports in NiFi?</h3><p>In NiFi, one or more processors are connected and combined into a process group. When you have a complex data flow, it’s better to combine processors into logical process groups. This helps in better maintaining the flows.</p><p>Process groups can have input and output ports which are used to move data between them.</p><p>The last and final term you should know is, ‘controller services.’</p><h3>What Is a Controller Service in NiFi?</h3><p>Controller services are shared services that can be used by processors. For example, a processor which gets and puts data into a SQL database can have a Controller Service with the required DB connection details.</p><p>The controller service is not limited to DB connections.</p><p>Happy learning!</p><p>To learn more about Apache NiFi, kindly visit my YouTube Channel. I have created a playlist, especially for beginners.</p><p><a href="https://www.youtube.com/watch?v=VVnFt54jUQ8&amp;list=PL55symSEWBbMBSnNW_Aboh2TpYkNIFMgb">NiFi Introduction — YouTube Playlist</a></p><p>After finishing my YouTube tutorial, if you wish to dive deep into the advanced topic, you can opt my Udemy course.</p><p><a href="https://www.udemy.com/course/apache-nifi-the-beginner-guide/?referralCode=3FCCF7F1DF404032EE74">Apache NiFi — The Complete Guide (Udemy Course)</a></p><p>You can learn the same course in Skillshare for FREE using the below referral link.</p><p><a href="https://www.skillshare.com/r/user/learnwithmanoj">Apache NiFi — The Complete Guide (Skillshare Course)</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f48ba731995" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>