<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Suraj Nathani on Medium]]></title>
        <description><![CDATA[Stories by Suraj Nathani on Medium]]></description>
        <link>https://medium.com/@surajnathani?source=rss-a2454c74033a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*4DjEAEiUWFhlc7byN7lFDA.png</url>
            <title>Stories by Suraj Nathani on Medium</title>
            <link>https://medium.com/@surajnathani?source=rss-a2454c74033a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 13:26:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@surajnathani/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Rebalancing Customer Groups in Kafka.]]></title>
            <link>https://medium.com/@surajnathani/rebalancing-customer-groups-in-kafka-1f6188e70f8a?source=rss-a2454c74033a------2</link>
            <guid isPermaLink="false">https://medium.com/p/1f6188e70f8a</guid>
            <category><![CDATA[kafka-consumer-group]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[kafka-consumer]]></category>
            <dc:creator><![CDATA[Suraj Nathani]]></dc:creator>
            <pubDate>Sat, 27 Jul 2024 20:14:47 GMT</pubDate>
            <atom:updated>2024-07-27T20:14:47.532Z</atom:updated>
            <content:encoded><![CDATA[<p>When diving into the world of Apache Kafka, one of the most perplexing aspects you’ll encounter is understanding consumer groups. Incorporating visual diagrams will make this concept easier to understand.</p><ol><li>Let’s start with a cluster of nodes hosting apache kafka, with one kafka topic on one of the nodes</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*YbRdoiH8R6rWNDQESjwfxQ.png" /></figure><p>2. Let’s add an Application(Consumer) connecting to the topic. This Application will be considered a part of a Consumer group. Since only one connection is made, this Application will be one consumer, consuming from both the partitions</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*Aw-p3a5vTX0DYhNbbfIWrw.png" /></figure><p>3. Now let’s add another Consumer(this consumer can be in the form of a new Application or new thread in the same Application) connecting to the same topic. This Application will use the same “group-id” as the previous one, enabling it to be a part of the same Consumer Group, with an additional consumer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*ofhjawTA3jqP3qyRC2CXYw.png" /></figure><p>4. Whenever a consumer is added, a series of steps is set in motion. The Kafka topic (co-ordinator) receives a signal that a new consumer is requesting to join. The Kafka topic will enter a rebalancing phase and wil ask all the existing consumers to disconnect and reconnect again</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*QYHFsEHCvOE02XTpZXfCng.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*xATstNJYvLjVocfrNR1XBA.png" /></figure><p>5. In the meanwhile, the kafka topic will re-assign the partitions to the consumers. When all h the consumers confirm that they will join, each consumer is given a number of partitions to consume from</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ieGmNgOlefKIuH3YCqYaHw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*2xh0akWXqU6-Wzw_8oBvtQ.png" /></figure><p>6. If a new consumer requests to join, the entire process will be followed again.</p><p>What does this imply? This helps us determine best practices to be followed when designing kafka solutions, which will be covered in the next blog.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f6188e70f8a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A peek into Sliding Window Problems]]></title>
            <link>https://medium.com/@surajnathani/a-peek-into-sliding-window-problems-eb67b8a46ca0?source=rss-a2454c74033a------2</link>
            <guid isPermaLink="false">https://medium.com/p/eb67b8a46ca0</guid>
            <category><![CDATA[sliding-windows]]></category>
            <category><![CDATA[arrays]]></category>
            <category><![CDATA[data-structures]]></category>
            <dc:creator><![CDATA[Suraj Nathani]]></dc:creator>
            <pubDate>Thu, 08 Feb 2024 20:49:10 GMT</pubDate>
            <atom:updated>2024-02-08T20:49:10.305Z</atom:updated>
            <content:encoded><![CDATA[<p>The Sliding Window algorithm involves two different types of problems:</p><ol><li>Sliding window with constant window size</li><li>Sliding window with variable window size</li></ol><p>We’ll try to understand the first type of problem, the one with the constant window size.</p><p>Some of the problems that use this approach include the below:</p><ol><li>Print all subarrays of size k with sum equal to x.</li><li>Print all subarrays of size k with sum equal to 0.</li><li>Number of subarrays of size k and average greater than x.</li><li>Find count of distinct elements in every subarray of size k.</li></ol><p>To solve the above problems:</p><ol><li>List all subarrays with size k</li><li>Eliminate some subarrays based on condition</li></ol><p>Lets try to understand the first part of the problem, which is listing all subarrays of size k</p><p>We’ll write the code one step at a time.</p><p>Step 1: Print all elements of the array with a while loop</p><pre>def step1(array_1):<br>    print(&quot;Step 1: Print all elements of the array in a while loop&quot;)<br>    i = 0<br>    while i &lt;= len(array_1):<br>       print(i)<br>       i += 1<br><br>array_1 = [0, 1, 2, 3, 4, 5, 6, 8, 8, 8]<br>step1(array_1)</pre><p>Step1: Output:</p><pre>Print all elements of the array in a while loop<br>0<br>1<br>2<br>3<br>4<br>5<br>6<br>7<br>8<br>9<br>10</pre><p>Step 2: Define a new subarray and add elements to this subarray</p><pre>def step2(array_1):<br> print(&quot;Step 2: Define a new subarray and add elements to this subarray&quot;)<br> i = 0<br> subarray = []<br> while i &lt;= len(array_1) - 1:<br>    print(i)<br>    subarray.append(array_1[i])<br>    print(subarray)<br>    i += 1<br><br>array_1 = [0, 1, 2, 3, 4, 5, 6, 7, 8]<br>step2(array_1)</pre><p>Step2: Output:</p><pre>Step 2: Define a new subarray and add elements to this subarray<br>0<br>[0]<br>1<br>[0, 1]<br>2<br>[0, 1, 2]<br>3<br>[0, 1, 2, 3]<br>4<br>[0, 1, 2, 3, 4]<br>5<br>[0, 1, 2, 3, 4, 5]<br>6<br>[0, 1, 2, 3, 4, 5, 6]<br>7<br>[0, 1, 2, 3, 4, 5, 6, 7]<br>8<br>[0, 1, 2, 3, 4, 5, 6, 7, 8]</pre><p>Step3: Add elements to subarray upto k (k = target subarray size = 3)</p><pre>def step3(array_1):<br>    print(&quot;Add elements to subarray upto only k&quot;)<br>    i = 0<br>    subarray = []<br>    k = 3<br>    while i &lt;= len(array_1) - 1:<br>       print(i)<br>       subarray.append(array_1[i])<br>       print(subarray)<br>       if i &lt; k - 1:<br>          i += 1<br>          continue<br>       break<br><br>array_1 = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]<br>step3(array_1)</pre><p>Step3: Output:</p><pre>Add elements to subarray upto only k<br>0<br>[0]<br>1<br>[0, 1]<br>2<br>[0, 1, 2]</pre><p>Step4: Introduce len_subarray variable which is separate than i</p><pre>def step4(array_1):<br>    print(&quot;Step 3&quot;)<br>    i = 0<br>    len_subarray = 0<br>    subarray = []<br>    k = 3<br>    while i &lt;= len(array_1) - 1:<br>       print(i)<br>       subarray.append(array_1[i])<br>       print(subarray)<br>       if len_subarray &lt; k - 1:<br>          i += 1<br>          len_subarray += 1<br>          continue<br>       break<br><br>array_1 = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]<br>step4(array_1)</pre><p>Step4: Output:</p><pre>Introduce len_subarray variable which is separate than i<br>0<br>[0]<br>1<br>[0, 1]<br>2<br>[0, 1, 2]</pre><p>Step5: Once we reach 3 elements, remove first element, and let the loop run</p><pre>def step5(array_1):<br> print(&quot;When we reach 3 elements, remove first element, and let the loop run&quot;)<br> i = len_subarray = 0<br> subarray = []<br> k = 3<br> while i &lt;= len(array_1) - 1:<br>    print(i)<br>    subarray.append(array_1[i])<br>  <br>    if len_subarray &lt; k - 1:<br>       i += 1<br>       len_subarray += 1<br>       continue<br>  <br>    print(subarray)<br>    subarray.pop(0)<br>    i += 1<br>  <br>array_1 = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]<br>step5(array_1)</pre><p>Step5: Output:</p><pre>When we reach 3 elements, remove first element, and let the loop run<br>0<br>1<br>2<br>[0, 1, 2]<br>3<br>[1, 2, 3]<br>4<br>[2, 3, 4]<br>5<br>[3, 4, 5]<br>6<br>[4, 5, 6]<br>7<br>[5, 6, 7]<br>8<br>[6, 7, 8]<br>9<br>[7, 8, 9]</pre><p>Once we are grasp the aforementioned logic, the remaining problems appear relatively straightforward to solve.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb67b8a46ca0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[YARN, Spark Cluster Tuning]]></title>
            <link>https://medium.com/@surajnathani/yarn-spark-cluster-tuning-2c461247754b?source=rss-a2454c74033a------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c461247754b</guid>
            <category><![CDATA[hadoop]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[yarn]]></category>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Suraj Nathani]]></dc:creator>
            <pubDate>Sun, 21 Jan 2024 02:06:56 GMT</pubDate>
            <atom:updated>2024-01-21T02:06:56.638Z</atom:updated>
            <content:encoded><![CDATA[<p>Throughout my extensive experience in managing Spark clusters and fine-tuning their performance, I’ve identified key parameters crucial for the seamless operation of Spark clusters. One can say that 9 out of 10 issues with Spark performance can be resolved by tweaking one or more of these parameters. When dealing with performance issues, take a thorough look if one of these parameters are the cause.</p><p>In a gist, the below are a list of the parameters:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*TpqsJDVzLdQRGrvnyHLuLQ.png" /></figure><p><strong>1. executor.memory:</strong> When you submit a Spark application, it is divided into tasks that are distributed across the cluster to be executed in parallel. Each executor is responsible for running a subset of these tasks. The executor memory is the memory allocated to each of these executor processes.</p><p><strong>2. executor.core:</strong> Each executor is allocated a certain number of CPU cores, and these cores determine the parallelism with which tasks can be executed.</p><p><strong>3. spark.memory.offHeap.enabled:</strong> When set to true, Spark uses off-heap memory to store certain data structures, reducing the pressure on the Java heap and potentially improving the memory management of Spark applications.</p><p><strong>4. spark.memory.offHeap.size:</strong> Specifies the amount of off-heap memory to allocate for Spark’s off-heap storage. When spark.memory.offHeap.enabled is set to true to enable off-heap memory usage, you can use spark.memory.offHeap.size to control the size of the allocated off-heap memory.</p><p><strong>5. spark.task.cpus:</strong> Specifies the number of CPU cores to allocate for each task in a Spark application.</p><p><strong>6. spark.submit.deployMode:</strong> In client mode, the driver program (the main Spark application) runs on the machine where you submit the Spark job<br>In cluster mode, the driver program runs on one of the worker nodes in the Spark cluster.</p><p><strong>7. spark.port.maxRetries:</strong> When a job is submitted to Spark, it tried to bind the job UI to a particular port, starting from 4040. If there are other jobs submitted, port 4041 may already be occupied, and Spark will try to bind to consecutive ports. If there are 32 applications already running, job submission will fail because 32 attempts will be exhausted, trying to bind to 32 ports which are already in use. Make sure to increase this number to a higher value if the number of simultaneous jobs expected on the cluster is high</p><p><strong>8. spark.scheduler.mode:</strong> Determines the scheduling mode for executing tasks in a Spark application.</p><p><strong>9. spark.task.maxFailures:</strong> Determines the maximum number of task failures allowed before the entire Spark job is considered as failed</p><p><strong>10. spark.dynamicAllocation.enabled:</strong> Allows Spark to dynamically adjust the number of executor instances based on the workload.</p><p><strong>11. spark.dynamicAllocation.initialExecutors:</strong> When you set spark.dynamicAllocation.enabled to true to enable dynamic allocation, you can also configure spark.dynamicAllocation.initialExecutors to control the initial number of executor instances when the application starts.</p><p><strong>12. spark.sql.broadcastTimeout:</strong> When executing a SQL query that involves a broadcast join, Spark might decide to broadcast a smaller DataFrame to all worker nodes to optimize the join operation. The spark.sql.broadcastTimeout parameter determines the maximum amount of time Spark will spend trying to broadcast the smaller DataFrame.</p><p><strong>13. spark.sql.files.maxPartitionBytes:</strong> Controls the size of partitions created during the reading process, which can impact the parallelism and efficiency of Spark SQL queries.</p><p><strong>15. spark.speculation:</strong> When spark.speculation is set to true, Spark will identify tasks that are running significantly slower than their expected completion time and launch speculative copies of those tasks on other executor nodes.</p><p><strong>16. yarn.nodemanager.pmem-check-enabled:</strong> Determines whether the NodeManager should perform physical memory checks to ensure that the allocated container’s memory usage does not exceed the specified physical memory limits on the node.</p><p><strong>17. yarn.nodemanager.vmem-check-enabled:</strong> Determines whether the NodeManager should perform virtual memory checks to ensure that the allocated container’s virtual memory usage does not exceed the specified virtual memory limits on the node.</p><p><strong>18. yarn.nodemanager.resource.memory-mb:</strong> Specifies the total physical memory (RAM) available for each NodeManager in a YARN-managed cluster, measured in megabytes.</p><p><strong>19. yarn.scheduler.minimum-allocation-mb:</strong> Sets the minimum amount of memory (in megabytes) that can be allocated to a container by the resource scheduler.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c461247754b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding RPC (vs REST API)]]></title>
            <link>https://medium.com/@surajnathani/understanding-rpc-vs-rest-api-d6dd092cb1fc?source=rss-a2454c74033a------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6dd092cb1fc</guid>
            <category><![CDATA[grpc]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[rpc]]></category>
            <dc:creator><![CDATA[Suraj Nathani]]></dc:creator>
            <pubDate>Mon, 03 Jul 2023 18:41:21 GMT</pubDate>
            <atom:updated>2023-07-03T18:41:21.687Z</atom:updated>
            <content:encoded><![CDATA[<p>Within the context of Server-Client Architecture, there are various communication protocols that exist.<br>The two main forms of Server-Client mode of communication is REST API and RPC. I’ll be ellaborating on RPC in this blog</p><p>REST API protocol works on a mechanism of a Web Server hosting multiple HTTP endpoints that serve execute different functions when the endpoints are called.</p><p>RPC (Remote Procedure Call) protocol is a communication protocol that allows one computer program to request function from another program on a different computer over a network. Remote stands for distant. Procedure stands for Function. Call stands for Request. It is similar to REST API, except that instead of HTTP endpoints being called/requested, Procedures/Functions are called/requested.</p><p>There are several types of RPC (Remote Procedure Call) protocols, including:</p><ol><li>Simple Object Access Protocol (SOAP): SOAP is a protocol for exchanging structured data over HTTP. It is an XML-based protocol that is commonly used for web services.</li><li>XML-RPC: XML-RPC is a simpler protocol than SOAP that uses XML to encode its messages. It is also commonly used for web services.</li><li>JSON-RPC: JSON-RPC is a protocol that uses JSON (JavaScript Object Notation) to encode its messages. It is a lightweight alternative to SOAP and XML-RPC.</li><li>gRPC: gRPC is a high-performance RPC protocol that uses Protocol Buffers (a binary serialization format) for message encoding. It is commonly used in modern microservices architectures.</li><li>CORBA (Common Object Request Broker Architecture): CORBA is a more complex RPC protocol that supports multiple programming languages and platforms. It is commonly used in enterprise applications.</li></ol><p>Below are examples of some of the above protocols:</p><ol><li>SOAP-RPC:</li></ol><p>SOAP (Simple Object Access Protocol) RPC protocol in Python using the suds library</p><p>Install the suds library using pip:</p><pre>pip install suds-jurko</pre><p>Create a WSDL (Web Services Description Language) file that defines a simple calculator service:</p><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;definitions xmlns=&quot;http://schemas.xmlsoap.org/wsdl/&quot; xmlns:soap=&quot;http://schemas.xmlsoap.org/wsdl/soap/&quot; xmlns:tns=&quot;http://example.com/calculator&quot; targetNamespace=&quot;http://example.com/calculator&quot;&gt;<br>    &lt;types&gt;<br>        &lt;xsd:schema xmlns:xsd=&quot;http://www.w3.org/2001/XMLSchema&quot; targetNamespace=&quot;http://example.com/calculator&quot;&gt;<br>            &lt;xsd:element name=&quot;add&quot;&gt;<br>                &lt;xsd:complexType&gt;<br>                    &lt;xsd:sequence&gt;<br>                        &lt;xsd:element name=&quot;x&quot; type=&quot;xsd:float&quot;/&gt;<br>                        &lt;xsd:element name=&quot;y&quot; type=&quot;xsd:float&quot;/&gt;<br>                    &lt;/xsd:sequence&gt;<br>                &lt;/xsd:complexType&gt;<br>            &lt;/xsd:element&gt;<br>            &lt;xsd:element name=&quot;addResponse&quot;&gt;<br>                &lt;xsd:complexType&gt;<br>                    &lt;xsd:sequence&gt;<br>                        &lt;xsd:element name=&quot;result&quot; type=&quot;xsd:float&quot;/&gt;<br>                    &lt;/xsd:sequence&gt;<br>                &lt;/xsd:complexType&gt;<br>            &lt;/xsd:element&gt;<br>        &lt;/xsd:schema&gt;<br>    &lt;/types&gt;<br>    &lt;message name=&quot;addRequest&quot;&gt;<br>        &lt;part name=&quot;parameters&quot; element=&quot;tns:add&quot;/&gt;<br>    &lt;/message&gt;<br>    &lt;message name=&quot;addResponse&quot;&gt;<br>        &lt;part name=&quot;parameters&quot; element=&quot;tns:addResponse&quot;/&gt;<br>    &lt;/message&gt;<br>    &lt;portType name=&quot;CalculatorPortType&quot;&gt;<br>        &lt;operation name=&quot;add&quot;&gt;<br>            &lt;input message=&quot;tns:addRequest&quot;/&gt;<br>            &lt;output message=&quot;tns:addResponse&quot;/&gt;<br>        &lt;/operation&gt;<br>    &lt;/portType&gt;<br>    &lt;binding name=&quot;CalculatorBinding&quot; type=&quot;tns:CalculatorPortType&quot;&gt;<br>        &lt;soap:binding style=&quot;rpc&quot; transport=&quot;http://schemas.xmlsoap.org/soap/http&quot;/&gt;<br>        &lt;operation name=&quot;add&quot;&gt;<br>            &lt;soap:operation soapAction=&quot;http://example.com/calculator/add&quot;/&gt;<br>            &lt;input&gt;<br>                &lt;soap:body use=&quot;encoded&quot; namespace=&quot;http://example.com/calculator&quot; encodingStyle=&quot;http://schemas.xmlsoap.org/soap/encoding/&quot;/&gt;<br>            &lt;/input&gt;<br>            &lt;output&gt;<br>                &lt;soap:body use=&quot;encoded&quot; namespace=&quot;http://example.com/calculator&quot; encodingStyle=&quot;http://schemas.xmlsoap.org/soap/encoding/&quot;/&gt;<br>            &lt;/output&gt;<br>        &lt;/operation&gt;<br>    &lt;/binding&gt;<br>    &lt;service name=&quot;CalculatorService&quot;&gt;<br>        &lt;port name=&quot;CalculatorPort&quot; binding=&quot;tns:CalculatorBinding&quot;&gt;<br>            &lt;soap:address location=&quot;http://localhost:8000/calculator&quot;/&gt;<br>        &lt;/port&gt;<br>    &lt;/service&gt;<br>&lt;/definitions&gt;</pre><p>This code defines a WSDL document that describes a CalculatorService service with a CalculatorPort endpoint. The service contains a single add operation that takes two float arguments and returns their sum.</p><p>Next, we’ll use the suds library to generate Python code from the WSDL file:</p><pre>from suds.client import Client<br><br>url = &#39;http://localhost:8000/calculator?wsdl&#39;<br>client = Client(url)<br><br>result = client.service.add(3.0, 4.0)<br>print(result)</pre><p>This code creates a SOAP client that connects to the CalculatorService endpoint using the URL <a href="http://localhost:8000/calculator?wsdl">http://localhost:8000/calculator?wsdl</a>. We then call the `add` function to add the two numbers.</p><p>2. XML-RPC</p><p>Using the RPC protocol in Python using the xmlrpc module in the Python standard library.</p><p>Create a server program that provides a function for adding two numbers:</p><pre>from xmlrpc.server import SimpleXMLRPCServer<br><br>def add(x, y):<br>    return x + y<br><br>server = SimpleXMLRPCServer((&#39;localhost&#39;, 8000))<br>server.register_function(add, &#39;add&#39;)<br>server.serve_forever()</pre><p>This code creates a simple XML-RPC server that listens on port 8000 of the local host. The add function takes two arguments and returns their sum. We then register the add function with the server and start the server loop.</p><p>Now, let’s create a client program that calls the add function on the server:</p><pre>import xmlrpc.client<br><br>server = xmlrpc.client.ServerProxy(&#39;http://localhost:8000&#39;)<br><br>result = server.add(3, 4)<br>print(result)</pre><p>This code creates a client that connects to the server running on the local host, using the ServerProxy class from the xmlrpc.client module. We then call the add method on the server object with the arguments 3 and 4, and print the result. The output should be 7, which is the sum of 3 and 4 computed by the server.</p><p>3. JSON-RPC:<br>JSON-RPC protocol in Python using the jsonrpcclient library:</p><p>Create a server program that provides a function for adding two numbers:</p><pre>from jsonrpcserver import method, serve<br><br>@method<br>def add(x, y):<br>    return x + y<br><br>serve(&#39;localhost&#39;, 8000)</pre><p>This code creates a simple JSON-RPC server that listens on port 8000 of the local host. The add function takes two arguments and returns their sum. We then use the serve function from the jsonrpcserver module to start the server loop.</p><p>Create a client program that calls the add function on the server:</p><pre>from jsonrpcclient import request<br><br>response = request(&#39;http://localhost:8000&#39;, &#39;add&#39;, x=3, y=4)<br><br>if response.is_error:<br>    print(response.error)<br>else:<br>    print(response.result)</pre><p>This code creates a client that connects to the server running on the local host, using the request function from the jsonrpcclient module. We then call the add method on the server object with the named arguments x and y, and print the result. The output should be 7, which is the sum of 3 and 4 computed by the server.</p><p>That’s a simple example of how you can use the JSON-RPC protocol in Python using the jsonrpcclient library.</p><p>4. COBRA</p><p>CORBA (Common Object Request Broker Architecture) protocol in Python using the omniORB library:</p><p>Create an IDL (Interface Definition Language) file that defines a simple interface for a calculator:</p><pre>module calculator {<br>    interface Calculator {<br>        float add(in float x, in float y);<br>        float subtract(in float x, in float y);<br>        float multiply(in float x, in float y);<br>        float divide(in float x, in float y);<br>    };<br>};</pre><p>This code defines a module called calculator that contains an interface called Calculator. The interface has four methods: add, subtract, multiply, and divide, each taking two float arguments and returning a float.</p><p>Next, we’ll use the omniidl tool from the omniORB library to generate Python stubs from the IDL file:</p><pre>omniidl -bpython calculator.idl</pre><p>This will generate several Python files, including calculator_idl.py, which contains the Python stubs for the Calculator interface.</p><p>Create a server program that implements the Calculator interface:</p><pre>import calculator_idl<br>from omniORB import CORBA, PortableServer<br><br>class CalculatorImpl(calculator_idl.Calculator):<br>    def add(self, x, y):<br>        return x + y<br><br>    def subtract(self, x, y):<br>        return x - y<br><br>    def multiply(self, x, y):<br>        return x * y<br><br>    def divide(self, x, y):<br>        return x / y<br><br>orb = CORBA.ORB_init()<br>poa = orb.resolve_initial_references(&quot;RootPOA&quot;)<br><br>calculator = CalculatorImpl()<br><br>poa.activate_object(calculator)<br>objref = poa.servant_to_reference(calculator)<br><br>ior = orb.object_to_string(objref)<br><br>with open(&quot;calculator.ior&quot;, &quot;w&quot;) as f:<br>    f.write(ior)<br><br>poa._get_the_POAManager().activate()<br>orb.run()</pre><p>This code creates a server that implements the Calculator interface by defining a class called CalculatorImpl. We then use the ORB_init function from the omniORB library to initialize the ORB (Object Request Broker), and the resolve_initial_references function to obtain a reference to the RootPOA (Portable Object Adapter), which is used to manage objects in the server.</p><p>We create an instance of the CalculatorImpl class, activate it with the POA (Portable Object Adapter), and convert it to a CORBA object reference using the servant_to_reference function. We then convert the object reference to a string and write it to a file called calculator.ior, which is used by the client to connect to the server.</p><p>Finally, we activate the POA manager and start the ORB event loop using the run method.</p><p>Now, let’s create a client program that uses the Calculator interface:</p><pre>import calculator_idl<br>from omniORB import CORBA<br><br>with open(&quot;calculator.ior&quot;, &quot;r&quot;) as f:<br>    ior = f.read()<br><br>orb = CORBA.ORB_init()<br>objref = orb.string_to_object(ior)<br><br>calculator = objref._narrow(calculator_idl.Calculator)<br><br>result = calculator.add(3.0, 4.0)<br>print(result)</pre><p>This code reads the object reference from the calculator.ior file, initializes the ORB, and converts the object reference back to a CORBA object using the string_to_object function.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6dd092cb1fc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Binary search in bi minutes]]></title>
            <link>https://medium.com/@surajnathani/binary-search-in-bi-minutes-5554af484492?source=rss-a2454c74033a------2</link>
            <guid isPermaLink="false">https://medium.com/p/5554af484492</guid>
            <category><![CDATA[binary-search]]></category>
            <category><![CDATA[data-structures]]></category>
            <category><![CDATA[search-algorithm]]></category>
            <dc:creator><![CDATA[Suraj Nathani]]></dc:creator>
            <pubDate>Sat, 24 Jun 2023 17:15:19 GMT</pubDate>
            <atom:updated>2023-06-24T17:15:19.572Z</atom:updated>
            <content:encoded><![CDATA[<p>I always thougt binary search was named that way because it took only two minutes to learn it.</p><p>The binary search algorithm involves the below steps</p><ol><li>Check if the middle element = target element. Return if positive.</li><li>If not, half the array based on a condition.</li><li>Repeat from Step 1</li><li>Add a part to stop the recursion</li></ol><p>Let’s try to code the same, in parts</p><p>We’ll use multiple arrays to understand the below:</p><p>Array 1: [2,5,7,8,10,11] | Target_element = 7</p><p>Array 2: [1,2,3,4,5] | Target_element = 5</p><p>Array 3: [11,15,20,21,89,97,99] | Target_element = 89</p><p>Step1 : Operation: Check if middle element is equal to target element</p><pre>def binary_search():<br>    middle_index = (left_index + right_index)//2<br>    middle_element = array[middle_index]<br>    if middle_element == target_element:<br>      return middle_index</pre><p>This operation is the core of the binary search. The rest of the code is support statements.</p><p>Note the return in the above function. This function will always return a value, so a value always has to be collected when this function is called.</p><p>Considering the examples above:</p><p>Array 1: [2,5,7,8,10,11] | Middle_index = 2 | Middle_element = 7. Return</p><p>Array 2: [1,2,3,4,5] | Middle_index = 2 | Middle_element = 3</p><p>Array 3: [11,15,20,21,89,97,99] | Middle_index = 3| Middle_element = 21</p><p>Step2: Half the array based on a condition.</p><p>We’ll take one step at a time.</p><p>2.1: Understand the condition for breaking the array to half</p><pre>if middle_element &gt; target_element:<br>    binary_search(first_half) # index 0 to index mid-1 = left array<br>else:<br>    binary_search(second_half) # mid to len(array) = right array</pre><p>First half = left_index(index 0) to middle_index-1</p><p>The first half goes from 0 to mid-1, because we have to use this array to calculate the middle element (and not use this array for iteration)</p><p>Right array = middle_index+1 to right_index</p><p>2.2: Understand the condition for breaking the array to half</p><pre>if middle_element &gt; target_element:<br>    binary_search(left_index, middle_index - 1, array, target_element)<br>else:<br>    binary_search(middle_index + 1, right_element, array, target_element)</pre><p>Array 2: [1,2,3,4,5] | Half = index 3 to index 4</p><p>Array 3: [11,15,20,21,89,97,99] | Half = index 4 to index 6</p><p>2.3: We have to add a return to this function (as mentioned in Step 1):</p><pre>if middle_element &gt; target_element:<br>    return binary_search(left_index, middle_index - 1, array, target_element)<br>else:<br>    return binary_search(middle_index + 1, right_element, array, target_element)</pre><p>Step 3: Repeat</p><p>Combine the above steps</p><pre>def binary_search(left_index, right_index, array, target_element):<br>    middle_index = (left_index + right_index)/2<br>    middle_element = arr_1[middle_index]<br>    if middle_element == target_element:<br>      return middle_index<br><br>    if middle_element &gt; target_element:<br>        return binary_search(left_index, middle_index - 1, array, target_element)<br>    else:<br>        return binary_search(middle_index + 1, right_element, array, target_element)</pre><p>Step 4: Add a part to stop the recursion</p><p>Since we half the array in every iteration, there comes a point when only one element is present in the array. After this point, the array becomes negative. Consider the below</p><p>Size of array = 8</p><p>Iteration 1: [0–3, {4}, 5–8]</p><p>Iteration 2: [0–1, {2}, 3, | 5, {6}, 7–8 |]</p><p>Iteration 3: [0–0 {0} 0–0, | 1–1 {1} 1–1, | 3–3 {3} 3–3, | 5–5 {5} 5–5, | 7–7 {7} 7–7, |8–8 {8} 8–8 |]</p><p>Iteration 4: [ +1- -1, 2–0, 4–2, 6–4, 8–6, 9–7]</p><p>At Iteration 4, since left_index &gt; right_index, this array does not make sense and we stop.</p><p>Add a None at the end, since nothing is received if ‘target_element’ is not found.</p><pre>def binary_search(left_index, right_index, array, target_element):<br>   if right_index &gt;= left_index:<br>      middle_index = (left_index + right_index) // 2<br>      middle_element = array[middle_index]<br>      if middle_element == target_element:<br>       return middle_index<br>    <br>      if middle_element &gt; target_element:<br>       return binary_search(left_index, middle_index - 1, array, target_element)<br>      else:<br>       return binary_search(middle_index + 1, right_index, array, target_element)<br>    return None<br>  <br>arr_1 = [1,2,3,4,5]<br>target_element = 4<br>print(binary_search(0, len(arr_1)-1, arr_1, target_element))</pre><p>That’s all, folks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5554af484492" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recur(Recur(Recursion)sion)sion): Part 2]]></title>
            <link>https://medium.com/@surajnathani/recur-recur-recursion-sion-sion-part-2-45a43e77e9b5?source=rss-a2454c74033a------2</link>
            <guid isPermaLink="false">https://medium.com/p/45a43e77e9b5</guid>
            <dc:creator><![CDATA[Suraj Nathani]]></dc:creator>
            <pubDate>Tue, 21 Mar 2023 23:08:44 GMT</pubDate>
            <atom:updated>2023-03-21T23:08:44.684Z</atom:updated>
            <content:encoded><![CDATA[<p>Several efforts have been made to present an explanation for recursion. For anyone who finds it difficult to imagine the idea of Recursion, this post may lead you to enlightment.</p><p>Approach 1: <a href="https://medium.com/@surajnathani/recur-recur-recursion-sion-sion-3ae05ba2cbbc">https://medium.com/@surajnathani/recur-recur-recursion-sion-sion-3ae05ba2cbbc</a></p><p>Approach 2:</p><p>This is one of the easiest ways to apply when solving a problem that requires Recursion.</p><p>We will try this approch with the below three problems</p><ol><li>Print integers from 1 to 5</li><li>Print integers from 5 to 0</li><li>Sum of odd elements of a list</li></ol><p>Let’s start with Problem 1: Print integers from 1 to 5</p><p>To start with, try to code the “Operation” part of the above problem. The “Operation” to implement in this Problem is “Print integer”. Here it is:</p><pre>def print_func_1():<br>  print(0)</pre><p>Now, since this is a Recursion problem, we need to call the same function within itself. So how do we do this? We define a new function with a different name, but with the same body. Here it is:</p><pre>def print_func_1():<br>  print(0)<br>  print_func_2()<br><br>def print_func_2():<br>  print(1)</pre><p>Repeat the above process. Multiple different functions, same body. As below:</p><pre>def print_func_1():<br>  print(0)<br>  print_func_2()<br><br>def print_func_2():<br>  print(1)<br>  print_func_2()<br><br>def print_func_3():<br>  print(1)<br>  print_func_4()<br><br>def print_func_4():<br>  print(1)<br>  print_func_5()<br><br>def print_func_5():<br>  print(1)</pre><p>This has to be halted at a point, so we stop at “print_func_5” above</p><p>Now, let’s add parameters and arguments to the above function.</p><p>How to determine what parameters to add? All you do is look at the variables that change in the flow, and add them as parameters. In this case, “integer” variable is the one that changes. Demonstrated below:</p><pre>integer = 0<br>def print_func_1(integer):<br>  print(integer)<br>  integer = integer + 1<br>  print_func_2(integer)<br><br>def print_func_2(integer):<br>  print(integer)<br>  integer = integer + 1<br>  print_func_2(integer)<br><br>def print_func_3(integer):<br>  print(integer)<br>  integer = integer + 1<br>  print_func_4(integer)<br><br>def print_func_4(integer):<br>  print(integer)<br>  integer = integer + 1<br>  print_func_5()<br><br>def print_func_5(integer):<br>  print(integer)</pre><p>In the last function, since we need a Stopping point, we add a condition. As below:</p><pre>def print_func_5(integer):<br>  print(integer)<br>  if integer == 5:<br>    print(&quot;End of function&quot;)<br>    return</pre><p>Now, we try to merge all elements of the above functions: Init, increment, parameters, End:</p><pre>integer = 0                  # Init<br>def print_func(integer):      <br>  print(integer)              # Operation<br>  integer = integer + 1       # Integer<br>  if integer == 5:            # End<br>    print(&quot;End of function&quot;)<br>    return</pre><p>To summarize, when solving a Recursion problem:</p><ol><li>Create one function, with the minimum Operation implemented by the function</li><li>Create multiple duplicates of the above function. All of them implement the same Operation, although on different values.</li><li>Determine:</li></ol><p>3.1. The parameters (dynamic variables) and pass them from one function to another</p><p>3.2 The End of the function.</p><p>4. Re-check the Operation part of the function body to add or modify any statements.</p><p>4. Merge all the functions.</p><p>This approach should take not more than 15–20 minutes to solve easy to moderate problems with Recursion.</p><p>Let’s try the next problem.</p><p>2. Print integers from 5 to 0</p><p>There are two ways to solve this problem</p><p>a. Start with 5, print, decrement 1, close function.</p><p>b. Start with 0, increment 1, when 5 is reached, print before end of function, end function. This second approach is what we will implement below.</p><p>As per the steps listed above in this post for solving Recursive problems ,</p><ol><li>Start with the body of one function:</li></ol><pre>def print_func_1(integer):<br>  if integer &lt; 5:          # If variable is not 5, increment to reach 5<br>    integer = integer + 1</pre><p>2. Duplicate the same function and call Recursively:</p><pre>def print_func_1():<br>  if integer &lt; 5:          # If variable is not 5, increment to reach 5<br>    integer = integer + 1<br>    print_func_2()<br><br>def print_func_2():<br>  if integer &lt; 5:          # If variable is not 5, increment to reach 5<br>    integer = integer + 1<br>    print_func_3()  <br>def print_func_3():<br>  if integer &lt; 5:          # If variable is not 5, increment to reach 5<br>    integer = integer + 1<br>    print_func_4()<br># and so on..</pre><p>3. Determine the parameters and End of function.</p><p>In this case, the parameters is the “integer” that is incrementing</p><p>The End of function is the “less than” condition, which is already present:</p><pre>integer = 0<br>def print_func_1(integer):<br>  if integer &lt; 5:          # If variable is not 5, increment to reach 5<br>    integer = integer + 1<br>    print_func_2(integer)<br><br>def print_func_2(integer):<br>  if integer &lt; 5:          # If variable is not 5, increment to reach 5<br>    integer = integer + 1<br>    print_func_3(integer) <br> <br>def print_func_3(integer):<br>  if integer &lt; 5:          # If variable is not 5, increment to reach 5<br>    integer = integer + 1<br>    print_func_4(integer)<br># and so on..</pre><p>4. Re-check the Operation part of the function body to add or modify any statements:</p><pre>integer = 0                # Init<br>def print_func_1(integer):<br>  if integer &lt; 5:          # If variable is not 5, increment to reach 5<br>    integer = integer + 1  # Increment<br>    print_func_2(integer)  # Recursive call<br>  print(Integer)           # Operation<br><br>def print_func_2(integer):<br>  if integer &lt; 5:          # If variable is not 5, increment to reach 5<br>    integer = integer + 1<br>    print_func_3(integer)  <br>  print(Integer)<br><br>def print_func_3(integer):<br>  if integer &lt; 5:          # If variable is not 5, increment to reach 5<br>    integer = integer + 1<br>    print_func_4(integer)<br>  print(Integer)<br><br>def print_func_4(integer):<br>  if integer &lt; 5:          # If variable is not 5, increment to reach 5<br>    integer = integer + 1<br>    print_func_5(integer)<br>  print(Integer)<br><br>def print_func_5(integer):<br>  if integer &lt; 5:          # If variable is not 5, increment to reach 5<br>    integer = integer + 1<br>    print_func_5(integer)<br>  print(Integer)</pre><p>In the above approach, we do manage to reach Integer 5, but we have nothing printed yet. That is because we were supposed to print AFTER we reach 5, so we add the “print” Operation:</p><p>5. Merge the functions to have one Recursive function:</p><pre>integer = 0                # Init<br>def print_func(integer):<br>  if integer &lt; 5:          # If variable is not 5, increment to reach 5<br>    integer = integer + 1  # Increment<br>    print_func(integer)    # recursive call<br>  print(Integer)           # print, after 5 is reached.</pre><p>The last problem</p><p>3. Sum of odd elements of a list, recursively.</p><p>That’s on you to try.</p><p>Taking a Break now, more explanations on Recursion coming soon.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=45a43e77e9b5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recur(Recur(Recursion)sion)sion): Part 1]]></title>
            <link>https://medium.com/@surajnathani/recur-recur-recursion-sion-sion-3ae05ba2cbbc?source=rss-a2454c74033a------2</link>
            <guid isPermaLink="false">https://medium.com/p/3ae05ba2cbbc</guid>
            <category><![CDATA[recursion-python]]></category>
            <category><![CDATA[recursive]]></category>
            <category><![CDATA[recursion]]></category>
            <category><![CDATA[data-structures]]></category>
            <category><![CDATA[recursive-function]]></category>
            <dc:creator><![CDATA[Suraj Nathani]]></dc:creator>
            <pubDate>Mon, 20 Mar 2023 21:29:48 GMT</pubDate>
            <atom:updated>2023-03-20T21:34:49.215Z</atom:updated>
            <content:encoded><![CDATA[<p>Several efforts have been made to provide an explanation for recursion. For everyone who find it difficult to imagine the idea of Recursion, this post may lead you to the root of Recursion</p><p>Multiple ways will be presented to visualize the notion of Recursion. Here’s the first take on it.</p><p><strong>Approach One</strong>: Recursion is For loops on steroids.</p><p><strong>Explanation:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iV3zpLEYZxi9KUVNG-qnZA.png" /></figure><p>Let’s understand the above with a slightly relevant example:</p><p>Problem: Sum of first 5 natural numbers with FOR LOOP vs RECURSION</p><ol><li>For loop:</li></ol><pre>for i in range(0,5): # Init, End, Increment<br>  sum = sum + i.     # Operation</pre><p>2. Recursion:</p><pre>init = 0              # Init<br>func_sum(init): <br>   sum = sum + init   # Operation<br>   if init == 5:      # End<br>     return            <br>   init = init + 1    # Increment<br><br>func_sum(init)</pre><p>To summarize, every loop has the below five main parts</p><p>1. Init</p><p>2. End</p><p>3. Increment</p><p>4. Operation</p><p>5. Break, if required</p><p>Any problem that can be solved with For loop, can be solved with Recursion as well. The sequence of the above five elements in the right order gives you the correct solution in Recursion. As problems become more complex, the above five parts have to be placed in the correct order to get the desired answer.</p><p>Taking a Break now, more explanations on Recursion coming soon.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3ae05ba2cbbc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simplifying context managers in python]]></title>
            <link>https://faun.pub/simplifying-context-managers-in-python-5bfe47f86977?source=rss-a2454c74033a------2</link>
            <guid isPermaLink="false">https://medium.com/p/5bfe47f86977</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[context-managers]]></category>
            <dc:creator><![CDATA[Suraj Nathani]]></dc:creator>
            <pubDate>Fri, 17 Jun 2022 06:19:58 GMT</pubDate>
            <atom:updated>2022-06-17T06:19:58.860Z</atom:updated>
            <content:encoded><![CDATA[<p>Lets start with an example of a context manager.</p><p>This is code without a context manager:</p><pre>f = open(&quot;test_file&quot;, &quot;w&quot;)<br>f.write(&quot;Test&quot;)<br>f.close()</pre><p>Converting it to context manager in python,</p><pre>with open(&quot;test_file&quot;, &quot;a&quot;) as f:<br>   f.write(&quot;test_write_from_context_manager\n&quot;)</pre><p>Lets create a custom context manager.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*-b__2pI9ZexAcNrfXDW33A.png" /></figure><p>Using the 4 step process in the diagram above above, any code can be converted in context manager.</p><p>Context managers can be used in situations when the code has the below structure:</p><ol><li>Open/Setup/Enter</li><li>Execute/Implement</li><li>Close/Teardown/Exit</li></ol><p>Examples of context managers:</p><ol><li>Open, read, close file</li><li>Open connection of database, execute query, close connection of database</li><li>Login, get data, logout</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*tcNZJi5GIXb6Bp7x.png" /></figure><h4>If this post was helpful, please click the clap 👏 button below a few times to show your support for the author 👇</h4><h4>🚀Developers: Learn and grow by keeping up with what matters, <a href="https://faun.to/8zxxd">JOIN FAUN.</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5bfe47f86977" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/simplifying-context-managers-in-python-5bfe47f86977">Simplifying context managers in python</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simplifying context managers in python]]></title>
            <link>https://medium.com/@surajnathani/simplifying-context-managers-in-python-37d87fa485e2?source=rss-a2454c74033a------2</link>
            <guid isPermaLink="false">https://medium.com/p/37d87fa485e2</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[context-managers]]></category>
            <category><![CDATA[file-operations]]></category>
            <dc:creator><![CDATA[Suraj Nathani]]></dc:creator>
            <pubDate>Sun, 12 Jun 2022 08:25:31 GMT</pubDate>
            <atom:updated>2022-06-12T08:25:31.834Z</atom:updated>
            <content:encoded><![CDATA[<p>Lets start with an example of a context manager.</p><p>This is code without a context manager:</p><pre>f = open(&quot;test_file&quot;, &quot;w&quot;)<br>f.write(&quot;Test&quot;)<br>f.close()</pre><p>Converting it to context manager in python,</p><pre>with open(&quot;test_file&quot;, &quot;a&quot;) as f:<br>   f.write(&quot;test_write_from_context_manager\n&quot;)</pre><p>Lets create a custom context manager.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*-b__2pI9ZexAcNrfXDW33A.png" /></figure><p>Using the 4 step process in the diagram above above, any code can be converted in context manager.</p><p>Context managers can be used in situations when the code has the below structure:</p><ol><li>Open/Setup/Enter</li><li>Execute/Implement</li><li>Close/Teardown/Exit</li></ol><p>Examples of context managers:</p><ol><li>Open, read, close file</li><li>Open connection of database, execute query, close connection of database</li><li>Login, get data, logout</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=37d87fa485e2" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>