<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Subex AI Labs - Medium]]></title>
        <description><![CDATA[We take Artificial Intelligence very seriously! Subex AI Labs leverages the latest and greatest in the field of AI and applies them to solve business challenges in the digital world. Our data scientists love massive and complex datasets, and this space is to showcase their work! - Medium]]></description>
        <link>https://medium.com/subex-ai-labs?source=rss----321f3b8a1fd1---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Subex AI Labs - Medium</title>
            <link>https://medium.com/subex-ai-labs?source=rss----321f3b8a1fd1---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 17:29:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/subex-ai-labs" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Working with Redis Streams in Python-Basic]]></title>
            <link>https://medium.com/subex-ai-labs/working-with-redis-streams-in-python-basic-30f97055f61a?source=rss----321f3b8a1fd1---4</link>
            <guid isPermaLink="false">https://medium.com/p/30f97055f61a</guid>
            <category><![CDATA[redis-stream]]></category>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Anirudh Pratap Singh]]></dc:creator>
            <pubDate>Tue, 08 Aug 2023 11:21:10 GMT</pubDate>
            <atom:updated>2023-08-08T11:21:10.494Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Optional:</strong></p><p><strong>What is redis?</strong> Follow this <a href="https://medium.com/better-programming/getting-started-with-redis-a-python-tutorial-3a18531a73a6">link</a> and get started with some python basics on different types of Redis datatypes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*h5TurAevLEnwc5Pf.png" /><figcaption>Redis Streaming data</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dil9-gZuB1ZcYnTVfjWSMg.jpeg" /><figcaption>Just an image for data storage :)</figcaption></figure><p>A Redis database has many data-types to store information in. A few of these include strings, lists, hashes, streams, etc. In this article we will spend some considerable time with Redis streams.</p><h4>Setting-up Redis</h4><p>We will use <a href="https://medium.com/@knoldus/docker-basic-understanding-and-terminology-7f240d9faf75">Docker</a> for our case to setup Redis server and database.</p><pre># Runs a docker container with lightweight redis <br>$ docker run -d -p 6005:6379 --name your_redis_name redis:alpine3.17  redis-server --bind 0.0.0.0<br><br># This gets us inside the container where we can connect via CLI <br>$ docker exec -it your_redis_name redis-cli</pre><p>The first code snippet above spins off a Redis container with the image redis:alpine3.17. The image being used is lightweight, feel free to use any image of your choosing. The second code snippet is to get into the Redis container and get into the command line interface(CLI) to communicate with the Redis database</p><h4><strong>Why Use Redis Stream</strong></h4><p>Redis stream is a unique data type that is used to tackle the deadlock situation. The deadlock situation arises when two or more transactions access the database at the same time. This can happen when multiple process are producing data and want to store in the same database at the same time instance. To circumvent this problem Redis streams comes to the rescue. The Redis stream gives a unique ID to each entry of the stream (more will be talked about later).</p><p>For example, suppose there are multiple weather data sensors that access the same database to dump their data into the same key at the same time. In this situation a deadlock occurs and this may lead to data loss. The same example of temperature and humidity sensors will be used to simulate weather in Bangalore.</p><h4><strong>Working With Redis Streams</strong></h4><p>Having setup the Redis database an object needs to be created to work with it in a python script. For this the following command needs to be used to make an object.</p><pre>import redis<br><br># Note the port 6005 has been arbitrarily chosen. Default port is 6379. <br># Can use any port number<br><br>redisCli = redis.Redis(host=host_ip, port=6005)</pre><p>For demonstration we will add some fake data for weather in Bangalore. Data to be added to Redis stream is done with XADD command.</p><pre># Example of adding data using Redis CLI<br><br>[In] 127.0.0.1:6379&gt; XADD weather_blr * temperature 24 humidity 62</pre><p>The ‘*’ after the stream name refers to unique ID generation from Redis. One can give own ID but, it is recommended to let Redis do it. The same data can be added to the stream using python as follows(used loop to add mutiple data in same milli-second).</p><pre># Simulating a multiple producers<br><br>[In]for _ in range(10):<br>      data = {&quot;temperature&quot;:random.randint(20,25),<br>              &quot;humidity&quot;: random.randint(60,80)}<br>      redisCli.xadd(&quot;weather_blr&quot;,data)</pre><p>This adds stream key “weather_blr” to the Redis database. “temperature” and “humidity” are the fields for each of the stream.</p><h3><strong>How to consume the data added to the stream</strong></h3><p>Moving forward two code snippets will be given, one for Redis CLI and the other for python command. The output will be common for both.</p><p><em>XLEN (returns the length of the stream)</em></p><pre>[In]127.0.0.1:6379&gt; XLEN weather_blr<br>[Out](integer) 10</pre><pre>[In]print(redisCli.xlen(&quot;weather_blr&quot;))<br>[Out] 10</pre><p><em>XRANGE (query the stream data)</em></p><pre>[In] 127.0.0.1:6379&gt; XRANGE weather_blr - + COUNT 2<br>[Out]  1) 1) &quot;1690179048588-0&quot;<br>       2) 1) &quot;temperature&quot;<br>          2) &quot;24&quot;<br>          3) &quot;humidity&quot;<br>          4) &quot;65&quot;<br>       2) 1) &quot;1690179048588-1&quot;<br>       2) 1) &quot;temperature&quot;<br>          2) &quot;20&quot;<br>          3) &quot;humidity&quot;<br>          4) &quot;64&quot;</pre><pre>[In] print(redisCli.xrange(&quot;weather_blr&quot;, max=&quot;+&quot;, min=&quot;-&quot;, count=2))<br>[Out] [(b&#39;1690179048588-0&#39;, {b&#39;temperature&#39;: b&#39;24&#39;, b&#39;humidity&#39;: b&#39;65&#39;}), (b&#39;1690179048588-1&#39;, {b&#39;temperature&#39;: b&#39;20&#39;, b&#39;humidity&#39;: b&#39;64&#39;})]</pre><p>The First ID “1690179048588–0” is the unique ID generated for each entry of the stream. The part before the hyphen is the timestamp in millisecond. It can be seen that there are two entries with same millisecond stamp. The number after hyphen is the serial ID which can be 64bit number, practically making it impossible to have deadlock situation. The ‘b’ before each entry in the output refers to bit storage type in Redis database.</p><p>The ‘-’ and ‘+’ denotes the smallest and largest ID in the “weather_blr” stream. The COUNT gives number of entries to show in the stream starting from the first one.</p><p><em>XREVRANGE(query the data in reverse)</em></p><p>Queries the data in the reverse order. This is very useful especially in the case of accessing the last item in the stream.</p><pre>[In] 127.0.0.1:6379&gt; XREVRANGE weather_blr + - COUNT 1<br>[Out] 1) 1) &quot;1690179048589-6&quot;<br>      2) 1) &quot;temperature&quot;<br>         2) &quot;25&quot;<br>         3) &quot;humidity&quot;<br>         4) &quot;68&quot;</pre><pre>[In] print(redisCli.xrevrange(&quot;weather_blr&quot;, max=&quot;+&quot;, min=&quot;-&quot;, count=1))<br>[Out] [(b&#39;1690179048589-6&#39;, {b&#39;temperature&#39;: b&#39;25&#39;, b&#39;humidity&#39;: b&#39;68&#39;})]</pre><p>The command is very similar to XRANGE and carry the same meaning except in reverse order.</p><p><em>XREAD(consuming the streaming data)</em></p><p>This is the best way to consume streaming data. This is extremely useful when producer continuously produces data. To read the latest entry into the stream the following command needs to be written.</p><pre>[In] 127.0.0.1:6379&gt; XREAD COUNT 2 BLOCK 5000 STREAMS weather_blr $<br>[Out](nil)<br>     (5.06s)</pre><pre>[In] print(redisCli.xread({&quot;weather_blr&quot;:&#39;$&#39;}, count = None, block = 5000))<br>[Out] []</pre><p>The XREAD command reads the streaming data.</p><p>The parameter COUNT reads that many entries that come in the same millisecond after the command is executed. If only 1 entry comes from the producer side then it reads the single entry. If multiple entries do come</p><p>If the producer doesn’t produce information in the next millisecond after the command goes then it waits for 5s for the stream to be updated using the BLOCK parameter. Since, in our case the producer wasn’t working thus, it shows nil output.</p><p>The ‘$’ sign after the stream name is to indicate after which timestamp does the consumer needs to get the information.</p><p><strong>Working with the gathered data</strong></p><p>This small section will help with working with the gathered data from Redis database</p><pre>[In] data =redisCli.xrevrange(&quot;weather_blr&quot;, max=&quot;+&quot;, min=&quot;-&quot;, count=1))<br><br># To get the value for humidity the following code needs to be given<br>[In] print(data[0][1][b&#39;humidity&#39;]<br>[Out] b&#39;68&#39;</pre><p>The ‘b’ refers to the bit encoding of the data that is stored in Redis Database. To convert it for normal usage we want to decode it. The stream data is stored as a list. The first element of the list contains the unique stream ID while the second contains key value pair of data stored. We need to decode this for our usage.</p><pre># Without decoding the data<br>[In] print(data[0][1])<br>[Out] {b&#39;temperature&#39;: b&#39;25&#39;, b&#39;humidity&#39;: b&#39;68&#39;}<br><br># After decoding the data<br>[In] data_decoded = {key.decode(): value.decode() for key, value in data[0][1].items()}<br>[In] print(data_decoded)<br>[Out] {&#39;temperature&#39;: &#39;25&#39;, &#39;humidity&#39;: &#39;68&#39;}</pre><p>Now, one can use the data in normal way without need of using ‘b’ as prefix.</p><p>To know more about complex features used in Redis stream watch the informative video from the Redis creator Salvatore Sanfilippo by following the <a href="https://www.youtube.com/watch?v=qXEyuUxQXZM&amp;t=856s">link</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30f97055f61a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/subex-ai-labs/working-with-redis-streams-in-python-basic-30f97055f61a">Working with Redis Streams in Python-Basic</a> was originally published in <a href="https://medium.com/subex-ai-labs">Subex AI Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Executing Raw and Aggregated Join in Spark Structured Streaming]]></title>
            <link>https://medium.com/subex-ai-labs/executing-raw-and-aggregated-join-in-spark-structured-streaming-466fa2b0c63a?source=rss----321f3b8a1fd1---4</link>
            <guid isPermaLink="false">https://medium.com/p/466fa2b0c63a</guid>
            <category><![CDATA[join]]></category>
            <category><![CDATA[spark-streaming]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Shubham Bathwal]]></dc:creator>
            <pubDate>Mon, 05 Jun 2023 13:07:54 GMT</pubDate>
            <atom:updated>2023-06-05T13:07:54.790Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/517/1*F6oqK0umxsipqHuzxITRTg.png" /></figure><h3>Introduction to Streaming</h3><p><strong>What is Stream Processing?</strong></p><p>Stream processing refers to a method of handling and analyzing data in real-time as it is continuously generated or received. It involves the processing of data streams in a sequential and continuous manner, rather than storing and processing data in batches or as static datasets.</p><p>Stream processing systems are designed to handle high-volume, high-velocity data streams from various sources such as sensors, social media feeds, logs, financial transactions, and more. These systems allow for the immediate processing, transformation, and analysis of the data as it flows, enabling real-time insights and decision-making.</p><p>Stream processing involves several key components, including data ingestion, data processing, and data output. Data is ingested from the stream source, processed using algorithms, filters, aggregations, or other operations, and then delivered to one or more destinations, such as databases, dashboards, or other applications.</p><p><strong>Why Streaming Processing ?</strong></p><p>Processing streaming data offers several advantages and opportunities compared to traditional batch processing. Here are some reasons why processing streaming data is beneficial:</p><blockquote><strong>Real-time insights: </strong>Streaming data processing enables real-time or near-real-time analysis of data as it is generated. This allows organizations to gain immediate insights and make timely decisions based on the most up-to-date information.</blockquote><blockquote><strong>Rapid detection and response:</strong> Streaming data processing facilitates the quick identification of critical events or anomalies. By continuously analyzing incoming data in real-time, organizations can promptly detect and respond to issues, such as fraud, security breaches, system failures, or any other events that require immediate attention. This proactive approach can help mitigate risks and minimize the impact of potential problems.</blockquote><blockquote><strong>Improved scalability:</strong> Stream processing systems can be designed to scale horizontally, meaning they can handle increasing data volumes by distributing the workload across multiple processing units. This scalability allows organizations to accommodate growing data streams and handle high data throughput effectively.</blockquote><blockquote><strong>Enhanced situational awareness: </strong>By processing streaming data in real-time, organizations can gain a better understanding of current situations and trends. They can monitor and analyze data as it unfolds, allowing for more accurate and timely decision-making. This situational awareness is valuable in various domains, such as finance, logistics, cybersecurity, IoT, and many others.</blockquote><blockquote><strong>Personalized and real-time experiences:</strong> Streaming data processing enables organizations to deliver personalized and dynamic experiences to users in real-time. For example, in e-commerce or digital advertising, real-time analysis of user behavior and preferences can be used to deliver personalized recommendations or targeted advertisements instantly.</blockquote><p>Overall, processing streaming data offers the advantage of real-time insights, rapid detection and response, continuous processing, scalability, enhanced situational awareness, and the ability to provide personalized and real-time experiences. These benefits make stream processing invaluable in numerous use cases across industries.</p><blockquote><strong>Streaming Use Cases:</strong></blockquote><blockquote>· Fraud detection</blockquote><blockquote>· Real-time stock trades</blockquote><blockquote>· Marketing, sales, and business analytics</blockquote><blockquote>· Customer/user activity</blockquote><blockquote>· Log Monitoring: Troubleshooting systems, servers, devices, and more</blockquote><blockquote>· Ride share matching</blockquote><p><strong>Real Life Example</strong></p><p>When a passenger calls Lyft, real-time streams of data join to create a seamless user experience. Through this data, the application pieces together real-time location tracking, traffic stats, pricing, and real-time traffic data to simultaneously match the rider with the best possible driver, calculate pricing, and estimate time to destination based on both real-time and historical data.</p><blockquote><strong>Challenges in Stream Processing:</strong></blockquote><blockquote>· Scalability</blockquote><blockquote>· Ordering</blockquote><blockquote>· Consistency and Durability</blockquote><blockquote>· Fault Tolerance &amp; Data Guarantees</blockquote><h3><strong>Spark Structured Stream</strong></h3><p>Structured Streaming is built on top of the Spark SQL engine, which is a distributed SQL query engine that allows you to query large datasets using SQL syntax. Structured Streaming extends the Spark SQL engine to support real-time streaming data processing by providing a unified API for batch and streaming data processing.</p><p>Spark Structured Streaming works by dividing the data stream into micro-batches and processing each batch using Spark’s batch processing engine. This allows Structured Streaming to process data streams in near real-time without having to write complex streaming code.</p><blockquote><strong>Why Spark Structured Streaming?</strong></blockquote><blockquote>·<strong> Ease of use :</strong> Developers can use SQL-like queries to process real-time data streams</blockquote><blockquote>·<strong> Scalability : </strong>Spark Structured Streaming is built on top of the Spark SQL engine, which is a distributed SQL query engine that can scale to handle large datasets</blockquote><blockquote>· <strong>Fault-tolerant : </strong>Spark Structured Streaming is fault-tolerant, which means that it can handle failures without losing data.</blockquote><blockquote>· <strong>Integration : </strong>Spark Structured Streaming integrates with a wide range</blockquote><h3><strong>Join Raw and Aggregated Streams</strong></h3><blockquote>Raw Stream : Input Data Stream</blockquote><blockquote>Aggregated Stream : Window Based Aggregation on Input Stream.</blockquote><p><strong>Why join Raw and Aggregated Streams?</strong></p><p>When using Spark Structured Streaming or similar stream processing frameworks, it is common to aggregate data based on a look-back window or sliding window interval. This aggregation helps in summarizing or condensing the data to derive useful statistics or metrics. However, the aggregated data alone may lack context or details about individual transactions or records.</p><p>By joining the raw data stream with the aggregated data stream, we can combine the summarized information with the corresponding original data. This allows us to obtain transaction or record-level results that provide a comprehensive understanding of the data.</p><p>Therefore, the relationship between the raw and aggregated streams is crucial for obtaining transaction-level details and gaining a more comprehensive understanding of the data.</p><p><strong>Example Use Case:</strong></p><p>Let’s consider a scenario where we have a raw stream of sales transactions and an aggregated stream that provides the total sales amount per product category for a specific time window. If we only had the aggregated data, we would know the total sales amount for each category, but we wouldn’t have insights into which individual transactions contributed to those totals. By joining the raw and aggregated streams, we can determine which specific sales transactions contributed to the aggregated amounts, enabling us to analyze the performance of individual products or identify patterns in customer behavior.</p><p><strong>Way to join input and agregated stream in Spark Strucutred Streaming join</strong></p><p>Input and Aggregated stream can be joined in spark structured streaming in append mode.</p><blockquote><strong>Understanding the modes :</strong></blockquote><blockquote>Spark Structured Stream provides ways to process the input stream.</blockquote><blockquote><strong>Complete </strong>: Entire result will be outputted after every trigger.</blockquote><blockquote><strong>Append </strong>: Only the new rows will be outputted after every trigger.</blockquote><blockquote><strong>Update </strong>: Only the updated result since the last trigger will be outputted.</blockquote><p><strong>Shortcoming of Append Output Mode:</strong></p><p><em>Joining two stream is only available for append mode</em>. But it has certain limitations.</p><p>In append mode the output for a records comes only once when spark is sure that the lifetime of the record is completed and it is not going to be updated in future. This time will be sliding interval plus the watermark duration — so there will be a delay of this duration in outputting the actual data.</p><h3><strong>Our Approach:</strong></h3><p><strong>Terminologies -</strong></p><p><strong>Foreach Batch:</strong> The foreachbatch operations allow you to apply arbitrary operations and writing logic on the output of a streaming query. This allows you to define a function that is executed on the output data of every micro-batch of a streaming query.</p><p><strong>Check Point:</strong> Checkpoint helps build fault-tolerant and resilient Spark applications. In Spark Structured Streaming, it maintains intermediate state on HDFS compatible file systems to recover from failures.</p><blockquote><strong>Checkpoint Directory Folder Structure :</strong></blockquote><blockquote>1. Commits — It maintains batch wise commits.</blockquote><blockquote>2. Offsets — The offset files contain information about the position of the last processed record in each partition of the input data source. This information is used to resume processing from the last checkpoint in case of a failure or a restart. Applicable in case of Kafka</blockquote><blockquote>3. Source — The offset files contain information about the position of the last processed record in each partition of the input data source. This information is used to resume processing from the last checkpoint in case of a failure or a restart. Applicable in case of file store</blockquote><blockquote>4. State — The state files contain the state of the streaming job, such as the state of the aggregations or the state of the window operations.</blockquote><blockquote>5. Metadata — The metadata files contain information about the data sources, schema, and other metadata related to the streaming job.</blockquote><p><strong>Architecture Diagram:</strong></p><figure><img alt="Join Raw and Aggregated Stream — Architectural Diagram" src="https://cdn-images-1.medium.com/max/961/1*oFxRtTKxEnbcNS9NH1j64Q.png" /></figure><p><strong>Flow:</strong></p><p>After Aggregating the values, we call the foreach batch function with output mode as update, Inside the foreach batch function we do the following things:</p><ol><li>Get the batch number from the foreach batch function. When we call the foreach batch function by default it passes two arguments — <em>Batch Dataframe</em> and <em>Batch Number.</em></li><li>Based on the batch number we will get the list of files that needs to be read again. From the checkpoint folder we can the list of files read in the current batch either from source or offset folder depending on the data source.<br>Sample File :<br>{“path”:”s3a://simulated-simbox/simulated-simbox/part-13037.parquet”,”timestamp”:1670486999000,”batchId”:665}{“path”:”s3a://simulated-simbox/simulated-simbox/part-13038.parquet”,”timestamp”:1670487000000,”batchId”:665}</li><li>Parse the file list and then read the data from the source as a batch dataframe inside the foreach batch function.</li><li>Apply the transformations again that was applied on streaming data before the aggregations.</li><li>Join the raw data frame with the aggregated streaming data frame and drop the duplicates.</li><li>Apply transformations on the joined data frame and write it to sink.</li></ol><p><strong>Using Hypersense AI to do stream operations:</strong></p><p>HyperSense AI, a powerful AI platform, offers advanced functionality specifically designed for processing streaming data and joining streams, revolutionizing data-driven decision-making. With its comprehensive set of features, Hypersense AI enables businesses to effortlessly incorporate real-time insights into their decision-making processes. By seamlessly integrating Hypersense AI into the flow of processing streaming data, organizations can efficiently aggregate, transform, and analyze data streams in real-time. Moreover, Hypersense AI provides robust capabilities for joining streams, allowing businesses to combine multiple data sources, perform complex join operations, and extract valuable insights from the unified view of the data. This seamless integration empowers organizations to gain immediate insights, make timely decisions, and uncover actionable intelligence from streaming data. With Hypersense AI’s powerful streaming data processing and joining capabilities, businesses can unlock new opportunities and gain a competitive advantage in the dynamic landscape of data-driven decision-making.</p><p><strong>About Hypersense AI:</strong></p><p>HyperSense AI is a cutting-edge AI Orchestration platform from Subex that provides the following benefits to our customers</p><ul><li>Orchestrate E2E machine learning model building, deployment and ML Ops, at least 3X faster than traditional methods</li><li>No Code platform requiring no knowledge of any programming languages to build ML models</li><li>Fully automated Auto ML capability for even business users to do simple AI/prediction tasks, thereby democratising AI adoption within enterprises</li><li>Explainability, What-if analysis, counterfactual analysis to ensure model transparency and remove bias ensuring that you can operationalize AI models with full confidence</li><li>Can be deployed on any Hyperscaler cloud platform, or on-prem. Built using distributed computing architecture and micro services for high scalability and support for huge volumes of batch and streaming data processing and analytics.</li><li>Subex professional services with deep telco knowledge to solve complex problems using AI/ML</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=466fa2b0c63a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/subex-ai-labs/executing-raw-and-aggregated-join-in-spark-structured-streaming-466fa2b0c63a">Executing Raw and Aggregated Join in Spark Structured Streaming</a> was originally published in <a href="https://medium.com/subex-ai-labs">Subex AI Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kohonen Self-Organizing Maps]]></title>
            <link>https://medium.com/subex-ai-labs/kohonen-self-organizing-maps-5e790fe65934?source=rss----321f3b8a1fd1---4</link>
            <guid isPermaLink="false">https://medium.com/p/5e790fe65934</guid>
            <category><![CDATA[self-organizing-map]]></category>
            <category><![CDATA[kohonen]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[worst.ai]]></dc:creator>
            <pubDate>Mon, 05 Jun 2023 13:02:00 GMT</pubDate>
            <atom:updated>2023-06-05T13:02:00.382Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong>Introduction</strong></blockquote><p>Kohonen Self-Organizing Maps (SOMs) are a type of artificial neural network used in machine learning and data analysis. The SOM algorithm is a type of unsupervised learning technique that is used to cluster and visualize high-dimensional data in a low-dimensional space. It is referred to as a neural network that is trained by competitive learning.</p><p>Competitive learning is a type of unsupervised learning technique used in artificial neural networks. It is based on the idea of competition between neurons in the network, where each neuron attempts to become the most active or “winning” neuron in response to a given input. Competitive learning can be used for a variety of tasks, such as pattern recognition, clustering, and feature extraction.</p><blockquote><strong>Architecture of KSOM</strong></blockquote><p>A Kohonen Self-Organizing Map consists of a single layer linear 2D grid of neurons. The nodes do not know the values of their neighbors. The architecture of Kohonen Self-Organizing Maps (KSOM) consists of a grid of neurons arranged in a two-dimensional lattice. Each neuron in the grid is connected to the input layer and receives input signals from the input data. The neurons in the grid are arranged in a way that preserves the topology of the input space, which means that neighboring neurons in the grid are more likely to respond to similar input data. The weights of links are updated as a function of the given inputs. However, all the nodes on the grid are directly linked to the input vectors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/489/0*_twIft8vX1jpr8dE.jpg" /><figcaption>Architecture of a Kohonen Self-Organizing Map</figcaption></figure><blockquote><strong>Algorithm of KSOM</strong></blockquote><p>The Kohonen SOM algorithm can be summarized in the following steps:</p><ol><li>Initialize the weights of the neurons in the grid to random values.</li><li>Present an input data to the network.</li><li>Calculate the activation level of each neuron in the grid in response to the input data.</li><li>Select the neuron with the highest activation level as the winning neuron.</li><li>Update the weights of the winning neuron and its neighbors, using a learning rate and a neighborhood function that decrease with distance from the winning neuron.</li><li>Repeat steps 2–5 for multiple iterations, or until convergence is reached.</li></ol><p>The resulting weight vectors of the neurons in the grid can be visualized as a low-dimensional representation of the high-dimensional input data.</p><blockquote><strong>Advantages of KSOM</strong></blockquote><p>Kohonen Self-Organizing Maps (KSOM) have several advantages that make them useful for a wide range of applications, including:</p><ol><li><strong>Nonlinear dimensionality reduction</strong>: KSOMs can be used to represent high-dimensional data in a low-dimensional space, while preserving the topological relationships between the data points. This can help to reveal underlying patterns and structure in the data, which may not be apparent in the high-dimensional space.</li><li><strong>Unsupervised learning:</strong> KSOMs are a type of unsupervised learning technique, which means that they do not require labeled data for training. This makes them useful for tasks where labeled data is not available or is too expensive to obtain.</li><li><strong>Clustering and visualization:</strong> KSOMs can be used for clustering and visualization of complex data. The resulting low-dimensional representation of the data can be used to identify clusters and patterns in the data, which can be useful for exploratory data analysis and data mining.</li><li><strong>Robustness to noise:</strong> KSOMs are relatively robust to noise and can still perform well even if the input data contains some level of noise or errors.</li><li><strong>Easy interpretation:</strong> The output of a KSOM can be easily visualized and interpreted, which can be useful for identifying trends and patterns in the data, and for communicating the results to others.</li><li><strong>Flexibility:</strong> KSOMs can be adapted to a wide range of data types, including continuous, discrete, and categorical data.</li></ol><blockquote><strong>Disadvantages of KSOM</strong></blockquote><p>While Kohonen Self-Organizing Maps (KSOM) have many advantages, there are also some limitations and disadvantages to using this technique, including:</p><ol><li><strong>Sensitivity to initial conditions:</strong> The performance of a KSOM can be sensitive to the initial conditions of the network, such as the initial weights of the neurons in the grid. This means that different initializations can result in different final solutions, and it may be necessary to run the algorithm multiple times to obtain a stable solution.</li><li><strong>Computational complexity:</strong> The computational complexity of KSOMs can be high, particularly for large datasets and complex network architectures. This can make training and testing the network time-consuming and computationally expensive.</li><li><strong>Difficulty in determining the optimal network size:</strong> Choosing the optimal network size, or the number of neurons in the grid, can be difficult and is often a trial-and-error process. Using too few neurons can result in poor representation of the input data, while using too many neurons can lead to overfitting.</li><li><strong>Limited to low-dimensional data:</strong> KSOMs are typically used for dimensionality reduction of high-dimensional data. However, the performance of KSOMs may degrade as the dimensionality of the input data increases, making them less effective for very high-dimensional datasets.</li><li><strong>Limited interpretability:</strong> While the output of a KSOM can be easily visualized, interpreting the resulting clusters or patterns in the data can be difficult. The meaning of the clusters or patterns may be unclear, and it may be necessary to combine KSOMs with other techniques to gain a deeper understanding of the data.</li></ol><p>To summarize, Kohonen SOMs can be used for image and signal processing, text and data mining, and bioinformatics. They have also been used to analyze gene expression data, identify patterns in social network data, and classify images based on visual features.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e790fe65934" width="1" height="1" alt=""><hr><p><a href="https://medium.com/subex-ai-labs/kohonen-self-organizing-maps-5e790fe65934">Kohonen Self-Organizing Maps</a> was originally published in <a href="https://medium.com/subex-ai-labs">Subex AI Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine Learning 101]]></title>
            <link>https://medium.com/subex-ai-labs/machine-learning-101-5fa20edbbc95?source=rss----321f3b8a1fd1---4</link>
            <guid isPermaLink="false">https://medium.com/p/5fa20edbbc95</guid>
            <category><![CDATA[intro-to-machine-learning]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[supervised-learning]]></category>
            <dc:creator><![CDATA[Rea]]></dc:creator>
            <pubDate>Fri, 02 Jun 2023 07:13:07 GMT</pubDate>
            <atom:updated>2023-06-02T07:13:07.853Z</atom:updated>
            <content:encoded><![CDATA[<p>Machine learning is a buzzword everyone sees floating around the news, but do you actually know what it is? It’s not something you should know but rather something worth knowing. Ever thought about the movie recommendations you get on Netflix or Amazon Prime? That’s machine learning right there. Machine learning is like having a super-smart assistant that learns from data to make predictions or decisions. It’s the technology behind personalized movie recommendations, voice assistants, and self-driving cars. Even if you aren’t a tech expert, knowing about machine learning could be interesting as it provides insights into the cutting-edge technologies shaping our world.</p><p>So, what is machine learning? Machines don’t learn the same way humans do. Machines learn through algorithms that analyze data, whereas humans learn through cognition, allowing us to generalize and apply knowledge to new situations. Machines, on the other hand, are sensitive to variations in inputs that differ from what they were trained on, which can lead to inaccurate outcomes since machines lack the broader contextual understanding humans possess.</p><p>Machine learning encompasses three types of learning: supervised, unsupervised, and reinforcement learning. For now, let’s dive into supervised and unsupervised learning. But in case you’re curious, reinforcement learning is like teaching a computer using a reward system; it learns by taking actions, receiving feedback, and adjusting its behavior to maximize rewards over time.</p><p>1. <strong>Supervised learning</strong>: Supervised learning algorithms learn from labeled examples. What do we mean by this? Suppose you have a collection of different fruits and want a computer to recognize each fruit. In that case, you would train the system with pictures of these fruits together with their corresponding <em>labels,</em> e.g., “apple”, “orange”, “kiwi”, etc. By repeatedly exposing the model to many labeled examples, it learns to identify patterns and relationships between pictures and their labels. Eventually, it can correctly classify new, unseen images of fruits on its own, even without explicitly being told of the labels. Supervised learning problems are of two types: regression and classification. Regression algorithms are used when the target (output) is a real value, such as housing prices in a pricing prediction problem. Classification algorithms are used when the target is a categorical value, such as “spam” or “not spam”, in email spam filtering.</p><p>2. <strong>Unsupervised learning: </strong>These learning algorithms are used to discover underlying patterns in the input data without being provided with explicit labels. Unsupervised learning algorithms have no prior knowledge about the expected output and try to identify patterns in the data independently; hence the name “unsupervised”. For example, in Google News, the algorithm automatically groups related news articles based on the topics they cover, without anyone telling it which words or issues to look for. Examples of unsupervised learning algorithms are clustering (groups similar data points together), anomaly detection (finding unusual data points, which is helpful in fraud detection), and dimensionality reduction (lets you take an extensive dataset and compress it to a much smaller dataset by reducing the number of features in the dataset while preserving the essential information)</p><p>If this sounds exciting to you, why not get in on the action yourself? Here are some helpful resources to get you started:</p><p>1. <a href="https://www.deeplearning.ai/courses/machine-learning-specialization/">https://www.deeplearning.ai/courses/machine-learning-specialization/</a></p><p>2. <a href="https://codewithmosh.com/p/python-programming-course-beginners">https://codewithmosh.com/p/python-programming-course-beginners</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5fa20edbbc95" width="1" height="1" alt=""><hr><p><a href="https://medium.com/subex-ai-labs/machine-learning-101-5fa20edbbc95">Machine Learning 101</a> was originally published in <a href="https://medium.com/subex-ai-labs">Subex AI Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Adversarial attacks: A detailed review — Part 1]]></title>
            <link>https://medium.com/subex-ai-labs/adversarial-attacks-a-detailed-review-part-1-20fac7ccfc1a?source=rss----321f3b8a1fd1---4</link>
            <guid isPermaLink="false">https://medium.com/p/20fac7ccfc1a</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ethical-ai]]></category>
            <category><![CDATA[adversarial-attack]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Vishal Ranjan]]></dc:creator>
            <pubDate>Mon, 09 Jan 2023 07:01:13 GMT</pubDate>
            <atom:updated>2023-01-09T07:01:13.486Z</atom:updated>
            <content:encoded><![CDATA[<h3>Adversarial attacks: A detailed review — Part 1</h3><p>Deep Learning has proven to be a very efficient tool in recent times when it comes to solve challenging problems across various fields such as Healthcare (computer-aided diagnosis, drug discovery), Finance (fraud detection), Automobiles (self-driving cars, robotics), Media (News Aggregation and Fake News Detection) and other day-to-day utilities (such as virtual assistants, language translation, information extraction)</p><p>However it is now known that Deep Learning is vulnerable to adversarial attacks that can manipulate its predictions by introducing almost indistinguishable perturbations in audio, images or videos. In this series of articles we will understand what is an adversarial attacks and how it tries to manipulate the deep learning models in order to get the desired output from the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/1*Uop7RjuZUAJydCNpmnkmqg.png" /><figcaption>Few examples of adversarial attacks (a) panda misclassified as gibbon by addition of little noise (b) stop sign mis-classified as speed limit sign (c) Person wearing a particular pattern not being detected</figcaption></figure><h3>Outline</h3><p>First we are going to specify the definitions of commonly appearing technical terminologies in publications, and then we will understand what is an adversarial attack and formal definition of the problem statement. In the next section, we will classify the attacks into different categories based on various attributes.</p><h3>Common terminologies and definitions:</h3><p>Here we will try to understand some of the most common terminologies that occur very commonly in papers related to this attack and will also be occurring frequently in this article.</p><p><strong><em>Adversarial example/image -</em> </strong>It<strong> </strong>is an example/image that is intentionally manipulated to cause incorrect model prediction. This example (or a series of such examples) are provided as input to the model.</p><p><strong><em>Adversarial perturbation:</em></strong> It is the component of an adversarial example/image that causes the incorrect prediction. Commonly, it is a low magnitude additive noise-like signal.</p><p><strong><em>Adversary: </em></strong>It is the agent (i.e. the attacker) creating an adversarial example. Alternatively, the adversarial signal/perturbation is also referred to as the adversary, albeit much less often.</p><p><strong><em>Defense/adversarial defense: </em></strong>It<strong><em> </em></strong>is a broader term used for any mechanism of introducing robustness in a model, or external/internal mechanisms to detect adversarial signals, or image processing to negate adversarial effects of input manipulations.</p><p><strong><em>Target image: </em></strong>It is the clean example/image being manipulated by the adversary.</p><p><strong><em>Target label: </em></strong>It is the (desired) incorrect label of the adversarial example. The term is more relevant for classification problems.</p><h3>What is an Adversarial attack?</h3><p>An adversarial attack tries to trick the deep learning model to get the desired output by making the minimum amount of alterations to the input image or extract some useful information from the model using various tactics. The attacker might have varying range of access to the the target deep learning models, its weights and training dataset. Based on the degree of access that the attacker has, attacks can be classified into various categories which have been described in the article later.</p><p>Formally the problem can be describe using the equation below:</p><figure><img alt="Formal problem statement describing advesarial attacks" src="https://cdn-images-1.medium.com/max/1024/1*THVj8-QEFpnjXZ5QmvcFcg.png" /></figure><p>It is a common practice to try and keep the pre-defined scalar threshold (η) to a minimum value, so that difference seems very minimal to human subject. Similarly, most common value of p is one or two, although it is not limited to that.</p><h3>Classification of adversarial attacks</h3><p>A Machine Learning system can be viewed with a generalized data processing pipeline (see Figure below). At inference, (a) input features are collected from sensors or data repositories, (b) processed in the digital domain, (c) used by the model to produce an output, and (d) the output is communicated to an external system or user and acted upon. To illustrate, consider a generic pipeline, autonomous vehicle, and network intrusion detection systems in Figure 1 (middle and bottom). Given that the attack can be of a varying range based on the goals of an adversary and his capabilities in accessing the model and data, we will try to understand various attacks based on their effect on this generic pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QHVDk5uf7cHC6vIBHzgduw.png" /><figcaption>A generic ML system pipeline (with examples)</figcaption></figure><p>Attacks are classified based on three properties: Attack surface, Adversarial capabilities, Adversarial goals.</p><h4>Attack surface</h4><p>Given a pipeline of steps, an attacker can choose to target a particular step (or surface) of the pipeline in order to achieve his/her goal. The main attack scenarios identified by the attack surface are sketched as follows:</p><ol><li><strong><em>Evasion attack</em></strong>: This is the most common type of attack in the adversarial setting. The adversary tries to evade the system by adjusting malicious samples during testing phase. This setting does not assume any influence over the training data.</li><li><strong><em>Poisoning attack:</em></strong> This type of attack, known as contamination of the training data, is carried out at training phase by injecting skillfully crafted samples to poison the system in order to compromise the entire learning process.</li><li><strong><em>Exploratory attack:</em></strong> These attacks do not influence training dataset. Given black‐box access to the model, they try to gain as much knowledge as possible about the learning algorithm of the underlying system and pattern in the training data.</li></ol><h4><strong><em>Adversarial capabilities:</em></strong></h4><p>It refer to the amount of information available to an adversary about the system. We explore the range of attacker capabilities by further dividing them into inference and training phases.</p><ol><li><strong><em>Training phase capabilities: </em></strong>Most of the attacks in the training phase are accomplished by learning, influencing or corrupting the model by direct alteration of the dataset. The attack strategies are broadly classified into the following three categories based on the adversarial capabilities:</li></ol><p>▹ <strong>Data injection: </strong>When adversary cannot access training data or learning algorithm, but has ability to augment a new data to the training set. He can corrupt the target model by inserting adversarial samples into the training. ▹ <strong>Data modification: </strong>The adversary does not have access to the learning algorithm but has full access to the training data. He poisons the training data directly by modifying the data before it is used for training the model ▹<strong> Logic corruption: </strong>The adversary is able to meddle with the learning algorithm. Apparently, it becomes very difficult to design counter strategy against these.</p><p>2. <strong><em>Testing phase capabilities: </em></strong>Adversarial attacks at the testing time do not interfere with the targeted model but rather forces it to produce incorrect outputs. These attacks can be categorized as white‐box or black‐box attack.</p><p>▹<strong><em>White-box attack:</em></strong> In white‐box attack on a machine learning model, an adversary has total knowledge about the model used (e.g. type of neural network along with number of layers, information about the algorithm used in training, parameters (θ) of the fully trained model architecture).The adversary utilizes this information to analyze the feature space where the model might be vulnerable, that is, for which the model has a high error rate. The access to internal model weights for a white‐box attack corresponds to a very strong adversarial attack.</p><p>▹<strong><em>Black-box attack: </em></strong>Black‐box attack assumes no knowledge about the model and uses information about the settings and prior inputs to exploit the model. Black‐box attacks are further subdivided into the three categories: Non‐adaptive black‐box attack, Adaptive black‐box attack, Strict black‐box attack.</p><h4>Adversarial goals</h4><p>Based on what is the objective of adversary, attacks can be divided into following four categories:</p><ol><li><strong>Confidence reduction</strong>: The adversary tries to reduce the confidence of prediction for the target model. For example, a legitimate image of a ‘stop’ sign can be predicted with a lower confidence having a lesser probability of class belongingness.</li><li><strong>Mis‐classification</strong>: The adversary tries to alter the output classification of an input example to some other class. For example, a legitimate image of a ‘stop’ sign will be predicted as any other class different from the class of stop sign.</li><li><strong>Targeted mis‐classification</strong>: The adversary tries to craft the inputs in such a way that the model produces the output of a particular target class.</li><li><strong>Source/target mis‐classification:</strong> The adversary tries to classify a particular input source to a predefined target class. For example, th input image of ‘stop’ sign will be predicted as ‘go’ sign by the classification model.</li></ol><p>All categories and sub-categories of adversarial attack can be sumarised via the flow-chart shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tqvpKGGn-oHHV-V--O0hGw.png" /><figcaption>Flow-chart for types of adversary attack</figcaption></figure><p>Thus, we have studied and understood what an adversarial attack is and how many ways can it be classified based on different properties. In the next parts, we will study what are some of the most common types of attacks and how adversarial attacks can scale beyond the task of image classification to object detection, object tracking, NLP and audio tasks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=20fac7ccfc1a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/subex-ai-labs/adversarial-attacks-a-detailed-review-part-1-20fac7ccfc1a">Adversarial attacks: A detailed review — Part 1</a> was originally published in <a href="https://medium.com/subex-ai-labs">Subex AI Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Preparation Hack to Deal with Duplicate and Unorganized Images]]></title>
            <link>https://medium.com/subex-ai-labs/data-preparation-hack-to-deal-with-duplicate-and-unorganized-images-3d7b1d349e4e?source=rss----321f3b8a1fd1---4</link>
            <guid isPermaLink="false">https://medium.com/p/3d7b1d349e4e</guid>
            <category><![CDATA[data-preprocessing]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[convolutional-network]]></category>
            <category><![CDATA[data-cleaning]]></category>
            <category><![CDATA[data-preparation]]></category>
            <dc:creator><![CDATA[Sanjay G]]></dc:creator>
            <pubDate>Tue, 11 Jan 2022 13:10:53 GMT</pubDate>
            <atom:updated>2022-01-11T13:10:53.566Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MD7ufAbhTjfQETX2" /><figcaption>Photo by <a href="https://unsplash.com/@hjrc33?utm_source=medium&amp;utm_medium=referral">Héctor J. Rivas</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Data preparation is considered one of the most important as well as the most time-consuming steps in building an ML/DL model. This article attempts to help you efficiently solve one of the image data preparation challenges.</p><p><strong>Problem:</strong></p><p>Let’s say, you’re building a deep learning model that needs to be trained on face image data for some application. You specifically want a single face image per person. But unfortunately, you find that the current data you’re given:</p><ol><li>Has got a lot of duplicate face images of the same person.</li><li>These duplicate images are not arranged or organized together. They’re spread out randomly in the data.</li><li>There is no proper naming convention for image names that helps you to identify unique individuals.</li></ol><p>A few reasons why we could end up in this situation could be that the data creation was generic and was not aimed for a specific use case; or that these images were sampled from a video containing multiple and random occurrences of unique individuals.</p><p>What do you do in such cases? It would be very naive and rather a tedious process to manually inspect and remove duplicate images. It would be a near-impossible task if the number of images is huge.</p><p>Let’s see how we can use deep learning to solve this problem!</p><p><strong>Solution Approach:</strong></p><ol><li>Use a pre-trained image classification model such as VGG-16 to extract image feature vectors for all the images.</li><li>Apply a clustering algorithm on these extracted image feature vectors. This will group feature vectors of near-identical or similar images under the same cluster.</li><li>Now, each cluster represents a unique individual. We can easily pick one image per cluster which would get our ultimate job done.</li></ol><p>This logic works because, the images that are similar or even identical, produce feature vectors that lie close together in that feature space. You could also consider these feature vectors as image embedding vectors.</p><p>Let’s look at the above steps in detail with code. The code is kept generic so that you can conveniently use it as per your needs.</p><h4>1. Feature extraction</h4><p>Pre-trained image classification models such as VGG-16 can be easily accessed through Keras. Here’s the code to get the model ready:</p><pre>from keras.applications.vgg16 import VGG16<br>from keras.applications.vgg16 import preprocess_input<br>from keras.preprocessing import image<br>image.LOAD_TRUNCATED_IMAGES = True<br>import numpy as np</pre><pre>model = VGG16(weights=&#39;imagenet&#39;, include_top=False)</pre><p>In the last line, we’ve passed an argument ‘include_top’ with the value ‘False’. This excludes the final classification or fully connected layer. We don’t need this layer since we’re only interested in the features extracted by the previous layers. The classification layer is only intended to make decisions and classify the image based on the features it receives.</p><p>Assume that the variable ‘all_image_paths’ is a list containing the full path for all the images for which you want to extract the features. We iterate through this list to load, preprocess and extract the image features and store them in another list called ‘feature_list’.</p><pre>feature_list = []<br>for image_path in all_image_paths:<br>    img = image.load_img(image_path, target_size=(128, 128))<br>    img_data = image.img_to_array(img)<br>    img_data = np.expand_dims(img_data, axis=0)<br>    img_data = preprocess_input(img_data)<br>    features = np.array(model.predict(img_data))<br>    feature_list.append(features.flatten())</pre><p>Note the ‘target_size’ argument passed to the ‘image.load_img’ function. In this case, it is (128, 128) but you can change this as per your requirements and all the images read, will be resized to the specified dimension before further processing. The ‘model.predict()’ call is what will give us the image features. We then flatten and append them in the ‘feature_list’.</p><h4>2. Clustering</h4><p>Let’s use K-means clustering on all the feature vectors of the images. The value of ‘K’ or the number of clusters will be the number of unique individuals (assuming you already know this). In case, you don’t know this, use the elbow method to determine the optimal value of K, and that will be the number of unique individuals. After that, you can again fit the K-means model with this optimum value of K.</p><pre>from sklearn.cluster import KMeans</pre><pre>kmeans = KMeans(n_clusters=100, random_state=0).fit(np.array(feature_list))</pre><pre>labels = kmeans.labels_</pre><p>In the above code, the variable ‘labels’ is an array that contains the label or cluster ID for each image feature vector. This array preserves the order of the input data given, so the labels correspond, in the same order, to the image paths present in the ‘image_paths’ list defined earlier.</p><p>Now with this information, you can either pick a single image for each cluster-ID, or you may also just choose to store all images under the same cluster-ID in different directories. The goal of separating multiple images of unique individuals is achieved!</p><p>What are some other variations you can try?</p><ol><li>You could also try and extract features from networks designed to perform well for faces.</li><li>In case if the dimension of the resulting feature vector is huge and also in cases where the number of images is huge, use PCA to reduce the dimensions and use these low dimensional vectors for clustering.</li><li>If the elbow method to find the value of K isn’t optimal for you, use other clustering methods like DBSCAN which doesn’t require any prior knowledge of the number of clusters.</li></ol><p>As you might have realized by now that this technique not only applies to face images but any image class in general.</p><p>I hope you found this image data preparation method useful. Do share the article if you liked it. Cheers!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d7b1d349e4e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/subex-ai-labs/data-preparation-hack-to-deal-with-duplicate-and-unorganized-images-3d7b1d349e4e">Data Preparation Hack to Deal with Duplicate and Unorganized Images</a> was originally published in <a href="https://medium.com/subex-ai-labs">Subex AI Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tips and Tricks for Efficient Python Programming]]></title>
            <link>https://medium.com/subex-ai-labs/tips-and-tricks-for-efficient-python-programming-cc13f63cded?source=rss----321f3b8a1fd1---4</link>
            <guid isPermaLink="false">https://medium.com/p/cc13f63cded</guid>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[efficiency]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Sanjay G]]></dc:creator>
            <pubDate>Tue, 10 Aug 2021 14:09:52 GMT</pubDate>
            <atom:updated>2021-08-10T14:09:52.002Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JUz_vzGGKj95psIV" /><figcaption>Photo by <a href="https://unsplash.com/@cdr6934?utm_source=medium&amp;utm_medium=referral">Chris Ried</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Python being one of the most rapidly growing programming languages is best known and loved by developers for its simplicity, vast library support, and improved productivity.</p><p>If you’ve been coding in Python for a while you would’ve come across several tips to write efficient code, right from using list comprehensions to using built-in functions that are highly optimized. This article will look at some more tips you can use right away in your development.</p><p><strong>1. Use generator expressions as an iterator to feed a <em>for</em> loop</strong></p><p>Consider a situation where you need to use a cartesian combination of all elements of two lists or create a list with a huge number of elements and iterate on them in a for loop. In such cases, generator expressions are better since it saves the expense of building the entire list and storing them in memory. They instead yield the items one by one on demand.</p><p>Generator expressions are created similar to list comprehensions but they use parenthesis “()” instead of square brackets. Consider the following illustrative example showing the use of generator expressions:</p><pre>brands = [&quot;ASUS&quot;, &quot;DELL&quot;, &quot;Lenovo&quot;, &quot;HP&quot;, &quot;Acer&quot;, &quot;Microsoft Surface&quot;, &quot;MSI&quot;, &quot;Toshiba&quot;, &quot;Samsung&quot;, &quot;Sony&quot;]</pre><pre>cores = [&quot;i3&quot;, &quot;i5&quot;, &quot;i7&quot;]</pre><pre>for laptop in (str(b)+&quot; &quot;+str(c) for b in brands for c in cores):<br>    # Your code here</pre><p>Let’s compare the memory usage between a list and a generator expression. We’ll use the getsizeof() method in the sys module which gives us the size of an object in bytes.</p><pre>import sys</pre><pre>my_list = [str(b)+&quot; &quot;+str(c) for b in brands for c in cores]<br>print(&quot;Size of my_list: &quot;, sys.getsizeof(my_list))</pre><pre>gen = (str(b)+&quot; &quot;+str(c) for b in brands for c in cores)<br>print(&quot;Size of gen: &quot;, sys.getsizeof(gen))</pre><pre><strong>&quot;&quot;&quot;<br>Output</strong></pre><pre><strong>Size of my_list: 352<br>Size of gen: 128<br>&quot;&quot;&quot;</strong></pre><p>Let’s add another nested for loop to increase the number of elements.</p><pre>my_list = [str(b)+&quot; &quot;+str(c)+&quot; &quot;+str(d) for b in brands for c in cores for d in brands]<br>print(&quot;Size of my_list: &quot;, sys.getsizeof(my_list))</pre><pre>gen = (str(b)+&quot; &quot;+str(c)+&quot; &quot;+str(d) for b in brands for c in cores for d in brands)<br>print(&quot;Size of gen: &quot;, sys.getsizeof(gen))</pre><pre><strong>&quot;&quot;&quot;<br>Output</strong></pre><pre><strong>Size of my_list: 2544<br>Size of gen: 128<br>&quot;&quot;&quot;</strong></pre><p>Clearly, the list uses more memory compared to the generator. As the length of the list increases, so does the memory usage. But this isn’t the case for the generator.</p><p><strong>2. Use <em>setdefault </em>to add new keys to a dictionary</strong></p><p>When you want to add a new key only if it is not present in the dictionary and assign values to it, we would write a logic something like this:</p><pre>if key not in dict_1:              # key search 1 <br>    dict_1[key] = []               # key search 2<br>dict_1[key].append(new_value)      # key search 3</pre><p>The problem with the above logic is that it needs to perform at least two key searches. Two searches (searches 1 and 3) if the key is present and 3 searches (searches 1, 2, and 3) if the key is not present.</p><p>It is possible to achieve the above goal with a single lookup using the <em>setdefault </em>method as shown below.</p><pre>dict_1.setdefault(key, []).append(new_value)</pre><p><strong>3. Use _ and * while unpacking tuple with multiple elements</strong></p><p>We may not be interested in all the values unpacked from a tuple. In such situations, we can use _ as a dummy variable to catch an unnecessary value. This is especially useful when assigning multiple values returned by the function as a tuple.</p><pre># Catch first tuple element with variable a.<br>a, _ = tuple([1, 2])<br>print(&quot;Value of a: &quot;, a)</pre><pre><strong>&quot;&quot;&quot;<br>Output</strong></pre><pre><strong>Value of a: 1<br>&quot;&quot;&quot;</strong></pre><pre># Catch second tuple element with variable b.<br>_, b, _ = tuple([3, 4, 5])<br>print(&quot;Value of b: &quot;, b)</pre><pre><strong>&quot;&quot;&quot;<br>Output</strong></pre><pre><strong>Value of b: 4<br>&quot;&quot;&quot;</strong></pre><p>We can use *&lt;variable_name&gt; to catch excess values as shown below:</p><pre>a, b, *other_values = tuple([1, 2, 3, 4, 5])<br>print(&quot;a = &quot;, a)<br>print(&quot;b = &quot;, b)<br>print(&quot;other_values = &quot;, other_values)</pre><pre><strong>&quot;&quot;&quot;<br>Output</strong></pre><pre><strong>a = 1<br>b = 2<br>other_values = [3, 4, 5]<br>&quot;&quot;&quot;</strong></pre><pre>a, *other_values, b = tuple([1, 2, 3, 4, 5])<br>print(&quot;a = &quot;, a)<br>print(&quot;other_values = &quot;, other_values)<br>print(&quot;b = &quot;, b)</pre><pre><strong>&quot;&quot;&quot;<br>Output</strong></pre><pre><strong>a = 1<br>other_values = [2, 3, 4]<br>b = 5<br>&quot;&quot;&quot;</strong></pre><p><strong>4. Use an ellipsis (…) in slicing notation to access all values in a multi-dimensional object</strong></p><p>Consider a 2-D array ‘a’. Now to select all columns in the 3rd row, we use the familiar syntax a[2, :]. But let’s say we wanted to apply a similar operation on a 5-dimensional array. In that case instead of doing a[3, :, :, :, :], we could simply use ellipses and write a[3, …]. This would indicate that we want to access all values from other dimensions.</p><p><strong>5. Use sets for containment checks</strong></p><p>On many occasions, we are required to do containment checks of the following nature</p><pre>item in item_collections</pre><p>Now if the item_collections holds a large number of values (and also contains repeated values), it is best to create a set out of it and then do the containment check.</p><pre>set_item_collections = set(item_collections)</pre><pre>item in set_item_collections</pre><p>Let’s go ahead and compare the time taken to do these containment checks on a list and a set.</p><pre>from time import perf_counter</pre><pre>my_list = [str(i) for i in range(1000000)]</pre><pre>t1 = perf_counter()<br>res = &quot;45639&quot; in my_list<br>list_time = perf_counter() - t1<br>print(&quot;Time taken on list: &quot;, list_time, &quot;s&quot;)</pre><pre>set_list = set(my_list)<br>t2 = perf_counter()<br>res = &quot;45639&quot; in set_list<br>set_time = perf_counter() - t2<br>print(&quot;Time taken on set: &quot;, set_time, &quot;s&quot;)</pre><pre>print(&quot;Ratio list_time/set_time: &quot;, list_time/set_time)</pre><pre><strong>&quot;&quot;&quot;<br>Output</strong></pre><pre><strong>Time taken on list:  0.0008049289972404949 s<br>Time taken on set:  4.453999281395227e-06 s<br>Ratio list_time/set_time:  180.72050451439426<br>&quot;&quot;&quot;</strong></pre><p>Note the e-06 (order of microseconds) in the time taken on set. Sets are optimized for very fast searches or containment checks. Thus, it will speed up your check. In the above case, it is about 180 times faster! Also, if possible build the set with set comprehension as it is faster than calling the set() method to build the set.</p><p><strong>6. Avoid augmented assignment with immutable datatypes</strong></p><p>The usage of+= and *= is called augmented assignment and is an example of syntactic sugar. This may seem convenient and we may be tempted to use it everywhere. But there is a catch! Let’s look at the following code example which uses augmented assignment for a list and tuple:</p><pre>&gt;&gt;&gt; a = [1, 2, 3]<br>&gt;&gt;&gt; print(id(a))<br><strong>140658173664576        # id of a</strong><br>&gt;&gt;&gt; a += [7, 8, 9]<br>&gt;&gt;&gt; print(id(a))<br><strong>140658173664576 </strong>       <strong># id of a is the same. It is the same object.</strong></pre><pre>&gt;&gt;&gt; b = (4, 5, 6)<br>&gt;&gt;&gt; print(id(b))<br><strong>140658173448768</strong>        <strong># id of b</strong><br>&gt;&gt;&gt; b *= 2<br>&gt;&gt;&gt; print(id(b))<br><strong>140658173561920        # id of b has changed! It is a new object.</strong></pre><p>Did you get the catch? If we have to repeatedly concatenate or add new data it is better to use mutable types such as lists since they simply append the new data in the same memory location or the same object. On the other hand, this is inefficient for immutable types such as tuples since for every new addition of data, there is an overhead for the interpreter to copy the entire target data to create a new object.</p><p><strong>7. Use bisect.insort() to add new elements to a sorted sequence</strong></p><p>It is inefficient and expensive to repeatedly sort a sequence when we add new elements to the same sequence. Because the addition of a new element to the end of the sequence may destroy the sort. The bisect.insort() function allows us to insert elements while retaining the sort. Here’s an illustrative example of the usage:</p><pre>&gt;&gt;&gt; import bisect<br>&gt;&gt;&gt; a = [9, 8, 7, 6]<br>&gt;&gt;&gt; a_sort = sorted(a)<br>&gt;&gt;&gt; print(a_sort)<br><strong>[6, 7, 8, 9]</strong><br>&gt;&gt;&gt; bisect.insort(a_sort, 4)<br>&gt;&gt;&gt; print(a_sort)<br><strong>[4, 6, 7, 8, 9]</strong><br>&gt;&gt;&gt; bisect.insort(a_sort, 7.5)<br>&gt;&gt;&gt; print(a_sort)<br><strong>[4, 6, 7, 7.5, 8, 9]</strong></pre><p><strong>8. Use sets and set operations wherever possible</strong></p><p>By definition, sets are a collection of unique objects. When we quickly want to remove duplicates or get the count of the unique items we can use the set function and pass our list to its constructor:</p><pre>&gt;&gt;&gt; a = [1, 3, 2, 1, 3, 5, 7, 6, 6]<br>&gt;&gt;&gt; set_a = set(a)<br>&gt;&gt;&gt; print(set_a)<br><strong>{1, 2, 3, 5, 6, 7}</strong><br>&gt;&gt;&gt; print(list(set_a))<br><strong>[1, 2, 3, 5, 6, 7]</strong><br>&gt;&gt;&gt; print(len(set_a))<br><strong>6</strong></pre><p>Sets also support mathematical operations such as unions, intersections, and set differences. For two sets a and b, they are respectively used as a|b, a&amp;b, and a-b.</p><p>Adroit use of sets reduces loops and conditional checkings. Which ultimately also results in a more readable code and reduced overall runtime. For example, consider a use case where there is a need to check the number of instances in ‘items’ that occur in a larger collection of all instances ‘collections’. It’s natural for most of us to jump straight in to write a logic something looks like this:</p><pre>count = 0<br>for item in items:<br>    if item in collections:<br>        count += 1</pre><p>But the usage of sets and set intersection makes it simpler:</p><pre>count = len(set(items) &amp; set(collections))</pre><pre># To actually find out the items that are present in the collections</pre><pre>items_in_collection = set(items) &amp; set(sollections)</pre><p>So that’s all for this article! I hope you’ve learned something new and you’d start using them in your daily Python programming.</p><p>Share this article if you found this useful and learned something new.</p><p>Happy coding. Cheers!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc13f63cded" width="1" height="1" alt=""><hr><p><a href="https://medium.com/subex-ai-labs/tips-and-tricks-for-efficient-python-programming-cc13f63cded">Tips and Tricks for Efficient Python Programming</a> was originally published in <a href="https://medium.com/subex-ai-labs">Subex AI Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Normalizing Flows and Its Use Case in Speech Synthesis (Part 2)]]></title>
            <link>https://medium.com/subex-ai-labs/understanding-normalizing-flows-and-its-use-case-in-speech-synthesis-part-2-3e19840e80b5?source=rss----321f3b8a1fd1---4</link>
            <guid isPermaLink="false">https://medium.com/p/3e19840e80b5</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[density-estimation]]></category>
            <category><![CDATA[text-to-speech]]></category>
            <category><![CDATA[normalizing-flow]]></category>
            <dc:creator><![CDATA[Sanjay G]]></dc:creator>
            <pubDate>Wed, 30 Jun 2021 10:47:57 GMT</pubDate>
            <atom:updated>2021-06-30T10:47:57.403Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eHa1O4GtmfFPGHJM" /><figcaption>Photo by <a href="https://unsplash.com/@adigold1?utm_source=medium&amp;utm_medium=referral">Adi Goldstein</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Welcome to the second part of the article on Normalizing Flows and its application! In the first part, we discussed what normalizing flows are along with the math behind it. Check out the first part if you haven’t already.</p><p><a href="https://medium.com/subex-ai-labs/understanding-normalizing-flows-and-its-use-case-in-speech-synthesis-part-1-5f805c2d43ce">Understanding Normalizing Flows and Its Use Case in Speech Synthesis (Part 1)</a></p><p>In this part, let’s see how we can use this concept to generate speech. Specifically, let’s look at a Text-to-Speech (TTS) system which usually has two networks. The first network — Feature Prediction Network (FPN) — takes in the English text(s) and predicts an intermediate feature representation like a spectrogram. The second network — Audio Generation Network (AGN) — uses this spectrogram to generate the audio samples of the speech (the speech for the English text provided as the input). Our focus is on the AGN.</p><p>There have been many deep network architectures proposed for the AGN, but here let’s consider <a href="https://arxiv.org/abs/1811.00002">WaveGlow</a>.</p><p>WaveGlow uses the concept of normalizing flows to obtain the audio samples from the samples drawn from a known distribution — standard Gaussian distribution in this case. The general idea is that, given samples from a known distribution, the samples from the desired distribution can be obtained by applying a series of sufficiently complex invertible transformations to the samples from the known distribution.</p><p>Here, the information about the spectrogram is included within the transformation that the samples undergo. Speaking in terms of probability theory, we predict audio samples from the input samples conditioned on the spectrogram.</p><p>Let ‘x’ and ‘z’ represent audio samples and standard Gaussian samples, respectively related by a sequence of ‘K’ invertible transformations ‘<em>f’. </em>The WaveGlow model is trained by maximizing the log-likelihood of ‘x’ obtained directly using the change of variables formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5Z5LVuYnIgj0ScKwo5LVrQ.png" /></figure><p>In the WaveGlow network, the invertible transforms consist of two layers <em>viz </em>Coupling layer and Invertible 1x1 convolutional layer. Let’s look at each of these layers separately and later see how these are combined in the WaveGlow architecture.</p><h4>Coupling layer</h4><p>The input vector ‘x’ to the coupling layer is split into two halves and the forward transforms are applied on each of these two halves as described by the equations below. The corresponding inverse transformations are also shown.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xgPGEYO64c4k7uYKomsAQg.png" /></figure><p>The first transform is an identity transform and the other one is an affine transform. The parameters <em>s </em>and <em>t </em>are obtained from another function <em>‘m’</em> (described later) which takes input as the first half of ‘x’ and the mel-scaled spectrogram corresponding to the input audio samples. The inverse transforms are possible because xa = ya which results in the same values for s<em> </em>and<em> t.</em></p><h4>Invertible 1x1 convolutional layer</h4><p>This is essentially a matrix transformation. The corresponding forward and inverse transforms are given below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mVpWr1EdwhcnDtiV0hcF0g.png" /></figure><p>The matrix <em>W</em> is initialized to be an orthonormal matrix and QR decomposition is applied on <em>W</em> for easy computation of its inverse.</p><h4>WaveGlow architecture</h4><p>Now that we know both layers of invertible transforms we can easily comprehend the block diagram of WaveGlow shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dqC-emm8pUUE-THW30P3Tg.png" /></figure><p>During the training phase, ‘x’ is the input (audio samples) and ‘z’ is the output. We’re mapping the samples in the audio space to the standard Gaussian space. The input goes through an invertible 1 x 1 convolutional layer and then the resulting output is fed as input to the coupling layer. This operation is called a ‘flow’. In total, there are 6 flows. The output of the coupling layer of the last flow is ‘z’.</p><p>The function <em>m </em>another deep network shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EcLa67lJeumjWOpGi1uXwQ.png" /></figure><p>As mentioned before, it takes the first half of the split in the coupling layer and spectrogram as the input to generate <em>s</em> and <em>t</em> values. These inputs essentially go through separate dilated convolutional layers followed by gated activation functions (a combination of tanh and sigmoid). This output then is given to a 1 x 1 convolutional layer with ReLU activation to get <em>s</em> and <em>t.</em></p><p>Now the final equation of log-likelihood to maximize becomes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hq89O1Fp0cPyRQ5SFWEW-g.png" /></figure><h4>Inference</h4><p>Finally, once the training is complete, we’re ready for inference. During the inference phase, we use the inverse transformation. We’re mapping samples from the standard Gaussian space to the audio space.</p><p>We simply feed in the spectrogram predictions for the given English text(s) from the FPN along with a randomly sampled vector from standard Gaussian to the WaveGlow network. And there you go, the Gaussian samples get transformed into audio samples conditioned on the spectrogram information. And now you have the speech for the input text given!</p><p>I hope now you know what normalizing flows are and you got a fair idea of how they could be applied in deep learning use cases.</p><p>Happy learning, Cheers!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e19840e80b5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/subex-ai-labs/understanding-normalizing-flows-and-its-use-case-in-speech-synthesis-part-2-3e19840e80b5">Understanding Normalizing Flows and Its Use Case in Speech Synthesis (Part 2)</a> was originally published in <a href="https://medium.com/subex-ai-labs">Subex AI Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Druid — Analytics DB for Timeseries Data]]></title>
            <link>https://medium.com/subex-ai-labs/apache-druid-analytics-db-for-timeseries-data-a143856b5fa0?source=rss----321f3b8a1fd1---4</link>
            <guid isPermaLink="false">https://medium.com/p/a143856b5fa0</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[timeseries]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[druid]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Rakshith S]]></dc:creator>
            <pubDate>Mon, 17 May 2021 10:16:32 GMT</pubDate>
            <atom:updated>2021-05-17T10:16:32.542Z</atom:updated>
            <content:encoded><![CDATA[<h3>Apache Druid — Analytics DB for Timeseries Data</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*N62fkiqTz-MqXDqq17kImg.png" /><figcaption>Source: Apache Druid</figcaption></figure><p>In this article, we will have a complete overview of Apache Druid and its salient features, starting from when and where to use Druid, how to deploy it and a deep dive into its architecture. All the details that are required to get a basic idea as to what exactly does Druid architecture looks like, have been covered. It is very much important to understand what’s happening beneath Apache Druid in order to get the maximum benefit to your use case.</p><blockquote><strong>Official definition</strong>:- <strong>Apache Druid</strong> is a real-time analytics database designed for fast slice-and-dice analytics (“<a href="http://en.wikipedia.org/wiki/Online_analytical_processing"><strong>OLAP</strong></a>” queries) on large data sets.</blockquote><p>In simple words, say your business or platform is generating huge amounts of data every second and you are sitting on a goldmine of data. This data by itself is of no value unless we extract some meaningful information and get insights from it, in order to understand the various characteristics of your business. To achieve this, one might have to execute multiple complex queries on this dataset. In most scenarios, while scaling up for large datasets, this will add up to the latency and the load on the servers increases at an exponential rate. In such analytical use cases, <strong>Apache Druid</strong> is one of the best databases available in the market today as it tackles these issues with ease.</p><blockquote><strong>Important Features:-</strong></blockquote><p>Druid is most often used as a database for powering use cases where real-time ingestion, fast query performance, and high uptime are important. Druid’s main value add is to reduce time to insight and action.</p><p>Druid excels in timestamp:- That means the way backend storage and querying are designed is purely based on timestamps, which makes it the best fit for time-related queries.</p><p>Column-oriented:- This means, it can have multiple columns but can scan through only selected columns, unlike traditional row-wise scan.</p><p>In perspective of UI, Used in dashboarding of powerful analytical applications. In perspective of backend, Used for fast aggregations and concurrent API’s executions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*Wc2O5mI91lZ8OtV_1waX8A.png" /><figcaption>Time-Series Chart showing daily trends of online purchases for an E-Commerce use case — Source: CrunchMetrics</figcaption></figure><blockquote><strong>When you should use druid:-</strong></blockquote><ol><li>For data having time component:- Druid is designed and optimized around timestamps and hence everything is built around ingesting and sorting data around timestamps</li><li>If your data has high cardinality data columns</li><li>If most of the queries are group by queries. (Also scan and search is supported)</li><li>You have your own data warehouse and if you are looking for a secondary DB for your analytics</li></ol><blockquote><strong>Cases where Druid is not a right fit:-</strong></blockquote><ol><li>If you need updates on your table rather than inserts</li><li>If your query has huge joins on multiple tables</li><li>If you are looking for a data warehouse, then its not a right fit, because Druid was specifically designed for data analytics and hence loading your raw data into Druid for storage purposes would be a bad idea</li><li>If you have huge data and do not have enough computational hardware(CPU and Memory). Apache Druid has been designed and developed solely with the agenda of getting low latency results and hence it requires more memory and CPUs. As always there is a trade-off between performance and hardware.</li><li>If you need an offline reporting system, and latency is not of concern.</li></ol><blockquote><strong>So, How do you get started with Druid deployment??</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*OmJbIxX8ZIGlpJWQFhkmHw.png" /><figcaption>Source:- lynda.com</figcaption></figure><p>Druid can be deployed as either a process or as a docker. The docker deployment is generally preferred as it gives you the flexibility of microservice architecture and moreover it’s easy to scale up and scale down. An important factor to consider while creating druid deployment is the configuration profiles. All the components in Druid are optimized to work based on your machine’s CPU and memory. So do make sure that you are using the right set of configurations based on your data load and machine specs. And it’s quite easy to pick the right set of configuration profile as Druid provides a bunch of configuration file for specific machine specs. You can pick the right one based on your machine. For single-server deployments following config files are provided by Druid: nano-quickstart, micro-quickstart, small, medium, large, xlarge</p><blockquote><strong>Druid Architecture — Simplified</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*PXN17RMxu-LzUr_ok3pVEA.png" /><figcaption>Source: Apache Druid</figcaption></figure><p>The Druid server in the core of Apache Druid is a unique system, unlike any of its competitors. Druid has multiple processes at the core of its design and each process can be configured and scaled individually.</p><p>To understand why and how druid is so optimal in its functioning, let us review its architecture. Before going into the technical nitty-gritty, let us explore the data lifecycle in druid first.</p><blockquote><strong>Lifecycle of data within Druid:-</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/0*3M25CP8mmgpkZrqA" /></figure><ol><li>Data Ingestion</li><li>Creation of segments for ingested data and storing these segments into deep storage</li><li>Load data from deep storage into historical cache and store as queryable data</li><li>Execute queries on these segments on historical</li><li>Drop segments based on retention rules</li></ol><p>Every component on Druid is given one of the above responsibility(Simple)</p><blockquote><strong>Druid has mainly 3 server types</strong></blockquote><p><strong>1. Master Servers</strong>: Manages data availability and ingestion.</p><p><strong>&gt; Coordinator</strong>:- (Responsibility:- Segments)</p><p>Manages the segment. Coordinator constantly communicates with historical to load/drop segments based on configurations and also create replications.</p><p><strong>&gt; Overload</strong>:- (Responsibility: Data Ingestion tasks)</p><p>The Overlord process is responsible for accepting tasks, coordinating task distribution, creating locks around tasks, and returning statuses to callers and hand them to coordinators.</p><p><strong>2. Query Servers</strong>: Receives queries from external clients and forwards queries to Data servers</p><p><strong>&gt; Brokers</strong>:- (Responsibility:- Queries)</p><p>The Broker is the process to route queries from external clients. It keeps track of where the data is i.e., on which historical process, using zookeeper. This process also merges the result sets from all the individual processes together.</p><p><strong>Question</strong>:- How zookeeper knows where the data is?</p><p>On startup, Historical processes announce themselves about the segments they are serving in Zookeeper</p><p>&gt; <strong>Routers</strong>:- (Responsibility:- API gateway in front of Druid Brokers, Overlords, and Coordinators. Also runs Druid console). The Apache Druid Router process can be used to route queries to different Broker processes. Druid Console is hosted by the Router process.</p><p><strong>3. Data Server</strong>: Executes ingestion jobs and stores all queryable data.</p><p>&gt; <strong>Historical</strong>:- (Responsibility:- Storing queryable data)</p><p>The data stored on historical is immutable</p><p><strong>&gt; MiddleManager</strong>:- (Responsibility: Ingesting data)</p><p>The MiddleManager process is a worker process that executes submitted tasks. Middle Managers forward tasks to Peons that run in separate JVMs. The reason we have separate JVMs for tasks is for resource and log isolation. Each Peon is capable of running only one task at a time, however, a MiddleManager may have multiple Peons.</p><p>Middle managers take data and analyze, build aggregations, build indexes, partition data, include dictionary data and write to segments.</p><p><strong>External Dependencies:-</strong></p><p><strong>4. Metadata storage:- </strong>(Responsibility: Stores metadata about druid system components)</p><p>Druid uses relational database like “postgres/mysql/Derby” to store various metadata about the system, but not to store the actual data. These include multiple details like where segments are, what’s in each segment, all the rules for querying and arranging data into segments, tasks details, runtime configurations about supervisors and data sources, etc.</p><p><strong>5. Deep storage</strong>:-</p><p>Druid uses deep storage to store any data that has been ingested into the system. It&#39;s like a backup and is a way to transfer data in the background between Druid processes. To respond to queries, Historical processes do not read from deep storage, but instead, read prefetched segments from their local disks before any queries are served. i.e., historical processes do not fetch data on demand. Below are the deep storage components.</p><p>In case of clustered deployment: s3/HDFS/network mounted filesystem<br>In case of single-server deployment:- Local disk</p><p><strong>ZooKeeper</strong>:-</p><p>Apache Druid uses Apache Zookeeper(ZK) for the management of the current cluster state. The operations that happen over ZK are</p><ol><li>Coordinator/Overload:- Leader election</li><li>Segment “publishing” protocol from Historical</li><li>Segment load/drop protocol between Coordinator and Historical</li></ol><blockquote><strong>Conclusion</strong></blockquote><p>We have tried to cover the overall architecture in a simplified way. However, there are a lot of other features which is responsible to boost the performance parameters of Apache Druid. For instance storage formats(Segments), Querying layer,(Historical), and so on. Also, Druid is horizontally scalable, which means you can keep increasing the servers as you feel the need. There are different data sources that Druid supports for both batch and live ingestion. We would love to cover more details on these topics in the coming days.</p><p>Reference:- <a href="https://druid.apache.org/">https://druid.apache.org/</a></p><p>Thanks for reading. Hope it was helpful:)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a143856b5fa0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/subex-ai-labs/apache-druid-analytics-db-for-timeseries-data-a143856b5fa0">Apache Druid — Analytics DB for Timeseries Data</a> was originally published in <a href="https://medium.com/subex-ai-labs">Subex AI Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an efficient input data pipeline for deep learning.]]></title>
            <link>https://medium.com/subex-ai-labs/building-an-efficient-input-data-pipeline-for-deep-learning-45300e59d8f6?source=rss----321f3b8a1fd1---4</link>
            <guid isPermaLink="false">https://medium.com/p/45300e59d8f6</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[tf-data]]></category>
            <category><![CDATA[tensorflow]]></category>
            <dc:creator><![CDATA[Mrinal Haloi]]></dc:creator>
            <pubDate>Mon, 03 May 2021 11:42:40 GMT</pubDate>
            <atom:updated>2021-05-03T11:42:40.917Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9QvAQu8J5eZ7Wv2hbzMTEw.jpeg" /><figcaption><a href="https://www.google.com/url?sa=i&amp;url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fpetroleum-pipelines&amp;psig=AOvVaw3BhZUwr7L4G9FYi1iDkB2e&amp;ust=1619983320725000&amp;source=images&amp;cd=vfe&amp;ved=0CAoQjRxqFwoTCOD46qWaqfACFQAAAAAdAAAAABA3">credit</a></figcaption></figure><blockquote><strong><em>Problem</em></strong></blockquote><p>When dealing with huge datasets of thousands/millions of files input data pipeline can be a game-changer or a bottleneck based on the design.</p><p>When datasets don’t fit in the RAM, using a python generator-based approach can become a huge bottleneck in training complex GPU compute-intensive models. GPU sitting idle waiting for data slow down model training by a huge margin. <br>That’s where Tensorflow TFRecord comes to the rescue.</p><blockquote>TFRecord</blockquote><p>TFRecord stores data in a sequence of protocol buffers serialized as a binary record. TFRecords files are very read efficient and light on hard disk space usage.</p><p>TFRecord does away with the need of reading each sample file from the disk for every epoch.</p><p>The efficiency of using TFRecord also comes with a cost, the requirement of writing a lot of complex code to create and read the records files.</p><p>Here we present a solution to this cost by introducing <a href="https://github.com/openAGI/datum">Datum</a> to manage TFRecords files.</p><blockquote><a href="https://github.com/openAGI/datum">Datum</a></blockquote><p><a href="https://github.com/openAGI/datum">Datum</a> is a library built on the top of TensorFlow to build an efficient fast input pipeline. Datum is designed to create/manage TFRecord datasets almost without writing complex codes.</p><p>Datum is designed to read/write TFRecord datasets and to build a fast input pipeline to be used for a single GPU or distributed training with just a few lines of code.</p><p>Datum makes use of tf.data and tfrecord build a fast input pipeline.</p><blockquote>Installation</blockquote><p>Datum can be installed from pypi</p><pre>pip install datum</pre><blockquote>Export to TFRecord</blockquote><p>Writing / exporting data to tfrecord format can get very complex if not using datum.</p><p>Datum makes it easier to export datasets to tfrecord format. Datum provides a few predefined problem types to create a dataset with a few lines of code without getting deep into the working of tfrecord and serialization.</p><p>Import TFRWriteConfigs to define datum configuration for writing/exporting data to tfrecord</p><pre>from datum.configs import TFRWriteConfigs</pre><p>Define the splits information in the configs, splits names are important for datum to automatically identify the splits data.</p><pre>write_configs = TFRWriteConfigs()<br>write_configs.splits = {<br>    &quot;train&quot;: {<br>        &quot;num_examples&quot;: &lt;num of train examples in the dataset&gt;<br>    },<br>    &quot;val&quot;: {<br>        &quot;num_examples&quot;: &lt;num of validation examples in the dataset&gt;<br>    },<br>}</pre><p>Import the export API and problem type to convert the datasets for.</p><p>Different datasets are used for different purposes. For example, an image classification dataset with only a class label can not be used for image segmentation or image detection. To make conversion easier datum defines separate problem types for classification, detection, and segmentation tasks.</p><pre>from datum.export.export import export_to_tfrecord<br>from datum.problem.types import IMAGE_CLF</pre><p>Suppose we want to build a tfrecord dataset for an image classification task, type for that is IMAGE_CLF</p><p>Convert the dataset to tfrecord format</p><pre>export_to_tfrecord(input_path, output_path, types.IMAGE_CLF, write_configs)</pre><p>Datum will convert the dataset and save the output .tfrecord files and dataset metadata files in the output path. The export tfrecord files can be easily loaded as tf.data.Dataset using Datum load API.</p><blockquote>Load as tf.data.Dataset</blockquote><p>Import the loading API for loading the tfrecord dataset as tf.data.Dataset</p><pre>from datum.reader import load</pre><p>Loading dataset is very simple, just pass the output_path from the previous export state.</p><pre>dataset = load(&lt;path to tfreord files folder&gt;)<br>train_dataset = dataset.train_fn(&#39;train&#39;, shuffle=True)<br>val_dataset = dataset.val_fn(&#39;val&#39;, shuffle=False)</pre><p>Examples/cases in the dataset can be augmented before feeding into the model. It’s easy to preprocess and post-process samples in the dataset using pre_batching_callbackand post_batching_callback .</p><p><em>pre_batching_callback: </em>Using this callback example can be processed before batching.</p><p>post_batching_callback: Using this callback example can be processed after batching. Examples are processed as a batch.</p><p>Suppose we want to augment the dataset, which can be achieved using the following pre_batching_callback</p><pre>def augment_image(example):<br>  image = tf.image.resize(example[&quot;image&quot;], IMG_SIZE)<br>  image = tf.image.random_flip_left_right(image)<br>  image = tf.image.random_flip_up_down(image)<br>  example.update({&quot;image&quot;: image})<br>  return example</pre><pre>dataset_configs = dataset.dataset_configs<br>datset_configs.pre_batching_callback = lambda example: augment_image(example)<br>train_dataset = dataset.train_fn(&#39;train&#39;, shuffle=True)</pre><blockquote>Try Datum in practice</blockquote><p>This <a href="https://colab.research.google.com/drive/1_r34J0MgdC7yCIVtH_EV0ne5q2y6EaH9?usp=sharing">notebook</a> demonstrates the use of datum for transfer learning an EfficientNet-B0 model for an image classification project.</p><p>Play with the <a href="https://colab.research.google.com/drive/1_r34J0MgdC7yCIVtH_EV0ne5q2y6EaH9?usp=sharing">Transfer Learning with Datum</a> notebook and use it to make your input pipeline fast.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=45300e59d8f6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/subex-ai-labs/building-an-efficient-input-data-pipeline-for-deep-learning-45300e59d8f6">Building an efficient input data pipeline for deep learning.</a> was originally published in <a href="https://medium.com/subex-ai-labs">Subex AI Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>