<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Manash Kumar Mandal on Medium]]></title>
        <description><![CDATA[Stories by Manash Kumar Mandal on Medium]]></description>
        <link>https://medium.com/@manashmandal?source=rss-6f5d8825e907------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*JeXrqgurSzjlMbi96zYQlw.jpeg</url>
            <title>Stories by Manash Kumar Mandal on Medium</title>
            <link>https://medium.com/@manashmandal?source=rss-6f5d8825e907------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 22:04:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@manashmandal/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[[Memoir: Chapter 4] Building a simple ETL Pipeline in C++ using Generator and Boost Asio — Part 2]]></title>
            <link>https://blog.manash.io/memoir-chapter-4-building-a-simple-etl-pipeline-in-c-using-generator-and-boost-asio-part-2-d8a7cd30c4a?source=rss-6f5d8825e907------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8a7cd30c4a</guid>
            <category><![CDATA[tcp-ip]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[c-plus-plus-language]]></category>
            <category><![CDATA[etl-pipeline]]></category>
            <dc:creator><![CDATA[Manash Kumar Mandal]]></dc:creator>
            <pubDate>Fri, 12 May 2023 18:04:09 GMT</pubDate>
            <atom:updated>2023-05-12T19:30:33.306Z</atom:updated>
            <content:encoded><![CDATA[<h3>[Memoir: Chapter 4] Building a simple ETL Pipeline in C++ using Generator and Boost Asio — Part 2</h3><h4>Leveraging C++20 ranges and view standard library to do the transformation part</h4><h4>Table of Contents</h4><ul><li><a href="https://blog.manash.io/memoir-chapter-3-building-a-simple-etl-pipeline-in-c-using-generator-and-boost-asio-part-1-5ae7a76d13d9">Building a simple ETL Pipeline in C++ using Generator and Boost Asio — Part 1</a></li><li>Building a simple ETL Pipeline in C++ using Generator and Boost Asio — Part 2 [You are here]</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NxIr3s44m6-4b0AP" /><figcaption>Photo by <a href="https://unsplash.com/@joshmillerdp?utm_source=medium&amp;utm_medium=referral">Josh Miller</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Last time, we attempted to construct a TCP server in C++ using Boost ASIO that could receive data. Today, we’re going to assemble the final piece of the puzzle. Specifically, we’ll transform the data and then insert the transformed data into MongoDB.</p><p>Why MongoDB? No particular reason — I simply wanted to experiment with the Mongo C++ driver.</p><p>The same prerequisite applies here too. You’ll need a compiler that supports the C++20 standard to run the code. While C++23 offers some impressive functionalities for handling various types of data transformation and aggregation, I’ll compose another piece when I get the opportunity to set up the g++13 compiler.</p><h3>Mistakes that I made</h3><p>I’ve made numerous errors, but let’s focus on rectifying those that are most critical.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oKvDdKB3YYmZnKLmSu4nFg.jpeg" /><figcaption>Credit: Me (Manash K Mandal), Location: Stralauer Allee 6, Berlin, Germany</figcaption></figure><h4>Didn’t do null termination of received data converted to string</h4><p>I neglected to terminate the string I was receiving. As I attempted to convert the received bytes into a string and then from a string to an integer, an extra 0 kept appearing at the end. After tinkering with the code and with the help of ChatGPT, I managed to identify and resolve the issue. Here’s the code snippet.</p><pre>...<br><br>boost::array&lt;char, 128&gt; local_buffer;<br>size_t length = socket.read_some(boost::asio::buffer(local_buffer));<br><br>// This makes sure that the string is null terminated, converting it to a number might add additional digits<br>local_buffer[length] = &#39;\0&#39;;<br><br>...</pre><h4>Didn’t handle `ConnectionRefusedError` on python client’s end</h4><p>This solution was quite straightforward; I simply added a try-catch block. Previously, the code would exit immediately if it couldn’t detect an operational TCP server.</p><pre>while True:<br>    try:<br>        data = str(random.randint(0, 100))<br>        # Create a TCP/IP socket<br>        with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as client_socket:<br>            client_socket.connect((server_host, server_port))<br><br>            # Send data to the server<br>            client_socket.sendall(data.encode())<br><br>            # Receive response from the server<br>            response = client_socket.recv(1024)<br>            print(f&quot;Received from server: {response.decode()}&quot;)<br>    except ConnectionRefusedError:<br>        print(f&quot;Tried to send: {data} ... but failed!&quot;)</pre><p>I likely made other errors, but as I continued working on subsequent parts, I didn’t encounter many.</p><h3>Final Steps to Complete the ETL Workflow</h3><ul><li>Create a MongoDB database using Docker/Docker Compose (See Appendix)</li><li>Install MongoDB C++ driver using vcpkg (See Appendix)</li><li>Develop a helper class to establish a connection to the database we created and add some methods for handling documents and collections</li><li>Write the transformation function</li></ul><h4>Modify the Python client to transmit numbers as strings only</h4><p>The updated snippet mentioned above includes some modifications in error handling and data transmission.</p><h4>Database Header File for MongoDB Interactions</h4><p>Please note, this is a basic, non-reviewed piece of code that works, but it’s not super optimized or refined. For the sake of brevity, I did not create a separate C++ file, and instead placed the implementation in the header file, which is generally considered bad practice.</p><pre>#ifndef TCPTASK_DATABASE_H<br>#define TCPTASK_DATABASE_H<br><br>#include &lt;bsoncxx/json.hpp&gt;<br>#include &lt;mongocxx/client.hpp&gt;<br>#include &lt;mongocxx/instance.hpp&gt;<br>#include &lt;string&gt;<br><br>using bsoncxx::builder::basic::kvp;<br>using bsoncxx::builder::basic::make_document;<br><br>class Database {<br>public:<br>  Database(std::string connection_string, std::string database_name)<br>      : database_name(database_name) {<br>    this-&gt;connection_string = static_cast&lt;mongocxx::uri&gt;(connection_string);<br>    mongocxx::client client{this-&gt;connection_string};<br>    this-&gt;client_ = std::move(client);<br>    db = this-&gt;client_[database_name];<br>  }<br><br>  mongocxx::collection get_collection(std::string collection_name) {<br>    return this-&gt;db[collection_name];<br>  }<br><br>  bsoncxx::document::value prepare_document(std::string key, double value) {<br>    return make_document(kvp(key, value));<br>  }<br><br>  std::optional&lt;mongocxx::result::insert_one&gt;<br>  insert_document(const bsoncxx::document::view &amp;document,<br>                  std::string collection) {<br>    return this-&gt;get_collection(collection).insert_one(document);<br>  }<br><br>  void ping() {<br>    const auto ping_cmd = make_document(kvp(&quot;ping&quot;, 1));<br>    try {<br>      this-&gt;db.run_command(ping_cmd.view());<br>      std::cout &lt;&lt; &quot;Database connection established successfully&quot; &lt;&lt; std::endl;<br>    } catch (const std::exception &amp;e) {<br>      std::cout &lt;&lt; &quot;Exception: &quot; &lt;&lt; e.what() &lt;&lt; std::endl;<br>    }<br>  }<br><br>private:<br>  mongocxx::uri connection_string;<br>  mongocxx::client client_;<br>  mongocxx::database db;<br>  std::string database_name;<br>};<br><br>#endif // TCPTASK_DATABASE_H</pre><p>Let’s break this code down to make it more digestible.</p><p>We have some private members which I actually didn’t need.</p><pre>mongocxx::uri connection_string;<br>mongocxx::client client_;<br>mongocxx::database db;<br>std::string database_name;</pre><p>Constructor to initialize the connection.</p><pre>Database(std::string connection_string, std::string database_name)<br>      : database_name(database_name) {<br>    this-&gt;connection_string = static_cast&lt;mongocxx::uri&gt;(connection_string);<br>    mongocxx::client client{this-&gt;connection_string};<br>    this-&gt;client_ = std::move(client);<br>    db = this-&gt;client_[database_name];<br>  }</pre><p>I developed a ‘ping’ function to evaluate the connectivity. Although I could have integrated this into the constructor, I chose to keep it distinct. This function attempts to send a command to MongoDB to verify whether communication is possible.</p><pre>void ping() {<br>    const auto ping_cmd = make_document(kvp(&quot;ping&quot;, 1));<br>    try {<br>      this-&gt;db.run_command(ping_cmd.view());<br>      std::cout &lt;&lt; &quot;Database connection established successfully&quot; &lt;&lt; std::endl;<br>    } catch (const std::exception &amp;e) {<br>      std::cout &lt;&lt; &quot;Exception: &quot; &lt;&lt; e.what() &lt;&lt; std::endl;<br>    }<br>  }</pre><p>We also have the prepare_document and insert_document functions to assist with creating a document and inserting it into the collection. These functions could be combined, but they are currently separate.</p><pre>bsoncxx::document::value prepare_document(std::string key, double value) {<br>    return make_document(kvp(key, value));<br>  }<br><br>  std::optional&lt;mongocxx::result::insert_one&gt;<br>  insert_document(const bsoncxx::document::view &amp;document,<br>                  std::string collection) {<br>    return this-&gt;get_collection(collection).insert_one(document);<br>  }</pre><h4>Transformer Function to Transform and Aggregate Incoming Data</h4><p>Admittedly, there’s room for improvements. For instance, I’m iterating the lazy transformation function twice, which isn’t efficient. I also didn’t account for the scenario where there are no odd numbers in the input vector.</p><p>However, the code is fairly self-explanatory. I utilized transform and filter functions to convert data types from string to integer and to discard even numbers.</p><pre>void compute_mean_odd_only(std::vector&lt;std::string&gt; inputs,<br>                           Database &amp;database) {<br>  std::cout &lt;&lt; &quot;Begin calculation &quot; &lt;&lt; std::endl;<br>  try {<br>    auto results = inputs | std::views::transform([](std::string value) {<br>                     return std::stoi(value);<br>                   }) |<br>                   std::views::filter([](int n) { return n % 2 != 0; });<br><br>    int sum = std::accumulate(results.begin(), results.end(), 0);<br>    int size = std::distance(results.begin(), results.end());<br><br>    double mean = static_cast&lt;double&gt;(sum) / size;<br>    database.insert_document(database.prepare_document(&quot;mean&quot;, mean).view(),<br>                             &quot;computed_mean&quot;);<br>    std::cout &lt;&lt; &quot;Mean: &quot; &lt;&lt; mean &lt;&lt; std::endl;<br><br>  } catch (std::exception &amp;e) {<br>    std::cout &lt;&lt; &quot;Exception: &quot; &lt;&lt; e.what() &lt;&lt; std::endl;<br>  }<br>  std::cout &lt;&lt; &quot;End calculation &quot; &lt;&lt; std::endl;<br>}</pre><h3>Final main.cpp code</h3><p>Here is the modified main.cpp file with the transformer and database instance.</p><pre>#include &lt;boost/asio.hpp&gt;<br>#include &lt;iostream&gt;<br>#include &lt;numeric&gt;<br>#include &lt;ranges&gt;<br>#include &lt;string&gt;<br>#include &lt;string_view&gt;<br>#include &lt;vector&gt;<br><br>#include &quot;Database.h&quot;<br>#include &quot;Generator.h&quot;<br><br>using boost::asio::ip::tcp;<br><br>void compute_mean_odd_only(std::vector&lt;std::string&gt; inputs,<br>                           Database &amp;database) {<br>  std::cout &lt;&lt; &quot;Begin calculation &quot; &lt;&lt; std::endl;<br>  try {<br>    auto results = inputs | std::views::transform([](std::string value) {<br>                     return std::stoi(value);<br>                   }) |<br>                   std::views::filter([](int n) { return n % 2 != 0; });<br><br>    int sum = std::accumulate(results.begin(), results.end(), 0);<br>    int size = std::distance(results.begin(), results.end());<br><br>    double mean = static_cast&lt;double&gt;(sum) / size;<br>    database.insert_document(database.prepare_document(&quot;mean&quot;, mean).view(),<br>                             &quot;computed_mean&quot;);<br>    std::cout &lt;&lt; &quot;Mean: &quot; &lt;&lt; mean &lt;&lt; std::endl;<br><br>  } catch (std::exception &amp;e) {<br>    std::cout &lt;&lt; &quot;Exception: &quot; &lt;&lt; e.what() &lt;&lt; std::endl;<br>  }<br>  std::cout &lt;&lt; &quot;End calculation &quot; &lt;&lt; std::endl;<br>}<br><br>int main() {<br>  std::string connection_uri{&quot;mongodb://localhost:27020&quot;};<br>  std::string database_name{&quot;data_engineering_in_cpp_with_manash&quot;};<br><br>  Database database(connection_uri, database_name);<br>  database.ping();<br><br>  try {<br>    boost::asio::io_context io_context;<br>    tcp::acceptor acceptor(io_context, tcp::endpoint(tcp::v4(), 5000));<br>    auto gen = iterateNextBatch(io_context, acceptor);<br><br>    for (;;) {<br>      gen.next();<br>      compute_mean_odd_only(gen.getValue(), database);<br>    }<br>  } catch (std::exception &amp;e) {<br>    std::cerr &lt;&lt; e.what() &lt;&lt; std::endl;<br>  }<br><br>  return 0;<br>}</pre><h4>Executing the Server and Client</h4><p>Just as before, initiate the C++ server and execute the client.py file. If all components operate without any error, you should observe data being inserted into the MongoDB collection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y2ilDEXTTw4NTtV_UW7D-w.png" /></figure><h4>Full Code</h4><p><a href="https://github.com/manashmandal/tcp-server-cpp-generator">Here you can find the full code.</a> Feel free to send PR to improve it.</p><h4>Limitations:</h4><p>This is a purely thread-based blocking I/O implementation. For each request, Boost actually creates a new thread, which consumes resources. Moreover, the thread remains blocked until the data is fully read. If we transition to a combination of asynchronous and multithreading implementation, it would be far more efficient, which I plan to explore in my upcoming articles.</p><p>However, the primary objective was to construct something, irrespective of whether it’s riddled with bugs, and to experiment with new C++ features.</p><h3>Appendix</h3><ol><li>Installing Mongo CXX Driver</li></ol><p>I used vcpkg as before to streamline dependency management. <a href="https://github.com/mongodb/mongo-cxx-driver">Here is the library I installed.</a> In <a href="https://github.com/manashmandal/tcp-server-cpp-generator/blob/main/CMakeLists.txt">CMakeLists.txt</a> you will find all the necessary cmake commands to compile the project. To add the toolchain to your IDE please follow the previous article.</p><p>2. Creating a local MongoDB Instance</p><p>Have docker installed and use docker-compose.yml file to spin up the container. The database should be available in port 27020 without any authentication needed.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8a7cd30c4a" width="1" height="1" alt=""><hr><p><a href="https://blog.manash.io/memoir-chapter-4-building-a-simple-etl-pipeline-in-c-using-generator-and-boost-asio-part-2-d8a7cd30c4a">[Memoir: Chapter 4] Building a simple ETL Pipeline in C++ using Generator and Boost Asio — Part 2</a> was originally published in <a href="https://blog.manash.io/">Manash’s blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Memoir: Chapter 3] Building a simple ETL Pipeline in C++ using Generator and Boost Asio — Part 1]]></title>
            <link>https://blog.manash.io/memoir-chapter-3-building-a-simple-etl-pipeline-in-c-using-generator-and-boost-asio-part-1-5ae7a76d13d9?source=rss-6f5d8825e907------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ae7a76d13d9</guid>
            <category><![CDATA[tcp]]></category>
            <category><![CDATA[c-plus-plus-language]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[asyncio]]></category>
            <category><![CDATA[generator]]></category>
            <dc:creator><![CDATA[Manash Kumar Mandal]]></dc:creator>
            <pubDate>Mon, 24 Apr 2023 22:47:01 GMT</pubDate>
            <atom:updated>2023-05-12T18:05:07.624Z</atom:updated>
            <content:encoded><![CDATA[<h3>[Memoir: Chapter 3] Building a simple ETL Pipeline in C++ using Generator and Boost Asio — Part 1</h3><h4>Taking C++20 Generator feature for a spin</h4><h3>Table of Contents</h3><ul><li><a href="https://blog.manash.io/memoir-chapter-3-building-a-simple-etl-pipeline-in-c-using-generator-and-boost-asio-part-1-5ae7a76d13d9">Building a simple ETL Pipeline in C++ using Generator and Boost Asio — Part 1</a> [You are here]</li><li><a href="https://blog.manash.io/memoir-chapter-4-building-a-simple-etl-pipeline-in-c-using-generator-and-boost-asio-part-2-d8a7cd30c4a">Building a simple ETL Pipeline in C++ using Generator and Boost Asio — Part 2</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9fGZj63MbDp1ZFnn" /><figcaption>Photo by <a href="https://unsplash.com/@brandi1?utm_source=medium&amp;utm_medium=referral">Brandi Redd</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Background</h4><p>It was a while back when I made something using C++. With new features stacking up, it’s really hard not to try it out again. For some reason, I couldn’t continue working; one of the reasons was dependency management. It was a nightmare. Every single time I tried to use some libraries, I ended up discarding the attempt. Another reason was the lack of resources to learn the language properly. There are a lot of books available, but you can easily find out that many are pretty much garbage. There is a <a href="https://stackoverflow.com/questions/388242/the-definitive-c-book-guide-and-list">StackOverflow FAQ</a> where it shows a plethora of books to learn from. Even after that, I found it pretty hard to stick to a book and try learning in-depth. Either the resources were too hard for me to follow, or I lacked the knowledge even to begin with.</p><p>But when I got to know about the C++ books by Rainer Grimm, one being The C++ Standard Library and another one Concurrency with Modern C++, both of these books were really easy for me to follow. The right amount of explanation and to-the-point code snippets were just amazing. This is not a sponsored post 😅.</p><p>Why did I pick up C++ again? It seems there is still a demand for C++ in the Rust-y world. Will I get a lawsuit for mentioning the name? However, the number of recruiter messages I receive nowadays indicates they are looking for C++ engineers. Maybe if I lose my job, I will have a shot at trying to land one of those positions.</p><p>Without further ado, let’s try to build a very basic ETL pipeline in C++. This will be a two-part story.</p><h4>Goal</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wfo9uqSvgPO7Cxip" /><figcaption>Photo by <a href="https://unsplash.com/@esteejanssens?utm_source=medium&amp;utm_medium=referral">Estée Janssens</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The goal is to write a TCP server that will receive data, run some transformations, and insert the transformed data into a database. Essentially, the generator will function as an infinite data stream, reading data from the socket and yielding the batch data vector.</p><p>For the first part, I will:</p><ol><li>Show how to set up the development environment (appendix).</li><li>Explain how to write a generator, which I will shamelessly copy from cppreference and Rainer Grimm’s book.</li></ol><p>Note: You must have a C++20-supported compiler.</p><h4>Writing the Generator Class</h4><p>Let’s write the generator class first. It’s kind of similar to the generator function we are accustomed to write in python. Instead of using return we have to use yield . In C++ we will be using co_yield . In C++20 we will use the coroutine framework to write it as there is no concrete implementation. But the basic building blocks are there to build which is already mentioned in cppreference.</p><p>The main concept of coroutine is that it can be suspended and resumed at will while maintaining their state.</p><p>But to write the class we have to follow some rules which are mandatory in C++20 standard.</p><ol><li>Include a nested struct promise_type</li><li>Add aget_return_object() method in the struct</li></ol><pre>// Generator Header file : Generator.h<br><br>#ifndef TASK_GENERATOR_H<br>#define TASK_GENERATOR_H<br><br>#include &lt;boost/array.hpp&gt;<br>#include &lt;boost/asio.hpp&gt;<br>#include &lt;coroutine&gt;<br>#include &lt;vector&gt;<br><br><br>using boost::asio::ip::tcp;<br><br>template &lt;typename T&gt; struct Generator {<br>  struct promise_type;<br><br>  using handle_type = std::coroutine_handle&lt;promise_type&gt;;<br><br>  Generator(handle_type h) : coro(h) {}<br><br>  handle_type coro;<br><br>  T getValue() { return coro.promise().current_value; }<br><br>  bool next() {<br>    coro.resume();<br>    return not coro.done();<br>  }<br><br>  struct promise_type {<br>    promise_type() = default;<br>    ~promise_type() = default;<br><br>    auto initial_suspend() { return std::suspend_always{}; }<br><br>    auto final_suspend() noexcept { return std::suspend_always{}; }<br><br>    auto get_return_object() {<br>      return Generator{handle_type::from_promise(*this)};<br>    }<br><br>    auto return_void() { return std::suspend_never{}; }<br><br>    auto yield_value(const T value) {<br>      current_value = value;<br>      return std::suspend_always{};<br>    }<br><br>    void unhandled_exception() { std::exit(1); }<br><br>    T current_value;<br>  };<br>};<br><br>Generator&lt;std::vector&lt;std::string&gt;&gt;iterateNextBatch(boost::asio::io_context &amp;context, tcp::acceptor &amp;acceptor);<br><br>#endif // TASK_GENERATOR_H</pre><p>In the code, I added the mandatory promise_type struct. But you can choose any class name instead of Generator .</p><p>Let’s write the implementation for the iterateNextBatch function.</p><pre>#include &quot;Generator.h&quot;<br>#include &lt;iostream&gt;<br><br>std::string make_daytime_string() {<br>  using namespace std; // For time_t, time and ctime;<br>  time_t now = time(0);<br>  return ctime(&amp;now);<br>}<br><br>Generator&lt;std::vector&lt;std::string&gt;&gt;<br>iterateNextBatch(boost::asio::io_context &amp;context, tcp::acceptor &amp;acceptor) {<br><br>  std::vector&lt;std::string&gt; buffer_vector;<br><br>  while (true) {<br>    tcp::socket socket(context);<br>    acceptor.accept(socket);<br><br>    // Enable TCP Keep-Alive<br>    boost::asio::socket_base::keep_alive option(true);<br>    socket.set_option(option);<br><br>    boost::array&lt;char, 128&gt; local_buffer;<br>    socket.read_some(boost::asio::buffer(local_buffer));<br>    <br>    boost::system::error_code ignored_error;<br>    std::string message = make_daytime_string();<br>    boost::asio::write(socket, boost::asio::buffer(message), ignored_error);<br><br>    buffer_vector.push_back(std::string(local_buffer.c_array()));<br><br>    if (buffer_vector.size() &gt; 5) {<br>      co_yield buffer_vector;<br>      buffer_vector.clear();<br>    }<br>  }<br>}</pre><p>This C++ code is a server application that uses the Boost.Asio library for asynchronous I/O operations. It listens for incoming TCP connections, reads incoming data, and sends back the current date and time as a response. It also uses a generator to yield the incoming data in batches.</p><p>Let’s break down the code:</p><ol><li>Header files are included:</li></ol><ul><li>“Generator.h”: Contains the definition for the Generator class.</li><li>&lt;iostream&gt;: Provides input-output streaming functionalities.</li></ul><p>2. make_daytime_string function:</p><ul><li>Returns a string containing the current date and time.</li><li>Uses time_t, time, and ctime from the standard namespace.</li></ul><p>3. iterateNextBatch function:</p><ul><li>Takes two arguments: boost::asio::io_context &amp;context and tcp::acceptor &amp;acceptor.</li><li>Returns a generator of type Generator&lt;std::vector&lt;std::string&gt;&gt;.</li><li>Inside the function:</li><li>A std::vector&lt;std::string&gt; called buffer_vector is created to store the incoming data.</li><li>An infinite while loop is used to continually accept incoming connections.</li><li>A new tcp::socket is created.</li><li>The acceptor accepts the incoming connection.</li><li>TCP Keep-Alive option is enabled for the socket.</li><li>A boost::array called local_buffer is used to read data from the socket.</li><li>The server generates the current date and time using the make_daytime_string function and sends it back to the client using boost::asio::write.</li><li>The received data is added to the buffer_vector.</li><li>If the buffer_vector size is greater than 5, the vector is yielded using co_yield and then cleared.</li></ul><p>This server accepts and processes incoming connections and stores the received data in a buffer. Once the buffer reaches a size greater than 5, it yields the data using a generator and clears the buffer for the next batch of incoming data.</p><p><em>Note: I am using syncrhonous read function to read from the socket. It can be further optimized using async, which I am too lazy to do that today.</em></p><h4>Main.cpp</h4><pre>#include &lt;boost/asio.hpp&gt;<br>#include &lt;iostream&gt;<br>#include &lt;string&gt;<br>#include &quot;Generator.h&quot;<br><br>using boost::asio::ip::tcp;<br><br><br>int main() {<br>  try {<br>    boost::asio::io_context io_context;<br>    tcp::acceptor acceptor(io_context, tcp::endpoint(tcp::v4(), 5000));<br>    auto gen = iterateNextBatch(io_context, acceptor);<br><br>    for (;;) {<br>      gen.next();<br>      for (auto v : gen.getValue()) {<br>        std::cout &lt;&lt; v &lt;&lt; std::endl;<br>      }<br>    }<br>  } catch (std::exception &amp;e) {<br>    std::cerr &lt;&lt; e.what() &lt;&lt; std::endl;<br>  }<br><br>  return 0;<br>}</pre><p>We are creating io_context and tcp::acceptor then passing the reference to the generator. And let generator consume data from the socket and pop out data vector of size 5.</p><h4>Time to write the TCP client which will push data to this C++ server</h4><pre>import socket<br>import random<br><br><br>def main():<br>    server_host = &quot;127.0.0.1&quot;<br>    server_port = 5000<br><br>    while True:<br>        # Create a TCP/IP socket<br>        with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as client_socket:<br>            client_socket.connect((server_host, server_port))<br><br>            # Send data to the server<br>            data = &quot;Hello, server! - &quot; + str(random.randint(0, 100))<br>            client_socket.sendall(data.encode())<br><br>            # Receive response from the server<br>            response = client_socket.recv(1024)<br>            print(f&quot;Received from server: {response.decode()}&quot;)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>Should wrap this code in try catch but will leave it for the reader 😉.</p><p><a href="https://github.com/manashmandal/tcp-server-cpp-generator">Here is the repository</a> containing all of the files and of course CMakeLists .</p><p>If you run both C++ and Python code the client should send data and on tcp server you’ll see that when vector size is 5 the generator returns data.</p><h3>Appendix — Configuring the project in Qt/CLion</h3><p>I used vcpkg for managing the dependencies. You can install it easily following <a href="https://vcpkg.io/en/getting-started.html">the instruction.</a></p><p>In both Qt or CLion, just add this flag in build configuration.</p><p>-DCMAKE_TOOLCHAIN_FILE=/prefix/vcpkg/..../vcpkg.cmake</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*N0kyaJKFa_baIq0zLy4D2g.png" /></figure><p>And in CLion,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/843/1*s6IaVol2GBoLTJZwhirbGA.png" /></figure><p>You can find the asio and other <a href="https://vcpkg.io/en/packages.html">packages here.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ae7a76d13d9" width="1" height="1" alt=""><hr><p><a href="https://blog.manash.io/memoir-chapter-3-building-a-simple-etl-pipeline-in-c-using-generator-and-boost-asio-part-1-5ae7a76d13d9">[Memoir: Chapter 3] Building a simple ETL Pipeline in C++ using Generator and Boost Asio — Part 1</a> was originally published in <a href="https://blog.manash.io/">Manash’s blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Memoir: Chapter 2] Running Puppeteer in a Background Process: Managing Out of Memory Errors]]></title>
            <link>https://blog.manash.io/memoir-chapter-2-running-puppeteer-in-a-background-process-managing-out-of-memory-errors-d938c7e418ad?source=rss-6f5d8825e907------2</link>
            <guid isPermaLink="false">https://medium.com/p/d938c7e418ad</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[puppeteer]]></category>
            <dc:creator><![CDATA[Manash Kumar Mandal]]></dc:creator>
            <pubDate>Tue, 11 Apr 2023 20:05:56 GMT</pubDate>
            <atom:updated>2023-04-11T20:05:56.200Z</atom:updated>
            <content:encoded><![CDATA[<p>I am working on an interesting project. The goal is to build a system where we can evaluate the quality of a Facebook post in a group before it goes through the manual moderation pipeline. However, to even begin, we needed to collect labeled data. So we started from scratch: first, building a data pipeline to gather data, and then constructing a multimodal machine learning model. Just like every other project, moving forward with one goal is challenging; side quests will inevitably arise. The question remains, “when?”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LXytJBsbdQoAPb-U" /><figcaption>Photo by <a href="https://unsplash.com/@jrkorpa?utm_source=medium&amp;utm_medium=referral">Jr Korpa</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Perhaps we can discuss the total system architecture in another post at another time, but for now, let’s focus on the problem at hand. The goal was to run the Puppeteer browser in a process and push tasks asynchronously through a REST API. I leveraged <a href="https://www.npmjs.com/package/faktory">Faktory</a>and <a href="https://hapi.dev/">Hapi</a>to accomplish this, exposing an endpoint through Hapi and pushing jobs via Faktory to a Faktory worker.</p><p>The constant issue I faced was the “out of memory” error in the worker process. I managed the browser using PM2, but it still required occasional restarts.</p><p>The first version of the code looked something like this. Please go easy on me, as I am not a JavaScript expert, but I do enjoy tinkering with it from time to time.</p><p>A simple endpoint to rule them all.</p><h4>REST API to Push Task</h4><pre>// app.js<br><br>const Hapi = require(&quot;@hapi/hapi&quot;);<br>const faktory = require(&quot;faktory-worker&quot;);<br>const Joi = require(&quot;joi&quot;);<br>const dotenv = require(&quot;dotenv&quot;);<br>const JOB_NAME = &quot;crawl&quot;;<br><br>dotenv.config();<br><br>const schema = Joi.object({<br>  cookie: Joi.array().required(),<br>  fromDate: Joi.string().required(),<br>  tillDate: Joi.string().required(),<br>  groupId: Joi.string(),<br>  moderationAction: Joi.string().default(&quot;posts_declined&quot;),<br>  retry: Joi.number().default(0),<br>});<br><br>const init = async () =&gt; {<br>  const server = Hapi.server({<br>    port: 3000,<br>    host: &quot;0.0.0.0&quot;,<br>  });<br><br>  const faktoryClient = await faktory.connect({<br>    url: process.env.FAKTORY_URL || &quot;tcp://localhost:7419&quot;,<br>  });<br><br>  server.route({<br>    method: &quot;POST&quot;,<br>    path: &quot;/crawling/start&quot;,<br>    handler: async (request, h) =&gt; {<br><br>      if (request.headers.authorization !== process.env.AUTH_TOKEN) {<br>        return h.response({ error: &quot;Unauthorized&quot; }).code(401);<br>      }<br><br>      const { payload } = request;<br><br>      let { error, value } = schema.validate(payload);<br><br>      if (error !== undefined &amp;&amp; error !== null) {<br>        return h.response({ error }).code(400);<br>      }<br><br>      const job = faktoryClient.job(JOB_NAME, { ...value });<br>      job.retry = value.retry;<br>      job.push();<br><br>      return value;<br>    },<br>  });<br><br>  await server.start();<br>  console.log(&quot;Server running on %s&quot;, server.info.uri);<br>};<br><br>process.on(&quot;unhandledRejection&quot;, (err) =&gt; {<br>  console.log(err);<br>  process.exit(1);<br>});<br><br>init();</pre><p>I understand that the code may not be the most organized, but it’s a starting point. Now, let’s take a look at the worker code for Puppeteer.</p><h4>Puppeteer Worker.js</h4><pre>//worker.js<br><br>const faktory = require(&quot;faktory-worker&quot;);<br><br>const puppeteer = require(&quot;puppeteer&quot;);<br>const dotenv = require(&quot;dotenv&quot;);<br>const fs = require(&quot;fs&quot;);<br>const getUnixTime = require(&quot;date-fns/getUnixTime&quot;);<br><br>dotenv.config();<br><br>const sleep = (ms) =&gt; new Promise((resolve) =&gt; setTimeout(resolve, ms));<br><br>const buildUrl = ({ fromDate, tillDate, moderationAction, groupId }) =&gt; {<br>  const action = moderationAction || &quot;posts_declined&quot;;<br>  const selectedGroupId = groupId;<br>  return `https://www.facebook.com/groups/${selectedGroupId}/admin_activities?moderation_action=${action}&amp;date_range[0]=${fromDate}&amp;date_range[1]=${tillDate}`;<br>};<br><br>const crawl = async ({<br>  fromDate,<br>  tillDate,<br>  cookie,<br>  moderationAction,<br>  groupId,<br>}) =&gt; {<br>  fromDate = getUnixTime(new Date(fromDate));<br>  tillDate = getUnixTime(new Date(tillDate));<br><br>  const browser = await puppeteer.launch();<br>  const page = await browser.newPage();<br>  await page.setViewport({ width: 1366, height: 768 });<br>  await page.setCookie(...cookie);<br>  await page.goto(buildUrl({ fromDate, tillDate, moderationAction, groupId }), {<br>    waitUntil: &quot;networkidle2&quot;,<br>  });<br><br>  while (true) {<br><br>    await page.keyboard.down(&quot;PageDown&quot;);<br><br>    // Secret sauce to get all posts<br>    <br>    // If end of page then<br>    await browser.close();<br>    break;<br>    }<br>  }<br>};<br><br>faktory.register(<br>  &quot;crawl&quot;,<br>  async ({<br>    fromDate,<br>    tillDate,<br>    cookie,<br>    moderationAction,<br>    groupId,<br>  }) =&gt; {<br>    await crawl({<br>      fromDate,<br>      tillDate,<br>      cookie,<br>      moderationAction,<br>      groupId,<br>    });<br>  }<br>);<br><br>faktory.work().catch((error) =&gt; {<br>  console.error(`worker failed to start: ${error}`);<br>  process.exit(1);<br>});</pre><p>There are several visible issues in this code. Firstly, closing and launching the browser again for each new task is not practical, as it wastes precious resources.</p><p>Secondly, the code is running in sandbox mode, where resources are already limited. To crawl something as resource-intensive as Facebook, more processing power is needed, but it’s not being provided in this case.</p><p>Thirdly, resource destruction is not being managed gracefully. At this point, I haven’t done any profiling, which I should have. What happens if the process is exited but the browser is still allocating memory? It seems we need to investigate further. For now, let’s make some assumptions and move on by addressing these issues.</p><p>Having identified the errors so far, let’s attempt to solve them.</p><pre>const faktory = require(&quot;faktory-worker&quot;);<br>const puppeteer = require(&quot;puppeteer&quot;);<br>const dotenv = require(&quot;dotenv&quot;);<br>const getUnixTime = require(&quot;date-fns/getUnixTime&quot;);<br>const AMQP = require(&quot;./rabbitmqclient&quot;);<br>const { v4: uuidv4 } = require(&quot;uuid&quot;);<br><br>const THREE_SECONDS_IN_MILLISECONDS = 3000;<br>const HALF_SECOND_IN_MILLISECONDS = 500;<br>const ONE_SECOND_IN_MILLISECONDS = 1000;<br>const DEFAULT_OCCURANCE_THRESHOLD = 10;<br>const JOB_NAME = &quot;crawl&quot;;<br><br>dotenv.config();<br><br><br>const sleep = (ms) =&gt; new Promise((resolve) =&gt; setTimeout(resolve, ms));<br><br>const buildUrl = ({ fromDate, tillDate, moderationAction, groupId }) =&gt; {<br>  const action = moderationAction || &quot;posts_declined&quot;;<br>  const selectedGroupId = groupId;<br>  return `https://www.facebook.com/groups/${selectedGroupId}/admin_activities?moderation_action=${action}&amp;date_range[0]=${fromDate}&amp;date_range[1]=${tillDate}`;<br>};<br><br>const initializeBrowser = async (headless) =&gt; {<br>  const browser = await puppeteer.launch({<br>    headless: headless,<br>    args: [<br>      &quot;--no-sandbox&quot;,<br>      &quot;--disable-setuid-sandbox&quot;,<br>      &quot;--disable-dev-shm-usage&quot;,<br>    ],<br>  });<br>  return browser;<br>};<br><br>const crawl = async ({<br>  fromDate,<br>  tillDate,<br>  cookie,<br>  moderationAction,<br>  groupId,<br>  browser,<br>}) =&gt; {<br><br>  fromDate = getUnixTime(new Date(fromDate));<br>  tillDate = getUnixTime(new Date(tillDate));<br>  const url = buildUrl({ fromDate, tillDate, moderationAction, groupId });<br><br>  const page = await browser.newPage();<br>  await page.setViewport({ width: 1366, height: 768 });<br>  await page.setCookie(...cookie);<br>  await page.goto(url, {<br>    waitUntil: &quot;networkidle2&quot;,<br>  });<br><br>  const { connection, channel } = await AMQP.createAmqpChannel(<br>    process.env.RABBITMQ_BROKER || &quot;amqp://localhost:5672&quot;<br>  );<br><br>  while (true) {<br><br>    await page.keyboard.down(&quot;PageDown&quot;);<br>    await sleep(ONE_SECOND_IN_MILLISECONDS);<br><br>    // If crawled to last page and collected all data then<br>    await page.close();<br><br>      break;<br>    }<br>  }<br><br>  connection.close();<br>};<br><br>(async () =&gt; {<br>  const browser = await initializeBrowser(true);<br><br>  faktory.register(<br>    JOB_NAME,<br>    async ({<br>      fromDate,<br>      tillDate,<br>      cookie,<br>      moderationAction,<br>      groupId,<br>    }) =&gt; {<br>      await crawl({<br>        fromDate,<br>        tillDate,<br>        cookie,<br>        moderationAction,<br>        groupId,<br>        browser,<br>      });<br>    }<br>  );<br><br>  process.on(&quot;exit&quot;, async () =&gt; {<br>    await browser.close();<br>  });<br><br>  faktory<br>    .work({<br>      host: process.env.FAKTORY_HOST || &quot;localhost&quot;,<br>      port: process.env.FAKTORY_PORT || 7419,<br>    })<br>    .catch(async (error) =&gt; {<br>      await browser.close();<br>      console.error(`worker failed to start: ${error}`);<br>      process.exit(1);<br>    });<br>})();</pre><h3>What I Tried as a Solution and What Worked:</h3><h4>1. Optimizing the Browser via launch configuration</h4><p>Now that we’ve addressed the issues we just discovered, we can start by running Chromium in non-sandboxed mode using the first two flags.</p><pre>&quot;--no-sandbox&quot;,<br>&quot;--disable-setuid-sandbox&quot;</pre><p>And disabling shared memory file usage to /dev/shm by using the third flag, --disable-dev-shm-usage, so that it forces the system to use /tmp for serving purposes instead.</p><pre>&quot;--disable-dev-shm-usage&quot;</pre><h4>2. Using IIFE (Immediately Invoked Function Expression) to Share the Initialized Browser Throughout the Faktory Worker Lifecycle</h4><p>As you can see, instead of re-launching the browser each time, I created one instance and shared it across all tasks. Now, every task should create a new page.</p><pre>(async () =&gt; {<br>  const browser = await initializeBrowser(true);<br><br>  faktory.register(<br>    JOB_NAME,<br>    async ({<br>      fromDate,<br>      tillDate,<br>      cookie,<br>      moderationAction,<br>      groupId,<br>    }) =&gt; {<br>      await crawl({<br>        fromDate,<br>        tillDate,<br>        cookie,<br>        moderationAction,<br>        groupId,<br>        browser,<br>      });<br>    }<br>  );<br>....)();</pre><h4>3. Releasing resource</h4><p>Now, upon exit, it is quite guaranteed that the browser resources will be released.</p><pre>process.on(&quot;exit&quot;, async () =&gt; {<br>    await browser.close();<br>});</pre><h4>One last thing, which I believe was the most beneficial:</h4><p>As I mentioned, I used the PM2 process manager. However, I tweaked the settings a bit to ensure that, if required, the Node.js process would use at most 4GB of memory. The ecosystem.config.js file looks something like this:</p><pre>module.exports = {<br>  apps: [<br>    {<br>      script: &quot;worker.js&quot;,<br>      watch: &quot;.&quot;,<br>      node_args: &quot;--max-old-space-size=4096&quot;,<br>    },<br>  ],<br>};</pre><p>Now, if I run three crawling jobs on a t2.large AWS EC2 instance, it can run without breaking anything.</p><p>Hopefully, this was somewhat useful. See you next time!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d938c7e418ad" width="1" height="1" alt=""><hr><p><a href="https://blog.manash.io/memoir-chapter-2-running-puppeteer-in-a-background-process-managing-out-of-memory-errors-d938c7e418ad">[Memoir: Chapter 2] Running Puppeteer in a Background Process: Managing Out of Memory Errors</a> was originally published in <a href="https://blog.manash.io/">Manash’s blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Memoir: Chapter 1] Handling unusual test case scenarios in PyTest]]></title>
            <link>https://blog.manash.io/memoir-chapter-1-handling-unusual-test-case-scenarios-in-pytest-5263d301a1d6?source=rss-6f5d8825e907------2</link>
            <guid isPermaLink="false">https://medium.com/p/5263d301a1d6</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pytest]]></category>
            <category><![CDATA[unit-testing]]></category>
            <dc:creator><![CDATA[Manash Kumar Mandal]]></dc:creator>
            <pubDate>Tue, 04 Apr 2023 22:12:05 GMT</pubDate>
            <atom:updated>2023-04-11T20:06:31.358Z</atom:updated>
            <content:encoded><![CDATA[<p>On August 22nd, 2022, I joined Wolt as a Senior Backend Software Engineer in Berlin. Before that, I worked at a couple of companies in Bangladesh, where I didn’t get the chance to engage in any formal engineering practices. Most of the time, I had to write software and validate it myself, which is a rather inefficient way to make progress. I was blinded by my own opinions. I read plenty of books but couldn’t apply the knowledge, so I was back at square one. I always longed for a team of engineers who could provide valuable feedback to help me grow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UtkFZ80ewstBehEm" /><figcaption>Photo by <a href="https://unsplash.com/@aronvisuals?utm_source=medium&amp;utm_medium=referral">Aron Visuals</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>As I mentioned earlier, writing tests was one of the least prioritized tasks, and in some cases, non-existent. I didn’t have much exposure to writing test cases, and I can’t say it was entirely the company’s fault. Consequently, I ended up delivering faulty software that required rigorous manual testing. Perhaps we can discuss that in more detail some other time.</p><p>Now, I need to focus more on writing tests rather than just writing service logic, which is a positive change. I haven’t written anything in a long time, so I thought now might be a good time to brush up on my forgotten skills.</p><p>Allow me to introduce two unique cases that I encountered and believe are worth writing about. Even if they aren’t, they hold importance for me.</p><h3>Write test when an exception is expected but when it’s not at the same parameterized test</h3><p>I had to struggle a bit with this one. I needed to test a service where sometimes an exception should be raised, and other times it shouldn’t. I discovered a nifty context manager in the contextlib module that suppresses the specified exception. It is, in fact, called suppress.</p><p>Here are some code samples directly from the official documentation:</p><pre>from contextlib import suppress<br><br>with suppress(FileNotFoundError):<br>    os.remove(&#39;somefile.tmp&#39;)<br><br>with suppress(FileNotFoundError):<br>    os.remove(&#39;someotherfile.tmp&#39;)</pre><p>Now about the test case scenario.</p><pre>import pytest<br>from contextlib import suppress<br><br><br>class AdditionException(Exception):<br>    pass<br><br><br>def add_numbers(a, b):<br>    if a &lt; 0 or b &lt; 0:<br>        raise AdditionException(&quot;Negative numbers not allowed.&quot;)<br>    return a + b<br><br><br>test_data = [<br>    (1, 2, 3, suppress()),<br>    (3, 4, 7, suppress()),<br>    (-1, 2, None, pytest.raises(AdditionException)),<br>    (5, -1, None, pytest.raises(AdditionException)),<br>]<br>@pytest.mark.parametrize(&quot;a, b, expected, expectation&quot;, test_data)<br>def test_add_numbers(a, b, expected, expectation):<br>    with expectation:<br>        result = add_numbers(a, b)<br>        if isinstance(expectation, suppress):<br>            assert result == expected</pre><h3>Mock random data generators to get the same result every single time</h3><p>For example, let’s say you are writing tests for your password generator algorithm, and the algorithm relies on a function that generates a pepper or salt. Testing this would be challenging unless you wrap the generator in another function and mock that function instead. Does it sound confusing? Let’s examine the code.</p><pre># service.py<br><br>import secrets<br>import hmac<br><br><br>def _generate_token(nbytes: int = 32) -&gt; str:<br>    return secrets.token_hex(nbytes)<br><br>def generate_hash_from_phone_number(phone_number: str) -&gt; str:<br>    return hmac.new(str.encode(_generate_token()), msg=str.encode(phone_number), digestmod=&quot;sha256&quot;).hexdigest()</pre><pre># test_hash.py<br><br>import pytest<br>from pytest import MagicMock<br>from service import generate_hash_from_phone_number<br><br>@pytest.fixture<br>def token_hex_generator_mock(monkeypatch) -&gt; MagicMock:<br>    mock = MagicMock(return_value=&quot;4813f55954e2e3bba724ba12bb62b67e0a30d07f5fdb9a3b30f6446edac913a9&quot;)<br>    monkeypatch.setattr(&quot;service._generate_token&quot;, mock)<br><br>    return mock<br><br><br>def test_generate_hash(token_hex_generator_mock, generate_hash_from_phone_number):<br>    hash = generate_hash_from_phone_number(&quot;+491781234112&quot;)<br>    assert hash == &quot;fc544356f8a0814aa783cdb6928d09ed01fa248a86d0096d27c7a6b201b93840&quot;<br><br>    hash = generate_hash_from_phone_number(&quot;+491781234112&quot;)<br>    assert hash == &quot;fc544356f8a0814aa783cdb6928d09ed01fa248a86d0096d27c7a6b201b93840&quot;</pre><p>Although the function generate_hash_from_phone_numberwas called twice and should use a different pepper each time, thanks to mocking, it will return the same pepper every time.</p><p>Until next time!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5263d301a1d6" width="1" height="1" alt=""><hr><p><a href="https://blog.manash.io/memoir-chapter-1-handling-unusual-test-case-scenarios-in-pytest-5263d301a1d6">[Memoir: Chapter 1] Handling unusual test case scenarios in PyTest</a> was originally published in <a href="https://blog.manash.io/">Manash’s blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to use $regex on aggregation using PyMongo]]></title>
            <link>https://blog.manash.io/how-to-use-regex-on-aggregation-using-pymongo-b9377935c721?source=rss-6f5d8825e907------2</link>
            <guid isPermaLink="false">https://medium.com/p/b9377935c721</guid>
            <category><![CDATA[regex]]></category>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[flask-pymongo]]></category>
            <category><![CDATA[pymongo]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Manash Kumar Mandal]]></dc:creator>
            <pubDate>Sat, 16 Feb 2019 16:59:19 GMT</pubDate>
            <atom:updated>2023-04-04T22:50:33.719Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hh-XlArIgmgOXLMD.png" /></figure><p>There is no straightforward documentation on how to use $regex operator on MongoDB aggregation using PyMongo driver.</p><p>But there is a slight <a href="http://api.mongodb.com/python/current/api/bson/regex.html">hint on this page</a> where you can get a basic idea.</p><p>Let’s assume MongoDB is installed in your computer and you are using Python. That’s the main reason you’re here, right?</p><p>Here is the simple code snippet.</p><h4>Using MongoDB Supported Regular Expressions</h4><pre>from bson import regex<br>from pymongo import MongoClient<br><br>client = MongoClient(&quot;localhost&quot;, 27017)<br>db = client[&quot;your_db_name&quot;]<br>collection = db[&quot;your_db_collection&quot;]<br>result = collection.aggregate(<br>    [{&quot;$match&quot;: {&quot;$key_to_apply_regex&quot;: {&quot;$regex&quot;: regex.Regex(&quot;regex|expression&quot;)}}}]<br>)<br>print(list(result))</pre><h4>Using Native Regular Expressions</h4><p>Everything remains similar, but there will be a couple of extra steps.</p><pre>import re<br>from bson import regex<br>from pymongo import MongoClient<br><br>client = MongoClient(&quot;localhost&quot;, 27017)<br>db = client[&quot;your_db_name&quot;]<br>collection = db[&quot;your_db_collection&quot;]<br>result = collection.aggregate(<br>    [<br>        {<br>            &quot;$match&quot;: {<br>                &quot;$key_to_apply_regex&quot;: {<br>                    &quot;$regex&quot;: regex.Regex.from_native(re.compile(&quot;.*&quot;))<br>                }<br>            }<br>        }<br>    ]<br>)<br>print(list(result))</pre><p>It will also work in Flask-Pymongo since it is based on the PyMongo driver. That’s it! Thanks for reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9377935c721" width="1" height="1" alt=""><hr><p><a href="https://blog.manash.io/how-to-use-regex-on-aggregation-using-pymongo-b9377935c721">How to use $regex on aggregation using PyMongo</a> was originally published in <a href="https://blog.manash.io/">Manash’s blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi-label Learning in Keras | Implementation of Multi-label Classification Loss]]></title>
            <link>https://blog.manash.io/multi-task-learning-in-keras-implementation-of-multi-task-classification-loss-f1d42da5c3f6?source=rss-6f5d8825e907------2</link>
            <guid isPermaLink="false">https://medium.com/p/f1d42da5c3f6</guid>
            <category><![CDATA[deep-le]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[multi-label]]></category>
            <category><![CDATA[multitask-learning]]></category>
            <dc:creator><![CDATA[Manash Kumar Mandal]]></dc:creator>
            <pubDate>Sun, 18 Feb 2018 15:58:50 GMT</pubDate>
            <atom:updated>2023-05-12T23:29:26.051Z</atom:updated>
            <content:encoded><![CDATA[<p>Using neural networks and with the help of high level tensor libraries we can build models that can handle classification, regression and other tasks easily. When I did coursera specialization on deep learning, watching a video on multi-label learning by Andrew Ng I quickly set up my mind to try this out. But the resources available on the internet were sort of a bit alien to me. I wanted to make this problem as simple as it was shown on the video.</p><p>This article might reflect the resource offered by the deeplearning.ai course.</p><p>Finally I could manage some time to try this idea and guess what it worked perfectly, otherwise I wouldn’t have been writing this article :P.</p><h3>What is Multi-label Learning?</h3><p>Let’s learn some basics before we move on. Multi-label learning means using one model, here using one neural network we will do simultaneously several tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rrIJOpJO8fkFECNHlwq-jQ.png" /><figcaption>Inspired from <a href="http://ruder.io/multi-task-learning-nlp/">Sebastian Ruder’s blog</a></figcaption></figure><h4>Example:</h4><p>Suppose you want to build a self-driving car and a part of the problem is you have to classify objects on the street.</p><p>Input will be an image and output will be a 1D vector.</p><p>Let’s consider four classes for now which are, <strong>Y={Pedestrian, Car, Stop Sign, Traffic Light}.</strong></p><p>We are not going to localize object we are just going to classify them. Our model will only say if there exists objects from the set Y or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*9TN4b0-uMvKf3yWd793fZg.png" /><figcaption>The image only contains stop sign and a car.</figcaption></figure><p>So the output vector of this image would be [0, 1, 1, 0], since we are considering the mentioned classes for the time being.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RXWO8pWJelvFJrGEr8sRrg.png" /><figcaption>Where X is the input image and y hat is the prediction</figcaption></figure><p>This image above will be the network which will solve these 4 tasks alone.</p><h3>What About the Loss Function?</h3><p>There is a slight problem, you can’t neither train the network with a binary crossentropy loss nor with a categorical cross entropy loss. We need to be a bit clever here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/0*QwzWIymQBxtJbQLH." /></figure><p>This is nothing but the log loss applied on each class separately. What we are doing here is aggregating all of the separate loss into one and then just computing average of it. So k in this loss function represents number of classes we are going to classify from, and rest bears the conventional meaning, such as m means number of training examples and y hat means predicted output.</p><h3>When to use Multi-label Learning?</h3><ul><li>Training on a set of tasks that could benefit from having shared lower-level features</li><li>Amount of data you have for each task should be quite similar</li><li>A big enough neural network can do well on all tasks</li></ul><p>It’s a better idea to use a single network to do all tasks instead of using separate models for separate tasks.</p><p>We know what multi-label learning is, we know what’s the loss function for the problem to solve. Let’s pour down everything in code to see if it actually works</p><h3>Downloading and Viewing the dataset</h3><p>Download the dataset from <a href="https://github.com/manashmndl/Multitask_Learning_Keras/releases/download/Data/dataset.h5">this URL</a>. If you want to work on my repository then <a href="https://github.com/manashmndl/Multitask_Learning_Keras">clone this repo</a> then put the h5 file on data folder.</p><pre>import matplotlib.pyplot as plt<br>import h5py<br>from sklearn.model_selection import train_test_split<br><br>def load():<br>    f = h5py.File(&quot;./data/dataset.h5&quot;)<br>    x = f[&#39;x&#39;].value<br>    y = f[&#39;y&#39;].value<br>    f.close()<br>    <br>    x_train , x_test, y_train, y_test = train_test_split(x,y,test_size=0.2,random_state=100)<br>    <br>    # Making the data channel last<br>    x_train = np.rollaxis(x_train, 1, 4)<br>    x_test = np.rollaxis(x_test, 1, 4)<br>    <br>    # Normalizing data<br>    x_train = x_train  / 255.0<br>    x_test = x_test / 255.0<br>   <br>    return x_train, x_test, y_train, y_test<br><br>x_train, x_test, y_train, y_test = load()<br><br>plt.imshow(x_train[0])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/254/1*lQgAb3UoXMaecSf913gPNA.png" /></figure><h4>Input data matrix</h4><pre>x_train.shape<br># (1600, 100, 100, 3)</pre><h4>Output or Target Vector</h4><p>If you visit the URL for the data, you can see there are 2k images with natural scene. And each image could be one of these classes.</p><p><strong><em>desert</em>, <em>mountains</em>, <em>sea</em>, <em>sunset</em> and <em>trees</em></strong><em>. </em>So the dimension of output vector should be like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/121/0*OLkMUKj_9LMF15db." /></figure><h3>Implementation of Multi-label Loss Function</h3><p>The implementation of multi-label loss function is like this.</p><pre>import keras.backend as K<br><br>def multitask_loss(y_true, y_pred):<br>    # Avoid divide by 0<br>    y_pred = K.clip(y_pred, K.epsilon(), 1 - K.epsilon())<br>    # Multi-label loss<br>    return K.mean(K.sum(- y_true * K.log(y_pred) - (1 - y_true) * K.log(1 - y_pred), axis=1))</pre><p>Importing numpy as np then replacing all K with np will yield the implementation in NumPy except the clipping part.</p><h4>Activation of Last Layer</h4><p>If you are wondering what kind of activation you should use on the last layer. Let me spoil it for you, it’s <strong>sigmoid</strong>.</p><h3>Model Building and Training</h3><p>There are only 2k images and I am going to use 1.6k as training set and rest will be test set. So my target would be to build an overfitted model instead of a well generalized one. <strong>The</strong> <strong>task is left as an exercise to the reader. :P</strong></p><p>Without thinking much, a CNN model is built then I fed the training and validation examples with the custom loss function.</p><pre>from keras.models import Sequential<br>from keras.layers import Dense, Dropout, Flatten<br>from keras.layers import Conv2D, MaxPooling2D, Activation<br><br>batch_size = 100<br>num_classes = 5<br>epochs = 50<br><br># input image dimensions<br>img_rows, img_cols = 100, 100<br>channels = 3<br><br>model = Sequential()<br>model.add(Conv2D(32, kernel_size=(3, 3),padding=&#39;same&#39;,input_shape=(img_rows, img_cols, channels)))<br>model.add(Activation(&#39;relu&#39;))<br>model.add(Conv2D(32, (3, 3)))<br>model.add(Activation(&#39;relu&#39;))<br>model.add(MaxPooling2D(pool_size=(2, 2)))<br>model.add(Dropout(0.25))<br><br>model.add(Conv2D(64,(3, 3), padding=&#39;same&#39;))<br>model.add(Activation(&#39;relu&#39;))<br>model.add(Conv2D(64, (3, 3)))<br>model.add(Activation(&#39;relu&#39;))<br>model.add(MaxPooling2D(pool_size=(2, 2)))<br>model.add(Dropout(0.25))<br><br>model.add(Flatten())<br>model.add(Dense(512))<br>model.add(Activation(&#39;relu&#39;))<br>model.add(Dropout(0.5))<br>model.add(Dense(num_classes))<br>model.add(Activation(&#39;sigmoid&#39;))<br><br><br>model.compile(loss=multitask_loss,<br>              optimizer=&#39;adam&#39;,<br>              metrics=[&#39;accuracy&#39;])<br><br>model.fit(x_train, y_train,<br>          batch_size=batch_size,<br>          epochs=epochs,<br>          verbose=1,<br>          validation_data=(x_test, y_test))</pre><h4>Code Output</h4><pre>Epoch 1/50<br>1600/1600 [==============================] - 6s 4ms/step - loss: 2.7115 - acc: 0.3275 - val_loss: 2.3993 - val_acc: 0.4275<br>Epoch 2/50<br>1600/1600 [==============================] - 2s 1ms/step - loss: 2.0483 - acc: 0.5375 - val_loss: 1.9218 - val_acc: 0.5050<br>Epoch 3/50<br>1600/1600 [==============================] - 2s 1ms/step - loss: 1.8621 - acc: 0.5888 - val_loss: 1.8554 - val_acc: 0.5925<br>....<br>....<br>....<br>1600/1600 [==============================] - 2s 1ms/step - loss: 0.0560 - acc: 0.8706 - val_loss: 4.4425 - val_acc: 0.6500<br>Epoch 47/50<br>1600/1600 [==============================] - 2s 1ms/step - loss: 0.0719 - acc: 0.8700 - val_loss: 4.2263 - val_acc: 0.6950<br>Epoch 48/50<br>1600/1600 [==============================] - 2s 1ms/step - loss: 0.0571 - acc: 0.8744 - val_loss: 4.4416 - val_acc: 0.6750<br>Epoch 49/50<br>1600/1600 [==============================] - 2s 1ms/step - loss: 0.0615 - acc: 0.8687 - val_loss: 4.1540 - val_acc: 0.6550<br>Epoch 50/50<br>1600/1600 [==============================] - 2s 1ms/step - loss: 0.0394 - acc: 0.8638 - val_loss: 4.3329 - val_acc: 0.6800</pre><p>The validation accuracy is 68% and training accuracy is almost 86% and you can see the loss is going down from first to last iteration.</p><h3>Inference</h3><p>Now the big problem is how do you infer using this model? Let’s solve the problem.</p><p>I will apply a element wise threshold here, if the output of a single node is greater than 0.5, then I’ll assign 1 otherwise 0.</p><pre>def infer(input_data, model=model):<br>    inference = []<br>    <br>    classes = np.array([&#39;desert&#39;, &#39;mountain&#39;, &#39;sea&#39;, &#39;sunset&#39;, &#39;trees&#39;])<br>    <br>    y_pred = model.predict(input_data)<br>    <br>    # Performing masking<br>    y_pred = (y_pred &gt; 0.5) * 1.0<br>    <br>    for i in range(y_pred.shape[0]):<br>        # select the indices<br>        indices = np.where(y_pred[i] == 1.0)[0]<br>        # Adding the results <br>        inference.append(classes[indices].tolist())<br>        <br>    return inference</pre><p>This function does the all.</p><p>Here is the demo output.</p><pre>infer(x_train, model=model)<br>&quot;&quot;&quot;<br>[[&#39;sea&#39;, &#39;sunset&#39;],<br> [&#39;sea&#39;],<br> [&#39;mountain&#39;],<br> [&#39;sea&#39;, &#39;sunset&#39;],<br> [&#39;sunset&#39;],<br> [&#39;trees&#39;],<br> [&#39;sea&#39;],<br> [&#39;trees&#39;],<br> [&#39;mountain&#39;, &#39;trees&#39;],<br> [&#39;sea&#39;],<br> [&#39;sunset&#39;],<br> [&#39;desert&#39;],<br> [&#39;sea&#39;],<br> [&#39;desert&#39;],<br> [&#39;desert&#39;],<br> [&#39;desert&#39;],<br> [&#39;desert&#39;],<br>.....<br>&quot;&quot;&quot;</pre><p>That’s it for today. I showed how to implement a custom loss function in keras and train it using a CNN. I hope you enjoyed the blog post.</p><h4>Edit</h4><p>I mistakenly put Multi-task instead of Multi-label. Apologies for that. I was naive when I started writing and also couldn’t take constructive criticism.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f1d42da5c3f6" width="1" height="1" alt=""><hr><p><a href="https://blog.manash.io/multi-task-learning-in-keras-implementation-of-multi-task-classification-loss-f1d42da5c3f6">Multi-label Learning in Keras | Implementation of Multi-label Classification Loss</a> was originally published in <a href="https://blog.manash.io/">Manash’s blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing PCA, Feedforward and Convolutional Autoencoders and using it for Image Reconstruction…]]></title>
            <link>https://blog.manash.io/implementing-pca-feedforward-and-convolutional-autoencoders-and-using-it-for-image-reconstruction-8ee44198ea55?source=rss-6f5d8825e907------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ee44198ea55</guid>
            <category><![CDATA[autoencoder]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[image-retrieval]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[convolutional-network]]></category>
            <dc:creator><![CDATA[Manash Kumar Mandal]]></dc:creator>
            <pubDate>Tue, 09 Jan 2018 14:05:11 GMT</pubDate>
            <atom:updated>2018-01-09T14:37:44.919Z</atom:updated>
            <content:encoded><![CDATA[<h3>Implementing PCA, Feedforward and Convolutional Autoencoders and using it for Image Reconstruction, Retrieval &amp; Compression</h3><p>My interaction with autoencoders is completely new. Naturally there will be some errors you might find in this blog post. I am actually going to implement some variants of autoencoders in Keras and write some theoretical stuffs along the way.</p><p>But first let’s get to know the first topic mentioned here. PCA or Principal Component Analysis.</p><h3>Principal Component Analysis (PCA)</h3><p>PCA is the most popular instance of second main class of unsupervised projection methods which is also known as Dimensionality Reduction methods.</p><p>If you find this writing about PCA dull. Go to this <a href="http://setosa.io/ev/principal-component-analysis/">interactive visualization website to get more intuition</a>.</p><p><strong>AIM: </strong>The aim of PCA is to find small number of ‘directions’ in input space that explain variations in input data. Representing data by projecting along those direction.</p><h4>Application</h4><p>One of the important questions is, what are the useful applications of PCA?</p><ul><li>Visualization</li><li>Preprocessing</li><li>Modeling — Good prior for new data</li><li>Data Compression</li></ul><h4>Intuition</h4><ul><li>Assuming input data X with number of samples N with dimension of D. Representing as,</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/158/0*XnqRkq0NUA5IFJP6." /></figure><ul><li>Suppose the lower dimensional space is represented by M, so the objective would be to represent X in lower dimensional space M from dimension D. We can write</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/148/0*a9HdkO7cDHzjVGPA." /></figure><ul><li>PCA would search for orthogonal directions in space with respect to highest variance and then project data into this M subspace.</li><li>Structure of data vectors is encoded in sample covariance.</li></ul><h4>Finding Principal Components</h4><ul><li>To find the principal component directions, we have to centralize the data. Meaning, subtracting the sample mean from each variable.</li><li>Then compute the empirical covariance matrix:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/0*1b7VU08J6-EJK3a3." /></figure><ul><li>Find the M eigenvectors with largest eigenvalues of C: These are the principal components</li><li>Assemble these eigenvectors into a D X M matrix called U</li><li>We can now express D-dimensional vectors x by projecting them to M-dimensional z</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/144/0*Is2AuWXDvl9qTSNO." /></figure><p>What it basically does is maximize variance while keeping the reconstruction error minimized.</p><h3>PCA: Minimizing Reconstruction Error</h3><p>Since we can think of PCA as projecting data onto a lower-dimensional subspace. ONe derivation is that we want to find the projection such that the best linear reconstructions of the data is as close as possible to the original data.</p><p>The loss function is simply MSE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/319/0*vmczj8ASrzPGDx2e." /></figure><p>Where,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/469/0*fLpvzrTwsHkxoPPs." /></figure><p>And we already know</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/162/0*xMIJCswPPJoUxL8X." /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/161/0*4RSZCBdohXLEms-x." /></figure><h4>PCA in a nutshell</h4><p>Here is a summary of PCA for the impatients.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LBs6_kJ4VXWLQwKA.jpg" /><figcaption>Credit: <a href="https://www.youtube.com/watch?v=QP43Iy-QQWY">Victor Lavrenko</a></figcaption></figure><h3>PCA : Autoencoders</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*r5O7C8J7t3lrHJo5trST4g.png" /></figure><p>I think this was enough to brush up the theories of PCA. Let’s get to the point, what is the relation between PCA and autoencoder, how could we define one and implement one in our favorite programming language Python and most favorite deep learning framework named Keras.</p><p>The goal is to encode the image information in lower dimensional space then reconstruct it again from encoded lower dimensional representation to original form.</p><p>Encoded space is lower dimensional so PCA has to learn the most important features from which it can decode it again to keep the Mean Square Error minimum.</p><p>Again some mathematical stuffs then we will get down to coding.</p><p>Thinking this with respect to image representation will help you to understand.</p><h4>Encoding</h4><p>Suppose we are doing some affine transformation on input data then encoding it to a latent space named z. The representation would look something like this one.</p><p>Here f is an activation function, we are keeping it linear for the time being.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/216/0*rSMXd-KUNo0PhDQp." /></figure><h4>Decoding</h4><p>Great, we encoded all information of X into latent space z. Now we need to decode it back by some affine transformation again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/178/0*eTXJSa47zfZkKURC." /></figure><p>z being a function of f we can write,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/324/0*2Ivq_Zi0DzG_IS-c." /></figure><p>Assuming g and f are linear activations. We can get rid of the functions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/218/0*1ZDChjv9yMvli45f." /></figure><h4>Objective</h4><p>The goal is to minimize this loss function with respect to W and V matrices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/0*jz0tt2ciVyt2fEhF." /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/0*EN56YRkL_JIIIaTO." /></figure><p>What if g() is not linear, then we are basically doing nonlinear PCA.</p><h3>Implementation of PCA Autoencoder</h3><p>Yeah finally, but first, we need to download some dataset to test the autoencoder. No! Enough of MNIST dataset, let’s try something else to train on.</p><p>Here is the link of <a href="http://vis-www.cs.umass.edu/lfw/lfw.tgz">image data</a>. And link of <a href="http://www.cs.columbia.edu/CAVE/databases/pubfig/download/lfw_attributes.txt">attributes</a>. Download both and put them in one folder.</p><h4>Project Structure</h4><pre>Autoencoders/<br>     |<br>     |---- lfw_dataset.py<br>     |---- Autoencoder.ipynb<br>     |<br>     |---- data/<br>             |<br>             |---- lfw.tgz<br>             |---- lfw_attributes.txt</pre><h4>Loading Dataset</h4><p>I am going to use this script to load the dataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/33b9427772cb31f77825135150c94c56/href">https://medium.com/media/33b9427772cb31f77825135150c94c56/href</a></iframe><pre>#Import Stuff<br>import numpy as np<br>from sklearn.model_selection import train_test_split<br>from lfw_dataset import load_lfw_dataset<br>import tensorflow as tf<br>import keras, keras.layers as L<br>s = keras.backend.get_session()</pre><pre># Loading and normalizing [Might take some time]<br>X, attr = load_lfw_dataset(use_raw=True,dimx=38,dimy=38)<br>X = X.astype(&#39;float32&#39;) / 255.0</pre><pre>img_shape = X.shape[1:]</pre><pre>X_train, X_test = train_test_split(X, test_size=0.1, random_state=42)</pre><pre># Checking out some images<br>%matplotlib inline<br>import matplotlib.pyplot as plt<br>plt.title(&#39;sample image&#39;)<br>for i in range(6):<br>    plt.subplot(2,3,i+1)<br>    plt.imshow(X[i])</pre><pre>print(&quot;X shape:&quot;,X.shape)<br>print(&quot;attr shape:&quot;,attr.shape)</pre><p><strong>Output</strong></p><pre>X shape: (13143, 38, 38, 3)<br>attr shape: (13143, 73)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/0*2e3ACOkQzXfrQInu.png" /></figure><h4>Coding the PCA Autoencoder</h4><p>We could actually implement the autoencoder in a couple of ways.</p><p>Here is one way,</p><pre>code_size = 32<br>pca_autoencoder = keras.models.Sequential()</pre><pre># Input layer<br>pca_autoencoder.add(L.InputLayer(img_shape))</pre><pre># Flattening the layer<br>pca_autoencoder.add(L.Flatten())</pre><pre># Encoded space<br>pca_autoencoder.add(L.Dense(code_size))</pre><pre># Output units should be image_size * image_size * channels <br>pca_autoencoder.add(L.Dense(np.prod(img_shape)))</pre><pre># Last layer <br>pca_autoencoder.add(L.Reshape(img_shape))</pre><p>Model summary</p><pre>Layer (type)                 Output Shape              Param #   <br>=================================================================<br>input_10 (InputLayer)        (None, 38, 38, 3)         0         <br>_________________________________________________________________<br>flatten_6 (Flatten)          (None, 4332)              0         <br>_________________________________________________________________<br>dense_11 (Dense)             (None, 32)                138656    <br>_________________________________________________________________<br>dense_12 (Dense)             (None, 4332)              142956    <br>_________________________________________________________________<br>reshape_5 (Reshape)          (None, 38, 38, 3)         0         <br>=================================================================<br>Total params: 281,612<br>Trainable params: 281,612<br>Non-trainable params: 0<br>_________________________________________________________________</pre><p>Training the model</p><pre>pca_autoencoder.compile(&#39;adamax&#39;, &#39;mse&#39;)<br>pca_autoencoder.fit(x=X_train,y=X_train,epochs=10, batch_size=500,<br>                validation_data=[X_test,X_test])</pre><p>Output</p><pre>Train on 11828 samples, validate on 1315 samples<br>Epoch 1/10<br>11828/11828 [==============================]<br>loss: 0.0413 - val_loss: 0.0130</pre><pre>Epoch 2/10<br>11828/11828 [==============================]<br>50us/step - loss: 0.0087 - val_loss: 0.0074</pre><pre>Epoch 3/10<br>11828/11828 [==============================]<br>49us/step - loss: 0.0072 - val_loss: 0.0070</pre><pre>Epoch 4/10<br>11828/11828 [==============================]<br>49us/step - loss: 0.0070 - val_loss: 0.0069</pre><pre>Epoch 5/10<br>11828/11828 [==============================]<br>50us/step - loss: 0.0069 - val_loss: 0.0069</pre><pre>Epoch 6/10<br>11828/11828 [==============================] <br>54us/step - loss: 0.0069 - val_loss: 0.0068</pre><pre>Epoch 7/10<br>11828/11828 [==============================] <br>54us/step - loss: 0.0069 - val_loss: 0.0068</pre><pre>Epoch 8/10<br>11828/11828 [==============================] <br>51us/step - loss: 0.0068 - val_loss: 0.0068</pre><pre>Epoch 9/10<br>11828/11828 [==============================]<br>53us/step - loss: 0.0068 - val_loss: 0.0068</pre><pre>Epoch 10/10<br>11828/11828 [==============================] <br>53us/step - loss: 0.0068 - val_loss: 0.0068</pre><h4>Coding the autoencoder in other way</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2db733ccfda64df6c3db6ba57180d90a/href">https://medium.com/media/2db733ccfda64df6c3db6ba57180d90a/href</a></iframe><h3>Visualization of Reconstructed Output and the Code itself</h3><p>We are going to need a helper function to visualize the codes along with the outputs.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/08495a030bec6c5cba6f6badaa8f8f24/href">https://medium.com/media/08495a030bec6c5cba6f6badaa8f8f24/href</a></iframe><p><strong>Output</strong></p><pre>Final MSE: 0.00678860718958</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/0*Pi3h5_y7WY1qnwOX.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/0*dqbL732BtqfU_vAn.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/0*pxtvNh1S1EhA8NIn.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/0*tEsvMGGy8pPTY09O.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/0*Er9edLU1mhjG7lhH.png" /></figure><h3>Making a Deep Autoencoder using Feedforward Neural Network</h3><p>Autoencoders may be though of as being a special case of feedforward networks and can be trained with all of the same techniques.</p><p>General structure of an autoencoder is given below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/229/1*aQH7OFy7IrTFFhA835_DYQ.png" /><figcaption>Figure 14.1 : From Deep Learning Book by Ian Goodfellow</figcaption></figure><p>Where x is input and h is the internal representation and r is the reconstructed output from the representation.</p><h4>Deep Feedforward Autoencoder</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/955/1*wtJCjAFJTScUDqYW9g1Ijw.png" /><figcaption>Borrowed Image</figcaption></figure><p>This image represents a rough idea, we are actually going to build an autoencoder deeper than the depicted image.</p><h4>Sanity Checks</h4><ul><li>There shouldn’t be any hidden layer smaller than bottleneck (encoder output)</li><li>Adding nonlinearities between intermediate dense layers yield good result.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/62175700d4788baf1ba6ad06a912e24c/href">https://medium.com/media/62175700d4788baf1ba6ad06a912e24c/href</a></iframe><p>These are all examples of <strong>Undercomplete Autoencoders</strong> since the code dimension is less than the input dimension. If the encoder and decoder are allowed too much capacity, the autoencoder can learn to perform the copying task without extracting useful information about the distribution of data.</p><h3>Deep Convolutional Autoencoder</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/0*ObdQKr4S5PrKXs61.png" /></figure><p>Author of Keras has already explained and implemented variations of AE in his post. <a href="https://blog.keras.io/building-autoencoders-in-keras.html">You can check this out.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TOJD69Y8dZsKFEW-21xUPg.png" /><figcaption>Stolen from Hacker Noon :P</figcaption></figure><p>Here’s another image from internet to visualize autoencoders in a more intuitive way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/0*Dw-cELLTzWY34hxE.png" /></figure><p>Building Convolutional Autoencoder is simple as building a ConvNet, the decoder is the mirror image of encoder. That’s basically it!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8ef43a27aadd3601ec55b3fdfbefd4d2/href">https://medium.com/media/8ef43a27aadd3601ec55b3fdfbefd4d2/href</a></iframe><h3>Regularized ‘X’ Autoencoder</h3><p>Simply add a kernel_regularizer to the last layer of encoder. You can make any autoencoder regularized by this way. Here is the general step.</p><pre>encoder.add(L.Dense(code_size, kernel_regularizer=keras.regularizer.l2(0.01))</pre><h3>Denoising Autencoder</h3><p>Here is the computational graph from Deep Learning Textbook.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/0*mKDbng0hD7yxa1-V.png" /></figure><p>At training time I will add random gaussian noise to training dataset.</p><pre>def apply_gaussian_noise(X,sigma=0.1):<br>    <br>    noise = np.random.normal(loc=0.0, scale=sigma, size=X.shape)<br>        <br>    return X + noise</pre><pre># Clipping the images for showing<br>plt.subplot(1,4,1)<br>plt.imshow(X[0])<br>plt.subplot(1,4,2)<br>plt.imshow( np.clip(apply_gaussian_noise(X[:1],sigma=0.01)[0], 0, 1))<br>plt.subplot(1,4,3)<br>plt.imshow(np.clip(apply_gaussian_noise(X[:1],sigma=0.1)[0], 0, 1))<br>plt.subplot(1,4,4)<br>plt.imshow(np.clip(apply_gaussian_noise(X[:1],sigma=0.5)[0],0, 1))</pre><p><strong>Output</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/0*agmXZYauvSnM9oB2.png" /></figure><p>Preparing the model again for new code size.</p><pre>encoder,decoder = build_deep_conv_autoencoder((44, 44, 3),code_size=512)</pre><pre>inp = L.Input(img_shape)<br>code = encoder(inp)<br>reconstruction = decoder(code)</pre><pre>autoencoder = keras.models.Model(inp,reconstruction)<br>autoencoder.compile(&#39;adamax&#39;,&#39;mse&#39;)</pre><pre># Training with noise<br>for i in range(50):<br>    print(&quot;Epoch %i/50, Generating corrupted samples...&quot;%i)<br>    X_train_noise = apply_gaussian_noise(X_train)<br>    X_test_noise = apply_gaussian_noise(X_test)<br>    <br>    autoencoder.fit(x=X_train_noise,y=X_train,epochs=1,<br>                    validation_data=[X_test_noise,X_test])</pre><pre># Evaluation using noisy input<br>denoising_mse = autoencoder.evaluate(apply_gaussian_noise(X_test),X_test,verbose=0)<br>print(&quot;Final MSE:&quot;, denoising_mse)<br>for i in range(5):<br>    img = X_test[i]<br>    visualize(img,encoder,decoder)</pre><h4>Output</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/0*MnQI0xV02jejw78N.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/0*yIpG5kwMtxaYYyNH.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/0*63xwkxKsXlvlXRuL.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/0*CjCo7I_XxV0aQqif.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/0*YjoNb5HL0CFNY8vy.png" /></figure><p>Great, we have implemented some of the autoencoders. So far autoencoders have not been useful to us. Let’s try to do some fun things using it.</p><p>This result was made using KNN with encoded size of 32. <strong>Not 512! To get similar result, you might have to train your autoencoder with this settings.</strong></p><pre>images = X_train</pre><pre># Hashing the image with encoder<br>codes = encoder.predict(images)</pre><pre><strong>def</strong> show_image(x):<br>    plt.imshow(np.clip(x + 0.5, 0, 1))</pre><pre><strong># Fitting the codes<br>from</strong> <strong>sklearn.neighbors.unsupervised</strong> <strong>import</strong> NearestNeighbors<br>nei_clf = NearestNeighbors(metric=&quot;euclidean&quot;)<br>nei_clf.fit(codes)</pre><pre><strong>def</strong> get_similar(image, n_neighbors=5):<br>    assert image.ndim==3,&quot;image must be [batch,height,width,3]&quot;</pre><pre>code = encoder.predict(image[None])<br>    <br>    (distances,),(idx,) = nei_clf.kneighbors(code,n_neighbors=n_neighbors)<br>    <br>    return distances,images[idx]</pre><pre><strong>def</strong> show_similar(image):<br>    <br>    distances,neighbors = get_similar(image,n_neighbors=3)<br>    <br>    plt.figure(figsize=[8,7])<br>    plt.subplot(1,4,1)<br>    show_image(image)<br>    plt.title(&quot;Original image&quot;)<br>    <br>    for i in range(3):<br>        plt.subplot(1,4,i+2)<br>        show_image(neighbors[i])<br>        plt.title(&quot;Dist=%.3f&quot;%distances[i])<br>    plt.show()</pre><pre># Cherry picked examples</pre><pre># smiles<br>show_similar(X_test[247])</pre><pre><em># ethnicity</em><br>show_similar(X_test[56])</pre><pre><em># glasses</em> <br>show_similar(X_test[63])</pre><p><strong>Output</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/0*s26pcvLHBT1FhQBr.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/0*d75zEDbA-cA7M_bh.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/0*8gMVRg5zA3BU4_AG.png" /></figure><p>That’s it for today! Codes will be uploaded to GitHub soon enough! I hope you had fun as much I had exploring autoencoders. Pretty neat thing, right? Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ee44198ea55" width="1" height="1" alt=""><hr><p><a href="https://blog.manash.io/implementing-pca-feedforward-and-convolutional-autoencoders-and-using-it-for-image-reconstruction-8ee44198ea55">Implementing PCA, Feedforward and Convolutional Autoencoders and using it for Image Reconstruction…</a> was originally published in <a href="https://blog.manash.io/">Manash’s blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Dead Simple Speech Recognition Engine using ConvNet in Keras]]></title>
            <link>https://blog.manash.io/building-a-dead-simple-word-recognition-engine-using-convnet-in-keras-25e72c19c12b?source=rss-6f5d8825e907------2</link>
            <guid isPermaLink="false">https://medium.com/p/25e72c19c12b</guid>
            <category><![CDATA[mfcc]]></category>
            <category><![CDATA[speech-recognition]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[keras]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Manash Kumar Mandal]]></dc:creator>
            <pubDate>Tue, 21 Nov 2017 16:40:50 GMT</pubDate>
            <atom:updated>2023-05-12T23:54:30.462Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aO4f4GijwPT1l4lL.jpg" /><figcaption>Image Courtesy (<a href="https://forums.creativecow.net/thread/2/1067998">https://forums.creativecow.net/thread/2/1067998</a>)</figcaption></figure><p>So you’ve classified MNIST dataset using Deep Learning libraries and want to do the same with speech recognition! Well continuous speech recognition is a bit tricky so to keep everything simple I am going to start with a simpler problem instead. Which is <strong>word</strong> recognition. I’ve seen a competition going on at <a href="https://www.kaggle.com/c/tensorflow-speech-recognition-challenge">Kaggle</a> and couldn’t help but downloading the dataset.</p><p>If you think this blog post will make you an expert in Speech Recognition field please feel free to skip it. I am going to show you some quick techniques to be up and running in speech recognition area rather going deeper.</p><h3>Before Beginning</h3><ul><li>Python 3.5 should be installed</li><li>Following libraries will be used throughout the article, make sure you’ve installed it before trying out the codes.</li><li>librosa, keras, tensorflow, scikit-learn, numpy, scipy</li></ul><h3>So what am I going to cook?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*KSIFLZgoNTv3W7qx.jpg" /><figcaption>I’m the one who cooks</figcaption></figure><p>Nothing special, basically this one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BT64gvpuxnNM54PFcZ6DRA.png" /></figure><p>Yes I know directly feeding a speech signal to a ConvNet model doesn’t seem right. There are some preprocessing steps which I’ll introduce soon enough.</p><h3>Let’s take a look at the data</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/1*_MNViMtVgfigiw5-PK-yxQ.png" /></figure><p>There are basically three classes to recognize from. You can add as much as you may wish. Just change the number of output at the softmax layer and you’ll be good to go. Don’t forget to download data from Kaggle if you are willing to classify more than these three.</p><p>Each folder contains approximately 1700 audio files. The name of the folder is actually the <strong>label</strong> of those audio files. You can play some audio files randomly to get an overall idea.</p><p>The task will be to classify an audio between <strong>bed, cat</strong> and <strong>happy — these three classes</strong>.</p><h3>First things first</h3><p>You can’t just feed an audio file to a ConvNet, that’s outrageous! We need to prepare a <strong>fixed size vector </strong>for each audio file for classification. What can we do here? Represent the audio files into one-hot encoding? That doesn’t make any sense, right?</p><h3>Embedding</h3><p>An <strong>embedding</strong> is a mapping from discrete objects, such as words, to vectors of real numbers. [From TensorFlow]</p><p>TensorFlow guide contains a very good guide on embedding. If you have no idea what I am talking about, check it out.</p><ul><li><a href="https://www.tensorflow.org/programmers_guide/embedding">Embeddings | TensorFlow</a></li><li><a href="https://www.tensorflow.org/tutorials/word2vec">Vector Representations of Words | TensorFlow</a></li></ul><p>But we are dealing with a specific domain of embedding. Which is audio embedding.</p><h3>Audio Embedding</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iCfvizPZ4_YznDupa0hRAw.png" /></figure><p>So here is the problem, what technique we should actually use to embed the audio into vector space? There are a lot of techniques you can find online, even there are some python packages solely for audio feature extraction. But to keep everything simple. I am going to use the most obvious and simple (!) one which is called <strong>MFCC</strong> encoding, which is super effective for working with speech signals.</p><h3>MFCC (Mel Frequency Cepstral Coefficients)</h3><p>You can know a lot from internet by searching a bit. Explaining MFCC is not the target of this article. In short,</p><blockquote>In <a href="https://en.wikipedia.org/wiki/Sound_processing">sound processing</a>, the <strong>mel-frequency cepstrum</strong> (<strong>MFC</strong>) is a representation of the short-term <a href="https://en.wikipedia.org/wiki/Power_spectrum">power spectrum</a> of a sound, based on a <a href="https://en.wikipedia.org/wiki/Cosine_transform">linear cosine transform</a> of a <a href="https://en.wikipedia.org/wiki/Power_spectrum">log power spectrum</a> on a <a href="https://en.wikipedia.org/wiki/Nonlinear_system">nonlinear</a> <a href="https://en.wikipedia.org/wiki/Mel_scale">mel scale</a> of frequency.</blockquote><p>Yeah a lot to process, you can get an overview how this is computed from an audio signal. Which is clearly written in Wikipedia’s wiki.</p><p><a href="https://en.wikipedia.org/wiki/Mel-frequency_cepstrum">Mel-frequency cepstrum - Wikipedia</a></p><p>For the time being, just assume that MFCC can create useful vectors from audio signal for our deep learning model to work with.</p><h3>Preprocessing Audio Files</h3><p>The whole code can <a href="https://github.com/manashmndl/DeadSimpleSpeechRecognizer">be found here</a>. So don’t worry about it. I am going to explain my workflow and explain some piece of codes.</p><ol><li>First step will be to read the audio file from given path</li><li>I am going to take audio input from one channel only</li><li>Then perform downsample operation [You can skip it if you have enough computational power]</li><li>Then compute <strong>MFCC </strong>using librosa library</li><li>MFCC vectors might vary in size for different audio input, remember ConvNets can’t handle sequence data so we need to prepare a fixed size vector for all of the audio files.</li><li>To overcome this problem all we need to do is to pad the output vectors with constant value (the one I used here is 0 ).</li></ol><p>Here is my function which reads an audio file and computes MFCC.</p><h4>wav2mfcc</h4><pre>import numpy as np<br>import librosa<br><br>def wav2mfcc(file_path, max_pad_len=11):<br>    wave, sr = librosa.load(file_path, mono=True, sr=None)<br>    wave = wave[::3]<br>    mfcc = librosa.feature.mfcc(wave, sr=16000)<br>    pad_width = max_pad_len - mfcc.shape[1]<br>    mfcc = np.pad(mfcc, pad_width=((0, 0), (0, pad_width)), mode=&#39;constant&#39;)<br>    return mfcc</pre><h3>Preparing MFCC for all available audio files and save the computed vectors in a NumPy file</h3><p>In get_labels function there are some unnecessary codes. I’ll fix those later if I get enough time. Let’s ignore it for now.</p><pre># get_labels.py<br><br>import numpy as np<br>import os<br><br>DATA_PATH = &quot;./data/&quot;<br><br># Input: Folder Path<br># Output: Tuple (Label, Indices of the labels, one-hot encoded labels)<br>def get_labels(path=DATA_PATH):<br>    labels = os.listdir(path)<br>    label_indices = np.arange(0, len(labels))<br>    return labels, label_indices, to_categorical(label_indices)</pre><pre># savedatatoarray.py<br><br>import numpy as np<br><br>def save_data_to_array(path=DATA_PATH, max_pad_len=11):<br>    labels, _, _ = get_labels(path)<br><br>    for label in labels:<br>        # Init mfcc vectors<br>        mfcc_vectors = []<br><br>        wavfiles = [path + label + &#39;/&#39; + wavfile for wavfile in os.listdir(path + &#39;/&#39; + label)]<br>        for wavfile in wavfiles:<br>            mfcc = wav2mfcc(wavfile, max_pad_len=max_pad_len)<br>            mfcc_vectors.append(mfcc)<br>        np.save(label + &#39;.npy&#39;, mfcc_vectors)</pre><p>save_data_to_array reads all audio files from each directory and save the vectors in a .npy file which is named after the name of the label.</p><p>Since computing MFCC is time consuming, we will do it only once. And for later times we will just load it from the saved files, which will buy us some time to do other things ;)</p><h3>Prepare Train set and Test set</h3><p>Using the previous function we can compute MFCC but now we need to prepare train set and test set based on the data we have. To do this, I’ll take advantage of sklearn ‘s nice function train_test_split which will automatically split the whole dataset.</p><pre>DATA_PATH = &quot;./data/&quot;<br><br>def get_train_test(split_ratio=0.6, random_state=42):<br>    # Get available labels<br>    labels, indices, _ = get_labels(DATA_PATH)<br><br>    # Getting first arrays<br>    X = np.load(labels[0] + &#39;.npy&#39;)<br>    y = np.zeros(X.shape[0])<br><br>    # Append all of the dataset into one single array, same goes for y<br>    for i, label in enumerate(labels[1:]):<br>        x = np.load(label + &#39;.npy&#39;)<br>        X = np.vstack((X, x))<br>        y = np.append(y, np.full(x.shape[0], fill_value= (i + 1)))<br><br>    assert X.shape[0] == len(y)<br><br>    return train_test_split(X, y, test_size= (1 - split_ratio), random_state=random_state, shuffle=True)</pre><h3>Building the CNN model!</h3><p>Preprocessing takes away most of the time and it’s a pretty tedious job. But when everything works out, it feels good, right?</p><h3>But Wait! This audio is not three dimensional! How can I possibly apply CNN on it?</h3><p>Good question, but when we computed MFCC we actually converted the audio into sort of image like data.</p><pre>X_train, X_test, y_train, y_test = get_train_test()</pre><pre>print(X_train.shape)<br>#(3112, 20, 11)</pre><p>First number in the array shows the number of rows or audio samples to be precise. For each audio file the vector we get are 20 by 11 dimensional matrix. Yeah you guessed right! We can treat it just like an image. When we embed things it actually don’t matter what kind of data we are dealing with. Since everything gets converted into vector.</p><p><strong>Wait again!</strong></p><p>Yes we need to reshape it to give a depth. Just imagine you are working with an image which has a single channel only. Reshaping yields.</p><pre># # Loading train set and test set<br>X_train, X_test, y_train, y_test = get_train_test()<br><br># # Feature dimension<br>feature_dim_1 = 20<br>channel = 1<br>epochs = 50<br>batch_size = 100<br>verbose = 1<br>num_classes = 3<br><br># Reshaping to perform 2D convolution<br>X_train = X_train.reshape(X_train.shape[0], feature_dim_1, feature_dim_2, channel)<br>X_test = X_test.reshape(X_test.shape[0], feature_dim_1, feature_dim_2, channel)</pre><p>Now we are ready to feed it into CNN.</p><h4>One last thing!</h4><p>So much hindrance! Sorry for that, train test split output actually gives the output tag like this [0, 1, 2] which each number means separate class. If you remember your machine learning theories, we need to encode this output vector into one-hot-encoded one to perform softmax .</p><p>Thanks to Keras we can just use the built in function,</p><pre>y_train_hot = to_categorical(y_train)<br>y_test_hot = to_categorical(y_test)</pre><p>That’s it!</p><h3>Finally Building the CNN model</h3><p>Instead of putting too much thought in model architecture, I am going to snatch an established one [I know I am lazy] and change a few things. This is the model I am going to use,</p><p><a href="https://github.com/fchollet/keras/blob/master/examples/mnist_cnn.py">fchollet/keras</a></p><pre>from preprocess import *<br>import keras<br>from keras.models import Sequential<br>from keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPooling2D<br>from keras.utils import to_categorical<br>X_train, X_test, y_train, y_test = get_train_test()<br>X_train = X_train.reshape(X_train.shape[0], 20, 11, 1)<br>X_test = X_test.reshape(X_test.shape[0], 20, 11, 1)<br>y_train_hot = to_categorical(y_train)<br>y_test_hot = to_categorical(y_test)<br>model = Sequential()<br>model.add(Conv2D(32, kernel_size=(2, 2), activation=&#39;relu&#39;, input_shape=(20, 11, 1)))<br>model.add(MaxPooling2D(pool_size=(2, 2)))<br>model.add(Dropout(0.25))<br>model.add(Flatten())<br>model.add(Dense(128, activation=&#39;relu&#39;))<br>model.add(Dropout(0.25))<br>model.add(Dense(3, activation=&#39;softmax&#39;))<br>model.compile(loss=keras.losses.categorical_crossentropy,<br>              optimizer=keras.optimizers.Adadelta(),<br>              metrics=[&#39;accuracy&#39;])<br>model.fit(X_train, y_train_hot, batch_size=100, epochs=200, verbose=1, validation_data=(X_test, y_test_hot))</pre><h3>Output</h3><pre>....<br>Epoch 197/200<br>3112/3112 [==============================] - 0s 100us/step - loss: 0.0598 - <strong>acc: 0.9788 - val_loss: 0.2071 - val_acc: 0.9398</strong><br>Epoch 198/200<br>3112/3112 [==============================] - 0s 98us/step - loss: 0.0619 - <strong>acc: 0.9762 - val_loss: 0.2212 - val_acc: 0.9398</strong><br>Epoch 199/200<br>3112/3112 [==============================] - 0s 97us/step - loss: 0.0596 -<strong> acc: 0.9759 - val_loss: 0.2102 - val_acc: 0.9422</strong><br>Epoch 200/200<br>3112/3112 [==============================] - 0s 98us/step - loss: 0.0599 - <strong>acc: 0.9785 - val_loss: 0.2158 - val_acc: 0.9441</strong></pre><p>Nice accuracy!</p><h3>How to infer using the model?</h3><ol><li>Create MFCC from an audio file</li><li>Feed the vector to the model using the function predict</li><li>Perform argmax to find the index of the label</li></ol><p>Here</p><pre># Getting the MFCC<br>sample = wav2mfcc(&#39;./data/happy/012c8314_nohash_0.wav&#39;)<br># We need to reshape it remember?<br>sample_reshaped = sample.reshape(1, 20, 11, 1)<br># Perform forward pass<br>print(get_labels()[0][<br>    np.argmax(model.predict(sample_reshaped))<br>])<br># Output: &#39;happy&#39;</pre><p>I know it looks ugly, but works though!</p><h3>What’s next?</h3><ul><li>Apply CNN-LSTM to see how it works</li><li>Increase dimension of MFCC vectors</li><li>Don’t downsample</li><li>Include Batch Normalization Layer</li><li>Add more classes!</li></ul><p>There are a lot of things to do. Here is the project <a href="https://github.com/manashmndl/DeadSimpleSpeechRecognizer">repository.</a> Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=25e72c19c12b" width="1" height="1" alt=""><hr><p><a href="https://blog.manash.io/building-a-dead-simple-word-recognition-engine-using-convnet-in-keras-25e72c19c12b">Building a Dead Simple Speech Recognition Engine using ConvNet in Keras</a> was originally published in <a href="https://blog.manash.io/">Manash’s blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementation of Gradient Descent in TensorFlow using tf.gradients]]></title>
            <link>https://blog.manash.io/implementation-of-gradient-descent-in-tensorflow-using-tf-gradients-c111d783c78b?source=rss-6f5d8825e907------2</link>
            <guid isPermaLink="false">https://medium.com/p/c111d783c78b</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[gradient-descent]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[tensorflow-gradient]]></category>
            <dc:creator><![CDATA[Manash Kumar Mandal]]></dc:creator>
            <pubDate>Mon, 18 Sep 2017 14:37:47 GMT</pubDate>
            <atom:updated>2023-05-12T23:47:44.062Z</atom:updated>
            <content:encoded><![CDATA[<p>One of the best things I like about TensorFlow that it can automatically compute gradient of a function. All we need to do is to setup the equation then run tf.gradients function to compute the gradients. We don’t need to go through a lot of pages to calculate the gradients of a loss function then convert it into code. We can simply take the advantage of TensorFlow to compute the gradient for us.</p><p>Let’s look at some examples to clarify a bit more.</p><h3>Softmax Regression on MNIST dataset using TensorFlow’s built in Optimizer</h3><p>Before diving into implementing the cost functions using tf.gradients , let’s try to refresh our memory by solving MNIST problem using TensorFlow’s built in optimizer such as GradientDescentOptimizer.</p><pre>import tensorflow as tf<br><br># Import MNIST data<br>from tensorflow.examples.tutorials.mnist import input_data<br>mnist = input_data.read_data_sets(&quot;/tmp/data/&quot;, one_hot=True)<br><br># Parameters<br>learning_rate = 0.01<br>training_epochs = 10<br>batch_size = 100<br>display_step = 1<br><br><br># tf Graph Input<br>x = tf.placeholder(tf.float32, [None, 784]) # mnist data image of shape 28*28=784<br>y = tf.placeholder(tf.float32, [None, 10]) # 0-9 digits recognition =&gt; 10 classes<br><br># Set model weights<br>W = tf.Variable(tf.zeros([784, 10]))<br>b = tf.Variable(tf.zeros([10]))<br><br># Construct model<br>pred = tf.nn.softmax(tf.matmul(x, W) + b) # Softmax<br><br># Minimize error using cross entropy<br>cost = tf.reduce_mean(-tf.reduce_sum(y*tf.log(pred), reduction_indices=1))<br><br>optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)<br><br># Start training<br>with tf.Session() as sess:<br>    sess.run(tf.global_variables_initializer())<br><br>    # Training cycle<br>    for epoch in range(training_epochs):<br>        avg_cost = 0.<br>        total_batch = int(mnist.train.num_examples/batch_size)<br>        # Loop over all batches<br>        for i in range(total_batch):<br>            batch_xs, batch_ys = mnist.train.next_batch(batch_size)<br>            # Fit training using batch data<br>            _, c = sess.run([optimizer, cost], feed_dict={x: batch_xs,<br>                                                       y: batch_ys})<br>            <br>#             print(__w)<br>            <br>            # Compute average loss<br>            avg_cost += c / total_batch<br>        # Display logs per epoch step<br>        if (epoch+1) % display_step == 0:<br>#             print(sess.run(W))<br>            print (&quot;Epoch:&quot;, &#39;%04d&#39; % (epoch+1), &quot;cost=&quot;, &quot;{:.9f}&quot;.format(avg_cost))<br><br>    print (&quot;Optimization Finished!&quot;)<br><br>    # Test model<br>    correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))<br>    # Calculate accuracy for 3000 examples<br>    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))<br>    print (&quot;Accuracy:&quot;, accuracy.eval({x: mnist.test.images[:3000], y: mnist.test.labels[:3000]}))<br>    <br>    <br>#### Output<br>    <br># Extracting /tmp/data/train-images-idx3-ubyte.gz<br># Extracting /tmp/data/train-labels-idx1-ubyte.gz<br># Extracting /tmp/data/t10k-images-idx3-ubyte.gz<br># Extracting /tmp/data/t10k-labels-idx1-ubyte.gz<br># Epoch: 0001 cost= 1.184285608<br># Epoch: 0002 cost= 0.665428013<br># Epoch: 0003 cost= 0.552858426<br># Epoch: 0004 cost= 0.498728328<br># Epoch: 0005 cost= 0.465593693<br># Epoch: 0006 cost= 0.442609185<br># Epoch: 0007 cost= 0.425552949<br># Epoch: 0008 cost= 0.412188290<br># Epoch: 0009 cost= 0.401390140<br># Epoch: 0010 cost= 0.392354651<br># Optimization Finished!<br># Accuracy: 0.873333</pre><p>So what we did here is, we used the built in optimizer and let it compute the costs. What if we want to compute the gradient and update the weights by ourselves. That’s where tf.gradients function comes into play.</p><h3>Softmax Regression on MNIST dataset using `tf.gradients`</h3><p>By gradient descent formula the weights are needed to be updated like this,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*_cSDs81aejJt4oO1.png" /></figure><p>To implement Gradient Descent I’m going to omit the optimizer code instead I’ll write my own weight updates.</p><p>Since there are weight matrix W and bias matrix b . We need to calculate the gradient for each weight then update it. So the implementation would look something like this.</p><pre># Computing the gradient of cost with respect to W and b<br>grad_W, grad_b = tf.gradients(xs=[W, b], ys=cost)</pre><pre># Gradient Step<br>new_W = W.assign(W - learning_rate * grad_W)<br>new_b = b.assign(b - learning_rate * grad_b)</pre><p>These three lines of codes are written instead of one, why did I go through a lot of trouble? Because if you need to implement gradient of a custom cost function and don’t want to write rigorous mathematics then you should definitely let TensorFlow to handle all the gradient computations.</p><p>We have built the computational graph, all I need to do is to run it in a session. Let’s try that out.</p><pre># Fit training using batch data<br>            _, _,  c = sess.run([new_W, new_b ,cost], feed_dict={x: batch_xs, y: batch_ys})</pre><p>We don’t need the output from new_W and new_b that’s why I omitted those.</p><p>The full code and output is shown below,</p><pre>import tensorflow as tf<br><br># Import MNIST data<br>from tensorflow.examples.tutorials.mnist import input_data<br>mnist = input_data.read_data_sets(&quot;/tmp/data/&quot;, one_hot=True)<br><br># Parameters<br>learning_rate = 0.01<br>training_epochs = 10<br>batch_size = 100<br>display_step = 1<br><br># Parameters<br>learning_rate = 0.01<br>training_epochs = 10<br>batch_size = 100<br>display_step = 1<br><br># tf Graph Input<br>x = tf.placeholder(tf.float32, [None, 784]) # mnist data image of shape 28*28=784<br>y = tf.placeholder(tf.float32, [None, 10]) # 0-9 digits recognition =&gt; 10 classes<br><br># Set model weights<br>W = tf.Variable(tf.zeros([784, 10]))<br>b = tf.Variable(tf.zeros([10]))<br><br># Construct model<br>pred = tf.nn.softmax(tf.matmul(x, W) + b) # Softmax<br><br># Minimize error using cross entropy<br>cost = tf.reduce_mean(-tf.reduce_sum(y*tf.log(pred), reduction_indices=1))<br><br>grad_W, grad_b = tf.gradients(xs=[W, b], ys=cost)<br><br><br>new_W = W.assign(W - learning_rate * grad_W)<br>new_b = b.assign(b - learning_rate * grad_b)<br><br># Initialize the variables (i.e. assign their default value)<br>init = tf.global_variables_initializer()<br><br># Start training<br>with tf.Session() as sess:<br>    sess.run(init)<br><br>    # Training cycle<br>    for epoch in range(training_epochs):<br>        avg_cost = 0.<br>        total_batch = int(mnist.train.num_examples/batch_size)<br>        # Loop over all batches<br>        for i in range(total_batch):<br>            batch_xs, batch_ys = mnist.train.next_batch(batch_size)<br>            # Fit training using batch data<br>            _, _,  c = sess.run([new_W, new_b ,cost], feed_dict={x: batch_xs,<br>                                                       y: batch_ys})<br>            <br>            # Compute average loss<br>            avg_cost += c / total_batch<br>        # Display logs per epoch step<br>        if (epoch+1) % display_step == 0:<br>#             print(sess.run(W))<br>            print (&quot;Epoch:&quot;, &#39;%04d&#39; % (epoch+1), &quot;cost=&quot;, &quot;{:.9f}&quot;.format(avg_cost))<br><br>    print (&quot;Optimization Finished!&quot;)<br><br>    # Test model<br>    correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))<br>    # Calculate accuracy for 3000 examples<br>    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))<br>    print (&quot;Accuracy:&quot;, accuracy.eval({x: mnist.test.images[:3000], y: mnist.test.labels[:3000]}))<br>    <br>    <br># Output<br># Epoch: 0001 cost= 1.183741399<br># Epoch: 0002 cost= 0.665312284<br># Epoch: 0003 cost= 0.552796521<br># Epoch: 0004 cost= 0.498697014<br># Epoch: 0005 cost= 0.465521633<br># Epoch: 0006 cost= 0.442611256<br># Epoch: 0007 cost= 0.425528946<br># Epoch: 0008 cost= 0.412203073<br># Epoch: 0009 cost= 0.401364554<br># Epoch: 0010 cost= 0.392398663<br># Optimization Finished!<br># Accuracy: 0.874</pre><h3>Softmax Regression Using Gradient Formula</h3><p>This is the gradient of Softmax function with respect to weight W .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/0*dRaRimsE_eBmi-PI.png" /></figure><p>As before, without using tf.gradients or using TensorFlow’s built in optimizer, this way the gradient equation can be implemented.</p><pre>import tensorflow as tf<br><br># Import MNIST data<br>from tensorflow.examples.tutorials.mnist import input_data<br>mnist = input_data.read_data_sets(&quot;/tmp/data/&quot;, one_hot=True)<br><br># Parameters<br>learning_rate = 0.01<br>training_epochs = 10<br>batch_size = 100<br>display_step = 1<br><br># Parameters<br>learning_rate = 0.01<br>training_epochs = 10<br>batch_size = 100<br>display_step = 1<br><br># tf Graph Input<br>x = tf.placeholder(tf.float32, [None, 784]) # mnist data image of shape 28*28=784<br>y = tf.placeholder(tf.float32, [None, 10]) # 0-9 digits recognition =&gt; 10 classes<br><br># Set model weights<br>W = tf.Variable(tf.zeros([784, 10]))<br>b = tf.Variable(tf.zeros([10]))<br><br># Construct model<br>pred = tf.nn.softmax(tf.matmul(x, W)) # Softmax<br><br># Minimize error using cross entropy<br>cost = tf.reduce_mean(-tf.reduce_sum(y*tf.log(pred), reduction_indices=1))<br><br><br>W_grad =  - tf.matmul ( tf.transpose(x) , y - pred) <br>b_grad = - tf.reduce_mean( tf.matmul(tf.transpose(x), y - pred), reduction_indices=0)<br><br>new_W = W.assign(W - learning_rate * W_grad)<br>new_b = b.assign(b - learning_rate * b_grad)<br><br>init = tf.global_variables_initializer()<br><br><br>with tf.Session() as sess:<br>    sess.run(init)<br><br>    # Training cycle<br>    for epoch in range(training_epochs):<br>        avg_cost = 0.<br>        total_batch = int(mnist.train.num_examples/batch_size)<br>        # Loop over all batches<br>        for i in range(total_batch):<br>            batch_xs, batch_ys = mnist.train.next_batch(batch_size)<br>            # Fit training using batch data<br>            _, _, c = sess.run([new_W, new_b, cost], feed_dict={x: batch_xs, y: batch_ys})<br>            <br>        <br>            # Compute average loss<br>            avg_cost += c / total_batch<br>        # Display logs per epoch step<br>        if (epoch+1) % display_step == 0:<br>            print (&quot;Epoch:&quot;, &#39;%04d&#39; % (epoch+1), &quot;cost=&quot;, &quot;{:.9f}&quot;.format(avg_cost))<br><br>    print (&quot;Optimization Finished!&quot;)<br><br>    # Test model<br>    correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))<br>    # Calculate accuracy for 3000 examples<br>    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))<br>    print (&quot;Accuracy:&quot;, accuracy.eval({x: mnist.test.images[:3000], y: mnist.test.labels[:3000]}))<br>    <br>    <br># Output<br># Extracting /tmp/data/train-images-idx3-ubyte.gz<br># Extracting /tmp/data/train-labels-idx1-ubyte.gz<br># Extracting /tmp/data/t10k-images-idx3-ubyte.gz<br># Extracting /tmp/data/t10k-labels-idx1-ubyte.gz<br># Epoch: 0001 cost= 0.432943137<br># Epoch: 0002 cost= 0.330031527<br># Epoch: 0003 cost= 0.313661941<br># Epoch: 0004 cost= 0.306443773<br># Epoch: 0005 cost= 0.300219418<br># Epoch: 0006 cost= 0.298976618<br># Epoch: 0007 cost= 0.293222957<br># Epoch: 0008 cost= 0.291407861<br># Epoch: 0009 cost= 0.288372261<br># Epoch: 0010 cost= 0.286749691<br># Optimization Finished!<br># Accuracy: 0.898</pre><h3>How TensorFlow computes Gradient?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cJ5Zaxhw6Q8mejww.png" /><figcaption>Image Courtesy: Wikipedia</figcaption></figure><p>You might be thinking, how does TensorFlow compute gradient of a given function?</p><p>The method TensorFlow uses is called <a href="https://en.wikipedia.org/wiki/Automatic_differentiation">Automatic Differentiation</a>. You can know a lot from wikipedia.</p><p>I hope this article will help a lot for them who want to update the weights of their model without using TensorFlow’s built in optimizers. ;)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c111d783c78b" width="1" height="1" alt=""><hr><p><a href="https://blog.manash.io/implementation-of-gradient-descent-in-tensorflow-using-tf-gradients-c111d783c78b">Implementation of Gradient Descent in TensorFlow using tf.gradients</a> was originally published in <a href="https://blog.manash.io/">Manash’s blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing L2-constrained Softmax Loss Function on a Convolutional Neural Network using…]]></title>
            <link>https://blog.manash.io/implementing-l2-constrained-softmax-loss-function-on-a-convolutional-neural-network-using-1bb7c0aab7b1?source=rss-6f5d8825e907------2</link>
            <guid isPermaLink="false">https://medium.com/p/1bb7c0aab7b1</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[cnn]]></category>
            <category><![CDATA[loss-function]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[l2-normalized-softmax]]></category>
            <dc:creator><![CDATA[Manash Kumar Mandal]]></dc:creator>
            <pubDate>Wed, 13 Sep 2017 15:33:30 GMT</pubDate>
            <atom:updated>2017-09-14T17:26:58.435Z</atom:updated>
            <content:encoded><![CDATA[<h3>Implementing L2-constrained Softmax Loss Function on a Convolutional Neural Network using TensorFlow</h3><p>When we build Neural Network based models to classify more than 2 things using machine learning models, normally we choose softmax function. Basically it assigns a probability to each class and we just take the class with the higher probability as a verdict of the model. That is how we infer using the model or it can be referred as a forward pass. To train the model, we just perform backpropagation with the softmax function. Normally the output is represented as a one-hot vector or by 1-K class representation.</p><p>Since I am not writing this post to explain softmax regression or CNN. The main objective of this blog post is to implement L2-constrained softmax loss function using TensorFlow on good old MNIST dataset. <a href="https://arxiv.org/abs/1703.09507">The full description and other information related to this function can be found on this paper</a>.</p><p>Let’s clear some topics before diving into implementation.</p><h3>Softmax Loss Function</h3><p>Softmax loss function is defined like this,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/705/1*kgPr2J5aTxnW_MI7_3rwiA.png" /></figure><p>Where,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/0*DNnFb6I1S3z8KTBK.png" /></figure><h3>L2-Constrained Softmax Loss Function</h3><p>The definition stays almost exactly the same, the objective is to minimize this loss function</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*kgPr2J5aTxnW_MI7_3rwiA.png" /></figure><p>But here, <strong>f(x)</strong> gets changed.</p><p>Instead of computing the dot product between the last layer weights and <strong>f(x)</strong> [which is the output from the <strong>N-1</strong> layer, given <strong>Nth</strong> is the last layer]. Each element of <strong>f(x)</strong> matrix gets normalized by the norm value of <strong>f(x), </strong>then the matrix get scaled up by a parameter named <strong>alpha</strong>. After performing this operations first which can be addressed as L2-Normalization then Scaling, finally the dot product is computed then the regular softmax function is applied.</p><p>So the loss function is subjected to,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/0*MzwaYLtGfYwb_7By.png" /></figure><h3>Implementation Details</h3><p>So the architecture looks something like this, [this is the model I am going to implement]</p><p><strong>C</strong> means Convolution Layer, <strong>P</strong> means Pooling Layer, <strong>FC</strong> as you guessed, Fully Connected Layer, and L2-Norm and Scale layers are the ones I am going to implement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YH2K8w9KZRHCLGyYw27Xig.png" /></figure><h4>Implementing using TensorFlow</h4><p>To implement the model, I have taken the MNIST classification code using CNN <a href="https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/convolutional_network.ipynb">from this repository</a>.</p><p>Before applying dropout, at first I normalized the output from N-1 th layer which is <strong>fc1</strong> then scaled it up by the parameter alpha. Then softmax was computed. Here is the line of code that does all of this,</p><pre>fc1 = alpha * tf.divide(fc1, tf.norm(fc1, ord=&#39;euclidean&#39;))</pre><p>If <strong>alpha</strong> is set to 0 then it’s just regular softmax loss, otherwise it’s the L2-Normalized one.</p><p>Here is the full code</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/77517216c1d54bea18885bd05bc15871/href">https://medium.com/media/77517216c1d54bea18885bd05bc15871/href</a></iframe><h3>Performance Evaluation</h3><p>Did it actually work? Yes it did, accuracy was indeed increased almost by 1%. Although I should’ve applied more iterations but I didn’t because I didn’t have a powerful computer, and second reason is laziness. If you don’t believe me or the paper, why don’t you experiment it by yourself?</p><p>Here is the performance of the model for different alpha values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*jmk2eBXk9Etl-L0TCcUzFQ.png" /></figure><p>Orange line stands for the regular softmax loss function and the blue one is the l2 normalized one.</p><p>As I said, to know more about it, I recommend going through the paper, there are a lot of information I haven’t mentioned. Thanks for reading the post!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1bb7c0aab7b1" width="1" height="1" alt=""><hr><p><a href="https://blog.manash.io/implementing-l2-constrained-softmax-loss-function-on-a-convolutional-neural-network-using-1bb7c0aab7b1">Implementing L2-constrained Softmax Loss Function on a Convolutional Neural Network using…</a> was originally published in <a href="https://blog.manash.io/">Manash’s blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>