<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by David Puiggròs on Medium]]></title>
        <description><![CDATA[Stories by David Puiggròs on Medium]]></description>
        <link>https://medium.com/@puiggrosd?source=rss-d53b7e942a5a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*K7-IZw4_aqK0FIdH5bALhQ.jpeg</url>
            <title>Stories by David Puiggròs on Medium</title>
            <link>https://medium.com/@puiggrosd?source=rss-d53b7e942a5a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 15:51:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@puiggrosd/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building an IQ Link Solver with Claude Code]]></title>
            <link>https://medium.com/@puiggrosd/building-an-iq-link-solver-with-claude-code-832d391175f0?source=rss-d53b7e942a5a------2</link>
            <guid isPermaLink="false">https://medium.com/p/832d391175f0</guid>
            <category><![CDATA[puzzle]]></category>
            <category><![CDATA[search-algorithm]]></category>
            <category><![CDATA[ai-assisted-coding]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[David Puiggròs]]></dc:creator>
            <pubDate>Sun, 25 Oct 2026 16:24:55 GMT</pubDate>
            <atom:updated>2026-10-25T16:24:55.363Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*liyO5DSSibq7ACnt2__y9w.png" /><figcaption>A puzzle piece in the digital model</figcaption></figure><p>I’ve always enjoyed solving puzzles with computers. Some time ago, I wrote about solving <a href="https://medium.com/@puiggrosd/solving-martins-menace-a-computer-scientist-s-approach-08ff31ee817e">Martin’s Menace</a> using breadth-first search and a genetic algorithm.</p><p>This time I picked a different puzzle, IQ Link, but also a different approach. Instead of writing the code myself, I decided to develop the solver with Claude Code.</p><p>Three days later, I had a working solver, a web interface, and a few surprises along the way.</p><h3>What is IQ Link?</h3><p><a href="https://smartgames.com/uk/one-player-games/iq-link-0">IQ Link</a> is a puzzle created by SmartGames. It consists of 12 translucent pieces, each made of three elements: balls, open rings, and closed rings.</p><p>The challenge is to fit all 12 pieces onto a board with only 24 positions. But wait, 12 pieces with 3 elements each makes 36 elements. How can they fit into just 24 positions?</p><p>The trick is that some elements can overlap. A ball from one piece can fit inside an open ring from another, provided their geometries are compatible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*0rKfCGs2SxgjKHTqgk_LNA.jpeg" /><figcaption>Some of the puzzle piece</figcaption></figure><h3>Starting with a specification, not code</h3><p>My first step wasn’t writing code. Instead, I prepared a CLAUDE.md file describing the rules of the game, the data model to support the geometry of the pieces, and a plan for implementing the solver.</p><p>My main interest was the data model. I wanted a good representation of the puzzle that would allow me to implement an efficient search algorithm. The visual representation wasn’t a priority at all. I was interested in solving the puzzle, not building a user interface.</p><p>I wanted to work incrementally. The first task I gave Claude ended with a very explicit instruction: <em>don’t implement the whole application yet.</em></p><p>The idea was simple: let Claude implement the basic model, review it myself, and only then move on to the next step. I didn’t want to end up with a complete application that I couldn’t understand or validate.</p><h3>The key idea: seven bits per position</h3><p>The most interesting part of the data model was how to represent the pieces and their possible overlaps.</p><p>A simple approach would be to consider each board position as either occupied or empty. But that doesn’t work for IQ Link: two pieces can share the same position, provided their geometries are compatible.</p><p>The solution was to divide each position into seven smaller zones, arranged in three rows (2 + 3 + 2). Each zone is represented by a single bit:</p><pre>   NW   NE          4   8<br>  W   C   E       2   1   16<br>   SW   SE         64   32</pre><p>Each segment of a piece is represented by a bitmask indicating which zones it occupies. Checking whether two segments can share a position becomes a simple bitwise operation:</p><pre>occupied &amp; candidate == 0</pre><p>If the result is zero, the segments don’t overlap and can coexist.</p><p>And that’s the beauty of this representation: the solver doesn’t need special rules for balls, open rings, or closed rings. Their compatibility is entirely determined by their bitmasks.</p><p>If a piece doesn’t behave as expected, the problem is in its geometric representation, not in the search algorithm.</p><p>Rotating a piece only requires moving its bitmasks to their new positions on the board, without modifying their values. With this representation, solving the puzzle becomes a matter of placing pieces on the board, exploring their positions and orientations, and checking for collisions using bitwise operations.</p><h3>How big is the search space?</h3><p>Once the pieces were represented, I wanted to get an idea of the size of the problem.</p><p>A rough estimate is quite simple. Each piece can be placed at any of the 24 positions on the board, rotated in 6 different ways, and possibly flipped over, depending on its symmetry. That gives us 144 or 288 candidate placements per piece.</p><p>Of course, some placements fall outside the board, and others are equivalent due to symmetry.</p><p>And that’s before checking collisions between pieces. With 12 pieces to place, the number of possible combinations quickly becomes enormous.</p><p>As with Martin’s Menace, the interesting part isn’t exploring every possibility, but finding ways to discard invalid combinations as early as possible.</p><h3>Where Claude surprised me: the user interface</h3><p>As I mentioned earlier, my goal was to develop a solver, not a graphical application. I would probably have been happy with a Python script printing solutions to the console.</p><p>But Claude took the project in a different direction.</p><p>React isn’t part of my usual toolkit, and I wouldn’t have considered building a web interface for a small personal project like this. Yet, within a few hours, I had an interactive SVG viewer where I could drag pieces onto the board and explore their possible placements.</p><p>And, unexpectedly, the visual interface turned out to be useful for understanding the problem itself.</p><p>While exploring the different orientations, I noticed that some pieces appeared more than once in exactly the same configuration. The reason was symmetry: for certain pieces, flipping them over produced a configuration that could already be obtained through rotation.</p><p>In other words, not every piece has 12 distinct orientations (6 rotations × 2 sides). Some are equivalent to their own mirror image after a rotation. These are the non-chiral pieces.</p><p>The solution was simple: avoid generating mirrored orientations for those pieces.</p><p>I might have eventually discovered this by inspecting the data, but seeing the pieces on the screen made the problem immediately obvious.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*35eXMxt00YUSRmrl65QBrQ.png" /><figcaption>Solutions found for a given configuration</figcaption></figure><h3>Where the project stands</h3><p>The solver works, but there are still some limitations.</p><p>The search uses a backtracking algorithm using a most-constrained-first heuristic: at each step, it tries to place the piece with the fewest valid options. This helps reduce the number of combinations that need to be explored.</p><p>Another interesting detail is that the solver ended up being implemented in TypeScript. The Python package contains the data model, rotations, and tests, but the search algorithm itself only exists in the browser version.</p><h3>Conclusions</h3><p>I started this project because I wanted to solve another puzzle. My main interest was finding a good data representation and implementing an efficient search algorithm. I wasn’t particularly interested in building a graphical interface.</p><p>Three days later, I had a working solver, an interactive web application, and an architecture quite different from what I had originally imagined.</p><p>What surprised me most wasn’t how quickly Claude could write code, but how it changed the scope of the project. Things I would normally have skipped, like building a React interface, suddenly became easy enough to try. And, unexpectedly, that interface helped me better understand the problem itself.</p><p>Of course, Claude didn’t eliminate the need to think. Understanding the puzzle, defining the model, checking the results, and deciding what made sense still required my involvement.</p><p>I still enjoy designing algorithms and solving puzzles. Now I also have a tool that makes experimenting with ideas outside my usual toolkit much easier.</p><p>Not bad for three days of playing with a puzzle.</p><p>Salut i intel·ligència!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=832d391175f0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Zero-Shot Image Matching: Playing Memory]]></title>
            <link>https://medium.com/@puiggrosd/zero-shot-image-matching-playing-memory-560bfa31c04d?source=rss-d53b7e942a5a------2</link>
            <guid isPermaLink="false">https://medium.com/p/560bfa31c04d</guid>
            <category><![CDATA[playing]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[segmentation]]></category>
            <category><![CDATA[vision]]></category>
            <category><![CDATA[image-matching]]></category>
            <dc:creator><![CDATA[David Puiggròs]]></dc:creator>
            <pubDate>Fri, 09 Aug 2024 21:12:14 GMT</pubDate>
            <atom:updated>2024-08-09T21:12:14.076Z</atom:updated>
            <content:encoded><![CDATA[<p>Remember the classic memory game? The one where you flip over cards to find matching pairs? Well, it’s back with a twist! This time, we’re letting AI do the hard work for us.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1k0AEZQL_TO2mnUvP8ymRg.png" /><figcaption>Ready to play Memory</figcaption></figure><p>My daughter and I used to play this game together when she was younger. This summer, we decided to bring back the fun and let AI join in on the action. So, we rolled up our sleeves and built an artificial vision algorithm to play the game for us. Spoiler alert: it was awesome!</p><p>Now, let me walk you through how we programmed a computer to outmatch us at finding Frozen cards — faster than you can say, “Let it go!”</p><h3>Plan</h3><p>We’re going to build a straightforward solution using Python, Jupyter notebooks, and some powerful artificial vision algorithms from OpenCV.</p><p>Here’s the game plan:</p><ol><li><strong>Preprocessing:</strong> We’ll start by converting our images to grayscale, applying a blur to reduce noise, and using thresholding to create clear, binary images.</li><li><strong>Finding Contours:</strong> Next, we’ll identify and outline the card pieces by tracing their contours in the binary images.</li><li><strong>Warping:</strong> Then, we’ll zoom in on each card piece, standardize their size, and correct any perspective distortions.</li><li><strong>Calculating Similarity:</strong> Finally, we’ll compare the images by calculating their similarity scores to identify matching pairs.</li></ol><p>You can find the source code in <a href="https://github.com/puiggrosd/playing-memory">https://github.com/puiggrosd/playing-memory</a></p><h3><strong>Preprocessing</strong></h3><p>First things first, let’s get our images ready for some serious AI analysis. We’ll start by performing a series of preprocessing steps. The goal here is to simplify the images so that our algorithm can more easily identify and segment the different zones containing the card pieces.</p><p>Here’s the plan:</p><ol><li><strong>Convert to Grayscale:</strong> Simplify the image by removing color information.</li><li><strong>Apply Blur:</strong> Smooth out the image to reduce noise and irrelevant details.</li><li><strong>Binary Thresholding:</strong> Create a binary image to clearly distinguish the card pieces from the background.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aegQrzZ0GBT569fvwv5O4Q.png" /><figcaption>Images preprocessed: original to gray and blur to make an easy binary threshold method work.</figcaption></figure><p>With these steps, our images will be prepped and primed for segmentation.</p><p><strong>Finding Contours</strong></p><p>Now that we have our segmented image, it’s time to find the contours. This is where the magic of classic computer vision algorithms comes into play.</p><p>We’ll use cv2.findContours to trace the borders of the objects detected in our binary image. In this simplified black-and-white representation, the border is defined where black meets white. By connecting neighboring pixels, we can construct polygons that outline our card pieces.</p><p>Here’s how we refine our approach:</p><ol><li><strong>Filter by Area:</strong> Discard polygons that are too small, since all card pieces should be of similar size.</li><li><strong>Simplify Polygons:</strong> Use cv2.approxPolyDP to approximate each contour with a simpler polygon, ideally with four sides (rectangles or trapezoids).</li></ol><p>With a bit of luck — good lighting and minimal overlaps — we’ll end up with a clean set of polygons representing our card pieces, ready for further analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*Qhmnahhtb2h9CkenE_d_xw.png" /><figcaption>The pieces identified and labeled</figcaption></figure><h3>Warping</h3><p>Now for the fun part — warping! We need to zoom in on each card piece and convert it to a uniform size. This standardization is crucial for consistent analysis. To achieve this, we’ll use cv2.getPerspectiveTransform to correct any perspective distortions.</p><p>Here’s the game plan:</p><ol><li><strong>Zoom In:</strong> Focus on each detected card piece.</li><li><strong>Standardize Size:</strong> Convert each piece to the same size for consistency.</li><li><strong>Correct Perspective:</strong> Use cv2.getPerspectiveTransform to adjust for any perspective distortions, ensuring each card is viewed head-on.</li></ol><p>This process will help us create a clean, uniform dataset of card images, perfectly prepped for our AI to work its matching magic!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2ENMpdQYj6u6OWd8Nl-pWQ.png" /><figcaption>The pieces zoomed and standarized</figcaption></figure><h3>Calculating Similarity</h3><p>With all our card pieces now standardized in size, it’s time to determine which images are similar to each other. We’ll use a straightforward yet effective approach to compare the images.</p><p>Here’s our plan:</p><ol><li><strong>Image Subtraction:</strong> Subtract one image from another to highlight the differences.</li><li><strong>Root Mean Square:</strong> Calculate the Root Mean Square of these differences across the entire image. This gives us a quantitative measure of similarity.</li></ol><p>Since we can’t assume the images are correctly oriented, we’ll need to account for this by checking all four possible orientations of each card piece. By following this method, we’ll determine which cards are pairs based on their similarity scores. Let’s jump into the calculations!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FHBq5ADYd47t8mxJVcWWVw.png" /><figcaption>Which images is similar to the first one? Look at the RMS.</figcaption></figure><h3>Results</h3><p>Drumroll, please! It’s time to reveal the results of our AI-powered memory game experiment. After processing and comparing all the cards, here’s what we found:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*M4I805vATXpk5RJMrOp8YA.png" /><figcaption>Pair matching labelling: we succeeded!</figcaption></figure><h3>Conclusions</h3><p>These results highlight the potential of using AI to tackle classic games like memory. Who knew that Elsa, Anna, and a bit of computer vision could make such a great team?</p><p>This project was a delightful summer endeavor shared between a father and daughter, combining play and learning. We had a blast experimenting with various tools and algorithms, discovering how they can solve puzzles and make games more fun.</p><p>While we could have explored more advanced algorithms like SSIM or refined our preprocessing methods, that wasn’t the goal.</p><p>The goal was to experiment with computers, programming, and AI in gaming while also creating wonderful memories. Not everyone enjoys making a computer solve puzzles, but I’m fortunate 🍀 to have someone who shares that enthusiasm.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=560bfa31c04d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automatic Task Planning]]></title>
            <link>https://medium.com/@puiggrosd/automatic-task-planning-e867f1d29452?source=rss-d53b7e942a5a------2</link>
            <guid isPermaLink="false">https://medium.com/p/e867f1d29452</guid>
            <category><![CDATA[order-management]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[scheduling]]></category>
            <dc:creator><![CDATA[David Puiggròs]]></dc:creator>
            <pubDate>Sun, 30 Jun 2024 21:28:23 GMT</pubDate>
            <atom:updated>2024-06-30T21:28:23.221Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OpA96LZsywu1yLDDguanYw.jpeg" /></figure><p>In recent talks and discussions, there’s been a noticeable surge of interest in a classic AI topic: optimization, particularly in the context of manufacturing orders. Optimization is a crucial aspect of AI with far-reaching implications across various industries. It involves finding the most efficient way to complete a set of tasks. In manufacturing, this translates to scheduling production orders in a way that maximizes productivity while minimizing costs.</p><p>Imagine a scenario where multiple jobs need to be scheduled, each consisting of several tasks that require specific machines and operators. The challenge lies in efficiently allocating these limited resources to complete all jobs within the shortest possible time without overbooking any resource. This is where automatic task planning powered by AI comes into play.</p><p>In this article, I will provide an overview of the problem and explore the possibilities that automation offers. Whether you are an AI enthusiast or a professional looking to streamline your operations, this discussion aims to shed light on the potential of automatic task planning to revolutionize manufacturing processes.</p><h3>Problem Statement</h3><p>Let’s consider a simple problem involving a list of jobs. Each job is defined as a collection of tasks, with each task having a specific duration. For instance, a job might consist of five tasks, each requiring different types of machines or operators.</p><p>To illustrate, imagine Job A includes the following tasks:</p><ul><li>Task 1: Cut. Duration 2 hours, requires Machine X, and an Operator</li><li>Task 2: Wash. Duration 5 hours, requires an Operator</li><li>Task 3: Sew. Duration 3 hours, requires Machine Z and an Operator</li><li>Task 4: Paint. Duration 2 hours, requires an Operator</li><li>Task 5: Pack. Duration 1 hour, requires an Operator</li></ul><p>Both the machines and operators have limited availability, which adds a layer of complexity to the scheduling process.</p><p>The challenge lies in efficiently allocating these limited resources to complete all jobs within the shortest possible time while ensuring that no resource is overbooked.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xYsMVkJwGc5DjHeH_drcUg.png" /><figcaption>8 Jobs without any planning, just all starting at same time. We need resources of type H, K, M to run each of the tasks types.</figcaption></figure><h3>Why We Can Solve This with AI</h3><p>This problem is a perfect candidate for AI because it involves making complex, non-trivial decisions automatically. If we attempt to solve it using brute force, we’ll quickly realize that the number of elements to calculate becomes excessively large.</p><p>In more detail, each job can be performed in various orders, and the number of possible combinations grows exponentially. The impact of job sequencing is difficult to anticipate in the allocation of limited resources.</p><p>AI algorithms, particularly those in the field of optimization and scheduling, can efficiently explore these vast solution spaces. They use sophisticated methods to find near-optimal solutions without having to examine every possible combination. This capability makes AI an essential tool for solving such complex scheduling problems.</p><h3>Initial Approach: The Baseline</h3><p>As a starting point, let’s consider a baseline approach: assigning each job according to the order of arrival. In this method, we schedule tasks sequentially, attempting to start each task as soon as the required resource becomes available. If a task cannot be performed immediately because the necessary machine or operator is occupied, the job is paused until the resource is freed.</p><p>Here’s how it works:</p><ol><li><strong>Job Arrival</strong>: Assign jobs in the order they arrive.</li><li><strong>Task Scheduling</strong>: For each job, start tasks as soon as their required resources (machines or operators) are available. When we start a task, we should finish that task without interruptions.</li><li><strong>Resource Blocking</strong>: If a task cannot be started because its resource is currently in use, pause the job until the resource is available.</li><li><strong>Parallel Tasks</strong>: Whenever possible, start all tasks that can be performed with the available resources simultaneously.</li></ol><p>This approach is simple and easy to implement, serving as a useful benchmark for more advanced methods. However, it often leads to suboptimal schedules, with many idle times and resource bottlenecks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P8ZolK2FuUY0QvuAJVZ_ow.png" /><figcaption>Previous example with Baseline approximation. We satisfy the resources limit constraint but we require 58 hours.</figcaption></figure><h3>Second Approach: Constraint Programming</h3><p>To enhance our initial approach, we can utilize constraint programming to optimize the scheduling process. This method allows us to re-order jobs and allocate resources in a way that minimizes the maximum planning time (makespan), ensuring a more efficient schedule.</p><p>Here’s the process:</p><p><strong>Define Constraints</strong>: Set up constraints to ensure:</p><ul><li>Each task is assigned to the required machine/operator.</li><li>No machine/operator is assigned more than one task at a time.</li><li>Tasks within a job must be completed in sequence.</li></ul><p><strong>Formulate the Problem</strong>: Use constraint programming to model the scheduling problem. Define the objective to minimize the makespan, which is the total time required to complete all jobs.</p><p><strong>Solve the Problem</strong>: Apply a constraint programming solver to explore possible schedules, reordering tasks and allocating resources to find the optimal solution.</p><p>Constraint programming systematically narrows down the search space by eliminating schedules that violate any constraints. This approach effectively reduces idle times and resource bottlenecks, leading to a more efficient and balanced schedule compared to the baseline method. By leveraging constraint programming, we can achieve significant improvements in job scheduling and resource utilization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HWMqLeLjkZF0hfPoGJfd7w.png" /><figcaption>Previous example with Constraint Programming approximation. We reorder the jobs to obtain a better solution: 44 hour with the same resources.</figcaption></figure><h3>Scalability</h3><p>Additionally, using this approach, we can scale to more complex problems that are extremely difficult to manage with a human workforce alone. AI can handle vast amounts of data and intricate scheduling requirements efficiently, making it possible to optimize even the most complicated manufacturing processes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m2plX0KGjY-JHe3UkvL3XA.png" /><figcaption>20 jobs scheduling</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xKFWAojcyl7AwOD8C4A8aA.png" /><figcaption>One hundred job scheduling</figcaption></figure><h3>Conclusions</h3><p>Both approaches are automated and can enhance the management of manufacturing orders. However, an AI-based approach offers additional benefits:</p><ul><li><strong>Optimize Business Variables</strong>: AI can minimize the maximum schedule duration or other specific business variables.</li><li><strong>Introduce New Orders Minimizing Impact</strong>: AI can incorporate pre-agreed delivery dates into the optimization function, allowing urgent tasks to be added with minimal disruption to existing schedules.</li><li><strong>Generate Different Scenarios</strong>: AI can simulate various scenarios, such as adding tasks or changing resource availability, and predict their impact on the schedule.</li><li><strong>Calculate Resource Bottlenecks and Underutilization</strong>: AI can identify resource bottlenecks and periods of underutilization, enabling more efficient use of resources.</li><li><strong>Optimize Resource Scenarios</strong>: AI can find a balance between time reduction and cost minimization by adjusting resources accordingly.</li><li><strong>Handle Increased Complexity</strong>: AI can manage additional constraints and data, such as varying performance of resources, date limitations, and dynamic resource availability.</li></ul><p>The most interesting thing about all this is that we have everything ready to incorporate AI into planning: the tools and the algorithms already exist. The data are already present in our ERPs. We just need to focus on our business: constraints, variables and set the objectives.</p><blockquote>The potential for improved efficiency and optimization in manufacturing planning is within reach, waiting to be utilized.</blockquote><p>As AI technology continues to evolve, its applications in task planning will only become more robust and versatile. For those in the manufacturing industry, now is the time to explore and adopt these innovative tools to stay ahead of the competition.</p><p>Have an optimal day! Salut i intel·ligència!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e867f1d29452" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Painting Maps like an artist]]></title>
            <link>https://medium.com/@puiggrosd/painting-maps-like-an-artist-88e323a9e8a4?source=rss-d53b7e942a5a------2</link>
            <guid isPermaLink="false">https://medium.com/p/88e323a9e8a4</guid>
            <category><![CDATA[computer-art]]></category>
            <category><![CDATA[maps]]></category>
            <category><![CDATA[graphs-are-fun]]></category>
            <dc:creator><![CDATA[David Puiggròs]]></dc:creator>
            <pubDate>Fri, 29 Mar 2024 15:50:23 GMT</pubDate>
            <atom:updated>2024-03-29T15:50:23.572Z</atom:updated>
            <content:encoded><![CDATA[<p>Over the past few months, I’ve been working into one of AI’s classic problems: route optimization. It’s been quite some time since I last engaged this issue, and during my search for tools, I stumbled upon a particularly interesting discovery: the OSMNX library.</p><p>The <a href="https://osmnx.readthedocs.io/en/stable/">OSMNX</a> library is a powerful tool that allows you to download spatial geometries and construct, project, visualize, and analyze street networks using OpenStreetMap’s API. What stood out to me about OSMNX was its simplicity and flexibility, which transform tasks that used to be cumbersome into surprisingly straightforward ones. Whether you’re plotting routes, analyzing the layout of a city, or simply painting maps for fun, OSMNX opens up a world of possibilities with just a few lines of code. It’s akin to having a map of the world at your fingertips, ready to be explored and manipulated in any way imaginable.</p><p>Additionally, I discovered a more creative use of this library, inspired by the article <a href="https://medium.com/towards-data-science/the-intersection-of-geocoding-and-art-with-open-street-map-and-networkx-d3cff5a56200">“The Intersection of Geocoding and Art with Open Street Map and NetworkX”</a>. This piece discusses creating beautiful visualizations from information on streets, buildings, and places of interest — an approach that really captivated my interest.</p><p>Starting this creative project, I knew I wasn’t great at art, but I really enjoyed playing with colors and making my maps look vibrant. Even though I’m not a pro, I had a lot of fun. I’m excited to show you some of my work, even if it’s just beginner’s art:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*LqLa8UtpSmQ5osPNSoUS_Q.png" /><figcaption>Igualada: roads and buildings</figcaption></figure><p>Lastly, I want to share a visualization of where I live. I shared it a few months ago with a very good friend. We often discussed these kinds of topics, and I thought he would appreciate seeing it. As always, showing his usual interest and enthusiasm for my ideas, he immediately contributed more ideas with great energy. It’s one of the last memories I have of him, and it’s very precious to me.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BY2B7FsfnbaxcYY3kUbaMQ.png" /><figcaption>Pujalt + Igualada</figcaption></figure><p>Thanks for all the moments we shared.</p><p><strong>Sempre endavant.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88e323a9e8a4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Solving Martin’s Menace: A Computer Scientist’s Approach]]></title>
            <link>https://medium.com/@puiggrosd/solving-martins-menace-a-computer-scientist-s-approach-08ff31ee817e?source=rss-d53b7e942a5a------2</link>
            <guid isPermaLink="false">https://medium.com/p/08ff31ee817e</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[optimization-algorithms]]></category>
            <category><![CDATA[puzzle]]></category>
            <category><![CDATA[genetic-algorithm]]></category>
            <dc:creator><![CDATA[David Puiggròs]]></dc:creator>
            <pubDate>Mon, 01 Jan 2024 16:27:45 GMT</pubDate>
            <atom:updated>2024-01-01T16:27:45.838Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Ap5IQubBtMnzXDJh0jTMw.jpeg" /></figure><p>Happy New Year! I’m excited to share with you a hobby project that I recently completed. A while back, a dear friend gifted me the Martin’s Menace puzzle. I initially spent several days trying to solve this dissection puzzle using only my brainpower, but I couldn’t crack it. It ended up on my shelf, gathering dust. However, just a few days ago, it struck me: While I may not excel at solving puzzles manually, my proficiency lies in using computers to solve problems. And here’s what I came up with.</p><blockquote>SPOILER ALERT: This article reveals the solution to the puzzle, or at least give you some clues. I really encourage you to try to solve the puzzle without any help. It’s a <a href="https://en.wikipedia.org/wiki/Stewart_Coffin">Stewart Coffin</a> puzzle (#217).</blockquote><h3>What is Martin’s Menace puzzle?</h3><p>Martin’s Menace is a challenging dissection puzzle, designed to test spatial reasoning and problem-solving skills. It consists of four uniquely shaped pieces that must fit together in a rectangle that always seems to small!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PNu5HNb86EoLe2_TzruSog.jpeg" /></figure><p>We can see the pieces shaped like: “s”, “r”, “f”, “T”</p><h3>How I tried to Solve That?</h3><p>I approached this problem like a classic search problem in artificial intelligence, exploring the state space to find the target solution using various strategies. Good Old Artificial Intelligence in this era of deep learning and LLM. Great!</p><p>In this context, it was crucial to:</p><ol><li>Define the state space and navigate it effectively.</li><li>Detect when a solution has been found.</li></ol><h3>State space definition</h3><p>In artificial intelligence, “state space” refers to the set of all possible states or configurations that a system or problem can have. For Martin’s Menace puzzle, the state space includes every possible arrangement and orientation of the puzzle pieces.</p><p>My initial hypothesis was that orthogonal rotations (0, 90, 180, or 270 degrees) would suffice. This approach aimed to minimize holes or gaps in the puzzle. Similarly, I aligned the pieces along an imaginary grid, avoiding half-square alignments for the same reason. Thus, my state space and my search excluded solutions requiring different rotations or alignments.</p><p>I conceptualized various state space definitions:</p><p>First one: For each of the four puzzle pieces, I considered their &lt;x, y&gt; coordinates, orientations and side. Example: my f piece, at position &lt;1,2&gt; rotated 90 degrees and flipped. Move around the state space implies change some of this attributes: flip the piece, rotate or change the coordinates.</p><p>Second one: An incremental version that works piece by piece until complete all the board:</p><ul><li>Starting with an empty board and adding the symmetrical “<strong>T”</strong> piece first. This piece is fixed in position &lt;0,0&gt; and all the other pieces will be around.</li><li>Placing the “<strong>f”</strong> piece in some position, orientation and side around the initial piece.</li><li>Repeating this process for the remaining pieces.</li></ul><p>Example:</p><pre>Board().add(piece_f, False, 90, 2, 4).add(piece_r, False, 0, 6, 6).add(piece_s, False, 0, 5, -3)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/337/1*Fes3ZnV0gqOzPcHpQ5h_Zg.png" /></figure><p>The size of this state space was considerable. Each piece could be placed in 2 sides, 4 orientations, and within a 10x10 grid, resulting in 800 different positions per piece. Therefore, exploring all possible combinations (800³ = 512,000,000) was a significant task, estimated to take around 60 hours on my computer using a single core.</p><h3>Detecting a Solution</h3><p>Identifying a solution was initially challenging but the final approximation is simple. The key lay in calculating the minimal rectangle covering all the pieces. This involved computing the convex hull of all polygons and then determining the smallest enclosing rectangle.</p><p>It’s easy to calculate that using the minimum_rotated_rectangle function from the <a href="https://shapely.readthedocs.io/en/stable/manual.html">Shapely</a> library.</p><p>If this rectangle fit within the available space, we’ve got it!.</p><h3>How to find the solution</h3><p>I explored two options:</p><ol><li>Breadth-First algorithm, with custom implementation</li><li>Genetics algorithm using <a href="https://deap.readthedocs.io/en/master/">DEAP</a> library</li></ol><h4>Breadth-First Algorithm</h4><p>This straightforward approach involved navigating the state space by adding one piece at a time, discarding invalid positions, and retaining unique solutions. This method yielded 76 candidate solutions with two pieces, 160 with three pieces, and eventually narrowed down to just two final solutions. It was surprisingly fast, taking less than a minute on a standard computer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*Csm2OiTx8qLtTXkmQJru6w.png" /><figcaption>76 candidates with two pieces.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NGKE7GCxQyu6SugxAAiQ0g.png" /><figcaption>160 candidates with three pieces</figcaption></figure><p>And finally only two candidates for the final solution.</p><h4>Genetic algorithm</h4><p>I also experimented with genetic algorithms, particularly interested in the process and potential solutions they might uncover. By the way, I have to confess, I’ve always had a soft spot for genetic algorithms — they’re my favorite go-to method! There’s something fascinating about watching an AI mimic nature’s way of evolving solutions.</p><p>A genetic algorithm is a computational method that mimics natural evolution to solve complex problems. It starts with a random population of solutions, evaluates their effectiveness, and then iteratively produces new generations through processes akin to biological reproduction and mutation. Over successive generations, the algorithm selects and combines the most effective solutions, gradually evolving towards an optimal or near-optimal solution.</p><p>The key components defined were:</p><ul><li>Individual: Consisting of the side, orientation, and position of each piece.</li><li>Fitness: Guided by the size of the enclosing rectangle. Smaller was better, with penalties for orthogonal rectangles, as they wouldn’t fit the Martin’s Menace dimensions.</li><li>Mutation: Random changes in parameters.</li><li>Crossover: Involving one piece from one parent and the rest from the other to generate offspring.</li></ul><p>This approach revealed several near-solutions and the two final solutions already identified. Details of this implementation are also available on my <a href="https://github.com/puiggrosd/martins-menace">GitHub</a>.</p><h3>Conclusions</h3><p>This holiday season, I combined leisure with my love for technology by tackling puzzles with my computer. Delving into classic AI algorithms like Breadth-First Search and genetic algorithms was not only funny but also incredibly effective. In the end, I cracked the Martin’s Menace puzzle using the tools that define my professional expertise. This experience was a delightful blend of traditional pastime and modern tech, showcasing how a computer can be a powerful ally in solving even the most intricate challenges.</p><p>Happy 2024! Salut i intel·ligència!</p><blockquote>Note on Solutions: I have chosen not to publish the exact solution out of respect for the puzzle’s integrity and to preserve the fun of solving it for others.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=08ff31ee817e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring Anomaly Detection in Images]]></title>
            <link>https://medium.com/@puiggrosd/exploring-anomaly-detection-in-images-350efe9130ca?source=rss-d53b7e942a5a------2</link>
            <guid isPermaLink="false">https://medium.com/p/350efe9130ca</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[anomaly-detection]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[image-processing]]></category>
            <dc:creator><![CDATA[David Puiggròs]]></dc:creator>
            <pubDate>Fri, 17 Nov 2023 06:06:48 GMT</pubDate>
            <atom:updated>2023-11-17T06:06:48.939Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In the field of image processing, anomaly detection is a pivotal and fascinating challenge. The purpose of this post is to delve into a particular technique, with the goal of deepening my understanding and applying the insights obtained to a specific area of interest.</p><p>The key idea involves using an autoencoder neural network to reconstruct images and use the error to detect anomaly zones.</p><h3>Requirements</h3><p>The technique employed in this study is based on three fundamental requirements:</p><ol><li>Utilize unsupervised learning, or at least with minimal supervision, to ensure the approach is as autonomous as possible.</li><li>Employ a minimal number of images, emphasizing the method’s efficiency and effectiveness even with limited data.</li><li>Ensure that the process is viable on a system with modest capabilities, facilitating validation even on less powerful systems.”</li></ol><h3>Solution</h3><p>Key idea: Autoencoders are neural networks trained to reconstruct their input data. By training an autoencoder on normal images, it has difficulty reconstructing anomalies, thereby allowing us to detect them based on reconstruction error.</p><p>For the solution, I utilized a high-resolution image without anomalies. This approach aligns with minimal supervision as I only needed to ensure that the image was anomaly-free.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/209/1*02dLiombgqKfgob3PdRArw.png" /><figcaption>Original image to test the technique: 1080x1920</figcaption></figure><p>From this image, I can generate as many smaller images as needed by rotating and slicing the original image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/153/1*9RBrllhVdiB7wXe_18rkyw.png" /><figcaption>Example tile: Random pieces from the previous image (128x128)</figcaption></figure><p>I trained an autoencoder model with this smaller images. This process is efficient and feasible, especially given the reduced size of the images and the availability of a diverse training dataset.</p><p>I tested different autoencoder model configuration based on UNET. In this example a 4 level UNET, with skip connections and a inner latent space of 512 dimension is enough. Training on an 8GB GPU took one hour, achieving low-level error and excellent image reconstruction performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*GD39t8xWAO0uJAHU_gvAsw.png" /><figcaption>Input and Predicted image. The reconstruction works as expected.</figcaption></figure><p>The model can now be applied to any segment of a new image. Based on the quality of reconstruction, I can categorize each segment as more or less similar to the initial image. Anomalous segments are identified by their lower reconstruction quality.</p><p>This is an example of reconstruction of the original image with artifacts included on it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bQ8uekuX-BJ1ovg1kPXbqA.png" /><figcaption>Image with anomalies, heatmap with detection and histogram with the error reconstruction per tile</figcaption></figure><h3>One more test</h3><p>For a final test of the technique, I conducted an additional experiment using two photographs of pavement. I went outside and took two photographs of my city’s pavement. One photograph captured the scene without any leaves, while the other featured a leaf on the ground — a nod to the autumn season. This test was designed to further validate the technique’s efficacy in a real-world scenario with visually distinct elements.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*NzawBXl8POVJZDFWKnkbzQ.png" /><figcaption>Boring street pavement without leaves</figcaption></figure><p>The results with the pavement with a leaf:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*7udTqwx2ZgEH-ajwflEyRA.png" /></figure><p>The heatmap clearly indicates the leaf’s location, as identified by the model.</p><p>So, the cool part is, even though the model had never seen a leaf before, it still nailed it in the heatmap, showing exactly where the leaf was. Pretty awesome, right?</p><h3>Conclusions</h3><p>In summary, after conducting this brief and straightforward analysis, the following conclusions were reached:</p><ol><li>The method consistently yields positive results, performing as expected in all reviewed cases.</li><li>Training with image fragments allows for the effective use of a few original high resolution images.</li><li>This approach seems suitable if the images to be analyzed are repetitive or exhibit patterns, and the anomalies do not depend on global aspects of the image.</li><li>The network topology is not overly problematic. It needs to be sufficiently large to minimize reconstruction error. In this case, generalization is not as critical as in other scenarios.</li><li>The choice of metric will be crucial in minimizing false positives.</li></ol><p>In conclusion, this technique offers a simple, cost-effective approach for preliminary image analysis and anomaly detection with minimal supervision.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=350efe9130ca" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Comparison between simple audio classification methods]]></title>
            <link>https://medium.com/@puiggrosd/comparison-between-simple-audio-classification-methods-f2989e9647a8?source=rss-d53b7e942a5a------2</link>
            <guid isPermaLink="false">https://medium.com/p/f2989e9647a8</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[convolutional-network]]></category>
            <category><![CDATA[audio-classification]]></category>
            <dc:creator><![CDATA[David Puiggròs]]></dc:creator>
            <pubDate>Sun, 19 Sep 2021 16:13:25 GMT</pubDate>
            <atom:updated>2021-09-19T21:44:02.044Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_F3qi5To6UdLpVos" /><figcaption>Photo by <a href="https://unsplash.com/@mattbotsford?utm_source=medium&amp;utm_medium=referral">Matt Botsford</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this post I want to show a comparison, neither exhaustive nor particularly complete, of various methods for classifying sounds using deep learning. I have long wanted to review some of these methods in order to compare their performance and how easy is to apply them.</p><p>The exercise I propose is quite simple. The main objective is get a basic comparison of these methods and picking up some key ideas.</p><p>Using a dataset of 1.500 audio files labeled with 5 categories, I will train two classifiers, each using a different technique, to predict their audio categories.</p><p>I will explore two options based on neural networks: use the signal audio directly (one dimension) or through an image (two dimensions) generated from a spectral analysis of the original signal.</p><h3>Dataset</h3><p>The dataset used in this experiment can be found in <a href="https://www.kaggle.com/c/freesound-audio-tagging/data">https://www.kaggle.com/c/freesound-audio-tagging/data</a> and was published by Eduardo Fonseca, Manoj Plakal, Frederic Font, Daniel P. W. Ellis, Xavier Favory, Jordi Pons, Xavier Serra. General-purpose Tagging of Freesound Audio with AudioSet Labels: Task Description, Dataset, and Baseline. In Proceedings of DCASE2018 Workshop, 2018. URL: <a href="https://arxiv.org/abs/1807.09902">https://arxiv.org/abs/1807.09902</a>.</p><p>All 9.473 audio files in the dataset are gathered from <em>Freesound</em> and are provided as uncompressed PCM 16 bit, 44.1 kHz, mono audio files. The ground truth data provided in this dataset has been obtained after a data labeling process in 41 categories of the <em>AudioSet Ontology</em>. Only 40% of data have been manually verified.</p><p>In this experiment I only use 5 categories associated to these sounds: Bark, Cough, Knock, Laughter, Trumpet.</p><p>The distribution between training, validation and test sets are showed in the next table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/289/1*bnLR66XqhL9ZbDkf1oIYiw.png" /><figcaption>Sample distribution between categories and datasets</figcaption></figure><p>There are between 239 and 300 sample per category. I use this distribution for each dataset type: 20% test, 16% validation and 64% training.</p><h3>Dataset regularization</h3><p>The audio files (in wav format) in the dataset have different duration and this requires some regularization that I apply the same way for each analyzed option in the same way. To accomplish this I used <a href="https://fastaudio.github.io/Introduction%20to%20Fastaudio.html">Fastaudio</a> library.</p><p>I will only use 5 seconds of audio data per file. When the audio is longer, I crop 5 seconds signal. If its smaller I repeat until the duration is 5 seconds.</p><p>Before that cropping I remove the silence, so I try to use the most important information of the audio signal.</p><p>The signal croping it’s also an augmentation, because when the signal lasts more than 5 seconds I crop different parts of the signal in any training epoch.</p><p>Some improvement could be done here using all the available data of the audio or determining a better window for each audio domain (5 second is quite arbitrary decision).</p><p>In the next images it is shown one audio sample, the removed silence version and a cropped/repeated version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*Fbx1CMBcloqda9RgozIL5w.png" /><figcaption>Original audio</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*CZbDlqCPtMMXxtQnnY_KFA.png" /><figcaption>Same audio without silence</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*8mSPzmwRdZlmcsdTD_NRxQ.png" /><figcaption>Same audio repeated until 5 seconds duration reached</figcaption></figure><h3>Strategy to solve the problem</h3><p>A possible strategy (a very interesting one), which I will not work on in this post, is to apply feature-engineering from the signal. For example, it’s possible that some classes use fixed frequencies, or some duration… Then it would be possible to generate custom attributes to facilitate classification.</p><p>I will not use this alternative as I don’t know this domain well enough, nor it is my intention to work this part of the analysis. What I’m looking for, is an approach that saves me from working this knowledge without a significant cost in accuracy.</p><p>The <strong>first alternative,</strong> <strong>raw classifier</strong>, is to build a deep learning model that use the signal audio as is. These neural network will have two components:</p><ul><li>A convolutional layer that it’s responsible to generate interesting features of the raw data</li><li>A full connected layer that will classify using the previous features</li></ul><p>The <strong>second alternative, image classifier,</strong> consists in representing the audio data as an image, using a spectrogram representation, and then try to classify it using a pre-existing deep learning model specialized in object recognition.</p><h3>Raw classifier</h3><p>For the raw classifier I use a 1-dimension convolutional layer. It’s not as common as it’s 2d colleague, but we can see some examples in audio classification problems like: h<a href="https://medium.com/@oknagg/gender-classification-from-raw-audio-with-1d-convolutions-969c82e6b3d1">ttps://medium.com/@oknagg/gender-classification-from-raw-audio-with-1d-convolutions-969c82e6b3d1</a></p><p>After some tests (not too many since the aim of this experiment is to compare the two strategies without applying too much effort or refinement) I’d got the following structure with the best results.</p><pre>Conv1dModule(<br>  (m): Sequential(<br>    (0): Sequential(<br>      (0): ConvLayer(<br>        (0): Conv1d(1, 8, kernel_size=(4096,), stride=(1000,), padding=(2047,), bias=False)<br>        (1): BatchNorm1d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>        (2): ReLU()<br>      )<br>    )<br>    (1): Flatten(full=False)<br>    (2): Linear(in_features=640, out_features=500, bias=True)<br>    (3): ReLU()<br>    (4): Dropout(p=0.4, inplace=False)<br>    (5): Linear(in_features=500, out_features=50, bias=True)<br>    (6): ReLU()<br>    (7): Dropout(p=0.4, inplace=False)<br>    (8): Linear(in_features=50, out_features=25, bias=True)<br>    (9): ReLU()<br>    (10): Dropout(p=0.4, inplace=False)<br>    (11): Linear(in_features=25, out_features=5, bias=True)<br>  )<br>)</pre><pre>Total trainable params: 2,718,091</pre><p>The results, after some training:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*sYQqVmd1TtNCPlGGNOkIJw.png" /></figure><p>These results are not really good. Some classes are predicted well (Trumpet), but it’s easy to get confused by Bark, Cough and Laughter.</p><p>I’m sure that this model has a lot of room for improvement. It’s possible to use a better architecture, augmentations, or fine-tuning parameters like convolutional size or stride.</p><h3>Image classifier</h3><p>The image classifier requires that we convert the 1-dimension audio data to a 2-dimensional data. We use the AudioToSpec class in fastaudio to generate the image spectrogram. The spectrogram is a representation that shows the power of the sound in each frequency, through time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*UHF3I7xWuCtliYUi9WdtAw.png" /><figcaption>Spectogram of audio file showed previously</figcaption></figure><p>To improve the performance of this model we can use a really simple augmentation technique. We can cover some parts of the spectrogram in each training sample. This technique will improve de model generalization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*rhDP3pcP-gGBt3Q7_lJszQ.png" /><figcaption>Augmentation covering some random time interval and frequencies</figcaption></figure><p>Here we can see some examples of the original audio data, and the spectrogram used to train de model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WUob5LRohO9WxdMHzd505Q.png" /></figure><p>I use transfer learning from a pretrained <strong>xresnet18</strong> model to classify the spectrogram images. The network has more than 11 million parameters available for this classification task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*BnJ4vlrfLwuptE6kJeRI8g.png" /></figure><p>Results are far better. There is some confusion between Bark, Cough and Laughter but the 86% accuracy it’s pretty nice.</p><p>I think it’s really interesting how it’s possible to fine-tune a vision model specialized in common objects recognition to work with something so particular as spectrograms.</p><h3>Conclusions</h3><p>It seems pretty obvious, using this experiment, that a first approximation using images is not a bad idea.</p><p>Probably, the most important features in audio data are the power frequency of the signal. And the spectrogram is providing this information ready to be consumed. The one-dimension model has to generate it, with no clues, and sure it requires more examples and more fine-tunning.</p><p>The number of trainable parameters in each model it’s substantially different and that can also explain the better performance of the image model.</p><h3>Next steps</h3><ul><li>Improve the raw classifier using augmentation, or using transfer learning techniques</li><li>Train the first layer of the raw classifier using autoencoders, to build some kind of “language model” (audio model) that improves the classification of the full-connected layer</li><li>Use LSTM or some other method that allows the use of all the signal (including silences), regardless of the duration</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f2989e9647a8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Combining Global and Local Deep Learning Architectures for Art Mediums Classifications Tasks]]></title>
            <link>https://medium.com/@puiggrosd/combining-global-and-local-deep-learning-architectures-for-art-mediums-classifications-tasks-a609bacaaed?source=rss-d53b7e942a5a------2</link>
            <guid isPermaLink="false">https://medium.com/p/a609bacaaed</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[high-resolution-images]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[image-classification]]></category>
            <dc:creator><![CDATA[David Puiggròs]]></dc:creator>
            <pubDate>Wed, 19 May 2021 16:25:03 GMT</pubDate>
            <atom:updated>2021-05-19T16:25:03.201Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/846/0*TFX8eh4UnSLPU0i8" /></figure><p>In July 2020, the High-Performance Artificial Intelligence Group at the Barcelona SuperComputing Center BSC released the <a href="https://hpai.bsc.es/MAMe-dataset/">MAMe Dataset</a>. The aim of this dataset is to foster Deep Learning research on high-resolution image processing.</p><p>The task consists of assessing for each image the associated art medium.<br>High-resolution image processing is a two-sided problem. On one end, it provides great detail allowing for improved image processing tasks, but there is a price to be paid, memory management and performance issues to name a few.</p><p>To overcome these difficulties, we’ve explored the feasibility of combining Global Deep Learning architectures based on low-resolution images and Local Deep Learning architectures based on different sampling techniques of the original images.</p><p>The results show that hybrid architectures significantly outperform the Global and Local Deep Learning ones.</p><p>The reader can find all the related work on the following <a href="https://www.kaggle.com/egimenez/deep-learning-for-art-mediums-classification">kaggle notebook</a>.</p><h4>Acknowledgements</h4><p>We would like to thank the work of Ferran Parés, Anna Arias-Duart, Dario Garcia-Gasulla, Gema Campo-Francés, Nina Viladrich, Eduard Ayguadé, Jesús Labarta in the preparation of the dataset and the article A Closer Look at Art Mediums: The MAMe Image Classification Dataset.</p><p>This work was jointly done by <a href="https://www.linkedin.com/in/eduard-gimenez-funes/">Eduard Gimenez</a> and <a href="https://www.linkedin.com/in/david-puiggros/">David Puiggròs</a> during last winter and we present here the main results.</p><h3>The MAMe Dataset</h3><p>The MAMe dataset contains 37.407 images representing 29 different art mediums. These images belong to The Metropolitan Museum of Art, The Cleveland Museum of Art and The Los Angeles County Museum of Art.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/0*d678BTQjrwEoLQpB" /><figcaption>Some images from dataset</figcaption></figure><p>These images represent 29 different art mediums: Albumen photograph, Bronze, Ceramic, Clay, Engraving, Etching, Faience, Glass, Gold, Graphite, Hand-colored engraving, Hand-colored etching, Iron, Ivory, Limestone, Lithograph, Marble, Oil on canvas, Pen and brown ink, Poly-chromed wood, Porcelain, Silk and metal thread, Silver, Steel, Wood, Wood engraving, Woodblock, Woodcut, Woven fabric.</p><p>Each class contains 850 high-resolution images. The images are also labelled with subsets to train, validate and test. All the image files weigh more than 220GB of data.</p><p>This Dataset was introduced in the paper <a href="https://arxiv.org/abs/2007.13693">“A Closer Look at Art Mediums: The MAMe Image Classification Dataset”</a> and explored some problems found in classifying high resolution images and variable-shaped images. (<a href="https://hpai.bsc.es/MAMe-dataset">MAMe Dataset</a>)</p><h3>Problem description</h3><p>Some common techniques to classify images are well suited for low-resolution images (224x224 or 512x512). Converting MAMe high-resolution images to low-resolution could remove important data for classification (some classes are only different at really low level) but work with high-resolution images requires too much computational power.</p><p>The referenced paper explains a lot of details about the problem and shows some results. For example, rescaling images to R360K (600x600) and using resnet18 achieve an accuracy of 88.15%. And using R65K (256x256) and resnet18 achieves an accuracy 84.59%.</p><h3>Mimicking the Human Expert</h3><p>As described in the article, when we re-scale the images sometimes we lose important information. In some categories, this kind of information is the detail that the human expert uses to classify the image.</p><p>When we humans want to classify art mediums, we proceed as follows:</p><ul><li>We look at the big picture. Most of the time, that’s enough.</li><li>If we need more detail, we focus on some interesting parts of the object, zooming in.</li><li>Sometimes we need to zoom into different points of the image.</li><li>Then we add the different zoomed parts insights and get a global idea that we combine with our first impression.</li></ul><p>We have inspired our self in this procedure to create the following classification system:</p><ol><li>Re-scale the images and build an image classification model (Global-Model). This model will distinguish with very good precision objects really different like “Ceramic” vs “Oil on canvas”.</li><li>Cut the original image into pieces (like a puzzle) to detect low-level details. Build a model (Local-Model) to classify these pieces. This model will be good at distinguishing classes that are really different at a low level like different drawing techniques.</li><li>Generate a new model, the texture model, that combines all the predictions done by the local model using all the pieces of one image and generates a new prediction.</li><li>Lastly, combine (ensemble) the global model and the texture model in the final model.</li></ol><p>This schema presents some advantages:</p><ul><li>Both models are based on light-weighted images. Low-resolution images for the Global model and high resolution / small size images for the Local model. This setup allows learning with low computational power.</li><li>The only image operations applied to the raw images are re-scaling and cropping, which is faster than any calculation with neural networks models.</li><li>This is a general schema, that can easily be generalized to other domains beyond art medium classification.</li></ul><p>But also presents some important issues:</p><ul><li>How do you cut up the images? How are the parts labelled?</li><li>How do you combine the models?</li></ul><p>The rest of the text is the resolution of these issues with some techniques that we found useful.</p><h3>The Global Model</h3><p>The <strong>Global Model</strong> takes the full images (rescaled and cropped at 224x224) and generates class prediction. You can read further details of this model in the following <a href="https://www.kaggle.com/egimenez/global-dl-for-art-mediums-classifications-tasks">notebook</a>.</p><p>The key points:</p><ul><li>Re-scale the images (in a batch process previous learning) to 224x224</li><li>Uses the base model resnet34 and fine-tunes with the dataset.</li><li>The resizing uses padding to correct the different image shapes</li></ul><p>The results are great: precision score 84.3% evaluated in the test dataset. We can see in the figure the confusion matrix obtained.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/0*Fa9iTsPPE9kW0m2k" /><figcaption>Global model confusion matrix</figcaption></figure><p>We can see here some classification examples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/0*Nf1RpHVreuHUaNUR" /></figure><p>The most confusing categories are:</p><ul><li>‘Oil on canvas’, ‘Polychromed wood’</li><li>‘Marble’, ‘Limestone’</li><li>‘Engraving’, ‘Etching’</li><li>‘Silk and metal thread’, ‘Woven fabric’</li></ul><h3>The Local Model</h3><p>The <strong>Local Model</strong> takes pieces of the full images of size 224x224 and generates class prediction. You can read further details of the following <a href="https://www.kaggle.com/egimenez/local-dl-for-art-mediums-classifications-tasks">notebook</a>.</p><p>The key points:</p><ul><li>No image rescale. Instead, we use pieces of the original image (with size 224x224) and label these images with the same class that the original image.</li><li>The local regions are not chosen randomly. Rather a simple algorithm is defined to assess regions of interest.</li><li>We choose 10 local images among the most interesting (using a criterion that discards pieces without enough information).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/0*vBcCKH6bqT1g24iR" /><figcaption>Some examples of pieces 224x224</figcaption></figure><p>The results are good: precision score 62% evaluated in the test dataset. The problem is more difficult than the first one, so the results are not so good. But the interesting point is that we can combine the two approximations, using this model to improve the flaws of the first one.</p><p>We can see in the figure the confusion matrix obtained. Perhaps we can improve categories like ‘Oil on canvas’ or ‘Marble’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/0*9zlyYK4vmECLAche" /></figure><h3>The Texture Model</h3><p>After generating the <strong>Local</strong> model described we generate a new dataset using this algorithm:</p><p>For each image in the original dataset:</p><ul><li>Generate 16x16 pieces of 224x224 like a puzzle of the image. The pieces locations are equally spaced (in a grid layout). For more details please refer to the following <a href="https://www.kaggle.com/egimenez/global-local-dl-for-art-mediums-classification">notebook</a>.</li><li>Generate a 29x16x16 matrix using the Local model in each piece and accumulating all the 29 output for each one of the 16x16 pieces</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/0*UCHNjVkR8ZLIHjGN" /><figcaption>Original image: 5527x6685</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/0*iODDW0pwdDKmyW9_" /><figcaption>16x16 pieces in grid layout equally spaced</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mTXVrE-puNpVO_0K" /><figcaption>29x16x16 predictions using local model</figcaption></figure><p>Using this dataset we train a model to predict the correct class using the 29x16x16 tensor on each image. This tensor uses the output of the Local model (high resolution) and some spatial information (the grid layout).</p><p>The key points:</p><ul><li>We built a model from scratch using CNN with inputs 16x16x29 and output 29. The target class is the original image target class.</li><li>We precalculate all the inputs. That’s a problem because we can’t introduce any augmentation in this process.</li></ul><p>We achieve a precision of 72%, in the middle of the Global and Local Model.</p><p>Let’s see some examples:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/0*aM1fPKuwVLigDNFz" /></figure><p>The first one, an error of the Texture Model, seems that a lot of pieces are labelled in ‘Oil on canvas’ and ‘Polychrome wood’. And the Texture Model fails interpreting which is the correct category.</p><p>In the second one the decision it’s easier and the Texture Model does it fine. We can see that some pieces (in the lower right part of the image) look like Marble.</p><h3>The good news</h3><p>The good news is that we can ensemble the Global and the Texture model to increase accuracy. We are using really different inputs (high resolution versus low resolution) so we can expect a good improvement.</p><h3>The Global-Local Model</h3><p>The <strong>Global-Local Model</strong> is an ensemble model between the Global and Texture Model (the texture Model is the spatial aggregation of the Local Model).</p><p>The key points of these model:</p><ul><li>We use the validation data to train this model. We could use the training data, but using it we are less robust to possible overfitting.</li><li>The validation data is new for the two base models like will be the test data.</li><li>We use a simple neural network to learn the relation between the 58 inputs per image (the probabilities of each model) and the same class as previous models. More details: here</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*X4jdGvq3mUP_1SOT" /><figcaption>Global-Local Model architecture</figcaption></figure><h4>Main results</h4><p>We achieve a precision of 85.68%, increasing the precision of the first Global model in 1.7 points. This is comparable to other methods that use images with high resolutions, but that requires more computational power.</p><p>We can see in the final confusion matrix that we achieve some better results in ‘Oil on canvas’ or ‘Polychromed wood’ categories.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/0*_wX5maz1yt7KHLpq" /></figure><p>We can see other interesting results if we compare the models (Global and Texture) per image:</p><ul><li>The two models coincide in the classification of 75% of images. For these images, the precision achieved is 96%.</li><li>When the two models doesn’t coincide(25% of images), a 49% the global model is correct, a 32% the texture model is correct and 19% the two models are wrong.</li></ul><h3>Conclusions</h3><ul><li>We show a modular approximation to classify high resolution images that is the combination of global image (with low resolution) and texture identification of image pieces.</li><li>This approximation requires less computational power than others because it uses low resolutions images and common neural networks architectures and sizes.</li><li>The model generated improves the global model by 2 percentage points.</li><li>We think there are a lot of points of possible improvement: augmentations in textured data, the best selection of data, specific work in difficult categories…</li></ul><h3>Next steps</h3><ul><li>Analyze other methods for local model combination: LSTM with texture models, …</li><li>Augmentations: apply augmentations in the learning process of the texture model</li><li>Piece cutting: improve the method for cutting and selecting interesting pieces. Build a model to predict how important a piece of the image will be for classification and select important pieces before, without context knowledge.</li><li>Piece labeling: improve the labeling of pieces (solving the problem of pieces with a small intersection of the relevant object on them)</li><li>Model to predict the confidence of a texture prediction, and use that model to improve the combination</li><li>Test the new <a href="https://www.kaggle.com/ferranpares/mame-dataset">MAMe dataset</a> published in Kaggle with a 256x256 resolution and ready to work!</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a609bacaaed" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>