<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nikhil Badveli on Medium]]></title>
        <description><![CDATA[Stories by Nikhil Badveli on Medium]]></description>
        <link>https://medium.com/@nikhilbadveli6?source=rss-9059cc379720------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ognfMeL9Xb42MEIiRGXUTw.jpeg</url>
            <title>Stories by Nikhil Badveli on Medium</title>
            <link>https://medium.com/@nikhilbadveli6?source=rss-9059cc379720------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 21:58:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nikhilbadveli6/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to find which WooCommerce products will run out (and which never sell) with plain SQL]]></title>
            <link>https://medium.com/@nikhilbadveli6/how-to-use-askshelf-woocommerce-plugin-for-sql-79bb9876b1c7?source=rss-9059cc379720------2</link>
            <guid isPermaLink="false">https://medium.com/p/79bb9876b1c7</guid>
            <category><![CDATA[woocommerce-plugins]]></category>
            <category><![CDATA[wordpress]]></category>
            <category><![CDATA[woocommerce]]></category>
            <category><![CDATA[ecommerce]]></category>
            <category><![CDATA[wordpress-plugins]]></category>
            <dc:creator><![CDATA[Nikhil Badveli]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 13:31:01 GMT</pubDate>
            <atom:updated>2026-11-16T13:31:01.707Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Three queries for stock that’s about to run out, how much to reorder, and the money sitting on your shelves.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OUW6Jc7cd_BiDqB-spFczw.jpeg" /></figure><p>WooCommerce tells you when a product is <em>already</em> low on stock. You set a threshold, say 5 units, and you get an email when stock drops below it. That’s useful, but it doesn’t tell you the thing you actually want to know: when is it going to run out?</p><p>5 units of something that sells once a month is fine. 5 units of something that sells 10 a day is a problem you needed to solve last week.</p><p>The opposite problem is just as expensive and a lot quieter. Products that haven’t sold anything in three months don’t send you an email. They just sit there with your money in them.</p><p>Both of these are pretty easy to work out with SQL, once you know which tables to look at. So in this post we’re going to write three queries:</p><ol><li>How fast each product sells</li><li>How many days of stock are left, and how much to reorder</li><li>Dead stock: what hasn’t sold in 90 days, and how much money is tied up in it</li></ol><h3>Which tables?</h3><p>If you’ve ever searched for WooCommerce SQL queries, most of the answers you find join wp_posts with wp_postmeta and look for post_type = &#39;shop_order&#39;. Please don&#39;t do that. On stores using HPOS (high-performance order storage, the default for new stores now), orders aren&#39;t stored in wp_posts anymore, so those queries return nothing. Even on older stores, they&#39;re painfully slow.</p><p>Instead, we’ll use the lookup tables WooCommerce Analytics keeps up to date for its own reports. They work the same way with or without HPOS:</p><ul><li>wp_wc_order_stats: one row per order, with its status and dates. Refunds are rows here too, with a parent_id pointing at the original order.</li><li>wp_wc_order_product_lookup: one row per product in each order, with the quantity sold. A refunded item shows up as a row with a negative quantity.</li><li>wp_wc_product_meta_lookup: one row per product and variation, with the SKU, price and current stock.</li></ul><p>(If your table prefix isn’t wp_, change it in all the queries below.)</p><h3>Which orders count as sales?</h3><p>Before writing anything, we need to decide which orders count. If you just sum everything in wp_wc_order_product_lookup, failed payments and cancelled orders will make it look like you sold a lot more than you did.</p><p>WooCommerce Analytics leaves out these statuses by default, so we’ll do the same:</p><pre>s.status NOT IN (&#39;wc-pending&#39;, &#39;wc-failed&#39;, &#39;wc-cancelled&#39;, &#39;wc-trash&#39;, &#39;wc-auto-draft&#39;)</pre><p>You can check (and change) the list in WooCommerce › Analytics › Settings, under “Excluded statuses”. If you’ve changed it there, change it in the queries too.</p><h3>Step 1: How fast does each product sell?</h3><p>The idea is simple. Count the units sold in the last 30 days and divide by 30. That’s your sales speed, in units per day.</p><p>There’s one catch with variable products, though. Stock can be managed on each variation (a T-shirt with its own stock per size) or on the parent product (one shared pool of stock for all sizes). We need to count sales at the same level the stock is kept, otherwise the numbers won’t line up.</p><p>So first, let’s collect every product and variation that has managed stock:</p><pre>WITH stock_items AS (<br>  SELECT m.product_id AS stock_id,<br>         p.post_title AS name,<br>         m.sku,<br>         m.stock_quantity AS stock<br>  FROM wp_wc_product_meta_lookup m<br>  JOIN wp_posts p<br>    ON p.ID = m.product_id<br>   AND p.post_status IN (&#39;publish&#39;, &#39;private&#39;)<br>  JOIN wp_postmeta ms<br>    ON ms.post_id = m.product_id<br>   AND ms.meta_key = &#39;_manage_stock&#39;<br>   AND ms.meta_value = &#39;yes&#39;<br>)</pre><p>(Products are still stored in wp_posts even with HPOS, it&#39;s only orders that moved. So this join is fine.)</p><p>Then, the units sold in the last 30 days, counted against the variation if the variation has its own stock, and against the parent product otherwise:</p><pre>, sold AS (<br>  SELECT IF(l.variation_id IN (SELECT stock_id FROM stock_items),<br>            l.variation_id, l.product_id) AS stock_id,<br>         SUM(l.product_qty) AS units<br>  FROM wp_wc_order_product_lookup l<br>  JOIN wp_wc_order_stats s ON s.order_id = l.order_id<br>  WHERE s.status NOT IN (&#39;wc-pending&#39;, &#39;wc-failed&#39;, &#39;wc-cancelled&#39;, &#39;wc-trash&#39;, &#39;wc-auto-draft&#39;)<br>    AND s.date_created &gt;= CURDATE() - INTERVAL 30 DAY<br>    AND s.date_created &lt;  CURDATE()<br>  GROUP BY stock_id<br>)</pre><p>A couple of things to notice here:</p><ul><li>The period ends at the start of today, not now. If you include today, a half-finished day drags the average down, especially in the morning.</li><li>SUM(product_qty) includes the negative rows from refunded items, so refunds are taken out automatically.</li><li>date_created in these tables is already in your store&#39;s time zone (there&#39;s a separate date_created_gmt if you ever need UTC). CURDATE(), however, uses the database server&#39;s time zone, so if your server isn&#39;t in the same time zone as your store, replace CURDATE() with the date written out, like &#39;2026-11-15&#39;.</li></ul><h3>Step 2: Days of stock left, and how much to reorder</h3><p>Now we can put the two together. Days left is just stock divided by units sold per day:</p><pre>SELECT i.name,<br>       i.sku,<br>       i.stock,<br>       sold.units                          AS sold_30d,<br>       ROUND(sold.units / 30, 2)           AS per_day,<br>       ROUND(i.stock / (sold.units / 30), 1) AS days_left,<br>       GREATEST(0, CEIL(sold.units / 30 * (7 + 30)) - i.stock) AS reorder_qty<br>FROM stock_items i<br>JOIN sold ON sold.stock_id = i.stock_id<br>WHERE sold.units &gt; 0<br>ORDER BY days_left;</pre><p>The reorder quantity needs a bit of explanation. 7 is the lead time, i.e. how many days it takes your supplier to deliver. 30 is how many days of stock you want to have after the delivery arrives. So the formula is &quot;enough to cover the wait plus the next 30 days, minus what&#39;s already on the shelf&quot;. Change both numbers to whatever fits your business.</p><p>Here’s what it looks like on my test store:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*u2XrhA1s9quguug8KhZbvQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H6CJn8A9TIQK_9oare_FAQ.png" /></figure><p>See the linen tote bag? 48 units is way above any low stock threshold you’d set, and no email would ever go out. But at 6 a day it runs out in a week, which is the same as the supplier’s lead time. That’s exactly the kind of product this query is for.</p><p>Also notice the shirt sizes are separate rows. Each size has its own stock, so the medium is about to run out even though the small is fine.</p><p>Add HAVING days_left &lt;= 14 just before the ORDER BY if you only want the urgent ones.</p><h3>Step 3: Dead stock</h3><p>Now for the quiet problem. We want products that:</p><ul><li>have stock on hand,</li><li>haven’t sold anything in the last 90 days,</li><li>and weren’t just added (a product you added last week hasn’t had a chance to sell yet).</li></ul><p>For each one, we want the last date it sold and how much the stock is worth.</p><pre>WITH stock_items AS (<br>  SELECT m.product_id AS stock_id,<br>         p.post_title AS name,<br>         p.post_date  AS added,<br>         m.sku,<br>         m.stock_quantity AS stock,<br>         m.min_price      AS price<br>  FROM wp_wc_product_meta_lookup m<br>  JOIN wp_posts p<br>    ON p.ID = m.product_id<br>   AND p.post_status IN (&#39;publish&#39;, &#39;private&#39;)<br>  JOIN wp_postmeta ms<br>    ON ms.post_id = m.product_id<br>   AND ms.meta_key = &#39;_manage_stock&#39;<br>   AND ms.meta_value = &#39;yes&#39;<br>  WHERE m.stock_quantity &gt; 0<br>),<br>last_sale AS (<br>  SELECT id, MAX(sold_at) AS last_sold<br>  FROM (<br>    SELECT l.product_id AS id, s.date_created AS sold_at<br>    FROM wp_wc_order_product_lookup l<br>    JOIN wp_wc_order_stats s ON s.order_id = l.order_id<br>    WHERE l.product_qty &gt; 0<br>      AND s.status NOT IN (&#39;wc-pending&#39;, &#39;wc-failed&#39;, &#39;wc-cancelled&#39;, &#39;wc-trash&#39;, &#39;wc-auto-draft&#39;)<br>    UNION ALL<br>    SELECT l.variation_id, s.date_created<br>    FROM wp_wc_order_product_lookup l<br>    JOIN wp_wc_order_stats s ON s.order_id = l.order_id<br>    WHERE l.product_qty &gt; 0<br>      AND l.variation_id &gt; 0<br>      AND s.status NOT IN (&#39;wc-pending&#39;, &#39;wc-failed&#39;, &#39;wc-cancelled&#39;, &#39;wc-trash&#39;, &#39;wc-auto-draft&#39;)<br>  ) x<br>  GROUP BY id<br>)<br>SELECT i.name,<br>       i.sku,<br>       i.stock,<br>       DATE(ls.last_sold)          AS last_sold,<br>       ROUND(i.stock * i.price, 2) AS value_at_price<br>FROM stock_items i<br>LEFT JOIN last_sale ls ON ls.id = i.stock_id<br>WHERE i.added &lt; CURDATE() - INTERVAL 90 DAY<br>  AND (ls.last_sold IS NULL OR ls.last_sold &lt; CURDATE() - INTERVAL 90 DAY)<br>ORDER BY value_at_price DESC;</pre><p>The last_sale part looks a bit odd with the UNION ALL, so let me explain. Every row in the order lookup table has a product_id, and variations also have a variation_id. By taking the latest sale under both ids, the query works whether the stock is kept on the parent product or on each variation, the same problem as in step 1.</p><p>last_sold IS NULL means it has never sold at all, at least not since the Analytics tables started tracking it. Those are usually the first ones worth looking at.</p><p>The list is sorted by value, so the products with the most money tied up come first. Selling price isn’t really what that stock cost you, though. If you’ve turned on WooCommerce’s Cost of Goods Sold feature (WooCommerce › Settings › Advanced › Features) and added costs to your products, you can value the stock at cost instead. Add this join to stock_items:</p><pre>  LEFT JOIN wp_postmeta c<br>    ON c.post_id = m.product_id<br>   AND c.meta_key = &#39;_cogs_total_value&#39;</pre><p>and select ROUND(i.stock * c.meta_value, 2) AS value_at_cost instead. (Variations without their own cost use the parent&#39;s cost, so you&#39;d need one more join on p.post_parent to handle that properly.)</p><h3>How do I run these?</h3><p>Any of these will work:</p><ul><li>phpMyAdmin or Adminer, if your host gives you one. Paste the query into the SQL tab.</li><li>WP-CLI: save the query to a file and run wp db query &lt; runway.sql.</li><li>TablePlus, Sequel Ace, DBeaver or whatever you normally use.</li></ul><p>They’re all read-only SELECT queries, so they won&#39;t change anything. Still, if your store is big, run them on a staging copy first or at a quiet time of day. WITH needs MySQL 8 or MariaDB 10.2 or newer, which almost every host has these days.</p><h3>A few gotchas</h3><p>The lookup tables have to be up to date. WooCommerce fills them in the background as orders come in. If you’ve just imported a lot of old orders, or Analytics shows zeros for periods you know had sales, go to WooCommerce › Analytics › Settings and use “Import historical data”. Otherwise the queries will say things never sold when they did.</p><p>30 days isn’t always the right window. A product that sells mostly in December will look like dead stock in October. For seasonal stock, compare with the same period last year instead of the last 30 or 90 days.</p><p>Stock that’s out already won’t show in step 2. If a product is at 0 and has been for a while, it didn’t sell in the last 30 days, so it has no sales speed. Look at those separately with stock_status = &#39;outofstock&#39; in wp_wc_product_meta_lookup.</p><p>Backorders. If you allow backorders, stock can go negative. The days-left query will give you a negative number of days, which is technically correct but a bit confusing.</p><h3>Wrapping up</h3><p>That’s it. Three queries that answer questions WooCommerce doesn’t answer out of the box:</p><ol><li>How fast each product sells, at the right level for variable products</li><li>How many days of stock are left and how much to reorder</li><li>What hasn’t sold in 90 days and how much money is sitting in it</li></ol><p>I’d suggest saving them somewhere and running them every Monday. Even better, put the numbers in a spreadsheet each week and you’ll start seeing your sales patterns pretty quickly.</p><p>If you don’t fancy running SQL every week, I built a free WordPress plugin, <a href="https://askshelf.in/?utm_source=medium.com&amp;utm_medium=post&amp;utm_campaign=article2">Askshelf</a>, that runs these same checks for you. Its dashboard shows what runs out in the next 14 days and what hasn’t sold in 60, and you can ask it things like “What should I reorder this week?” in plain English. It’s on <a href="https://wordpress.org/plugins/askshelf/">WordPress.org</a> if you want to try it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WIhlfHM2jAO0k_Yz8LKeVA.png" /></figure><p>Let me know in the comments if these queries saved you from an empty shelf, or if you find a case they get wrong!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79bb9876b1c7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Can an LLM answer questions about my WooCommerce store without writing SQL?]]></title>
            <link>https://medium.com/@nikhilbadveli6/how-i-built-askshelf-woocommerce-plugin-dc1331db007a?source=rss-9059cc379720------2</link>
            <guid isPermaLink="false">https://medium.com/p/dc1331db007a</guid>
            <category><![CDATA[woocommerce]]></category>
            <category><![CDATA[wordpress-plugins]]></category>
            <category><![CDATA[woocommerce-plugins]]></category>
            <category><![CDATA[ecommerce]]></category>
            <category><![CDATA[wordpress]]></category>
            <dc:creator><![CDATA[Nikhil Badveli]]></dc:creator>
            <pubDate>Sun, 15 Nov 2026 13:37:54 GMT</pubDate>
            <atom:updated>2026-11-15T13:37:54.324Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Text-to-SQL looked like the obvious answer. It wasn’t.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*azKiIt-qOouyJtYKrAR59g.jpeg" /></figure><p>The title is pretty much the question I asked myself when I started this. Every WooCommerce store has the data to answer questions like “How did sales this month compare with last month?” or “What’s going to run out next week?”, but getting the answer means clicking through Analytics, setting date filters, exporting a CSV and opening it in a spreadsheet.</p><p>LLMs are good at reading questions in plain English. MySQL is good at answering them. So surely you just put the two together, right?</p><p>Well, kind of. In this post I’ll walk through the three versions I went through, what broke in each one, and the parts I’d keep if I were starting again.</p><h3>Attempt 1: Text-to-SQL</h3><p>This is the approach every tutorial shows. You give the model your database schema, ask it to write a query for the question, run the query and pass the rows back so it can write the answer.</p><p>WooCommerce already keeps its Analytics data in a handful of neat lookup tables, so I didn’t even need the messy wp_postmeta table. The prompt looked something like this:</p><pre>$schema = &quot;<br>wp_wc_order_stats(order_id, parent_id, date_created, date_paid, status,<br>                  num_items_sold, total_sales, net_total, customer_id)<br>wp_wc_order_product_lookup(order_id, product_id, variation_id,<br>                           product_qty, product_net_revenue, date_created)<br>wp_wc_product_meta_lookup(product_id, sku, stock_quantity, min_price)<br>&quot;;<br><br>$prompt = &quot;You write MySQL queries for a WooCommerce store.<br>Tables:\n{$schema}<br>Write one SELECT query that answers: {$question}&quot;;</pre><p>And it worked! On the first try I asked “What were my sales in October?” and got back a perfectly valid query:</p><pre>SELECT SUM(total_sales)<br>FROM wp_wc_order_stats<br>WHERE date_created BETWEEN &#39;2026-10-01&#39; AND &#39;2026-10-31&#39;;</pre><p>The number it gave me was also wrong. Not wildly wrong, just wrong enough that if you compared it with the WooCommerce Analytics page, you’d never trust the thing again. Can you spot the problems? There are four of them.</p><ol><li>Order status. There’s no status filter, so failed, cancelled and pending orders are all counted as sales. WooCommerce Analytics leaves those out by default (and the store owner can change the list in the settings).</li><li>“Sales” means different things. total_sales includes tax and shipping. What WooCommerce calls &quot;Net sales&quot; is net_total. The model picked the column whose name sounded right.</li><li>The last day goes missing. date_created is a DATETIME, so BETWEEN ... &#39;2026-10-31&#39; stops at midnight at the <em>start</em> of October 31st. A whole day of orders gone.</li><li>Which date? Analytics can report by the date the order was created, paid or completed, depending on a setting. The model always went with date_created.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b1dwteiCQQoOKqH2zga5PQ.png" /></figure><p>Refunds are another trap. In wp_wc_order_stats a refund is its own row with a parent_id and negative totals. Sometimes the model filtered on parent_id = 0 to &quot;only count real orders&quot;, which quietly threw the refunds away.</p><p>And when I tried an older model, it ignored my schema completely and wrote the classic wp_posts WHERE post_type = &#39;shop_order&#39; query from 2015-era Stack Overflow answers. On a store with HPOS (high-performance order storage, the default for new stores now), orders don&#39;t live in wp_posts anymore, so it returned nothing at all.</p><p>I could keep adding rules to the prompt (“always filter status”, “use net_total”, “use &lt; next day instead of BETWEEN”…), and I did for a while. But every rule I added was one more thing the model could forget on a long question. Plus, there’s the bigger problem: I’m letting an LLM write SQL that runs against a production database that also has customer emails and addresses in it. Not great.</p><h3>Attempt 2: Let WooCommerce do the maths</h3><p>The thing I eventually realised is that WooCommerce already solved all four of those problems. The Analytics pages in wp-admin are powered by a REST API (/wc-analytics/...) that applies the status filter, the date setting, the store&#39;s time zone and refunds exactly the way the store owner expects.</p><p>So instead of asking the model to write SQL, I gave it a small set of tools (function calling) that wrap those Analytics endpoints. The model’s job changes from “write a correct query” to “pick the right tool and fill in the dates”. That’s a much easier job, and LLMs are actually good at it.</p><p>Here’s what one tool looks like. The first part is the JSON schema the model sees:</p><pre>$tools = array(<br>    array(<br>        &#39;type&#39;     =&gt; &#39;function&#39;,<br>        &#39;function&#39; =&gt; array(<br>            &#39;name&#39;        =&gt; &#39;get_sales_summary&#39;,<br>            &#39;description&#39; =&gt; &#39;Net sales, orders, average order value and refunds for a date range, as WooCommerce Analytics reports them.&#39;,<br>            &#39;parameters&#39;  =&gt; array(<br>                &#39;type&#39;       =&gt; &#39;object&#39;,<br>                &#39;properties&#39; =&gt; array(<br>                    &#39;start_date&#39; =&gt; array( &#39;type&#39; =&gt; &#39;string&#39;, &#39;description&#39; =&gt; &#39;YYYY-MM-DD, store time zone&#39; ),<br>                    &#39;end_date&#39;   =&gt; array( &#39;type&#39; =&gt; &#39;string&#39;, &#39;description&#39; =&gt; &#39;YYYY-MM-DD, inclusive&#39; ),<br>                ),<br>                &#39;required&#39;   =&gt; array( &#39;start_date&#39;, &#39;end_date&#39; ),<br>            ),<br>        ),<br>    ),<br>);</pre><p>And the second part is the PHP that actually runs when the model calls it. The neat trick here is rest_do_request(), which calls a REST route in the same PHP process, so there&#39;s no HTTP request and no API keys to deal with. It runs as the logged-in user, so whoever asks the question needs permission to view WooCommerce reports, which is exactly what you want anyway:</p><pre>function get_sales_summary( array $args ): array {<br>    $request = new WP_REST_Request( &#39;GET&#39;, &#39;/wc-analytics/reports/revenue/stats&#39; );<br>    $request-&gt;set_query_params( array(<br>        &#39;after&#39;  =&gt; $args[&#39;start_date&#39;] . &#39;T00:00:00&#39;,<br>        &#39;before&#39; =&gt; $args[&#39;end_date&#39;] . &#39;T23:59:59&#39;,<br>    ) );<br><br>    $response = rest_do_request( $request );<br>    $data     = rest_get_server()-&gt;response_to_data( $response, false );<br>    // Some Analytics controllers return objects inside the data; make it all arrays.<br>    $data     = json_decode( wp_json_encode( $data ), true );<br>    $totals   = $data[&#39;totals&#39;];<br><br>    return array(<br>        &#39;period&#39;          =&gt; $args,<br>        &#39;net_sales&#39;       =&gt; $totals[&#39;net_revenue&#39;],<br>        &#39;orders&#39;          =&gt; $totals[&#39;orders_count&#39;],<br>        &#39;avg_order_value&#39; =&gt; $totals[&#39;avg_order_value&#39;],<br>        &#39;refunds&#39;         =&gt; $totals[&#39;refunds&#39;],<br>    );<br>}</pre><p>Notice the end date is inclusive and goes until 23:59:59, so October 31st is back. And because the numbers come from the same API as the Analytics page, they match it, which is honestly the only thing a store owner cares about.</p><p>I ended up with about a dozen of these: sales summary, top products, sales by category/country/weekday, stock running out, stock that isn’t selling, coupons, new vs returning customers and so on. Each one is boring on its own, but together they cover most of the questions people actually ask.</p><h3>The loop</h3><p>Now the model needs a way to call these tools. If you’ve used the OpenAI-style chat completions API, this is the standard tool-calling loop. I’m using OpenRouter here since it lets you switch between models without changing the code, but any OpenAI-compatible endpoint works the same.</p><pre>$handlers = array( &#39;get_sales_summary&#39; =&gt; &#39;get_sales_summary&#39; /* , ...the rest */ );<br><br>$messages = array(<br>    array( &#39;role&#39; =&gt; &#39;system&#39;, &#39;content&#39; =&gt; $system_prompt ),<br>    array( &#39;role&#39; =&gt; &#39;user&#39;, &#39;content&#39; =&gt; $question ),<br>);<br><br>for ( $step = 0; $step &lt; 6; $step++ ) {<br>    $response = wp_remote_post( &#39;https://openrouter.ai/api/v1/chat/completions&#39;, array(<br>        &#39;timeout&#39; =&gt; 60,<br>        &#39;headers&#39; =&gt; array(<br>            &#39;Authorization&#39; =&gt; &#39;Bearer &#39; . OPENROUTER_API_KEY,<br>            &#39;Content-Type&#39;  =&gt; &#39;application/json&#39;,<br>        ),<br>        &#39;body&#39;    =&gt; wp_json_encode( array(<br>            &#39;model&#39;    =&gt; LLM_MODEL,<br>            &#39;messages&#39; =&gt; $messages,<br>            &#39;tools&#39;    =&gt; $tools,<br>        ) ),<br>    ) );<br><br>    $message    = json_decode( wp_remote_retrieve_body( $response ), true )[&#39;choices&#39;][0][&#39;message&#39;];<br>    $messages[] = $message;<br><br>    // No tool calls means the model has written its answer.<br>    if ( empty( $message[&#39;tool_calls&#39;] ) ) {<br>        return $message[&#39;content&#39;];<br>    }<br><br>    foreach ( $message[&#39;tool_calls&#39;] as $call ) {<br>        $name   = $call[&#39;function&#39;][&#39;name&#39;];<br>        $args   = json_decode( $call[&#39;function&#39;][&#39;arguments&#39;], true ) ?: array();<br>        $result = isset( $handlers[ $name ] )<br>            ? call_user_func( $handlers[ $name ], $args )<br>            : array( &#39;error&#39; =&gt; &quot;Unknown tool {$name}&quot; );<br><br>        $messages[] = array(<br>            &#39;role&#39;         =&gt; &#39;tool&#39;,<br>            &#39;tool_call_id&#39; =&gt; $call[&#39;id&#39;],<br>            &#39;content&#39;      =&gt; wp_json_encode( $result ),<br>        );<br>    }<br>}</pre><p>The $step &lt; 6 limit is there because a comparison question (&quot;this month vs last month&quot;) needs at least two calls, and occasionally the model gets stuck calling the same tool again and again. Six was plenty in my testing.</p><p>One thing that tripped me up: the model has no idea what “today” is, or which time zone the store is in. Ask it about “last month” and it will happily guess based on its training data. So the system prompt has to tell it:</p><pre>$now = new DateTimeImmutable( &#39;now&#39;, wp_timezone() );<br><br>$system_prompt = sprintf(<br>    &#39;You answer questions about a WooCommerce store using only the tools. &#39; .<br>    &#39;Today is %s, %s. The store time zone is %s and the currency is %s. &#39; .<br>    &#39;Weeks start on Monday. Never guess a number a tool did not return.&#39;,<br>    $now-&gt;format( &#39;l&#39; ),<br>    $now-&gt;format( &#39;Y-m-d&#39; ),<br>    wp_timezone_string(),<br>    get_woocommerce_currency()<br>);</pre><p>That last sentence (“never guess a number”) made a surprising difference. Without it, when a tool didn’t return something, the model would sometimes just make up a plausible-looking figure. Not ideal for a sales report :)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*broRtXsMw68zSbpaf4nbfA.png" /></figure><h3>Attempt 3: Bringing SQL back, carefully</h3><p>Tools cover maybe 90% of the questions. The other 10% are the weird ones: “What’s the average number of items in orders that used a coupon?”, “Which products are usually bought together?”. You can’t write a tool for every possible question.</p><p>So I did add SQL back, but as one more tool (run_sql_query) that the model only reaches for when nothing else fits, with a lot of guard rails around it. Here&#39;s what a query has to pass before it runs:</p><ul><li>One statement, starting with SELECT or WITH. No comments, no ; in the middle, no @variables.</li><li>No dangerous keywords: nothing that writes, locks, sleeps, reads files or changes the session.</li><li>An allowlist of tables: the Analytics lookup tables, products and categories. Anything else, like wp_users or wp_options, is refused, even inside a join.</li><li>No personal columns: emails, names, phone numbers, addresses and IPs are refused wherever they show up, and so is SELECT *.</li><li>At most 200 rows come back, and anything that still looks like an email address gets replaced.</li></ul><p>And even after all that, the query runs inside a read-only transaction with a time limit, so in the worst case it just fails:</p><pre>$wpdb-&gt;query( &#39;SET SESSION MAX_EXECUTION_TIME = 5000&#39; ); // MariaDB: max_statement_time = 5<br>$wpdb-&gt;query( &#39;START TRANSACTION READ ONLY&#39; );<br>$rows  = $wpdb-&gt;get_results( $sql, ARRAY_A );<br>$error = $wpdb-&gt;last_error;<br>$wpdb-&gt;query( &#39;ROLLBACK&#39; );</pre><p>If the database rejects the query, I send the error message back to the model as the tool result. Most of the time it reads the error, fixes the query and tries again on the next step, which is kind of fun to watch.</p><p>I also check the query for the problems from attempt 1. If it reads wc_order_stats but never mentions status, a note goes back with the result saying which statuses Analytics leaves out, and the model usually corrects itself.</p><h3>Show your working</h3><p>The last piece wasn’t technical at all, but I think it mattered the most. Under every answer, I show how it was worked out: which tools were called, with which dates and settings, and any caveats the tool returned (for example “Cost of Goods Sold is off, so profit can’t be calculated”). If the custom SQL tool was used, the query is shown too.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HVRgehAVFaOcYFIr0hD-ig.png" /></figure><p>When the number looks surprising, the store owner can check <em>why</em> instead of just not trusting it. And half the time it turned out the question was ambiguous (“this week” meant Sunday to Saturday for them, not Monday to Sunday), not that the answer was wrong.</p><h3>What gets sent to the model?</h3><p>Something I didn’t think about at first: with this design, the model never sees the database. It only sees the question and the small JSON results the tools return, which are totals like “net sales: 4,812.50” or a list of product names with units sold. No customer names, emails or addresses ever leave the server, because no tool returns them.</p><p>That’s a much easier conversation to have with a store owner than “the AI can run SQL on your database”. It’s also cheaper, since a JSON result with 20 rows uses a lot fewer tokens than a schema plus raw query results.</p><h3>So, can it?</h3><p>Yes, mostly! But not by writing SQL, at least not as the main path. The version that worked:</p><ol><li>Let the platform do the maths. WooCommerce’s own Analytics API already knows about statuses, refunds, time zones and HPOS.</li><li>Give the model a small set of well-described tools instead of a schema.</li><li>Tell it today’s date, the time zone and the currency, and tell it not to guess.</li><li>Keep SQL as a guarded, read-only fallback for the odd questions.</li><li>Show how every answer was worked out.</li></ol><p>A few things are still not perfect. Smaller models sometimes pick a tool that’s close but not quite right (top products by revenue when the question was about units), and long comparison questions can take three or four round trips, so it’s a few seconds rather than instant.</p><p>I ended up turning all of this into a free WordPress plugin called <a href="https://askshelf.in/?utm_source=medium.com&amp;utm_medium=post&amp;utm_campaign=article1">Askshelf</a>, since I kept wanting it on my own test stores. If you want to see how the tools and the “how I got this” part behave on real data without building any of it, it’s on <a href="https://wordpress.org/plugins/askshelf/">WordPress.org</a>, and there’s a live demo that runs a sample store in your browser.</p><p>If you’re building something similar for another platform (Shopify, Magento, your own app), I’d love to hear whether you went with text-to-SQL or tools, and what broke. Let me know in the comments!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dc1331db007a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Good Stuff]]></title>
            <link>https://medium.com/@nikhilbadveli6/the-good-stuff-fe4df538fcdb?source=rss-9059cc379720------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe4df538fcdb</guid>
            <dc:creator><![CDATA[Nikhil Badveli]]></dc:creator>
            <pubDate>Sun, 07 Apr 2024 23:22:39 GMT</pubDate>
            <atom:updated>2024-04-07T23:22:39.649Z</atom:updated>
            <content:encoded><![CDATA[<p>This is a list of all the good stuff, at least according to my opinion.</p><h3>Movies</h3><ul><li>Dune 2 (2024), Sci-Fi</li><li>2001: A Space Odyssey (1968)</li><li>Blade Runner 2049 (2017)</li><li>Star Wars: The Empire strikes back (1980)</li><li>I, Robot (2004)</li><li>I am Legend (2007)</li><li>The Matrix (1999)</li><li>The Martian (2016)</li><li>Jurassic Park (1993)</li><li>The Godfather (1972)</li><li>Lord of the rings: The fellowship of the ring (2001)</li><li>Shawshank Redemption (1994), Feel good</li><li>Good Will Hunting (1997)</li><li>Pursuit of Happyness (2006)</li><li>Jack the Giant Slayer (2013)</li><li>Ella Enchanted (2004)</li><li>Enchanted (2007)</li><li>Frozen (2013)</li><li>Tangled (2010)</li><li>Just like Heaven (2005)</li></ul><h3>Games</h3><ul><li>Red dead redemption 2 (2018)</li><li>The Last of Us: Part 2 (2020)</li><li>Uncharted 4: A Thief’s End (2016)</li><li>Horizon Zero Dawn (2017)</li><li>Grand Theft Auto 5 (2013)</li><li>Assassin’s Creed Unity (2014)</li><li>God of War (2018)</li><li>The Witcher 3 (2015)</li><li>Ghost of Tsushima (2020)</li><li>Detroit: Become Human (2018)</li></ul><h3>TV Shows</h3><ul><li>Breaking Bad (2008)</li><li>The Big Bang Theory (2007)</li><li>Game of Thrones (2011)</li><li>Million Yen Women (2017)</li><li>Alice in Borderland (2020)</li><li>Squid Game (2021)</li><li>All of us are dead (2022)</li><li>Sweet Tooth (2021)</li><li>The Mandalorian (2021)</li><li>Power Rangers Dino Thunder (2004)</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe4df538fcdb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Indian Art and Culture — Architecture]]></title>
            <link>https://medium.com/@nikhilbadveli6/indian-art-and-culture-architecture-e4222a5b5943?source=rss-9059cc379720------2</link>
            <guid isPermaLink="false">https://medium.com/p/e4222a5b5943</guid>
            <category><![CDATA[temple-architecture]]></category>
            <category><![CDATA[temples-of-india]]></category>
            <dc:creator><![CDATA[Nikhil Badveli]]></dc:creator>
            <pubDate>Wed, 14 Feb 2024 20:33:49 GMT</pubDate>
            <atom:updated>2024-02-14T20:35:18.873Z</atom:updated>
            <content:encoded><![CDATA[<h3>Indian Art and Culture — Architecture</h3><p>I have recently started reading a book, very aptly titled as “<em>Indian Art and Culture</em>” by Nitin Singhania. I got the motivation to start reading this particular book, after watching the movie <em>12th fail</em>.</p><p>It kind of reminded me of my days, years ago when I used to cram every single day for my JEE examination. Ah, the good old days. The movie put this ridiculous notion in my mind to apply for the UPSC exam and just give prelims a shot, just for fun.</p><p>Then, I started researching about the syllabus for the exam and man, I realized it’s way too much for me to casually glean through everything and give the exam a go. If things are that easy, then it is as if I am looking down on all those people who go to Delhi and waste away their good prime years on prepping.</p><p>Anyways, I came across this book on one of the reddit posts which gave a detailed list of the content and some preparation strategies too.</p><p>I am pretty sure it’s this one —</p><p><a href="https://www.reddit.com/r/UPSC/comments/141nzc8/a_guide_to_getting_started_with_upsc_cse_2024/?utm_source=share&amp;utm_medium=web2x&amp;context=3">https://www.reddit.com/r/UPSC/comments/141nzc8/a_guide_to_getting_started_with_upsc_cse_2024/</a></p><p>I guess, enough of the backstory for today’s post. I just want to say how amazed I was to learn all these new things about India and then I figured whatever I was taught in school doesn’t even come close to the true depth of Indian history, culture, art and architecture.</p><p>I felt like documenting all the things I am learning from that book and that is the purpose of this post. Without further ado, let me dive into the amazing historical architecture of our country and peek into how life must’ve been in those days.</p><h3>Architecture</h3><h4>Temples</h4><p>I am a Hindu by religion, a very reluctantly chosen path for my life (not that I had much of a choice). Everytime, my mom drags me into temples, I always wondered even from my childhood, why are temples the way they are?</p><p>Apparently, temples didn’t always look the way they do now. Imagine, the most prominent feature of a temple, its <strong><em>Shikhara</em></strong>, the tower over the sacred sanctum (where the idol of the god lies, also known as <strong><em>Garbha Griha</em></strong>), not being present at all.</p><p>So, they used to just have the pillars and a flat roof and the whole temple complex were typically just square in shape. See below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-W3DUn-aLNhyFMRea1QEMg.jpeg" /><figcaption>Buddhist Monument — Temple No. 17, Sanchi, Madhya Pradesh</figcaption></figure><p>And then they started going a little bit up, more raised so to say. (See below). Additionally, around the central sanctum, a sort of passageway started to appear, which is still now the norm, where we do the <strong><em>Pradakshina</em></strong>. I always get confused with how many rounds I am supposed to go around. Different people say different things. Some say three, some say five and then there are others highly devoted, sometimes roll on the ground to make these rounds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e1IEksvkbdUjLMDWjw2sQw.jpeg" /><figcaption>Parvati temple — Nachna Kuthara in Madhya Pradesh</figcaption></figure><p>Now, I am not really sure if these can be called as the oldest temples or the earliest ones to be built over the entirety of the Indian subcontinent. These temple images I am sharing are only a reference to understand the evolution of our temple architecture.</p><p>Moving on, the next stage is when the towers started to show up on the top of the central sanctum, albeit relatively small. One other notable development, is the emergence of <strong><em>Mandapa</em></strong>, the pillared structure in front of the sanctum, attached to it, giving the full structure a rectangular shape now.</p><p>During this time, some temples were built in the <em>Panchayatan</em> style, with the idea of adding four more shrines, like subsidiaries to the main shrine in the center.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kxx05HN4AcsF6s-ZLAy5_g.jpeg" /><figcaption>Durga temple — Aihole, Karnataka</figcaption></figure><p>I am sure, there were lots of temples that came in every different shape and every little variation you can imagine, but the above mentioned features (<em>Garbha Griha, Shikhara, Mandapa</em>) remain common to the most of the temples you see in the Indian landmass.</p><p>Also, if you look at things deeply, you can study the same thing by asking different set of questions. Like what kinds of stones were used? or the sculptures, the design and the patterns carved out of the pillars, the walls? or are there more interesting features, like the <strong><em>Dhvajasthamba</em></strong> (the flagstaff in the front of the temples) and <strong><em>Vahana</em></strong> (the mount of the main deity), which is placed just before the central sanctum.</p><p>A detailed explanation of all these variations is out of scope for this post. Instead, let’s see what different styles of temple architecture can be seen.</p><blockquote>Broadly, we have two styles — <strong><em>Nagara</em> </strong>(mostly North and West India) and <strong><em>Dravidian</em> </strong>(mostly seen in South India).</blockquote><h4>Nagara style</h4><p>Some common features you can see with temples following this style are :-</p><ul><li>At the top end of Shikhara, a horizontal fluted disc (called as <strong><em>Amalaka</em></strong>) shaped stone is placed and a spherical shaped ball with pointy spire (called as <strong><em>Kalasha</em></strong>) becomes the highest point. Sometimes, a flag is hoisted here.</li><li>Shikharas and Mandapa, are built in such a way that it looks as if it is a continuous structure, with what I like to call mini-shikharas over the top of the mandapa.</li><li>And finally, the inner walls of the temple are shaped in a way that it follow a repetitive geometrical pattern on all three sides of the idol, except for the front which is where you enter obviously. These are called <strong><em>Rathas </em></strong>— like Triratha, Pancharatha, Saptharatha etc., (See below)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/955/1*5fb6seSMBO3u98GxwO8y_g.jpeg" /><figcaption>Anatomy of a temple in Nagara style</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*dl2HdISxfftpaVPehxGZHQ.jpeg" /><figcaption>Types of Rathas — when viewed from the top</figcaption></figure><p>Some of the most famous temples that are built in this style are <em>Vishvanatha Temple in Khajuraho</em>, <em>Konark Sun Temple</em> in Odisha and <em>Somnath Temple</em> in Gujarat.</p><blockquote>One fun fact here. The sun temple in Konark, is actually built to emulate a ratha (which means a ritual cart) and so it has wheels on the sides of the temples. One of these wheels is featured on the backside of the 10 rupee currency note.</blockquote><h4>Dravidian style</h4><p>This was the style developed in the southern regions of India, mostly under the Pallava and the Chola rulers. Some distinct features of this style are :-</p><ul><li>The main temple complex is surrounded by <em>high boundary walls</em> on all four sides.</li><li>And the main entrance, features another tower known as <strong><em>Gopuram</em></strong>. There can be multiple of these, each at a different entrance to the temple.</li><li>The shape of the shikhara is completely different from the one you see in the Nagara style. It is a sort of a stepped pyramid rising up linearly as opposed to the curved rise in the other style. Also, this tower is called as <strong><em>Vimana</em></strong>.</li><li>You can also see a <em>water tank</em> inside the temple, where pilgrims take their ritual baths before going to offer their prayers.</li></ul><blockquote>Some of the best temples built in this style are (generally present in Tamil Nadu) <strong><em>Brihadeeswara temple</em></strong> at Thanjavur and <strong><em>Meenakshi temple</em></strong> in Madurai.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CC0iW2lhi0ZrwknP_sTubQ.jpeg" /><figcaption>Meenakshi temple in Madurai, Tamil Nadu</figcaption></figure><h3>References</h3><p>To do.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e4222a5b5943" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What I learned about Investing]]></title>
            <link>https://medium.com/@nikhilbadveli6/what-i-learned-about-investing-8fadd56a9b7?source=rss-9059cc379720------2</link>
            <guid isPermaLink="false">https://medium.com/p/8fadd56a9b7</guid>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[investing]]></category>
            <dc:creator><![CDATA[Nikhil Badveli]]></dc:creator>
            <pubDate>Sat, 23 Sep 2023 16:12:16 GMT</pubDate>
            <atom:updated>2023-09-24T10:08:04.689Z</atom:updated>
            <content:encoded><![CDATA[<p>Just like everyone who has ever dreamed of becoming a successful investor and be flithy rich by a certain age, I too have always wanted to achieve this.</p><h3>2017 — One share?</h3><p>I remember quite clearly the first time I have invested in the stock market. It was back in 2017, when there was only Zerodha as the popular stock broker. Nowadays, it is quite easy to open a demat account and buy a stock, probably all within the app itself.</p><p>But, back then, we had to post our KYC documents and then some agent had to physically come to our house to verify the identity. The whole thing took weeks. Even after all that, I was stupid to never seriously study about this whole investing business.</p><p>All I did was buy few hundred rupees worth of shares in a company and thought it will miraculously make me rich. I mean we did try to do some “research” by filtering stocks that have been big gainers in the last trading session or maybe filter them by industry or whatever.</p><p>There was no trading strategy, no reasonable expectation, had no idea of a stop loss or take profit. We were just a couple of stupid kids, imagining ourselves becoming billionaires.</p><p>I was naive enough to think that if I buy some shares of a penny stock worth only 0.5 Rs., which keeps fluctuating between 0.5 to 1 rupees and I assumed I can gain 100% profit if I could buy at low and sell at high and repeat the same thing over and over. Oh, how wrong I was!</p><p>That’s when I learned about the importance of <em>trading volume </em>and <em>liquidity </em>of a stock. And most importantly shares are not some abstract things. When you’re buying a share in the stock market, someone has to be selling at the price you’re trying to buy. And that’s what a trading platform essentially does, they match the buyers with sellers by looking at the orderbook.</p><p>I think I must’ve been embarassed by my stupidity. After that incident, I haven’t touched anything for at least one year.</p><h3>2018 — Mutual funds are better?</h3><p>Fast forward to September of the year 2018 and I must’ve come across mutual funds from someone or maybe it was because of the excessive marketing by Groww.in app.</p><p>Whatever the case might be, I have installed the app, signed up and did my KYC all within the app and it felt wonderful.</p><p>I wanted to start an SIP and invest Rs. 1000 a month since I was getting that stipend from my university. So, after a little bit of research somehow I came across the Axis Long term fund, which was an ELSS tax saving fund.</p><p>Without reading what it means, I just started the SIP, knowing that I can always stop it whenever I want. What I didn’t check though is the fact that you can take your money out only after 3 years.</p><p>I stopped that SIP after 7 months and took out the money after 3 years. It actually generated good returns, I think my money got doubled somehow.</p><h3>Almost bought Bitcoin @ 5000$</h3><p>Once again, this was 2017/2018 and there was a lot of buzz around bitcoin and cryptocurrency. I remember Elon Musk saying he’ll give away a bunch of bitcoins to anybody who does a set of tasks mentioned in the tweet. Thankfully, I did not get scammed for that, I didn’t have that much money anyway, at least in those days. Later, we found out that it was someone else impersonating Musk on Twitter.</p><p>Regardless of this little incident, we were thinking of buying Bitcoin from either Zebpay or Unocoin. We didn’t go ahead with that plan because we observed that these platforms are just looting people and not giving Bitcoin worth their money. Instead they are just taking commission whenever someone buys/sells from their platform.</p><p>Although Binance had better pricing, it was not officially supported. Instead we were asked to send money via P2P, which we couldn’t trust with transferring to some stranger on the internet.</p><h3>Numerous other ways I tried</h3><p>Over the years, I can’t even begin to tell how many different apps/websites/opportunities I might have tried to realize my dream.</p><p>There was forex trading in 2020, to be more precise it was more of doing copy trading in OctaFX. Since, I was not familiar with this, I have put only 5000 rupees and I may have lost it all. I don’t remember it now.</p><p>And then, there was trading Binary options in 2020, which is nothing but pure gambling in my opinion. I mean, come on, how can you predict whether something will go up/down in the next 60 freakin’ seconds?</p><p>Sure, I had some good profits too. But, nothing from these quick ways of becoming rich. It was only in 2021, when I may have calmed my horses down and realized there’s only the slow way.</p><p>Even then, it was because I was just following my brother’s suggestions on which stock to buy and when to sell. And he was following some influencers on the internet.</p><p>Cryptocurrency futures, or more binary options stuff on Deriv.com, or something as ridiculous as buying a script off the internet, that when you run will be as good as printing you free money. Ridiculous stuff!</p><p>I had even tried to mine Bitcoin or any other cryptocurrency back in December 2021. I had just learned about some Machine learning and I was trying to predict the nonce for the next Bitcoin block. It’s a random number for God’s sake. As if there is any pattern in the 700,000 blocks that were mined so far.</p><h3>2022 — the year I gave my hope up for good</h3><p>Well, even after that setback I hadn’t completely given up on this dream. After a few months into my course in AI &amp; its applications, I got this idea of building an automated trading bot, that uses machine learning to predict the price of a crypto and then executes trade based on that.</p><p>God knows how many different things I would’ve tried building that. But, I never knew there was a fundamental flaw in my thinking or maybe with my code or calculations or whatever.</p><p>It doesn’t matter what fancy algorithm I find to make the predictions. In the long run, it can never beat the index or the people who just hold onto the securities. It’s only an illusion, a false hope.</p><p>Because, imagine for a second of having such a bot for yourself. It’s like having a duck shitting Golden eggs everyday for you. As if life is that easy.</p><h3>2023 — path ahead</h3><p>And now I’m in 2023. During the start of this year, I came across these YouTube videos by Ankur Warikoo about things like “How much money you need to retire now?” and more importantly, I liked the excel sheet he shared for doing the calculations yourself.</p><p>After putting down my numbers, it suddenly felt like, I don’t need to struggle this hard to get what I want out of my life. Everything made sense.</p><p>And now I have this crazy plan of retiring in the next 10–12 years by building a sort of retirement fund.</p><p>That’s my biggest long-term goal for now.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8fadd56a9b7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What I learned about Cancer]]></title>
            <link>https://medium.com/@nikhilbadveli6/what-i-learned-about-cancer-1f75aa9eb733?source=rss-9059cc379720------2</link>
            <guid isPermaLink="false">https://medium.com/p/1f75aa9eb733</guid>
            <category><![CDATA[cancer]]></category>
            <dc:creator><![CDATA[Nikhil Badveli]]></dc:creator>
            <pubDate>Tue, 25 Jul 2023 21:12:32 GMT</pubDate>
            <atom:updated>2023-09-23T14:43:17.313Z</atom:updated>
            <content:encoded><![CDATA[<p>After one of my relatives found out that he got Colo Rectal Cancer, I realized my knowledge of the matter is woefully inadequate.</p><p>And the book “Emperor of all Maladies by Siddarth Mukharjee” had been lying in my Goodreads want-to-read shelf for way too long.</p><p>So, naturally it prompted me to read. I was totally hooked right from the beginning. There’s something in the way he explained things.</p><p>I’m hoping this article will be like a notes to self in the future.</p><p>And so, here are things I learnt:-</p><h3>What is Cancer?</h3><p>Is it a single disease? If so, then why are there so many different manifestations of it, with universally familiar names such as Lung and Breast Cancer to obscure names like Acute Lymphoblastic Leukemia?</p><p>The reason for so many variations will become clear, if you understand that Cancer at its core, is unlike any other disease, caused by <em>uncontrolled</em> cell division.</p><p>Now, for those who don’t know what cell division is, or may have forgotten about it. Here’s a quick explainer. It is the process that is quite central to life itself. It is how every cell in your body is created from and will create more cells.</p><h3>Okay, so what causes Cancer?</h3><p>The technical term for this is called <em>Carcinogenesis, </em>the process by which Cancer is formed.</p><p>I was surprised to learn that doctors, scientists and everyone in the medical world did not put their efforts into understanding this, but instead put much more emphasis on finding a so called universal cure for Cancer.</p><p>As such, there’s no clear explanation as to what exactly causes Cancer. Although, there is a very strong statistical correlation between Smoking and Lung Cancer, it does not guarantee that you as an individual will never get it if you don’t smoke.</p><p>This is because there might be some other unknown or poorly understood causes of it and it is beyond Science’s ability to find every possible Carcinogen.</p><p>According to Statistics, some very disturbing numbers I might add at that, it looks like 1 in every 2 people living on our planet, will be diagnosed with Cancer in their lifetime. Let that sink in. It is very likely that I might get it sometime in the future.</p><p>Other than avoiding Smoking, there’s not much you can do differently in your life. Maybe if you’re a woman whose mother had breast cancer, that too a very specific type of it, then there’s a chance of diagnosing it early and saving your life.</p><h3>Treatment Options</h3><p>Now, this entirely depends on the type of cancer you got. But, broadly speaking they fall into four categories.</p><ol><li>Radiation Therapy</li><li>Chemotherapy</li><li>Surgery</li><li>Immunotherapy</li></ol><p>Each of these treatment options couldn’t be any more different.</p><p>Radiation Therapy is nothing but bombarding the tumours with high energy radio waves (same as Electromagnetic waves, more frequency = more energy) or maybe actual radiation emitted from radioactive materials. I do not quite recall right now.</p><p>The problem with this therapy, as you might have guessed is, Radiation also causes cancer and so it is a sort of double-edged sword so to speak. Hence, it is entirely possible that if you’re treated with this option, your cancer might just relapse in a few years.</p><p>Chemotherapy, probably the most misunderstood of all and yet the most common approach to combating cancer. While radiation can kill any cell, it is not the same with chemicals.</p><p>They are designed after painstakingly identifying a powerful combination of drugs that can kill the cancer cells. But, these drugs almost never specifically target the cancer cells, instead they just kill every damn cell in your body.</p><p>It is like experiencing artificially induced death and that is why patients need intense care during chemotherapy, as they might experience a number of unintended symptoms like hairfall, loss of appetite etc.,</p><p>Moving on to Surgery, well, there’s nothing much to say here than that in most of the cases, this treatment option is pretty useless, unless you’re pretty lucky to find out about your cancer early on.</p><p><em>Radical Mastectomy </em>was a surgical technique that was practiced until quite recently to treat breast cancer. I was baffled to learn that, women go through such a horrible procedure and get their bodies disfigured almost unrecognisably.</p><p>Finally, the last and the newest entrant into the battle arsenal for fighting cancer is the Immunotherapy. The premise is quite simple, make your immune system kill the cancer cells by tagging/marking them somehow.</p><p>This is not always easy and works well with only certain cancer types. It is still very much part of ongoing research, just like how all the treatment options are evolving over time.</p><p>There’s always breakthrough therapies like <a href="https://www.youtube.com/watch?v=FEA6BQARqE8">this one using poliovirus</a> (<em>oncolytic virotherapy</em>) to kill cancer cells in brain and most of other solid tumour cancers.</p><h3>Common Cancers</h3><p>There are some cancers that everyone must’ve heard in their life, either because someone they know got diagnosed or maybe through news.</p><p>Here are some:- <em>Lung cancer, Breast cancer, Colorectal cancer, Prostate cancer, Cervical cancer, Liver cancer, Blood cancer (Leukaemia), Brain cancer (GlioBlastoma)</em></p><h3>References</h3><p><a href="https://www.amazon.co.uk/Emperor-All-Maladies-Biography-Cancer/dp/0007250924">The Emperor of All Maladies: A Biography of Cancer</a></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FUSmu0scNcSs%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DUSmu0scNcSs&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FUSmu0scNcSs%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/37434dfd05e8b3878eb554b9fbd5dab8/href">https://medium.com/media/37434dfd05e8b3878eb554b9fbd5dab8/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f75aa9eb733" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Can we summarize a video with AI?]]></title>
            <link>https://medium.com/@nikhilbadveli6/can-we-summarize-a-video-with-ai-5ae1b172bf2d?source=rss-9059cc379720------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ae1b172bf2d</guid>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[image-captioning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[videos]]></category>
            <dc:creator><![CDATA[Nikhil Badveli]]></dc:creator>
            <pubDate>Mon, 31 Oct 2022 17:03:48 GMT</pubDate>
            <atom:updated>2022-10-31T17:09:53.312Z</atom:updated>
            <content:encoded><![CDATA[<p>The title aptly tells the question I asked myself before I started exploring the idea further. I figured there’s already AI models that can generate captions for a given image with decent performance.</p><p>So, I quickly thought of a system that can accomplish this. Let me outline the three major components of this system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jITXjSEkJIDtqI8kBDXp5A.png" /><figcaption>Sketch generated by DALLE-2</figcaption></figure><blockquote><strong>an AI model that can summarize a video based on its frames, shown as a quick sketch on a paper</strong> — this prompt was used to generate the above image.</blockquote><h3><strong>Frame Extractor</strong></h3><p>The first component of this system is the Frame Extractor which as the name implies retrieves the frames of a given video. I used OpenCV’s python package to implement this functionality.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/57709339041e56e418a0e44676e45aea/href">https://medium.com/media/57709339041e56e418a0e44676e45aea/href</a></iframe><p>Since most videos are <strong>30 fps</strong> which means that if you extract all the frames in a video of just 1 min length, you’ll still end up with <em>30 * 60 = 1800 images</em>.</p><p>That is way too many images to process, plus there’s probably little change between any two consecutive frames. So, I thought of skipping frames and selecting only one frame for every ’n’ frames where ’n’ is the parameter of the function <em>extract_frames</em>.</p><p>I also gave one additional parameter <em>max_num_frames </em>in case there’s a need for limiting the total number of frames to be extracted.</p><h3><strong>Image Captioner</strong></h3><p>Now that the frames are available for us to process further and generate a summary, let’s build our next component — Image Captioner.</p><p>I used Huggingface for this part and selected a model I feel that’s reasonably well in generating captions and also reasonably fast since we would need to generate a lot of times.</p><p>In case you don’t have Huggingface transformers already installed, it’s very easy to do so with pip.</p><pre>pip install transformers</pre><p>I went ahead with <a href="https://huggingface.co/nlpconnect/vit-gpt2-image-captioning">ViT — GPT2 model</a> trained in Flax using the CoCo-en-ckpts dataset. If you’ve previously worked with Huggingface transformers python package, then this step is relatively straightforward.</p><p>Anyways, here’s the code for the second component in my Video Summarizer system.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/578fd3448a6a76bc14a95ea7779acd0d/href">https://medium.com/media/578fd3448a6a76bc14a95ea7779acd0d/href</a></iframe><p>Huggingface has this wonderful function called <em>AutoTokenizer </em>that can automatically figure out which tokenizer is used based on the model string given.</p><p>Briefly speaking, the way this model generates captions is by first using ViT (Vision Transformer) as a feature extractor on the given image and then use this latent space vectors as input to a decoder that will spit the tokens which are further decoded into output text.</p><p>There are two parameters available here — <em>max_length </em>and<em> num_beams. </em>The first one is to limit the size of the output captions and the second one is used with the <a href="https://towardsdatascience.com/an-intuitive-explanation-of-beam-search-9b1d744e7a0f">beam search strategy</a> whenever any text generation is done.</p><h3><strong>Text Summarizer</strong></h3><p>Finally, the last component is a Text Summarizer model. You might wonder why we need this additional component. Well, I didn’t think of adding it either at first.</p><p>But, the output of the image captioner for the frames still tended to be repetitive and kind of long to call it as a meaningful summary of the video.</p><p>So, once again I turned to Huggingface for help. This time I chose the model based on the qualitative summary of a long paragraph text, generated by these models. I went ahead with <a href="https://huggingface.co/facebook/bart-large-cnn"><em>facebook/bart-large-cnn</em></a><em>.</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/007dffba6944ab72ebc844bf8705f17b/href">https://medium.com/media/007dffba6944ab72ebc844bf8705f17b/href</a></iframe><p>This code looks unbelievably short, but that’s the beauty of using Huggingface transformers. You just need to build a pipeline of what you want to achieve and then select a model and finally adjust the parameters according to your needs.</p><p>Now, it’s time to put together all the three components described above.</p><h3>Combined system</h3><p>Let me directly show you the code of the combined system, I am sure it is easier to understand this way.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/21b22fa4515cafb6c6b620a10e145b54/href">https://medium.com/media/21b22fa4515cafb6c6b620a10e145b54/href</a></iframe><p>All I did in the above code is importing the three components and passing the outputs of one component as inputs to the other and finally we are ended up with the summary of the given video.</p><h3>Example video</h3><p>Now, let’s see how well our system performs with a small YouTube shorts video — by Karl Rock (often mistaken to be PewDiePie in India).</p><p><a href="https://www.youtube.com/shorts/BDLJENRaV7k">Scam Shops in Pakistan &amp; India - Don&#39;t Be Fooled!</a></p><blockquote>A person is cooking some food on a pan. Two men are holding up donuts in front of them. A man is standing next to a man on a subway. A street scene is filled with graffiti.</blockquote><p>Not bad at all… while this summary doesn’t capture the essence of the video, we’ve to understand the limitation of the information the model had.</p><p>All it saw was a bunch of images, we didn’t consider the fact they form a time-series of kind. And we didn’t consider the audio part of the video which has much more information of what the subject is talking in the video.</p><h3>Deploying the system</h3><p>I tried deploying this system on Streamlit but I couldn’t make it work because it is failing to download the huggingface models needed. Probably due to the limitations of the free account.</p><p>Here’s my deployed app on Streamlit although it doesn’t work.</p><p><a href="https://nikhilbadveli-videosummarizer-streamlit-app-5tpfbv.streamlitapp.com/">https://nikhilbadveli-videosummarizer-streamlit-app-5tpfbv.streamlitapp.com/</a></p><p>I would still like to share the code I’ve used to do this deployment. It’s in my Github repo for this project.</p><p><a href="https://github.com/NIkhilbadveli/VideoSummarizer/blob/master/streamlit_app.py">VideoSummarizer/streamlit_app.py at master · NIkhilbadveli/VideoSummarizer</a></p><p>You can find the rest of the code as well in this repo. I will add proper documentation to the repo in some time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ae1b172bf2d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to extract social media content for free?]]></title>
            <link>https://medium.com/@nikhilbadveli6/how-to-extract-social-media-content-for-free-6e8edefbea7f?source=rss-9059cc379720------2</link>
            <guid isPermaLink="false">https://medium.com/p/6e8edefbea7f</guid>
            <category><![CDATA[scraping]]></category>
            <category><![CDATA[instagram]]></category>
            <category><![CDATA[social-media]]></category>
            <category><![CDATA[tweepy]]></category>
            <category><![CDATA[facebook]]></category>
            <dc:creator><![CDATA[Nikhil Badveli]]></dc:creator>
            <pubDate>Fri, 05 Aug 2022 14:35:26 GMT</pubDate>
            <atom:updated>2022-08-05T14:40:17.755Z</atom:updated>
            <content:encoded><![CDATA[<p>Who doesn’t use social media these days? In fact, I bet that the ones who never used any form of these social networks such as Facebook, Instagram, Twitter, Tiktok etc., is the <em>minority.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WzXmh_dL4tvO74Gi" /><figcaption>Photo by <a href="https://unsplash.com/@unarchive?utm_source=medium&amp;utm_medium=referral">Jeremy Bezanger</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Among the data science community, the process of extracting information from the internet in general is termed as <em>Data Mining. </em>And so what we’re trying to build is a Social Media Data Miner.</p><p>As part of this post, we’re going to build code to fetch data from 3 platforms — Facebook, Instagram and Twitter. Despite TikTok being more popular these days, I didn’t choose to do it because it’s mostly video content.</p><h3>FacebookScraper</h3><p>As everyone knows about how evil FB is (just kidding), they did not provide any official API to retrieve content from its platforms. Hence, I was forced to look for alternatives from the reliable python community.</p><p>Luckily I found this library — <a href="https://github.com/kevinzg/facebook-scraper">FacebookScraper</a> on Github. The way these libraries work is by <em>scraping </em>the official facebook website, usually known as Web Scraping.</p><p>I could’ve used tools like BeautifulSoup and build a web scraper myself but I didn’t want to reinvent the wheel, so I chose not to. Anyways, since this code is from a third party library, beware of its limitations.</p><p>At any point of time they could stop maintaining the repo or simply FB might choose to update their website rendering this library useless. So, there’s no <strong><em>Future Compatability</em></strong> so to say.</p><p>Anyways, here’s the code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5f8bc3a2c4887c9ca571def3ff8cf473/href">https://medium.com/media/5f8bc3a2c4887c9ca571def3ff8cf473/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b573e81f542ab7eda417fd91ed9191fb/href">https://medium.com/media/b573e81f542ab7eda417fd91ed9191fb/href</a></iframe><p>Don’t forget to install the package using pip, just like how you install any other python package. Feel free to check the package’s page for more options for your needs.</p><h3>InstaLoader</h3><p>Moving on to Instagram now. While Instagram is not exactly a text-first platform like Twitter is, we could still extract some valuable information from post captions, stats and comments.</p><p>Once again as with Facebook, there’s no official API and so I found an alternative that’s good enough, actually with awesome documentation I would say. It is called as <a href="https://instaloader.github.io/as-module.html">InstaLoader</a>.</p><p>I tried to follow similar code structure as Facebook and here it is.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/46cd02cf8bb059a4839a835ee0d8265d/href">https://medium.com/media/46cd02cf8bb059a4839a835ee0d8265d/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d1b4621e80a7e419578c154b45a94fe9/href">https://medium.com/media/d1b4621e80a7e419578c154b45a94fe9/href</a></iframe><p>Please remember these things when you’re working on either of these two packages I’ve mentioned above.</p><p>First, never write code such that you’re logging into FB or Insta multiple times. Simply put, never put the login code inside a loop. You might get <strong><em>banned</em></strong><em> </em>for this.</p><p>And, always remember to limit the number of posts/comments you’re trying to scrape if possible. Since these packages work by scraping FB or Insta, they’ve to repeatedly hit the same page if they want to extract more data which will trigger their <em>anti-scraping</em> tech.</p><h3>Tweepy</h3><p>Finally, coming to Twitter which is the only platform here that provided with an official API and generously enough for us, we can request for up to <strong>500K</strong> tweets per month for free.</p><p>Although you can directly hit Twitter API once you <a href="https://developer.twitter.com/en/docs/twitter-api/getting-started/getting-access-to-the-twitter-api">register </a>and get their API key, I’m using a python package called <a href="https://docs.tweepy.org/en/stable/getting_started.html">Tweepy </a>that makes my life easier with functions according to my requirement.</p><p>Here’s the code I wrote for fetching from Twitter.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/856dffbe6adaf0ca8eb56ea3059db097/href">https://medium.com/media/856dffbe6adaf0ca8eb56ea3059db097/href</a></iframe><p>I’m sure if you read about Twitter API you will find that it has almost everything that you might need in case if you have some special requirement.</p><p>That’s all folks. I did not cover <strong>YouTube </strong>and <strong>TikTok</strong>, the other two major social media platforms. Maybe I will in a follow-up post but until then, I hope this is enough for everyone else to get started on whatever they’re working on.</p><p>Here’s the full repository for the complete code. I’m calling it SocialMediaDataMiner.</p><p><a href="https://github.com/NIkhilbadveli/SocialMediaDataMiner">GitHub - NIkhilbadveli/SocialMediaDataMiner: Using different libraries for various social media platforms such as Twitter, Facebook, Instagram, this project has the functionality to extract data for a given username.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6e8edefbea7f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Did DeepMind crack the surface of Artificial General Intelligence (AGI)?]]></title>
            <link>https://medium.com/@nikhilbadveli6/did-deepmind-crack-the-surface-of-artificial-general-intelligence-agi-5cb878611c28?source=rss-9059cc379720------2</link>
            <guid isPermaLink="false">https://medium.com/p/5cb878611c28</guid>
            <category><![CDATA[agi]]></category>
            <category><![CDATA[deepmind]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[multimodal-learning]]></category>
            <dc:creator><![CDATA[Nikhil Badveli]]></dc:creator>
            <pubDate>Mon, 16 May 2022 12:13:04 GMT</pubDate>
            <atom:updated>2022-05-16T12:13:04.109Z</atom:updated>
            <content:encoded><![CDATA[<p>DeepMind, yet again published a crazy paper and this time it involves a robot in a virtual environment behaving according to human instructions expressed in natural language through a chat like interface. See the below video for yourself and judge how good it is!</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FZFgRhviF7mY%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DZFgRhviF7mY&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FZFgRhviF7mY%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ef754b659979610e469c16ee88dbe66/href">https://medium.com/media/7ef754b659979610e469c16ee88dbe66/href</a></iframe><h3>AGI… the ultimate dream of every AI researcher</h3><p>Artificial General Intelligence (AGI), the holy grail of the modern AI research is when we reach a state where the AI is capable of learning any previously unseen task without requiring large amount of data and time for training, and eventually become better at it than humans.</p><blockquote>Essentially, a very powerful generalized learning machine, hopefully without its own schemes of taking over the world. <em>Just kidding :)</em></blockquote><p>After watching the above demo, it really felt like DeepMind had created a very primitive version of an AGI. Maybe a couple of papers down the line, we could see a more capable and advanced version of this agent.</p><h3>So, what exactly is there in this paper?</h3><p>The researchers have created a virtual environment of a model <strong>playhouse </strong>with the usual rooms you see in a typical house, like a bedroom, bathroom, hall etc., And they have randomized the layout of this playhouse for every time they test the capability of the agent.</p><p>Speaking of the agent itself, they’ve named it <strong>Multimodal Interactive Agent</strong> (MIA)… let’s call the agent Mia from now on. Mia is capable of interacting with the users/humans through two modes — Visual and Textual. Mia is so amazing at recognizing objects in its field of vision that it can answer questions about what it sees presently.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*BWh6IpTSZYUM8Hq396FX3g.jpeg" /><figcaption>Example tasks from the paper. Image taken from DeepMind’s blog.</figcaption></figure><p>See the third task from the above image and you’ll be shocked how good it is at answering questions like these. Some of the other tasks, the researchers asked it to do are — tidying up the room, picking up objects, placing them at a different place etc.,</p><p>What’s more amazing is that it takes just a few hours for Mia to learn about a previously unseen object or an unseen command. Essentially, in these few hours Mia is able to learn to recognize the new object, manipulate it according to the task and answer any questions about it. Such rich learning behavior!</p><h3>What can we expect from this in the future?</h3><p>Whatever the pessimists might say about bringing AGI into life, I’m still very optimistic and looking forward to see this possibility come to fruition. Obviously, there’s still a ton of work to be done before we get to see that day.</p><p>I’m not saying that this paper is the foundation for achieving AGI. Maybe they’re showing us hand picked examples of tasks where Mia performed well… maybe there are still some drawbacks the researchers haven’t thoroughly tested. But, this is a continuous process and we have to appreciate the amazing progress we have made already.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gKF9j7-wJGdybPl6uCea3A.jpeg" /><figcaption>Red Dead Redemption 2 — a massive Open World game by Rockstar Games.</figcaption></figure><p>One possibility I always had in my mind was teaching an agent like Mia in an Open World Game setting. I know it is computationally a lot more intensive than learning inside a toy playhouse, but maybe in a couple of years this might be possible. Who knows how much more Mia could do then.</p><p>As the Prof. Károly Zsolnai-Fehér from <a href="https://www.youtube.com/c/K%C3%A1rolyZsolnai">Two Minute Papers</a> say, “<strong><em>what a time to be alive!</em></strong>”.</p><h3>References</h3><p><a href="https://arxiv.org/abs/2112.03763">Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learning</a></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FL9kA8nSJdYw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DL9kA8nSJdYw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FL9kA8nSJdYw%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e3ad22ac95821c66a99e61e19fc8d1d/href">https://medium.com/media/6e3ad22ac95821c66a99e61e19fc8d1d/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5cb878611c28" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Chatbot — 1, Natural Language Processing (NLP)]]></title>
            <link>https://medium.com/@nikhilbadveli6/chatbot-1-natural-language-processing-nlp-36d510477ef1?source=rss-9059cc379720------2</link>
            <guid isPermaLink="false">https://medium.com/p/36d510477ef1</guid>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[retrieval]]></category>
            <category><![CDATA[intent]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Nikhil Badveli]]></dc:creator>
            <pubDate>Wed, 11 May 2022 12:27:02 GMT</pubDate>
            <atom:updated>2022-05-11T12:27:02.981Z</atom:updated>
            <content:encoded><![CDATA[<h3>Chatbot — 1, Natural Language Processing (NLP)</h3><p>This is the first post in the chatbot series, which itself is part of a bigger series on NLP. I’m not going to explain the basics of NLP, a quick google search will point you to an infinite amount of resources on that. Like the below article, for instance.</p><p><a href="https://towardsdatascience.com/a-gentle-introduction-to-natural-language-processing-e716ed3c0863">A Gentle Introduction to Natural Language Processing</a></p><h3>Introduction</h3><p>Chatbots are an interesting application in the NLP domain. I mean, who doesn’t want to feel like they’re talking to an intelligent AI? But, did you know that there are actually two very different type of chatbots.</p><ol><li>One is called a <strong><em>retrieval-based chatbot</em> </strong>that can only output from a fixed set of responses. Essentially, we try to classify the user input into a set of pre-defined intent tags (ex: greeting, thanks etc.,) and choose a response at random, based on the intent we think the user has expressed.</li><li>The other is called a <strong><em>generative chatbot </em></strong>that uses a seq2seq model or an encoder-decoder architecture in its most general form. First the user input<br>is encoded and then the decoder takes this encoded output, processes it and produces the output response of the chatbot.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wAR3r0zwrYRQ6VMNJCSltg.png" /><figcaption>A visual overview of a retreival based chatbot. Image taken from Drift blog.</figcaption></figure><blockquote>Surprisingly, it looks like the popular digital assistants like Google Assistant, Siri etc., uses the first approach. And the second type is apparently still under research phase. I guess there’s more control in the first type, at least if you consider business needs.</blockquote><p>Anyways, for this post we’re going to implement the first type of the chatbot, which will support the customers in answering their queries related to food delivery.</p><h3>Training the intents data</h3><p>We need to build a multi-class classifier that can predict the intent tag given an user input. We could choose any classic ML classifiers like SVM, kNN, Naive-bayes. But, we will use a simple neural network based model with 2 hidden layers.</p><p>Let’s look at how the training data (<em>.json file</em>) looks like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/45d972dfd4d44f50a973fe60c5a0aed9/href">https://medium.com/media/45d972dfd4d44f50a973fe60c5a0aed9/href</a></iframe><p>As you can see, each item in the list has a <em>tag</em>, a list of <em>patterns, </em>and a list of <em>responses </em>to choose from. For the classifier our input is the patterns data and the output classes are the tags.</p><h4>Preprocessing the data</h4><p>We can’t directly feed this to the neural network since this is text data. We need to preprocess this data, which involves removing stop words, lemmatizing etc., The below code uses the popular NLTK package for cleaning the data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1359f35bc55f0062f8a0d208681036a0/href">https://medium.com/media/1359f35bc55f0062f8a0d208681036a0/href</a></iframe><p>Now, let’s use a <em>Tf-Idf vectorizer </em>to transform this cleaned data into a set of vectors and a one hot encoder for the output class labels.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/70269e4372a28bf766ca9d0ad428f225/href">https://medium.com/media/70269e4372a28bf766ca9d0ad428f225/href</a></iframe><h4>Build and fit the model</h4><p>We will use the <em>keras </em>library that uses tensorflow backend for building our neural network model. We will add 2 hidden layers with ReLU activation and an output layer with a Softmax activation and a couple of dropout layers for regularization.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/66863387434eca82345063060a94dc6e/href">https://medium.com/media/66863387434eca82345063060a94dc6e/href</a></iframe><p>Since our data is very small, I’ve used a <em>batch_size</em> of 1, choose this depending on your data’s size. Similarly, for other hyperparameters.</p><p>Hooray! Now we have a classifier that can tell the intent of the user (hopefully). The next step is to use this model and put together everything else to generate a response.</p><h3>Chatting loop</h3><p>We will get the user input in an infinite loop fashion, generating response for every query the user has. Once we get the input text from the user, we have to preprocess it in exactly the same way as before we did while training.</p><p>And then give this to the model, which generates probabilities for all the output intent tags, out of which we select the one with the highest probability.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1387f9f83ff9422e56b1d2f6fcefe135/href">https://medium.com/media/1387f9f83ff9422e56b1d2f6fcefe135/href</a></iframe><p>Note that we used a helper function <em>get_intent </em>that gives the corresponding intent to the predicted tag and we are using this intent to randomly select the response. In the case of a <em>goodbye</em> intent, we are breaking out of the chat loop.</p><p>That’s all folks, you’ve successfully built a retrieval based chatbot. I’m sure you might see the bot giving repeated responses which is pretty annoying I know. Wait for me to post about the <strong>generative chatbot</strong> and you’ll be amused by its responses.</p><p>If you’re interested in getting started with <strong>Reinforcement learning</strong>, read my articles on the same.</p><ul><li><a href="https://medium.com/@nikhilbadveli6/reinforcement-learning-1-79cdec3599f9">Reinforcement Learning — 1</a></li><li><a href="https://medium.com/@nikhilbadveli6/reinforcement-learning-2-8e525986b3df">Reinforcement Learning — 2</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36d510477ef1" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>