<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Anindya Chowdhury on Medium]]></title>
        <description><![CDATA[Stories by Anindya Chowdhury on Medium]]></description>
        <link>https://medium.com/@41x3n?source=rss-5b49ff83b7a4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*hOOBKvGSx_UZd6ck9RjVCw.jpeg</url>
            <title>Stories by Anindya Chowdhury on Medium</title>
            <link>https://medium.com/@41x3n?source=rss-5b49ff83b7a4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 01:46:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@41x3n/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[My Retriever Was an LLM. That Was the Bug.]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@41x3n/my-retriever-was-an-llm-that-was-the-bug-31c19bad4b8b?source=rss-5b49ff83b7a4------2"><img src="https://cdn-images-1.medium.com/max/1200/1*We35HbuUpvVS3QtQw_9u6A.jpeg" width="1200"></a></p><p class="medium-feed-snippet">A couple of weeks ago I wrote about building Samvad, a pipeline that answers questions about Indian news coverage and shows its sources. I&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@41x3n/my-retriever-was-an-llm-that-was-the-bug-31c19bad4b8b?source=rss-5b49ff83b7a4------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@41x3n/my-retriever-was-an-llm-that-was-the-bug-31c19bad4b8b?source=rss-5b49ff83b7a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/31c19bad4b8b</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[india]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Anindya Chowdhury]]></dc:creator>
            <pubDate>Mon, 13 Jul 2026 15:19:33 GMT</pubDate>
            <atom:updated>2026-07-13T15:19:33.784Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[I Built a News Engine That Cites Every Claim]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@41x3n/i-built-a-news-engine-that-cites-every-claim-704ea65f5d7a?source=rss-5b49ff83b7a4------2"><img src="https://cdn-images-1.medium.com/max/1600/1*nWSAIZy6hUfdIRehvjSfqQ.jpeg" width="1600"></a></p><p class="medium-feed-snippet">I kept running into the same small frustration. I&#x2019;d ask an LLM about a news story and get back something confident and fluent. ChatGPT and&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@41x3n/i-built-a-news-engine-that-cites-every-claim-704ea65f5d7a?source=rss-5b49ff83b7a4------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@41x3n/i-built-a-news-engine-that-cites-every-claim-704ea65f5d7a?source=rss-5b49ff83b7a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/704ea65f5d7a</guid>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[india]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Anindya Chowdhury]]></dc:creator>
            <pubDate>Mon, 13 Jul 2026 15:11:02 GMT</pubDate>
            <atom:updated>2026-07-13T15:24:46.359Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[A quick guide to creating Azure Function with Golang]]></title>
            <link>https://blog.stackademic.com/a-quick-guide-to-creating-azure-function-with-golang-4c22b4f90e68?source=rss-5b49ff83b7a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c22b4f90e68</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Anindya Chowdhury]]></dc:creator>
            <pubDate>Mon, 02 Oct 2023 08:14:28 GMT</pubDate>
            <atom:updated>2023-10-03T05:40:38.320Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Quick Guide to Creating Azure Function with Golang</h3><figure><img alt="AI generated image showing a cute gopher in a stormy day" src="https://cdn-images-1.medium.com/max/1024/1*QT3i1lAxkpwbJx24qVoK5w.jpeg" /></figure><p>Azure Functions, a serverless computing service by Microsoft Azure, supports various programming languages like C#, JavaScript, Python, and Java. It seamlessly integrates with Azure services, offering event-driven functions with automatic scaling based on demand. This guide simplifies the process of deploying your Golang backend service to Azure Function as a custom runtime.</p><p><strong>Prerequisites:</strong></p><ul><li>Basic knowledge of creating an HTTP server using Golang.</li><li>Azure setup on your desktop.</li></ul><p><strong>Installation:</strong></p><ol><li><a href="https://go.dev/doc/install">Download and install — The Go Programming Language</a></li><li><a href="https://learn.microsoft.com/en-us/cli/azure/install-azure-cli">How to install the Azure CLI | Microsoft Learn</a></li><li><a href="https://gin-gonic.com/docs/quickstart/">Quickstart | Gin Web Framework (gin-gonic.com)</a></li></ol><h3>Step 1: Create a Golang Server</h3><p>Create a server.go file and add the following code:</p><pre>package main<br><br>import (<br> &quot;net/http&quot;<br> &quot;os&quot;<br><br> &quot;github.com/gin-gonic/gin&quot;<br>)<br><br>func get_port() string {<br> port := &quot;:8080&quot;<br> if val, ok := os.LookupEnv(&quot;FUNCTIONS_CUSTOMHANDLER_PORT&quot;); ok {<br>  port = &quot;:&quot; + val<br> }<br> return port<br>}<br><br>func main() {<br> router := gin.Default()<br><br> router.GET(&quot;/api/hello&quot;, func(c *gin.Context) {<br>  c.JSON(http.StatusOK, gin.H{<br>   &quot;message&quot;: &quot;Hello, World!&quot;,<br>  })<br> })<br><br> port := get_port()<br> router.Run(port)<br>}</pre><p>Ensure the getPort function is present in your code. The FUNCTIONS_CUSTOMHANDLER_PORT is essential for Azure Functions to function properly. I lost many hours trying to figure this out as just like any other developer I do not enjoy reading docs even though I should be.</p><p>Azure Functions automatically assigns a port value to FUNCTIONS_CUSTOMHANDLER_PORT when a custom handler is used. The platform dynamically assigns a port to ensure proper communication between the Azure Functions runtime and the custom handler, simplifying the configuration process for developers.</p><h3>Step 2: Set Up Golang Dependencies</h3><pre>go mod init &lt;service name&gt;<br>go mod tidy<br><br># To run it locally<br>go run server.go</pre><h3>Step 3: Configure Azure Function Files</h3><pre>func init --worker-runtime custom<br>func new -l Custom -t HttpTrigger -n hello -a anonymous </pre><ul><li>func init — worker-runtime custom: Initializes a new Azure Functions project with a custom worker runtime. It creates a host.json file for configuring the Azure Functions host.</li><li>func new -l Custom -t HttpTrigger: Creates a new function with a custom language worker and an HTTP trigger. This command generates necessary files, including function.json that defines the function&#39;s trigger and bindings.</li></ul><p>Modify host.json:</p><pre># Add this field inside &quot;customhandler<br>&quot;enableForwardingHttpRequest&quot;: true<br># Update the value of &quot;defaultExecutablePath&quot; to your build file path <br>&quot;defaultExecutablePath&quot;: &quot;server&quot;</pre><p>Modify hello/function.json:</p><pre># Add this field below &quot;methods&quot; inside &quot;bindings&quot;<br>&quot;route&quot;: &quot;hello&quot; # We are using &quot;hello&quot; as value since our endpoint is /hello</pre><h3>Step 4: Run Locally with Azure CLI</h3><pre> func start --verbose </pre><h3>Step 5: Create Function in Azure Console</h3><ul><li>Go to Azure Console and create a function with the specified settings.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JephBbqPiRSkurcwj_x14Q.png" /></figure><h3>Step 6: Deployment</h3><p>In your terminal, build your project:</p><pre>GOOS=linux GOARCH=amd64 go build server.go</pre><p>Publish your function to Azure:</p><pre>func azure functionapp publish &lt;your function name&gt;</pre><p>Make sure to use GOOS and GOARCH values while building, especially if you are on macOS, to avoid platform mismatch issues after deployment in the Linux environment of Azure Functions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E6wHm1gGvfVOFpLwLvMuqg.png" /></figure><p>Upon successful deployment, you will receive a confirmation message in your terminal and see the changes reflected inside the Azure console for the function. Congratulations, your Golang backend service is now deployed on Azure Functions!</p><p>You can take a look at the source file at <a href="https://github.com/41x3n/Medium-Golang-Azure">Github</a>.</p><h3>Stackademic</h3><p><em>Thank you for reading until the end. Before you go:</em></p><ul><li><em>Please consider </em><strong><em>clapping</em></strong><em> and </em><strong><em>following</em></strong><em> the writer! 👏</em></li><li><em>Follow us on </em><a href="https://twitter.com/stackademichq"><strong><em>Twitter(X)</em></strong></a><em>, </em><a href="https://www.linkedin.com/company/stackademic"><strong><em>LinkedIn</em></strong></a><em>, and </em><a href="https://www.youtube.com/c/stackademic"><strong><em>YouTube</em></strong></a><strong><em>.</em></strong></li><li><em>Visit </em><a href="http://stackademic.com/"><strong><em>Stackademic.com</em></strong></a><em> to find out more about how we are democratizing free programming education around the world.</em></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4c22b4f90e68" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/a-quick-guide-to-creating-azure-function-with-golang-4c22b4f90e68">A quick guide to creating Azure Function with Golang</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using a custom domain with Azure Static Web App]]></title>
            <link>https://medium.com/@41x3n/using-a-custom-domain-with-azure-static-web-app-a48f05980b21?source=rss-5b49ff83b7a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/a48f05980b21</guid>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[azure]]></category>
            <dc:creator><![CDATA[Anindya Chowdhury]]></dc:creator>
            <pubDate>Fri, 17 Feb 2023 08:08:39 GMT</pubDate>
            <atom:updated>2023-02-17T08:08:39.961Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="A random image of a server room." src="https://cdn-images-1.medium.com/max/1024/0*noHA1Rpy8NXHUaVL" /><figcaption>Photo by <a href="https://unsplash.com/es/@tvick?utm_source=medium&amp;utm_medium=referral">Taylor Vick</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote>Initialy published <a href="https://41x3n.hashnode.dev/using-a-custom-domain-with-azure-static-web-app">here</a></blockquote><p>Recently I finished my university degree and thought of creating a portfolio website for myself. I looked into the <a href="https://education.github.com/pack/offers?sort=popularity">GitHub Student Developer Pack</a>, and saw Azure giving free credits and also Namecheap giving 1 year of free registration of the .me TLD. In this short article, I am gonna show how you can easily use a domain bought from Namecheap with the Azure static web app.</p><p>I used <a href="https://marketplace.visualstudio.com/items?itemName=ms-vscode.vscode-node-azure-pack">Azure Pack extension for VS Code</a> to scaffold the project. I won’t be going much deeper into how to deploy a web app with Azure in this article as it is out of scope but it was really easy to set it up with the extensions as they took care of the CI/CD automatically (Maybe in another article if anyone is interested 🙂).</p><p>Okay, I am now assuming, you have already deployed your web app and you got a ***.azurestaticapps.net domain. Now you want to use the custom domain you brought from Namecheap (or any other seller). Let&#39;s click on the Custom domains tab on the sidebar in your Azure Web app settings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*dxkPQv3sqFCnUMxG" /></figure><p>Clicking on the<strong> + Add</strong> button will give you two options. Choose <strong>Custom domain on other DNS</strong>. <strong>Add custom domain</strong> pane will open up and there you put the domain you own.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/0*RwtdtVg9X36Ycbaj" /></figure><p>Considering your domain is valid and you click <strong>Next</strong>, it will ask you to generate a code to <strong>Validate domain ownership</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/0*WWjsDvuZG2NOCdCN" /></figure><p>It will take some time to generate the code, once it gets generated, go to <strong>Advance DNS</strong> settings of your domain inside Namecheap (or any other provider) and enter the information the way Azure has asked to.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/0*fKkWuzPchDjwx3Bx" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*Sj1B5txPgpVy-P7h" /></figure><p>It takes a good amount of time to get the domain validated so you should now go eat a sandwich or maybe exercise for a while and then come back to the screen.</p><p>Okay, I am back after 15 minutes. Azure tells me that my domain has been validated. Time to set up apex domain and www subdomain. What I mean by those terms is that now I need to tell Namecheap, where to point if someone visits <a href="https://anindyac.me/">anindyac.me</a> or <a href="https://www.anindyac.me/">www.anindyac.me</a>.</p><p>Fill up the <strong>Host Records</strong> as shown in the pictures with your ***.azurestaticapps.net domain inside the <strong>value </strong>option. Make sure you don&#39;t have http:// or https:// prefix in that azurestaticapps.net URL.</p><p>The <strong>ALIAS </strong>record lets you access your web app through your custom domain and the <strong>CNAME </strong>record will let people access it even if they use www subdomain before the domain name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*lO8Z3nP1tZkVMU45" /></figure><p>Now to make your www.yourdomain.com work alongside yourdomain.com, you need to go back to <strong>Custom domains</strong> settings inside Azure once again. Create a custom domain again like before. But this time add <strong>www </strong>to the domain name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/0*yokZY3Dao-UiUXOU" /></figure><p>Since we have already added the CNAME record in Namecheap like it is showing in the below screenshot, we can just click on <strong>Add </strong>now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/0*KZCVu8Zp8iGfG_Kl" /></figure><p>Now if everything went fine, the <strong>Custom domains</strong> page will show up like this after a few minutes and your domains will work. 😊</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*q9M3QqbHGWA6w6w7" /></figure><p>Let me know in the comments if you have any queries or if you have any feedbacks to give to me. Thank you for reading and have a good day or night!</p><p><strong>References -</strong></p><ul><li><a href="https://docs.microsoft.com/en-us/azure/static-web-apps/getting-started?tabs=vanilla-javascript">Azure Documentation on building a static web app.</a></li><li><a href="https://docs.microsoft.com/en-us/azure/static-web-apps/custom-domain">Azure Documentation on Custom Domain</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a48f05980b21" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scraping a website and downloading files with Node.js using Axios and Cheerio]]></title>
            <link>https://medium.com/@41x3n/scraping-a-website-and-downloading-files-with-node-js-using-axios-and-cheerio-7bbb003f6064?source=rss-5b49ff83b7a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/7bbb003f6064</guid>
            <category><![CDATA[axios]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[cheerios]]></category>
            <category><![CDATA[nodejs]]></category>
            <dc:creator><![CDATA[Anindya Chowdhury]]></dc:creator>
            <pubDate>Sun, 08 Dec 2019 06:32:20 GMT</pubDate>
            <atom:updated>2019-12-08T06:32:20.495Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gr6If5AmWNLQh2Psm3om8g.jpeg" /><figcaption>A scrapyard</figcaption></figure><p>Usually, when people mention web scraping, the first thing that comes into mind is Python. However, Node.js has various libraries that can perform web scraping. From basic Request method to more complex solutions like Puppeteer and Nightmare. But in this post, we will explore an alternative way using <a href="https://github.com/axios/axios">Axios </a>and <a href="https://github.com/cheeriojs/cheerio">Cheerio </a>library.</p><h3>Axios</h3><blockquote><em>Promise based HTTP client for the browser and node.js</em></blockquote><h3>Cheerio</h3><blockquote><em>Fast, flexible &amp; lean implementation of core jQuery designed specifically for the server.</em></blockquote><p>One of the most important thing in writing codes is reading the documentation of the tools you are gonna use. If you have already done that then I assume you have understood what are the purposes of the above-mentioned tools. Basically, Axios lets us make HTTP requests from the node.js kinda like how we use <a href="https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API">fetch API</a> and Cheerio lets us manipulating the response that we get from Axios (which basically contains the markup of a particular webpage).</p><h3>Our Target…</h3><p><a href="https://books.goalkicker.com/"><strong>books.goalkicker.com</strong></a><strong> </strong>is a website where you can find lots of ebooks containing newbie friendly (this can be debatable) materials for many programming languages. Our aim is to scrap through the webpages and download all the PDFs available for download. So create a folder for your project. Inside it create an app.js and setup package.json.</p><h3>Okay, let’s get into coding…</h3><pre>const axios = require(&#39;axios&#39;);<br>const cheerio = require(&#39;cheerio&#39;);<br>const fs = require(&#39;fs&#39;)</pre><p>We have already talked about the first two libaries above. From Node.js website -</p><blockquote><em>The fs module provides an API for interacting with the file system in a manner closely modeled around standard POSIX functions.</em></blockquote><pre>const url = &#39;https://books.goalkicker.com/&#39;<br>let linkList = []<br>let dlinkList = []</pre><p>Here we assigned the link to a constant variable and also declared two array for using later.</p><pre>(async () =&gt; {<br>  await getWebsiteLinks(url)<br>  await downloadLinks(linkList)<br>  await downloadFiles(dlinkList)<br>})()</pre><p>Throughout the script, we will be using <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function">Async/Await </a>as I prefer it over Callback Hell or bewildering Promise Chaining. However, Axios themselves use promise chaining in their documentation so you can use that too.</p><p>Here we wrote an asynchronous function which will get invoked immediately. This is <a href="https://2ality.com/2016/10/async-function-tips.html">a nice article</a> I found while writing this article where you can learn more about Async/Await.</p><p>Inside the function, we will be calling three more async functions with different purposes.</p><ul><li><strong>getWebsiteLinks(url)</strong> — This will scrap the links of all webpages which contain the ebooks and store them in an array called linklist.</li><li><strong>downloadLinks(linkList)</strong> — This will scrap download links of the files from each page and store them in an array called dlinklist.</li><li><strong>downloadFiles(dlinkList)</strong> — This will download the files using the links stored in dlinklist.</li></ul><pre>const getWebsiteLinks = async (url) =&gt; {<br>  try {<br>    const response = await axios.get(url)<br>    const $ = cheerio.load(response.data)<br>    $(&#39;div.bookContainer&#39;).each(function (i, elem) {  <br>      let link = $(elem).find(&#39;a&#39;).attr(&#39;href&#39;)<br>      linkList.push(url+link)<br>    });<br>  } catch (error) {<br>    console.error(error)<br>  }<br>}</pre><p>Using Axios we sent a get request to <a href="https://hashnode.com/util/redirect?url=http://books.goalkicker.com"><strong>books.goalkicker.com</strong></a> and store the response to a variable called <strong>response</strong>. If you console.log the response, you will see a lot of information, most of which we do not need. This is what basically our browser gets when we visit a link. We will be focusing the information stored under <strong>response.data</strong>. It contains the HTML structure of the page which we can see when we use <strong>Inspect Element</strong> tool in browsers. We now load the markup into a Cheerio variable to parse and manipulate it.</p><p>When we were inspecting elements of the page in the browser, we found that links of each book were inside divs with class name — <strong>“bookContainer”</strong>. So we use <strong>each </strong>method (you can call it forEach of jQuery) on all the divs having the same class name. Then inside each occurrence, we pick the <strong>a tag </strong>and store what the <strong>href </strong>attribute of that tag contains, into a variable called link. Then we push it into an array called <strong>linkList</strong>.</p><pre>const downloadLinks = async (linkList) =&gt; {<br>  for (const link of linkList) {<br>    const response = await axios.get(link)<br>    const $ = cheerio.load(response.data)<br>    let name = $(&#39;.download&#39;).attr(&quot;onclick&quot;)<br>    name = name.match(/location\.href\s*=\s*[&#39;&quot;]([^&#39;&quot;]*)[&#39;&quot;]/)<br>    let dlink = link + name[1]<br>    dlinkList.push({<br>      name: name[1],<br>      dlink: dlink<br>    })<br>  }<br>}</pre><p>Now that we have all the links of the pages, we travel to each of them using a for loop while making get requests using Axios and parse the response using Cheerio. Again using <strong>Inspect Element </strong>tool, we found out that the download link is inside a button tag but attached with some other stuff (for example — <em>“location.href=’DotNETFrameworkNotesForProfessionals.pdf”</em>). For that, we used regex <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions">(Regular Expressions)</a> to clean the string. Once we got the desired download link, we push it into an array called dlinkList.</p><pre>const downloadFiles = async (dlinkList) =&gt; {<br>  for (const link of dlinkList) {<br>    let name = link.name + &#39;.pdf&#39;<br>    let url = link.dlink<br>    let file = fs.createWriteStream(name)<br>    const response = await axios({<br>      url,<br>      method: &#39;GET&#39;,<br>      responseType: &#39;stream&#39;<br>    })<br>    response.data.pipe(file)<br>  }<br>}</pre><p>We are almost close to the end of our scraping journey. Now we need to download the files using the links we extracted above. This is a bit confusing so I would suggest you read <strong>fs</strong> documentation thoroughly. First, we create a file name with the proper extension. With the current setup, the files will be downloaded into the folder where app.js is stored. Using <strong>fs.createWriteStream()</strong>, we create a writable stream where we can pipe the data of the file we are gonna download. Using Axios, we send a request to download the file and with <strong>response.data.pipe(file)</strong> we save the file.</p><p>Once we run the whole script, within a few minutes, all the books the site have will get downloaded into your machine. You can see the full code in <a href="https://github.com/41x3n/GoalKicker-Books-Script">my GitHub repository</a>.</p><p>This article was first published in <a href="https://41x3n.hashnode.dev/scraping-a-website-and-downloading-files-with-nodejs-using-axios-and-cheerio-cjzxoamre00040ns1hv7xr85d">my Hashnode blog</a>.</p><p><strong>Photo by sergio souza on Unsplash</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7bbb003f6064" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>