<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Oleh Teslenko on Medium]]></title>
        <description><![CDATA[Stories by Oleh Teslenko on Medium]]></description>
        <link>https://medium.com/@teslenkooleg2017?source=rss-2e5f5b703d04------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*0HiEll_JZqjl4vDSVKtKXw.png</url>
            <title>Stories by Oleh Teslenko on Medium</title>
            <link>https://medium.com/@teslenkooleg2017?source=rss-2e5f5b703d04------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 23:39:58 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@teslenkooleg2017/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building Video Chat in 3 Days with LiveKit]]></title>
            <link>https://levelup.gitconnected.com/building-video-chat-in-3-days-with-livekit-da74e6cfd144?source=rss-2e5f5b703d04------2</link>
            <guid isPermaLink="false">https://medium.com/p/da74e6cfd144</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[chat]]></category>
            <category><![CDATA[meetings]]></category>
            <category><![CDATA[nodejs]]></category>
            <dc:creator><![CDATA[Oleh Teslenko]]></dc:creator>
            <pubDate>Wed, 18 Jun 2025 03:49:40 GMT</pubDate>
            <atom:updated>2025-06-18T03:49:40.526Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>How to built a Slack-like video communication system using JavaScript, Node.js, and LiveKit</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YKuAHpZX-dv-V90w" /><figcaption>Photo by <a href="https://unsplash.com/@cwmonty?utm_source=medium&amp;utm_medium=referral">Chris Montgomery</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Hi! I’m Oleh, a developer at Bidscube, a product company. In my spare time, I work on personal <a href="https://t-slen.com/">projects</a>, and one of my key challenges was implementing video communication between users — something like Slack or Teams, but maximally customized for our specific needs.</p><p>At first glance, the task seemed daunting: WebRTC, signaling, STUN/TURN servers, and more. But thanks to LiveKit, the entire process took just one week instead of months of development. In this article, I’ll share my experience and show you how to build your own video chat using JavaScript (Angular) and Node.js.</p><h4>Why Build Your Own Video Chat Instead of Using Ready-Made Solutions?</h4><p>Before starting development, I considered various options:</p><ul><li><strong>Jitsi Meet</strong> — free, but difficult to integrate into custom UI</li><li><strong>Agora</strong> — good API, but pricing gets expensive at scale</li><li><strong>LiveKit</strong> — open source, flexible, with self-hosting capabilities</li><li><strong>OpenVidu</strong> — open source, built on top of LiveKit and fully compatible with it</li></ul><p>A custom solution provides:</p><ul><li><strong>Full UI/UX control</strong> — interface tailored exactly to your product</li><li><strong>Business logic integration</strong> — call recordings, analytics, CRM integration</li><li><strong>Data control</strong> — especially important for B2B applications</li><li><strong>Long-term cost savings</strong> — no per-minute billing</li></ul><h4>Who Can Benefit from This Approach?</h4><p>From my experience, custom video chat is particularly valuable for:</p><ul><li><strong>EdTech projects</strong> — online schools, tutoring platforms needing integration with schedules, homework, and whiteboards</li><li><strong>Medical applications</strong> — telemedicine requiring special security and medical record integration</li><li><strong>HR/CRM systems</strong> — internal communications with call recording and contact integration</li><li><strong>Dating platforms</strong> — video introductions with moderation and special logic</li><li><strong>B2B products</strong> — consultations, demos, technical support with screen sharing</li></ul><h4>Solution Architecture</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0wm1by1jFUnD2gVEhaZ-PA.png" /></figure><p>Our tech stack:</p><ul><li><strong>JavaScript (Angular)</strong> — UI and LiveKit SDK integration</li><li><strong>Node.js (Express)</strong> — backend for token generation</li><li><strong>LiveKit</strong> — WebRTC communication server (local or cloud)</li></ul><h4>What is LiveKit?</h4><p><a href="https://livekit.io/">LiveKit</a> is an open-source platform for developers building real-time media applications. It simplifies the integration of audio, video, text, data, and AI models, offering scalable real-time infrastructure built on WebRTC.</p><p>You can deploy LiveKit server locally (self-hosted) or use LiveKit Cloud — eliminating the need for your own server entirely. LiveKit Cloud offers a free plan that includes an informative dashboard, 50GB traffic, and 5,000 minutes — sufficient for getting started.</p><h4>Brief Overview of WebRTC Technology</h4><p><a href="https://webrtc.org/">WebRTC</a> enables you to add real-time communication capabilities to your application based on an open standard. It supports video, voice, and general data transmission between nodes, allowing developers to create powerful voice and video communication solutions. The technology is available in all modern browsers and on native clients for all major platforms.</p><h4>Local LiveKit Installation Example (Ubuntu):</h4><pre>curl -sSL https://get.livekit.io | bash<br>livekit-server --dev</pre><p>After launch, the server will be available at ws://localhost:7880. You can specify API keys, ports, TLS, etc., in .env or livekit.yaml. More details in the <a href="https://docs.livekit.io/">documentation</a>.</p><h4>Or Simply Use Free LiveKit Cloud:</h4><ol><li>Register at <a href="https://cloud.livekit.io/">cloud.livekit.io</a></li><li>Get your:</li></ol><ul><li>API Key</li><li>API Secret</li><li>WSS URL (like wss://your-app.livekit.cloud)</li></ul><p>3. Monitor metrics through the informative dashboard</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KEvEHiMQgyH2Q2pDYrBUCA.png" /></figure><h4>JavaScript Implementation</h4><p>This can be fully adapted to any JS framework. Don’t forget to replace wsURL and token with your own data. The token must be generated on the backend. You can find a ready Angular example <a href="https://github.com/TeslenkoOleg/meet-tslen-livekit">here</a>.</p><pre>// Install NPM package<br>npm install livekit-client --save</pre><pre>// Change URL and token with your values<br>import { Room } from &#39;livekit-client&#39;;<br>const wsURL = &#39;wss://your-app.livekit.cloud&#39;;<br>const token = &#39;your-generated-token-here&#39;;<br>const room = new Room();<br>await room.connect(wsURL, token);<br>console.log(&#39;connected to room&#39;, room.name);<br>// Publish local camera and mic tracks<br>await room.localParticipant.enableCameraAndMicrophone();</pre><h4>Node.js Implementation</h4><p>The backend is needed for token generation, which must be inserted on the frontend side. Don’t forget to replace LIVEKIT_API_KEY and LIVEKIT_API_SECRET with your own data. You can also consider serverless solutions (e.g., Firebase functions). Ready Node.js example available <a href="https://github.com/TeslenkoOleg/meet-tslen-livekit">here</a>.</p><pre># yarn:<br>yarn add livekit-server-sdk<br># npm:<br>npm install livekit-server-sdk --save</pre><pre>// server.js<br>import express from &#39;express&#39;;<br>import { AccessToken } from &#39;livekit-server-sdk&#39;;const createToken = async () =&gt; {<br>  // If this room doesn&#39;t exist, it&#39;ll be automatically created when the first<br>  // participant joins<br>  const roomName = &#39;quickstart-room&#39;;<br>  <br>  // Identifier to be used for participant.<br>  // It&#39;s available as LocalParticipant.identity with livekit-client SDK<br>  const participantName = &#39;quickstart-username&#39;;<br>  <br>  const at = new AccessToken(<br>    process.env.LIVEKIT_API_KEY, <br>    process.env.LIVEKIT_API_SECRET, <br>    {<br>      identity: participantName,<br>      // Token to expire after 10 minutes<br>      ttl: &#39;10m&#39;,<br>    }<br>  );<br>  <br>  at.addGrant({ roomJoin: true, room: roomName });<br>  return await at.toJwt();<br>};<br>const app = express();<br>const port = 3000;<br>app.get(&#39;/getToken&#39;, async (req, res) =&gt; {<br>  res.send(await createToken());<br>});<br>app.listen(port, () =&gt; {<br>  console.log(`Server listening on port ${port}`);<br>});</pre><h4>Key Takeaways</h4><p>Creating a custom video chat turned out to be much simpler than I initially thought — LiveKit allows you to implement basic functionality in literally a day or two. I’m particularly impressed by the choice between a cloud solution for quick startup and a self-hosted option for complete data control.</p><p>Thanks to open source code and excellent documentation, even complex customizations become accessible without deep diving into WebRTC internals. I recommend starting with LiveKit Cloud for prototyping, then gradually adding custom features — this approach allows you to quickly get an MVP and scale the project according to specific business needs.</p><p>The beauty of this solution lies in its flexibility: you get enterprise-grade video infrastructure without the complexity of building WebRTC from scratch. Whether you’re building the next EdTech platform or adding video calls to your B2B SaaS, LiveKit provides the foundation you need to focus on what makes your product unique.</p><h4>Useful Links</h4><ul><li><a href="https://github.com/TeslenkoOleg/meet-tslen-livekit">GitHub</a> repository of demo project</li><li><a href="https://meet.t-slen.com/">Live demo</a></li><li><a href="https://docs.livekit.io/">LiveKit documentation</a></li><li><a href="https://cloud.livekit.io/">LiveKit Cloud</a></li><li><a href="https://docs.openvidu.io/">OpenVidu documentation</a></li></ul><p><em>Thank you for reading. If you have any questions you can find me on </em><a href="https://www.linkedin.com/in/oleh-teslenko-720443161/"><em>LinkedIn</em></a><em>. Feel free to leave comments. You can also read my previous articles:</em></p><ul><li><a href="https://levelup.gitconnected.com/ip-detection-in-the-digital-world-a92940076c95">IP Detection in the Digital World</a></li><li><a href="https://levelup.gitconnected.com/deploy-nest-js-application-on-google-cloud-run-759ecd1a0ac3">Deploy Nest.js Application on Google Cloud Run</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da74e6cfd144" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/building-video-chat-in-3-days-with-livekit-da74e6cfd144">Building Video Chat in 3 Days with LiveKit</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[IP Detection in the Digital World]]></title>
            <link>https://levelup.gitconnected.com/ip-detection-in-the-digital-world-a92940076c95?source=rss-2e5f5b703d04------2</link>
            <guid isPermaLink="false">https://medium.com/p/a92940076c95</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[development]]></category>
            <dc:creator><![CDATA[Oleh Teslenko]]></dc:creator>
            <pubDate>Fri, 28 Mar 2025 02:02:19 GMT</pubDate>
            <atom:updated>2025-03-28T02:02:19.559Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to check the actual user IP address?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cpzgaUASWYB7zU4L" /><figcaption>AI generated image</figcaption></figure><p>Digital advertising constantly surrounds us in our lives. All these video and display ads in smartphones, tablets, laptops, and even TV. So how come they know exactly our region and city to show relevant advertisements? Is that some kind of conspiracy theory? Of course, no, but let’s go through everything step by step. My name is Oleh, and I’m a developer/team lead at <a href="https://bidscube.com/">Bidscube</a>. For the past 5 years, I’ve been working with real-time digital advertising, where identifying and verifying user locations is an important part of the overall process.</p><p>In this article, we’ll examine a real-world example of how to find out the actual user’s IP address. Additionally, we’ll develop our own mini IP address detector that you can use independently for your needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*IHinHjyvZYwFczKV" /><figcaption>DEMO (ip.t-slen.com)</figcaption></figure><blockquote>Note: The code examples in this article are written in Javascript (Node.js) using serverless solutions — Firebase Functions (part of the Google Cloud Run ecosystem).</blockquote><p>Let’s get started.</p><p>Let’s assume we own a mobile application (Application) and want to establish the actual address of a user (User). The User is not using a VPN or other methods of hiding their IP.</p><h4>Step 1. Determining the IP</h4><p>The first and simplest step is obtaining the user’s public digital address. You can be confident that almost all applications store the addresses of their users. And this is not done to find you, but primarily to know their audience and possibly retarget active visitors (for example, to show relevant advertisements). This is particularly important in the digital advertising ecosystem, where user location data significantly impacts ad targeting and bid valuation.</p><p>How does this happen?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1TkFa_PXS61NVp9y" /></figure><p>When a User interacts with the Application (reads news, plays games, etc.), they’re actually sending requests to the server. Each such request from the client to the server contains certain information, including your IP address, and as a result, can easily identify and store it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EDJ9hcli0DX6wtyy" /></figure><p>In the screenshot above, you can see the aggregated logs provided by Google Cloud Run. We’re primarily interested in the remoteIp. This is the User’s IP, so we save it in our imagine database.</p><p>On the backend side, this can be implemented something like this:</p><pre>function getClientIp(request) {<br>  return request.connection.remoteAddress || request.client.remoteAddress;<br>}</pre><p>If you want to check your address yourself, you can use free APIs, for example, <a href="https://api.ipify.org/?format=json">https://api.ipify.org?format=json</a></p><p>As a result, we’ll get a response in JSON format:</p><pre>{&quot;ip&quot;:&quot;104.28.205.18&quot;}</pre><h4>Step 2. Establishing Geolocation</h4><p>Now that we have the User’s public IP address, the next step is to determine the city where they are located. In the modern world, this isn’t the most difficult task either. There are many services, both paid and free, that completely legally provide their databases of IP addresses linked to countries/cities. We’ll use the free version of Maxmind’s database. Although it shows worse accuracy than the paid version, it’s quite suitable for our purposes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WxVaABA3NudwCuZw" /></figure><p>In the OpenRTB context, accurate geolocation is crucial for serving localized ads and meeting campaign targeting requirements.</p><p>We’ll use the GeoLite2City database, as it contains information about both the country and city where the required IP address is located.</p><p>To implement the Node.js backend, you need to install the corresponding <a href="https://www.npmjs.com/package/maxmind">NPM library</a>:</p><pre>npm i maxmind</pre><p>Here’s the implementation of connecting maxmind:</p><pre>import { Reader } from &#39;maxmind&#39;;<br>import * as fs from &quot;node:fs&quot;;<br>const cityDBPath = process.cwd() + &quot;/maxmind/GeoLite2-City.mmdb&quot;;<br>const cityBuffer = fs.readFileSync(cityDBPath);<br>export const cityDB = new Reader(cityBuffer);</pre><p>Next, you need to write the code for your backend. In my case, I use Cloud Functions for <a href="https://firebase.google.com/docs/functions">Firebase</a>.</p><blockquote>This is a serverless framework that allows you to automatically run server code in response to events triggered by background events or HTTPS requests. Your code is stored in Google Cloud infrastructure and runs in a managed environment. There’s no need to manage and scale your own servers.</blockquote><p>We’ll launch the Firebase function through an HTTPS request using a request handler. This allows you to call the function using supported HTTP methods: GET, POST, PUT, DELETE, and OPTIONS. Additionally, you can configure additional options such as CORS and timeout.</p><pre>&#39;use strict&#39;;<br>import { onRequest } from &quot;firebase-functions/https&quot;;<br>// Load MaxMind databases<br>import { cityDB } from &quot;../maxmind/index.js&quot;;<br>export const checkLocation = onRequest({cors: process.env.ALLOWED_ORIGINS,timeoutSeconds: 5},<br>      (req, res) =&gt; {<br>      const ip = req.query.ip;<br>      if (!ip) {<br>        return res.status(400).json({ error: &quot;IP address is required&quot; });<br>      }<br>      try {<br>        const cityInfo = cityDB.get(ip);<br>        if (!cityInfo) {<br>          return res.status(404).json({ error: &quot;Location not found for the given IP&quot; });<br>        }<br>      const result = {<br>          country: cityInfo.country?.names?.en || &quot;Unknown&quot;,<br>          city: cityInfo.city?.names?.en || &quot;Unknown&quot;,<br>          latitude: cityInfo.location?.latitude || &quot;Unknown&quot;,<br>          longitude: cityInfo.location?.longitude || &quot;Unknown&quot;,<br>        };<br>        return res.json(result);<br>      } catch (error) {<br>        console.error(&quot;Error fetching location:&quot;, error);<br>        return res.status(500).json({ error: &quot;Error fetching location data&quot; });<br>      }<br>});</pre><p>After deploying the function, you’ll receive a link to public access to it, however, if desired, these settings can be changed. Example link: <a href="https://checklocation-g2dxjhdf6a-uc.a.run.app/?ip=${ip}">https://checklocation-g2dxjhdf6a-uc.a.run.app?ip=${ip}</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*IHinHjyvZYwFczKV" /><figcaption>ip.t-slen.com</figcaption></figure><p>So, the result of determining the geolocation for a random IP (194.60.254.173) is a specific city. But this is only the city where the internet service provider operates.</p><h4>Step 3. Identifying the Internet Service Provider</h4><p>Next comes a slightly more complicated situation, as it’s impossible to proceed without investing money. The database containing information on providers that directly provide Users with internet services is already paid (about $100). However, such a service is only needed by those who want to build their own IP detector. You can easily find this information on the internet for free. But if you still want to do everything yourself without involving third parties, here’s the implementation:</p><pre>import { Reader } from &#39;maxmind&#39;;<br>import * as fs from &quot;node:fs&quot;;<br>const ispDBPath = process.cwd() + &quot;/GeoIP2-ISP.mmdb&quot;;<br>const ispBuffer = fs.readFileSync(ispDBPath);<br>export const ispDB = new Reader(ispBuffer);<br>const ip = &#39;194.60.254.173&#39;;<br>const info = ispDB.get(ip);<br>console.table(info);</pre><p>The result of determining the provider by IP is information about the ISP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-KOmvz5_z_6r4M0i" /></figure><p>From the information above, we can see that the provider is likely an individual entrepreneur. Further, we can go to business registry services and enter this data in the search and obtain contact information for the legal entity.</p><p>Now we know who provides the services and to whom the User provides information about their actual address.</p><p>You can view a <a href="https://ip.t-slen.com/">demo</a> of the IP detector from this article.</p><p>Useful links:</p><ol><li>Demo Github <a href="https://github.com/TeslenkoOleg/ip-checker">repository</a>.</li><li>Firebase function examples Github <a href="https://github.com/TeslenkoOleg/firebase-functions-ip.t-slen.com">repository</a>.</li></ol><p>Thank you for reading. If you have any questions you can find me on <a href="https://www.linkedin.com/in/oleh-teslenko-720443161/">LinkedIn</a>. Feel free to leave comments. You can also read my previous articles:</p><ul><li><a href="https://levelup.gitconnected.com/deploy-nest-js-application-on-google-cloud-run-759ecd1a0ac3">Deploy Nest.js Application on Google Cloud Run</a></li><li><a href="https://levelup.gitconnected.com/how-to-build-responsive-layouts-using-angular-react-or-vue-23968710aab8">How to Build Responsive Layouts using Angular, React, or Vue</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a92940076c95" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/ip-detection-in-the-digital-world-a92940076c95">IP Detection in the Digital World</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploy Nest.js Application on Google Cloud Run]]></title>
            <link>https://levelup.gitconnected.com/deploy-nest-js-application-on-google-cloud-run-759ecd1a0ac3?source=rss-2e5f5b703d04------2</link>
            <guid isPermaLink="false">https://medium.com/p/759ecd1a0ac3</guid>
            <category><![CDATA[google]]></category>
            <category><![CDATA[nestjs]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[nodejs]]></category>
            <dc:creator><![CDATA[Oleh Teslenko]]></dc:creator>
            <pubDate>Thu, 06 Feb 2025 19:50:02 GMT</pubDate>
            <atom:updated>2025-02-06T19:50:02.215Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/555/1*Dmks_nbeNZP-l4xJ2IEGfQ.png" /></figure><p>Google Cloud Run is a great way to host your backend application. Its main advantage is that you pay only for real usage. If your service needs to be run occasionally or does not have many active users yet, it is the best solution for you. For instance, I have a demo application for my pet project that has not been used for a long time. With Cloud Run, I can keep it available without worrying about unnecessary costs.</p><p>In this article, I will guide you through the trickiest moments and the most important steps to deploy your Nest.js application successfully.</p><h4>Prerequisites</h4><ul><li>A Google Cloud project (see <a href="https://cloud.google.com/run?hl=uk">documentation</a>)</li><li>A Nest.js application repository on GitHub (see <a href="https://nestjs.com/">documentation</a>)</li><li>Basic knowledge of Docker and Cloud Run (see <a href="https://www.docker.com/">documentation</a>)</li></ul><h4>Prepare Dockerfile</h4><p>One of the most important steps is creating an appropriate Dockerfile for your project. A properly structured Dockerfile ensures that your application runs efficiently and securely. Below is a working example, followed by a step-by-step breakdown.</p><pre>FROM node:22<br><br>WORKDIR /usr/src/app<br><br>COPY package*.json ./<br><br>RUN npm install -g @nestjs/cli<br><br>RUN npm install<br><br>COPY . .<br><br>RUN npm run build<br><br>CMD npm start<br><br>EXPOSE 8080</pre><p>Let’s break it down step by step:</p><ul><li>Uses the official <strong>Node.js 22</strong> image as the base.</li><li>Sets the working directory inside the container to /usr/src/app.</li><li>Copies package.json and package-lock.json from the host machine to the container.</li><li>Installs the NestJS CLI globally inside the container.</li><li>Installs project dependencies defined in package.json.</li><li>Copies <strong>all files</strong> from the host machine to the container’s /usr/src/app directory.</li><li>Compiles the NestJS application.</li><li>Defines the command to run when the container starts.</li><li>Informs Docker that the application inside the container will listen on <strong>port 8080</strong>.</li></ul><p>I highly recommend testing the Dockerfile on your local machine before deploying it:</p><pre><br>docker run -p 8080:8080 your-app-name<br>docker build -t your-app-name .</pre><h4>Deploy to Google Cloud Run</h4><p>Firstly, open Cloud Run in your project and choose “Deploy container ”— “Service”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DUiIfRTWIa5OGwWDZQzROg.png" /></figure><p>Next, select “Continuously deploy from GitHub repository” and click “Set up with cloud build”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wIoFcc2xfOLC7XufEPvV_Q.png" /></figure><p>Init integration with GitHub API and select a repository you wish.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cYEtoWwuwDUd5t83QSCaHA.png" /></figure><p>The build type should be a “Dockerfile”. Click the “Save” button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P6_pMw7dAwuiHJ2jXeK_0g.png" /></figure><p>After a few minutes, your service is ready and you can find the backend URL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BLQw-dQpYkM_L4dweKzd2g.png" /></figure><h4>Environment variables</h4><p>Environment variables are crucial for any application, as they store sensitive information such as database credentials, API keys, and other configurations. You can easily set them up in Google Cloud Run.</p><p>Navigate to your deployed service in Cloud Run and<strong> </strong>click “Edit &amp; deploy new revision” in service details.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yWfnW-R1tctiGuvfj446eA.png" /></figure><p>There you can find the “Variables&amp;Secrets” tab and click the “Deploy” button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aHKeMPfpWOh1xgRAshHP0g.png" /></figure><p>All these secrets are referred to a <strong>.env</strong> file in your nest.js application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/251/1*nyPphJjdpZvEvy9LGa1mbg.png" /></figure><h4>Monitor Deployment Logs</h4><p>During deployment, you can observe the build logs to check for any issues. Google Cloud Run provides real-time logs, making it easy to debug and troubleshoot problems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rwym5AuDnKEv1tgmazWXXQ.png" /></figure><p>As soon as your application is successfully deployed you can use it for your needs.</p><h4>Conclusion</h4><p>Google Cloud Run is a powerful and flexible hosting solution for backend applications. It offers automatic scaling, security, and cost-effectiveness, making it an excellent choice for applications with variable traffic. Whether you are deploying a demo application, a production-ready service, or an API backend, Cloud Run simplifies deployment while keeping costs minimal. By following this guide, you can ensure a smooth and efficient deployment of your Nest.js application. Start leveraging Cloud Run today and experience the benefits of a fully managed serverless environment.</p><p>Thank you for reading. If you have any questions you can find me on <a href="https://www.linkedin.com/in/oleh-teslenko-720443161/">LinkedIn</a>. Feel free to leave comments. You can also read my previous articles:</p><ul><li><a href="https://levelup.gitconnected.com/dsl-with-node-js-a-hands-on-guide-7656c7e9209a">DSL with Node.js: A Hands-On Guide</a></li><li><a href="https://medium.com/t-slen/google-auth-using-nestjs-2643af563f8e">Google Auth using NestJs</a></li><li><a href="https://medium.com/t-slen/no-code-needed-efficiently-download-clean-and-insert-files-into-databases-using-command-line-973e7a485f96">No Code Needed: Efficiently Download, Clean, and Insert Files into Databases Using Command Line</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=759ecd1a0ac3" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/deploy-nest-js-application-on-google-cloud-run-759ecd1a0ac3">Deploy Nest.js Application on Google Cloud Run</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DSL with Node.js: A Hands-On Guide]]></title>
            <link>https://levelup.gitconnected.com/dsl-with-node-js-a-hands-on-guide-7656c7e9209a?source=rss-2e5f5b703d04------2</link>
            <guid isPermaLink="false">https://medium.com/p/7656c7e9209a</guid>
            <category><![CDATA[domain-specific-languages]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[development]]></category>
            <dc:creator><![CDATA[Oleh Teslenko]]></dc:creator>
            <pubDate>Mon, 09 Dec 2024 06:37:36 GMT</pubDate>
            <atom:updated>2025-01-07T17:21:59.614Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building a Domain Specific Language using Node.js and YAML</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*Xq6dACcFI-gNNTtR2fAk8Q.png" /></figure><p>A DSL is a language focused on a specific domain. Unlike general-purpose languages, DSLs simplify niche tasks. If you’ve worked with SQL, you’ve used a DSL tailored to databases. In this article, I’ll walk you through building a DSL using Node.js and YAML, based on an example workflow that processes data from a CSV file, runs a script, and inserts results into a database.</p><h4>How It Works</h4><p>I have created a project on <a href="https://github.com/TeslenkoOleg/node-yaml-dsl/tree/main">GitHub</a> to show you how it works. The goal of the project is to execute workflows defined in YAML. The repository has three main parts:</p><ol><li>main.js: Parses the YAML and runs the actions.</li><li>stepActions.js: Maps actions like reading files or executing scripts to JavaScript functions.</li><li>scenarios/: Contains YAML files that define the workflows.</li></ol><h4>Step-by-Step Implementation</h4><p>1. scenarios/users.yaml: YAML workflow</p><pre>workflow:<br>  - step: &quot;file&quot;<br>    action: &quot;read&quot;<br>    path: &quot;users/users.csv&quot;<br><br>  - step: &quot;process data&quot;<br>    action: &quot;script&quot;<br>    script: &quot;users/users.js&quot;<br><br>  - step: &quot;insert results&quot;<br>    action: &quot;insertIntoDB&quot;<br>    database: &quot;mysql&quot;<br>    fields: &quot;id, age, name&quot;<br>    query: &quot;INSERT INTO users (id, age, name) VALUES (?, ?, ?)&quot;</pre><p>This workflow reads a CSV file, processes its content via a script, and inserts the processed data into a MySQL database.</p><p>2. main.js: Orchestrating the Workflow</p><p>This script is the entry point. It loads YAML files, parses workflows, and executes them:</p><ul><li>import all dependencies:</li></ul><pre>&#39;use strict&#39;;<br><br>import fs from &#39;node:fs&#39;;<br>import path from &#39;node:path&#39;;<br>import { fileURLToPath } from &#39;node:url&#39;;<br>import yaml from &#39;js-yaml&#39;;<br>import { STEP_ACTIONS } from &#39;./stepActions.js&#39;;<br><br>const __filename = fileURLToPath(import.meta.url);<br>const __dirname = path.dirname(__filename);</pre><ul><li>load YAML files from the directory:</li></ul><pre>function loadYamlFiles(directory) {<br>  return fs.readdirSync(directory)<br>    .filter(file =&gt; file.endsWith(&#39;.yaml&#39;))<br>    .map(file =&gt; {<br>      const filePath = path.join(directory, file);<br>      const fileContents = fs.readFileSync(filePath, &#39;utf8&#39;);<br>      return yaml.load(fileContents);<br>    });<br>}</pre><ul><li>execute workflow:</li></ul><pre>async function executeWorkflow(workflow) {<br>  let previousStepResult = null;<br>  for (const step of workflow) {<br>    const action = STEP_ACTIONS.get(step.action);<br>    if (!action) {<br>      continue;<br>    }<br>    previousStepResult = await action(step, previousStepResult);<br>  }<br>}</pre><ul><li>run main function:</li></ul><pre><br>async function main() {<br>  const scenarioDir = path.join(__dirname, &#39;scenarios&#39;);<br>  const workflows = loadYamlFiles(scenarioDir);<br><br>  for (const workflowData of workflows) {<br>    await executeWorkflow(workflowData.workflow);<br>  }<br>}<br><br>await main();</pre><p>This script:</p><ul><li>Loads YAML workflows from the scenarios/ directory.</li><li>Iterates through workflow steps, running actions defined in STEP_ACTIONS.</li></ul><p>2. stepActions.js: Mapping Actions</p><p>Each workflow step maps to a function in this file:</p><pre>&#39;use strict&#39;;<br>import fs from &quot;node:fs&quot;;<br>import path from &quot;node:path&quot;;<br>import { fileURLToPath } from &quot;node:url&quot;;<br>const __filename = fileURLToPath(import.meta.url);<br>const __dirname = path.dirname(__filename);<br><br>export const STEP_ACTIONS = new Map([<br>    [&#39;read&#39;, async (step, previousStepResult) =&gt; read(step)],<br>    [&#39;script&#39;, async (step, previousStepResult) =&gt; script(step, previousStepResult)],<br>    [&#39;insertIntoDB&#39;, async (step, previousStepResult) =&gt; insertIntoDB(step, previousStepResult)],<br>    [&#39;write&#39;, async (step, previousStepResult) =&gt; write(step, previousStepResult)],<br>]);<br><br>const read = async (step, previousStepResult) =&gt; {<br>    return fs.readFileSync(step.path, &#39;utf8&#39;);<br>};<br>const script = async (step, previousStepResult) =&gt; {<br>    const scriptModule = await import(path.join(__dirname, step.script));<br>    return scriptModule.default(previousStepResult);<br>};<br>const insertIntoDB = async (step, previousStepResult) =&gt; {<br>    console.log(`Inserting data into database: ${step.database} ` + previousStepResult);<br>    if (step.database === &#39;mysql&#39;) {<br>        //TODO: MySQL insertion logic<br>    }<br>};<br>const write = async (step, previousStepResult) =&gt; {<br>    console.log(`Writing data to file: ${step.path}`);<br>    fs.writeFileSync(step.path, previousStepResult);<br>};</pre><p>This file uses a Map to associate step actions (e.g., read, script) with their corresponding functions.</p><h4>Customizing the DSL</h4><p>To add new actions, modify stepActions.js. For example, to log messages:</p><pre>STEP_ACTIONS.set(&#39;log&#39;, async (step, previousStepResult) =&gt; {<br>  console.log(step.message);<br>});</pre><p>Then, define it in YAML:</p><pre>- step: &quot;log message&quot;<br>  action: &quot;log&quot;<br>  message: &quot;Workflow started!&quot;</pre><h4>Conclusion</h4><p>Building a DSL like this makes complex workflows easier to manage and execute. It’s a powerful way to automate tasks while keeping configurations simple and readable.</p><p>The full source of the code is <a href="https://github.com/TeslenkoOleg/node-yaml-dsl/tree/main">here</a>.</p><p>Thank you for reading. If you have any questions you can find me on <a href="https://www.linkedin.com/in/oleh-teslenko-720443161/">LinkedIn</a>. Feel free to leave comments. You can also read my previous articles:</p><ul><li><a href="https://medium.com/t-slen/google-auth-using-nestjs-2643af563f8e">Google Auth using NestJs</a></li><li><a href="https://medium.com/t-slen/no-code-needed-efficiently-download-clean-and-insert-files-into-databases-using-command-line-973e7a485f96">No Code Needed: Efficiently Download, Clean, and Insert Files into Databases Using Command Line</a></li><li><a href="https://medium.com/t-slen/breaking-down-big-data-efficient-search-strategies-for-10-gb-files-2a6f14522853">Breaking Down Big Data: Efficient Search Strategies for 10+GB Files</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7656c7e9209a" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/dsl-with-node-js-a-hands-on-guide-7656c7e9209a">DSL with Node.js: A Hands-On Guide</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Google Auth using NestJs]]></title>
            <link>https://medium.com/t-slen/google-auth-using-nestjs-2643af563f8e?source=rss-2e5f5b703d04------2</link>
            <guid isPermaLink="false">https://medium.com/p/2643af563f8e</guid>
            <category><![CDATA[authentication]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[nestjs]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[google]]></category>
            <dc:creator><![CDATA[Oleh Teslenko]]></dc:creator>
            <pubDate>Fri, 01 Nov 2024 16:56:52 GMT</pubDate>
            <atom:updated>2024-11-01T16:56:52.336Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to use Google Authentification with NestJs</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/1*qrFpqjvyQyAc3aNxESI6VA.png" /></figure><p>Integrating Google Authentication into a NestJS application provides users with a secure and straightforward way to log in using their Google accounts. This article walks you through setting up Google Authentication using NestJS, including creating an OAuth2 client, generating an authentication URL, and handling the authentication callback to access user information.</p><h4>How It Works</h4><ol><li>A user initiates Google authentication by sending a request to your backend server, it generates an OAuth2 URL and redirects the user to Google’s authentication page.</li><li>Once the user authenticates, Google redirects them to your preset URL, passing a code as a query parameter. This code is used to fetch user details and tokens.</li><li>Using the code, the backend retrieves tokens and user information, such as email, refresh token, and access token. You can then handle this data as needed, for example, creating a new user or logging in to an existing user.</li></ol><h4>Prerequisites</h4><p>Ensure you have the following:</p><ul><li>Node.js and NestJS</li><li>Google application credentials (<a href="https://cloud.google.com/nodejs/docs/reference/google-auth-library/latest#application-default-credentials">documentation</a>)</li></ul><h4>Setting Up</h4><ol><li>Install Required Packages:</li></ol><pre>npm i googleapis<br>npm i google-auth-library</pre><p>2. Environment Configuration</p><p>Create .env file and add Google OAuth credentials and scope configuration:</p><pre>GOOGLE_CREDENTIALS_PATH=path/to/credentials.json<br>GOOGLE_SCOPES_API=email,profile<br>REDIRECT_TO_LOGIN=http://yourapp.com/dashboard</pre><p>3. Creating the GoogleService:</p><p>The GoogleService class handles the interaction with Google&#39;s OAuth2 API, generating an authorization URL and handling the callback to retrieve user data.</p><pre>import { Injectable } from &#39;@nestjs/common&#39;;<br>import { google } from &#39;googleapis&#39;;<br>import { ConfigService } from &#39;@nestjs/config&#39;;<br>import { OAuth2Client } from &#39;google-auth-library&#39;;<br>import * as fs from &#39;fs&#39;;<br>import * as path from &#39;path&#39;;<br><br>@Injectable()<br>export class GoogleService {<br>  private readonly scopesAPI: string[];<br>  private readonly credentialsPath: string;<br>  constructor(private configService: ConfigService) {<br>    this.credentialsPath = path.join(<br>      process.cwd(),<br>      this.configService.get(&#39;GOOGLE_CREDENTIALS_PATH&#39;),<br>    );<br>    this.scopesAPI = this.configService.get(&#39;GOOGLE_SCOPES_API&#39;).split(&#39;,&#39;);<br>  }<br><br>  readCredentials(filePath: string): IGoogleAuthCredentials {<br>    const credentialsPath = path.join(filePath);<br>    const content: string = fs.readFileSync(credentialsPath, &#39;utf-8&#39;);<br>    return JSON.parse(content);<br>  }<br>  async getOAuth2ClientUrl(): Promise&lt;{ url: string }&gt; {<br>    const authClient = this.getAuthClient();<br>    return this.getAuthUrl(authClient);<br>  }<br>  getAuthClient(): OAuth2Client {<br>    const keys: IGoogleAuthCredentials = this.readCredentials(<br>      this.credentialsPath,<br>    );<br>    const authClient = new OAuth2Client(<br>      keys.web.client_id,<br>      keys.web.client_secret,<br>      keys.web.redirect_uris[0],<br>    );<br>    return authClient;<br>  }<br>  getAuthUrl(authClient: OAuth2Client): { url: string } {<br>    // Generate the url that will be used for the consent dialog.<br>    const authorizeUrl = authClient.generateAuthUrl({<br>      access_type: &#39;offline&#39;,<br>      scope: this.scopesAPI,<br>      prompt: &#39;consent&#39;,<br>      include_granted_scopes: true,<br>    });<br>    return { url: authorizeUrl };<br>  }<br>  async getAuthClientData(<br>    code: string,<br>  ): Promise&lt;{ email: string; refreshToken: string; accessToken: string }&gt; {<br>    const authClient = this.getAuthClient();<br>    const tokenData = await authClient.getToken(code);<br>    const tokens = tokenData.tokens;<br>    const refreshToken = tokens?.refresh_token || &#39;&#39;;<br>    const accessToken = tokens?.access_token || &#39;&#39;;<br><br>    authClient.setCredentials(tokens);<br><br>    const googleAuth = google.oauth2({<br>      version: &#39;v2&#39;,<br>      auth: authClient,<br>    } as any);<br><br>    const googleUserInfo = await googleAuth.userinfo.get();<br>    const email = googleUserInfo.data.email;<br>    return { email, refreshToken, accessToken };<br>  }<br>}<br>export interface IGoogleAuthCredentials {<br>  web: {<br>    client_id: string;<br>    client_secret: string;<br>    redirect_uris: string[];<br>    auth_uri: string;<br>    token_uri: string;<br>    auth_provider_x509_cert_url: string;<br>    javascript_origins: string[];<br>  };<br>}</pre><p>4. Creating the AppService and AppController:</p><p>The AppService coordinates authentication and data retrieval with GoogleService, while the AppController defines the routes for Google Authentication.</p><ul><li>AppService:</li></ul><pre>import { Injectable } from &#39;@nestjs/common&#39;;<br>import { GoogleService } from &#39;./google.service&#39;;<br><br>@Injectable()<br>export class AppService {<br>  constructor(private googleService: GoogleService) {}<br><br>  async googleAuth(): Promise&lt;{ url: string }&gt; {<br>    return this.googleService.getOAuth2ClientUrl();<br>  }<br><br>  async getAuthClientData(code: string): Promise&lt;{ email: string; refreshToken: string; accessToken: string }&gt; {<br>    return this.googleService.getAuthClientData(code);<br>  }<br>}</pre><ul><li>AppController:</li></ul><pre>import { Controller, Get, Redirect, Query } from &#39;@nestjs/common&#39;;<br>import { AppService } from &#39;./app.service&#39;;<br><br>@Controller()<br>export class AppController {<br>  constructor(private readonly appService: AppService) {}<br><br>  @Get(&#39;google-auth&#39;)<br>  @Redirect()<br>  async googleAuth(): Promise&lt;{ url: string }&gt; {<br>    return this.appService.googleAuth();<br>  }<br><br>  @Get(&#39;google-callback&#39;)<br>  @Redirect()<br>  async googleAuthCallback(@Query(&#39;code&#39;) code: string): Promise&lt;{ url: string }&gt; {<br>    const { email, refreshToken, accessToken } = await this.appService.getAuthClientData(code);<br>    // Implement additional sign-in logic here<br>    return { url: process.env.REDIRECT_TO_LOGIN };<br>  }<br>}</pre><p>5. Configuring the AppModule:</p><ul><li>Include ConfigModule for managing environment variables and AppService and GoogleService for handling authentication.</li></ul><pre>import { Module } from &#39;@nestjs/common&#39;;<br>import { AppController } from &#39;./app.controller&#39;;<br>import { AppService } from &#39;./app.service&#39;;<br>import { GoogleService } from &#39;./google.service&#39;;<br>import { ConfigModule, ConfigService } from &#39;@nestjs/config&#39;;<br><br>@Module({<br>  imports: [ConfigModule.forRoot({ isGlobal: true })],<br>  controllers: [AppController],<br>  providers: [AppService, GoogleService, ConfigService],<br>})<br>export class AppModule {}</pre><p>This setup provides a robust foundation for Google Authentication in NestJS. You can now extend it by adding additional routes, user sessions, and database integration for storing tokens or user information as needed.</p><p>The full source of the code is <a href="https://github.com/TeslenkoOleg/nest-google-auth/tree/main">here</a>. More details about Google Auth can be found in the official <a href="https://cloud.google.com/nodejs/docs/reference/google-auth-library/latest">documentation</a> for Node.js.</p><p>Thank you for reading. If you have any questions you can find me on <a href="https://www.linkedin.com/in/oleh-teslenko-720443161/">LinkedIn</a>. Feel free to leave comments. You can also read my previous articles:</p><ul><li><a href="https://levelup.gitconnected.com/automatic-application-deployment-using-bitbucket-pipelines-9fa8680e4371">Automatic Application Deployment Using Bitbucket Pipelines</a></li><li><a href="https://levelup.gitconnected.com/angular-google-maps-api-11af43f2fea4">Angular Google Maps API</a></li><li><a href="https://medium.com/t-slen/breaking-down-big-data-efficient-search-strategies-for-10-gb-files-2a6f14522853">Breaking Down Big Data: Efficient Search Strategies for 10+GB Files</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2643af563f8e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/t-slen/google-auth-using-nestjs-2643af563f8e">Google Auth using NestJs</a> was originally published in <a href="https://medium.com/t-slen">T-slen</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Aerospike UDF (User-Defined Function)]]></title>
            <link>https://medium.com/t-slen/aerospike-udf-user-defined-function-4274c27e1bde?source=rss-2e5f5b703d04------2</link>
            <guid isPermaLink="false">https://medium.com/p/4274c27e1bde</guid>
            <category><![CDATA[aerospike]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[nodejs]]></category>
            <dc:creator><![CDATA[Oleh Teslenko]]></dc:creator>
            <pubDate>Sat, 24 Aug 2024 07:37:30 GMT</pubDate>
            <atom:updated>2024-08-24T07:37:30.191Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to create and use Aerospike UDF using Node.js</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*oUOLpgbmUimL5A-PSFAUww.jpeg" /><figcaption>The image from Aerospike LinkedIn <a href="https://www.linkedin.com/company/aerospike-inc-/posts/?feedView=images">page</a></figcaption></figure><p>Aerospike User-Defined Functions (UDFs) allow developers to execute custom logic directly within the Aerospike database. This is particularly useful for tasks such as filtering, aggregation, and more complex data processing that would otherwise require multiple round trips between the client and server.</p><blockquote>An Aerospike <a href="https://aerospike.com/docs/guide/udf">User-Defined Function</a> is a piece of code, written by a developer in <a href="http://www.lua.org/about.html">Lua programming language</a> (or C called from Lua) that runs inside the Aerospike database server.</blockquote><p>In this article, we’ll explore how to create and use Aerospike UDFs using Node.js. We’ll cover the basics of Lua and demonstrate how to register and execute these functions within Aerospike.</p><h4>Lua: The Language Behind Aerospike UDFs</h4><p>Lua is a powerful, efficient, lightweight, and embeddable scripting language. It supports various programming paradigms, including procedural, object-oriented, functional, and data-driven programming. Lua’s flexibility makes it an excellent choice for writing Aerospike UDFs.</p><p>When you write UDFs for Aerospike, you’ll use Lua to define the logic that will run on the server. This logic can range from simple record filtering to more complex data manipulation.</p><h4>Setting Up Aerospike and Node.js</h4><p>To follow along with this guide, you’ll need the following versions:</p><ul><li>Node.js: v8–14 with Aerospike client v3.16 — Aerospike server v3</li><li>Node.js: v20–22 with Aerospike client v5 — Aerospike server v7.1</li></ul><h4>Writing a Lua UDF for Aerospike</h4><p>Let’s write a simple Lua UDF that filters records based on a specific condition. For instance, imagine that we have loads of records like the next one: {age: 25, isActive:1}. Create a file named test_udf.lua and define your function:</p><pre>local function my_filter_fn(rec)<br>    local isActive = rec[&#39;isActive&#39;] or 0<br>    if isActive == 1 then<br>        return true<br>    else<br>        return false<br>    end<br>end<br><br>local function my_map_fn(rec)<br>   local result = map()<br>        result[&#39;isActive&#39;] = rec[&#39;isActive&#39;]<br>   return result<br>end<br><br>function test_fn(stream)<br>    return stream : filter(my_filter_fn) : map(my_map_fn)<br>end</pre><p>This Lua code defines a set of functions designed to filter and map records within an Aerospike stream using a User-Defined Function (UDF). Here’s a detailed breakdown of each part:</p><p>1. my_filter_fn</p><pre>local function my_filter_fn(rec)<br>    local isActive = rec[&#39;isActive&#39;] or 0<br>    if isActive == 1 then<br>        return true<br>    else<br>        return false<br>    end<br>end</pre><p>This function filters records based on the value of the isActive field. It first attempts to retrieve the isActive field from the record (rec). If isActive doesn&#39;t exist, it defaults to 0.</p><p>2. my_map_fn</p><pre>local function my_map_fn(rec)<br>   local result = map()<br>   result[&#39;isActive&#39;] = rec[&#39;isActive&#39;]<br>   return result<br>end</pre><p>This function transforms records by creating a new map that includes only the isActive field from the original record. It makes a new, empty map named result. The isActive value from the original record (rec) is added to the result map. The function returns the result map, which now contains just the isActive field.</p><p>3. test_fn</p><pre>function test_fn(stream)<br>    return stream : filter(my_filter_fn) : map(my_map_fn)<br>end</pre><p>This function processes a stream of records by filtering them using my_filter_fn and then mapping the filtered records using my_map_fn. The function receives a stream of records as its input. The filter method applies my_filter_fn to the stream, keeping only the records where isActive equals 1. The map method then applies my_map_fn to each of the filtered records, transforming them into a simplified map containing just the isActive field. <strong><em>The final result is a stream of records that have been both filtered and transformed.</em></strong></p><h4>Registering the Lua UDF with Aerospike</h4><p>Once you’ve written your Lua script, you need to register it with Aerospike. You can do this using the Aerospike Query Language (AQL).</p><p>Open AQL:</p><pre>aql</pre><p>Register the Lua module:</p><pre>REGISTER MODULE &#39;test_udf.lua&#39;</pre><p>Verify the registration:</p><pre>SHOW MODULES</pre><h4>Creating a Secondary Index</h4><p>Before using the UDF to filter data, you need to create a secondary index on the field you plan to filter by. For our example, we’ll create an index on the age field. More about it <a href="https://aerospike.com/docs/server/guide/query">here</a>.</p><h4>Using the UDF in Node.js</h4><p>Now that the UDF is registered and the index is created, you can execute the UDF using Node.js.</p><ol><li>Create an index.js file:</li></ol><pre>const Aerospike= require(&#39;aerospike&#39;);<br><br>const AerospikeClient = Aerospike.client({<br>    hosts: &#39;127.0.0.1:3000&#39;, // aerospike server<br>    log: { level: Aerospike.log.DEBUG }, // show logs<br>    modlua: {userPath: &#39;./udf&#39;} // local path to udf<br>});<br><br>try {<br>    await AerospikeClient.connect();<br>    await getKeys();<br>    await AerospikeClient.close();<br>} catch (e) {<br>    console.log(&#39;Error:&#39;, e);<br>}<br><br>async function getKeys(){<br>    return new Promise((resolve, reject) =&gt; {<br>        try{<br>            let query = AerospikeClient.query(&#39;namespace&#39;, &#39;set&#39;);<br>            // add filter with secondary index<br>            query.where(Aerospike.filter.equal(&#39;age&#39;, 18));<br>            // set udf to be used (it should be present in the udf folder and registered in the aerospike)<br>            query.setUdf( &#39;test_udf&#39;, &#39;test_fn&#39;);<br>            let stream = query.foreach();<br>            let data = [];<br>            stream.on(&#39;data&#39;, (record) =&gt; {<br>                    data.push(record);<br>            });<br>            stream.on(&#39;error&#39;, (error) =&gt; {<br>              console.error(&#39;Stream error:&#39;, error);<br>            });<br>            stream.on(&#39;end&#39;, () =&gt; {<br>                console.log(&#39;Stream end - &#39;, data.length);<br>              return resolve();<br>            });<br>        } catch (err) {<br>            console.log(&#39;getKeys err - &#39;, err);<br>            return reject(err);<br>        }<br>    })<br>}</pre><p>2. Run the script:</p><pre>node index.js</pre><h4>Conclusion</h4><p>Aerospike UDFs offer a powerful way to execute custom logic directly within the database, reducing the need for complex queries or multiple round trips between client and server. By following the steps outlined in this article, you can easily create, register, and execute UDFs using Node.js.</p><h4>Useful links:</h4><ol><li>The full <a href="https://github.com/TeslenkoOleg/node-aerospike-udf/tree/main">code</a> of the example in this article.</li><li>Aerospike UDF <a href="https://aerospike.com/developer/udf/udf_guide">documentation</a></li></ol><p>Thank you for reading. If you have any questions you can find me on <a href="https://www.linkedin.com/in/oleh-teslenko-720443161/">LinkedIn</a>. Feel free to leave comments. You can also read my previous articles:</p><ul><li><a href="https://medium.com/t-slen/no-code-needed-efficiently-download-clean-and-insert-files-into-databases-using-command-line-973e7a485f96">No Code Needed: Efficiently Download, Clean, and Insert Files into Databases Using Command Line</a></li><li><a href="https://levelup.gitconnected.com/handling-100gb-data-in-real-time-with-node-js-and-bloom-filters-in-redis-bdb80746cd96">Handling 100GB Data in Real Time with Node.js and Bloom Filters in Redis</a></li><li><a href="https://medium.com/t-slen/breaking-down-big-data-efficient-search-strategies-for-10-gb-files-2a6f14522853">Breaking Down Big Data: Efficient Search Strategies for 10+GB Files</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4274c27e1bde" width="1" height="1" alt=""><hr><p><a href="https://medium.com/t-slen/aerospike-udf-user-defined-function-4274c27e1bde">Aerospike UDF (User-Defined Function)</a> was originally published in <a href="https://medium.com/t-slen">T-slen</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[No Code Needed: Efficiently Download, Clean, and Insert Files into Databases Using Command Line]]></title>
            <link>https://medium.com/t-slen/no-code-needed-efficiently-download-clean-and-insert-files-into-databases-using-command-line-973e7a485f96?source=rss-2e5f5b703d04------2</link>
            <guid isPermaLink="false">https://medium.com/p/973e7a485f96</guid>
            <category><![CDATA[development]]></category>
            <category><![CDATA[bash]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[shell]]></category>
            <dc:creator><![CDATA[Oleh Teslenko]]></dc:creator>
            <pubDate>Fri, 26 Jul 2024 14:51:36 GMT</pubDate>
            <atom:updated>2024-07-26T14:51:36.788Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to write data from large files into a database?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jv4P3d1io-PLvT3-" /><figcaption>Photo by <a href="https://unsplash.com/@john_matychuk?utm_source=medium&amp;utm_medium=referral">John Matychuk</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Although it sounds like a common task for modern developers, I want to show another view on this point. Instead of immediately starting to write code using your favorite language, importing libraries, and “reinventing the wheel”, I suggest thinking about alternative ways and trying to do it by writing none of the lines of code. Let me show how to use only the command line tools for this task. You would be surprised how powerful it could be sometimes.</p><h4>Benefits of Using Command Line for Data Tasks</h4><ol><li><strong>Speed and Efficiency</strong>: Command-line tools execute tasks directly and with minimal overhead, often faster than equivalent scripts in Python or JavaScript.</li><li><strong>Simplicity</strong>: Many command-line commands are concise and straightforward, allowing you to accomplish tasks with minimal code.</li><li><strong>Resource Usage</strong>: Command-line tools are lightweight and consume less memory, eliminating the need to manage external libraries and dependencies.</li></ol><h4><strong>Download the File</strong></h4><p>The easiest way to do this is to use the command curl. We use curl to download the file from a specified URL:</p><pre>curl -o output_file.tar.gz &quot;https://s3.amazonaws.com/&quot;</pre><p>As curl params, we set -o output_file.tar.gz (specifies the output filename) and URL.</p><h4><strong>Extract the File</strong></h4><p>We use tar command to extract the file:</p><pre>tar -xzf output_file.tar.gz</pre><p>As tar params, we set -xzf (to extract a gzipped tar file) and the output file path.</p><h4><strong>Clean the Data</strong></h4><p>We use sed (a stream editor) to remove single quotes from the CSV file:</p><pre>sed &quot;s/&#39;//g&quot; test.csv &gt; cleaned_test.csv</pre><p>As sed params, we set s/&#39;//g (a regular expression, which you can change for your purpose) command that removes all single quotes from test.csv. As an output file for cleaned data, we use cleaned_test.csv.</p><p><strong>Insert Data into MySQL</strong></p><ol><li>The mysql command is used to load data into a MySQL database:</li></ol><pre>mysql -u user_name -p your_database_name -e &quot;LOAD DATA LOCAL INFILE &#39;/home/user/cleaned_test.csv&#39; INTO TABLE test FIELDS TERMINATED BY &#39;,&#39; ENCLOSED BY &#39;\&quot;&#39; LINES TERMINATED BY &#39;\n&#39; IGNORE 1 ROWS;&quot;</pre><p>Replace user_name, your_database_name, and other placeholders with your MySQL credentials and database details.</p><h3>Automation</h3><p>But how could it work together as well as our hand-made written code? The answer is a <em>Bash (Unix shell)</em>.</p><blockquote>As a command processor, Bash operates within a text window where users input commands to execute various tasks. It also supports the execution of commands from files, known as <a href="https://en.wikipedia.org/wiki/Shell_script">shell scripts</a>, facilitating automation — <a href="https://en.wikipedia.org/wiki/Bash_(Unix_shell)">Wikipedia</a></blockquote><p>Here’s a full code snippet of the shell script that performs the tasks outlined:</p><pre>#!/bin/bash<br><br># URL of the file to download<br>URL=&quot;https://s3.amazonaws.com/&quot;<br><br># Desired output file name<br>OUTPUT_FILE=&quot;archive.tar.gz&quot;<br><br># Download the file with the specified name<br>curl -o $OUTPUT_FILE &quot;$URL&quot;<br><br># Extract the file<br>tar -xzf $OUTPUT_FILE<br><br># Use stream editor to remove single quotes from the file<br>sed &quot;s/&#39;//g&quot; test.csv &gt; cleaned_test.csv<br><br># Insert the data into the database CH<br>clickhouse-client --host=200.12.12.12 --user=user_name --password=&#39;password&#39; --query=&quot;INSERT INTO test.test_csv_table FORMAT CSVWithNames&quot; &lt; cleaned_test.csv<br><br># Insert the data into the database MySQL<br>mysql -u user_name -p your_database_name -e &quot;<br>LOAD DATA LOCAL INFILE &#39;/home/user/cleaned_test.csv&#39;<br>INTO TABLE test<br>FIELDS TERMINATED BY &#39;,&#39;<br>ENCLOSED BY &#39;\&quot;&#39;<br>LINES TERMINATED BY &#39;\n&#39;<br>IGNORE 1 ROWS;&quot;</pre><p>This shell script provides a straightforward way to automate the process of downloading a file, cleaning its content, and inserting it into the MySQL database. To execute this script run the next commands:</p><pre>chmod +x index.sh<br>./index.sh</pre><p>Useful links:</p><ol><li>My GitHub <a href="https://github.com/TeslenkoOleg/bash-curl-stream-file">repository</a> with the complete script and additional details.</li><li><a href="https://en.wikibooks.org/wiki/Bash_Shell_Scripting">Bash Shell Scripting</a> by Wikibooks Contributors.</li></ol><h4>Conclusion</h4><p>By leveraging bash scripting and command-line tools, you can efficiently handle tasks like downloading, cleaning, and inserting data into a database without writing a single line of application code. This method not only saves development time but also utilizes the powerful features of Unix-based systems, making your workflow simpler and more efficient. So, the next time you face a similar challenge, consider reaching for your terminal instead of your IDE.</p><p>Thank you for reading. If you have any questions you can find me on <a href="https://www.linkedin.com/in/oleh-teslenko-720443161/">LinkedIn</a>. Feel free to leave comments. You can also read my previous articles:</p><ul><li><a href="https://medium.com/t-slen/breaking-down-big-data-efficient-search-strategies-for-10-gb-files-2a6f14522853">Breaking Down Big Data: Efficient Search Strategies for 10+GB Files</a></li><li><a href="https://levelup.gitconnected.com/avoid-common-mistakes-using-a-docker-container-b661233d3905">Avoid Common Mistakes Using a Docker Container</a></li><li><a href="https://levelup.gitconnected.com/handling-100gb-data-in-real-time-with-node-js-and-bloom-filters-in-redis-bdb80746cd96">Handling 100GB Data in Real Time with Node.js and Bloom Filters in Redis</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=973e7a485f96" width="1" height="1" alt=""><hr><p><a href="https://medium.com/t-slen/no-code-needed-efficiently-download-clean-and-insert-files-into-databases-using-command-line-973e7a485f96">No Code Needed: Efficiently Download, Clean, and Insert Files into Databases Using Command Line</a> was originally published in <a href="https://medium.com/t-slen">T-slen</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Breaking Down Big Data: Efficient Search Strategies for 10+GB Files]]></title>
            <link>https://medium.com/t-slen/breaking-down-big-data-efficient-search-strategies-for-10-gb-files-2a6f14522853?source=rss-2e5f5b703d04------2</link>
            <guid isPermaLink="false">https://medium.com/p/2a6f14522853</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[mapreduce]]></category>
            <category><![CDATA[algorithms]]></category>
            <dc:creator><![CDATA[Oleh Teslenko]]></dc:creator>
            <pubDate>Sun, 16 Jun 2024 17:45:06 GMT</pubDate>
            <atom:updated>2024-06-16T17:45:06.628Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MvH6f73KgVTOCoKM" /><figcaption>Photo by <a href="https://unsplash.com/@nexgenfx?utm_source=medium&amp;utm_medium=referral">Luca Campioni</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>How to find a unique string in a big-sized file? Imagine that you work at Google and you have a huge data set of unique users stored in files. One day you come to the office and find out the new task — to create a service that can find any user by email and return the user’s name. The main condition is that this service should be as fast as possible and can work in real-time (10K requests per second).</p><p>There are several different solutions depending on your needs. Let’s discuss possible ways to do this:</p><h4>Relational Database</h4><p>Obviously, relation databases such as <a href="https://www.mysql.com/">MySQL</a> or <a href="https://www.postgresql.org/">PostgreSQL</a> could manage to do this task. All we need is to use an index for one column, don’t we? But would it be as effective and fast as we want for 10K requests per second? The short answer is no.</p><h4>Non-relational Database</h4><p>We can use key-value databases to store all data in them. For example, it could be Aerospike or Redis. These systems are often used for working with data in real-time. They efficiently store data in RAM that helps them to return data with high speed. However, there could be one drawback. Our files could take hundreds of GB and our memory space would increase significantly.</p><h4>Bloom Filter</h4><p>Bloom Filters are probabilistic data structures that offer space-efficient membership testing. They use space cleverly, allowing you to check if an item <em>might</em> be present in a dataset without storing the entire dataset itself. This approach could be a very effective solution if we can tolerate at least 1% false positives. More about this approach is below in the article More about this approach is below in the article:</p><p><a href="https://levelup.gitconnected.com/handling-100gb-data-in-real-time-with-node-js-and-bloom-filters-in-redis-bdb80746cd96">Handling 100GB Data in Real Time with Node.js and Bloom Filters in Redis</a></p><h4>Binary Search</h4><blockquote>Binary search is a search algorithm that finds the position of a target value within a sorted array — Wikipedia</blockquote><p>We a little bit rearrange this definition for a sorted file. So the main idea is that if we have sorted data structure we can use this type of algorithms. In this article, I want to describe this way to find a unique string in a large file.</p><h4>How Does It Work?</h4><p>First, we need to create a sorted file. The best way to do this is to use the <a href="https://en.wikipedia.org/wiki/MapReduce">MapReduce</a> algorithm.</p><blockquote><strong>MapReduce</strong> is a programming model and an associated implementation for processing and generating big data sets with a parallel, distributed algorithm on a cluster. — Wikipedia</blockquote><p>So we need to divide our process into two steps — map and reduce. To simplify, we only use the “map” part in this article.</p><h4>Unsorted file (data.csv)</h4><pre>apple,25,Ukraine<br>grape,345,France<br>pineapple,15,France<br>mandarin,22,Spain<br>mango,45,Ecuador<br>strawberry,12,Spain<br>blackberry,10,France<br>raspberry,20,Poland<br>cherries,50,Spain</pre><h4>Map</h4><p>We use the map method for formatting a data in file.</p><pre>// map.js<br><br>import fs from &#39;fs/promises&#39;;<br><br>const filename = &#39;./data.csv&#39;;<br><br>async function main() {<br>  const fileHandle = await fs.open(filename, &#39;r&#39;);<br>  for await (const line of fileHandle.readLines()) {<br>    const splitLine = line.split(&#39;,&#39;);<br>    const ip = splitLine[0];<br>    const fraudPercentage = splitLine[2];<br>    process.stdout.write(ip + &#39;\t&#39; + fraudPercentage + &#39;\n&#39;)<br>  }<br>}<br><br>main();</pre><p>Next, run the script in your terminal:</p><pre>node map.js &gt;./pairs.data</pre><p>After that, we will have the next file — <em>pairs.data</em>:</p><pre>apple   Ukraine<br>grape   France<br>pineapple   France<br>mandarin    Spain<br>mango   Ecuador<br>strawberry  Spain<br>blackberry  France<br>raspberry   Poland<br>cherries    Spain</pre><p>Next, we need to sort it out. Run the next command in your terminal:</p><pre>sort -k1,1 ./pairs.data &gt;./sorted.data</pre><p>Let’s look at our binary search script:</p><pre>// index.js<br><br>import fs from &#39;fs/promises&#39;;<br><br>const args = process.argv.slice(2);<br><br>if (args.length !== 2) {<br>  console.log(&quot;No all arguments provided. See README.md for more information.&quot;);<br>  process.exit();<br>}<br>const SEARCH_VALUE = args[0];<br>const DEFAULT_CHUNK_SIZE = +args[1]; // depends on the data size and the system memory<br>const DEFAULT_COMPARE_FN = async (searchValue, candidateValue) =&gt; {<br>  if (searchValue &gt; candidateValue) {<br>    return 1;<br>  } else if (searchValue &lt; candidateValue) {<br>    return -1;<br>  } else {<br>    return 0;<br>  }<br>};<br><br><br>const DEFAULT_DELIMITER = &quot;\n&quot;;<br>const FILENAME = &#39;sorted.data&#39;;<br><br>console.time(&#39;binarySearchInFile&#39;);<br>binarySearchInFile({<br>  filename: FILENAME,<br>  searchValue: SEARCH_VALUE,<br>})<br>    .then(console.log)<br>    .then(() =&gt; console.timeEnd(&#39;binarySearchInFile&#39;));<br><br>export async function readChunk({ fileHandle, chunkSize, position }) {<br>  const { buffer, bytesRead } = await fileHandle.read({<br>    buffer: Buffer.alloc(Number(chunkSize)),<br>    length: Number(chunkSize),<br>    position: Number(position)<br>  });<br><br>  if (bytesRead === chunkSize) {<br>    return buffer;<br>  } else {<br>    const bufferCorrectSize = Buffer.allocUnsafe(bytesRead);<br>    buffer.copy(bufferCorrectSize, 0, 0, bytesRead);<br>    return bufferCorrectSize;<br>  }<br>}<br><br>export async function binarySearchInFile({<br>  filename,<br>  searchValue,<br>  compareFn = DEFAULT_COMPARE_FN,<br>  chunkSize = DEFAULT_CHUNK_SIZE,<br>  delimiter = DEFAULT_DELIMITER,<br>} = {}) {<br>  if (!filename) throw new Error(&#39;&quot;filename&quot; required&#39;);<br>  if (!searchValue) throw new Error(&#39;&quot;searchValue&quot; required&#39;);<br><br>  const fileHandle = await fs.open(filename, &#39;r&#39;);<br>  const stats = await fileHandle.stat();<br><br>  let start = 0;<br>  let end = stats.size;<br><br>  while (start &lt; end) {<br>    let mid = Math.floor((start + end) / 2);<br><br>    let readMore = true;<br>    let data = &#39;&#39;;<br>    let parts = [];<br>    let position = mid;<br><br>    while (readMore) {<br>      data += await readChunk({ fileHandle, chunkSize, position });<br>      parts = data.split(delimiter);<br>      position += chunkSize;<br>      readMore = (mid === 0 &amp;&amp; parts.length &lt; 2) || parts.length &lt; 3;<br>      if (position &gt; stats.size) {<br>        await fileHandle.close();<br>        return { found: false };<br>      }<br>    }<br><br>    const midLine = mid === 0 ? parts[0] : parts[1];<br>    const cmpResult = await compareFn(searchValue, midLine.split(&#39;\t&#39;)[0]);<br><br>    if (cmpResult === 0) {<br>      await fileHandle.close();<br>      return { found: true, record: midLine, searchValue };<br>    } else if (cmpResult === 1) {<br>      start = mid;<br>    } else if (cmpResult === -1) {<br>      end = mid;<br>    } else {<br>      throw new Error(&#39;&quot;compareFn&quot; should return 0|1|-1&#39;)<br>    }<br><br>    if (start === mid &amp;&amp; (end - start) === 1) {<br>      await fileHandle.close();<br>      return { found: false, searchValue };<br>    }<br>  }<br><br>  await fileHandle.close();<br>  return { found: false, record: null };<br>}</pre><p>Run <em>index.js</em> to find the key passing arguments — “apple” is the key and 4 — the number of chunks bytes in the file, depending on the data size and the system memory:</p><pre>node index.js apple 4</pre><p>Useful links:</p><ol><li>Original <a href="https://github.com/koorchik/jabascript-materials/tree/main/inverted-index">code</a> example</li><li>The <a href="https://github.com/TeslenkoOleg/node-binary-search-in-file">code</a> provided in this article (there you can also find examples of typescript and using callbacks instead of async/await)</li><li>Apache <a href="https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html">MapReduce</a> documentation</li></ol><p>Thank you for reading. If you have any questions you can find me on <a href="https://www.linkedin.com/in/oleh-teslenko-720443161/">LinkedIn</a>. Feel free to leave comments. You can also read my previous articles:</p><ul><li><a href="https://levelup.gitconnected.com/how-to-build-responsive-layouts-using-angular-react-or-vue-23968710aab8">How to Build Responsive Layouts using Angular, React, or Vue</a></li><li><a href="https://levelup.gitconnected.com/avoid-common-mistakes-using-a-docker-container-b661233d3905">Avoid Common Mistakes Using a Docker Container</a></li><li><a href="https://towardsdev.com/ram-in-node-memory-leak-5cfca356ddcc">Memory Leak in Node.js Apps</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a6f14522853" width="1" height="1" alt=""><hr><p><a href="https://medium.com/t-slen/breaking-down-big-data-efficient-search-strategies-for-10-gb-files-2a6f14522853">Breaking Down Big Data: Efficient Search Strategies for 10+GB Files</a> was originally published in <a href="https://medium.com/t-slen">T-slen</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Build Responsive Layouts using Angular, React, or Vue]]></title>
            <link>https://levelup.gitconnected.com/how-to-build-responsive-layouts-using-angular-react-or-vue-23968710aab8?source=rss-2e5f5b703d04------2</link>
            <guid isPermaLink="false">https://medium.com/p/23968710aab8</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[angular]]></category>
            <category><![CDATA[react]]></category>
            <category><![CDATA[typescript]]></category>
            <category><![CDATA[vuejs]]></category>
            <dc:creator><![CDATA[Oleh Teslenko]]></dc:creator>
            <pubDate>Tue, 09 Apr 2024 16:35:44 GMT</pubDate>
            <atom:updated>2024-04-09T22:03:09.283Z</atom:updated>
            <content:encoded><![CDATA[<h4>Creating Interactive Dashboards in Minutes</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OsU5GimE3tW9dEiY" /><figcaption>Photo by <a href="https://unsplash.com/@olloweb?utm_source=medium&amp;utm_medium=referral">Agence Olloweb</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Although creating a nice interactive dashboard sounds like a common task for frontend developers sometimes it could be really tricky. Building functionalities like resizing, dragging, and rearranging layout elements can quickly become a complex endeavor. This article explores how to simplify this process using an open-source library that lets you create stunning draggable, resizable, and responsive layouts with minimal code.</p><p>First of all, let’s take a quick look at the desired result and behavior:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*aNjht7rkgikQHQc5IU_YVw.gif" /><figcaption><a href="https://teslenkooleg.github.io/angular-interactive-dashboard/">DEMO</a></figcaption></figure><h4><strong>The Power of Gridstack.js</strong></h4><p>This magic we can achieve this using an open-source project — <a href="https://gridstackjs.com/"><strong>gridstack.js</strong></a><strong> </strong>(5.8K stars on <a href="https://github.com/gridstack/gridstack.js">GitHub</a>).</p><blockquote>gridstack.js is a modern Typescript library designed to help developers create beautiful draggable, resizable, responsive layouts with just a few lines of code.<br>You can create more advanced grids: allow users to persist changes, create a sidebar of widgets to drag into your grid, nested grids, and much more.</blockquote><p>Here is what this library provides:</p><ul><li>pure TS can be used in any UI framework</li><li>mobile support</li><li>draggable, resizable items</li><li>drag-drop from the sidebar to insert/delete</li><li>responsive layouts, column-based</li><li>save/restore</li><li>drag between multiple grids</li><li>nested grids, including creation on the fly to any depth</li><li>Angular (included), demo and external Vue, React, Ember, Knockout.js bindings, and more</li></ul><h4><strong>Getting Started with Gridstack.js in Angular</strong></h4><p>I would like to show how to use gridstack.js in an Angular application. First of all, you need to install gridstack.js from a package manager:</p><pre>yarn add gridstack<br>// or<br>npm install --save gridstack</pre><p>Import <em>gridstack</em> styles into your application:</p><pre>@import &quot;gridstack/dist/gridstack.min.css&quot;;</pre><p>Add the next code to your <em>app.component.html</em>:</p><pre><br>  &lt;gridstack [options]=&quot;gridOptions&quot; (changeCB)=&quot;onChange($event)&quot;&gt;<br>    @for (gridItem of items; track gridItem.id){<br>      &lt;gridstack-item [options]=&quot;gridItem&quot;&gt;<br>        &lt;div class=&quot;item-content&quot;&gt;<br>          Item {{gridItem.id}}<br>        &lt;/div&gt;<br>      &lt;/gridstack-item&gt;<br>    }<br>  &lt;/gridstack&gt;</pre><p>If you want you can add some styles to your <em>app.component.scss</em>:</p><pre>gridstack {<br>  height: 500px;<br>  background: #FAFAD2;<br>  margin: 10px;<br>  padding: 40px;<br>}<br>.item-content {<br>  text-align: center;<br>  background-color: #c9f199;<br>  border: 1px solid #000;<br>  border-radius: 5px;<br>  height: 100%;<br>}</pre><p>And finally, add typescript logic in your app.component.ts:</p><pre>import { Component } from &#39;@angular/core&#39;;<br>import { RouterOutlet } from &#39;@angular/router&#39;;<br>import {GridStackOptions, GridStackWidget} from &quot;gridstack&quot;;<br>import {GridstackModule, nodesCB} from &quot;gridstack/dist/angular&quot;;<br><br>@Component({<br>  selector: &#39;app-root&#39;,<br>  standalone: true,<br>  imports: [RouterOutlet, GridstackModule],<br>  templateUrl: &#39;./app.component.html&#39;,<br>  styleUrl: &#39;./app.component.scss&#39;<br>})<br>export class AppComponent {<br>  public gridOptions: GridStackOptions = { margin: 5 }<br>  public items: GridStackWidget[] = [<br>    {x:0, y:0, w:4, id:&#39;1&#39;},<br>    {x:1, y:0, w:3, id:&#39;2&#39;},<br>    {x:0, y:1, h:4, w:4, id:&#39;3&#39;},<br>    {x:3, y:1, w:3, id:&#39;4&#39;},<br>    {x:0, y:2, w:2, id:&#39;5&#39;},<br>  ];<br><br>  public onChange(data: nodesCB) {<br>    console.log(&#39;change &#39;, data.nodes.length &gt; 1 ? data.nodes : data.nodes[0]);<br>  }<br><br>}</pre><p>Let’s take a closer look at GridStackOptions. It defines options for the parent grid and provides a huge amount of customization. For example, you can set <em>cellHeight </em>option that defines a minimum step for scaling your grid item.</p><p>GridStackWidget is an interface for gridstack children. You can set minimal and maximum weight, height, moving options, and others. For instance, you can define the grid item position (x, y, w, h). You should be aware that the <em>X position</em> has 12 cells by default.</p><p>You can also track changes with the <em>onChange</em> method that provides information about changed grids:</p><pre>public onChange(data: nodesCB) {<br>    console.log(&#39;change &#39;, data.nodes.length &gt; 1 ? data.nodes : data.nodes[0]);<br>  }</pre><h4><strong>Conclusion</strong></h4><p>Gridstack.js empowers developers to create dynamic and user-friendly dashboards with minimal effort. Its versatility and cross-framework compatibility make it a valuable tool for building modern and responsive web applications. By utilizing gridstack.js, you can significantly reduce development time and deliver exceptional user experiences.</p><h4>Useful sources</h4><ol><li>Gridstack <a href="https://gridstackjs.com/demo/">demos</a> for React, Vue, and others</li><li>Gridstack API <a href="https://github.com/gridstack/gridstack.js/tree/master/doc">documentation</a></li><li>Angular wrapper <a href="https://github.com/gridstack/gridstack.js/tree/master/angular">documentation</a></li><li>The application code source of this article is on my <a href="https://github.com/TeslenkoOleg/angular-interactive-dashboard">GitHub</a></li><li>The <a href="https://teslenkooleg.github.io/angular-interactive-dashboard/">demo</a> of my application</li></ol><p>Thank you for reading. If you have any questions you can find me on <a href="https://www.linkedin.com/in/oleh-teslenko-720443161/">LinkedIn</a>. Feel free to leave comments. You can also read my previous articles:</p><ul><li><a href="https://levelup.gitconnected.com/avoid-common-mistakes-using-a-docker-container-b661233d3905">Avoid Common Mistakes Using a Docker Container</a></li><li><a href="https://levelup.gitconnected.com/handling-100gb-data-in-real-time-with-node-js-and-bloom-filters-in-redis-bdb80746cd96">Handling 100GB Data in Real Time with Node.js and Bloom Filters in Redis</a></li><li><a href="https://levelup.gitconnected.com/how-to-create-npm-library-using-existing-angular-code-f6adc491693a">How to Create NPM Library Using Existing Angular Code</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23968710aab8" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/how-to-build-responsive-layouts-using-angular-react-or-vue-23968710aab8">How to Build Responsive Layouts using Angular, React, or Vue</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Avoid Common Mistakes Using a Docker Container]]></title>
            <link>https://levelup.gitconnected.com/avoid-common-mistakes-using-a-docker-container-b661233d3905?source=rss-2e5f5b703d04------2</link>
            <guid isPermaLink="false">https://medium.com/p/b661233d3905</guid>
            <category><![CDATA[development]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Oleh Teslenko]]></dc:creator>
            <pubDate>Mon, 11 Mar 2024 02:11:44 GMT</pubDate>
            <atom:updated>2024-03-11T02:11:44.567Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cPfapjQpQOIco7WG" /><figcaption>Photo by <a href="https://unsplash.com/@carrier_lost?utm_source=medium&amp;utm_medium=referral">Ian Taylor</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Understanding Docker’s core concepts, particularly image creation, container management, and file relationships is crucial for leveraging its full potential. At the same time, careless use of Docker’s commands could increase your machine&#39;s memory significantly. This article delves into key aspects and represents how to create, stop, and restart node.js applications avoiding common mistakes.</p><h4><strong>Prerequisites:</strong></h4><ul><li><strong>Node.js application:</strong> You can either create a new Node.js project or use one found on <a href="https://github.com/TeslenkoOleg/docker-node-app">GitHub</a>.</li><li><strong>Docker:</strong> Ensure you have Docker version <a href="https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-18-04">18.04</a> or <a href="https://docs.docker.com/engine/install/ubuntu/">later</a> installed.</li></ul><p>To work with Docker you need to add settings files:</p><ul><li>Dockerfile.</li><li>compose.yaml (docker-compose.yaml in early Docker versions).</li><li>.dockerignore (optional).</li></ul><h4><strong>Creating an image:</strong></h4><p>The cornerstone of building a Dockerized Node.js application lies in the <strong>Dockerfile</strong>. This text file specifies the instructions Docker follows to create an image. Let’s dissect a basic example:</p><pre><br>FROM node:18  # Base image with Node.js 18<br><br>WORKDIR /app  # Working directory within the container</pre><p>Explanation of Dockerfile lines:</p><ol><li>FROM node:18<strong>:</strong> Defines the base image, inheriting the operating system and Node.js version (18 in this case).</li><li>WORKDIR /app<strong>:</strong> Sets the working directory within the container, typically the root directory of your application.</li></ol><p>Another possible commands, <em>but we won’t use them in this article</em>:</p><ol><li>COPY package*.json ./<strong>:</strong> Copies package.json and package-lock.json files into the container&#39;s working directory.</li><li>RUN npm install<strong>:</strong> Installs the dependencies listed in package.json.</li><li>COPY . .<strong>:</strong> Copies the remaining application code from the current directory into the container&#39;s working directory.</li><li>EXPOSE 3000<strong>:</strong> Exposes port 3000 from the container, allowing external communication with the application.</li><li>CMD [&quot;npm&quot;, &quot;start&quot;]<strong>:</strong> Sets the default command that runs when a container instance is created based on this image.</li></ol><h4><strong>Building an image:</strong></h4><p>To create an image based on your Dockerfile, use the following command in your terminal, assuming your Dockerfile is named Dockerfile:</p><pre>docker build .</pre><p>This command instructs Docker to use the current directory (.) as the build context (location of Dockerfile and application files).</p><p><strong><em>Note:</em></strong> <em>You should be careful with the command above because it creates a new image every time you run it.</em></p><p>You also may want to build the image without cache:</p><pre>docker build --no-cache .</pre><h4><strong>Create a container :</strong></h4><p>While building images is essential, managing complex applications often involves multiple containers and services. Here’s where <strong>docker compose (docker-compose) </strong>comes in. It allows you to define and orchestrate multi-container applications using a YAML file called compose.yaml or docker-compose.yaml.:</p><pre>version: &#39;3.7&#39;<br>services:<br>  node-app:<br>    build:<br>      context: .<br>      dockerfile: Dockerfile<br>      target: base<br>    volumes:<br>      - .:/app<br>    container_name: node-app<br>    ports:<br>      - &#39;3001:3001&#39;<br>    command: sh -c &quot;npm install &amp;&amp; node ./build/index.js&quot;</pre><p>Explanation of each command:</p><ol><li>version: &#39;3.7&#39;: Specifies the version of the Docker Compose file syntax being used. In this case, it&#39;s version 3.7.</li><li>services:: Indicates the start of the services definition block.</li><li>node-app:: Defines a service named &quot;node-app.&quot; This service is essentially a containerized Node.js application.</li><li>build:: Specifies the build configuration for the Docker image of the &quot;node-app&quot; service.</li></ol><ul><li>context: .: Sets the build context to the current directory, meaning the build process will use the current directory as the source for building the Docker image.</li><li>dockerfile: Dockerfile: Specifies the Dockerfile to use for building the image. In this case, it is assumed that there is a file named &quot;Dockerfile&quot; in the build context.</li><li>target: base: Sets the target build stage to &quot;base.&quot; This is useful when the Dockerfile has multiple build stages, and it specifies which stage to use for the final image.</li></ul><p>5. volumes:: Mounts a volume from the host machine to the &quot;/app&quot; directory inside the container.</p><ul><li>- .:/app: Maps the current directory (&quot;.&quot;) on the host machine to the &quot;/app&quot; directory inside the container.</li></ul><p>6. container_name: node-app: Specifies the name for the container, which will be &quot;node-app.&quot;</p><p>7. ports:: Maps host machine ports to container ports.</p><ul><li>- &#39;3001:3001&#39;: Maps port 3001 on the host machine to port 3001 inside the container.</li></ul><p>8. command:: Overrides the default command to be executed when the container starts.</p><ul><li>sh -c &quot;npm install -f &amp;&amp; npm run tsc &amp;&amp; node ./build/index.js&quot;: Runs a shell command within the container. The command installs npm dependencies (npm install) and finally executes the application (node ./build/index.js).</li></ul><p><strong><em>Note</em></strong><em>: Running node.js application using </em>node<em>instead of </em>pm2<em>is recommended.</em></p><p><strong>Running a Docker Container:</strong></p><p>Once you have an image, you can create a container based on it using the docker compose up -d command.</p><p><strong><em>Note:</em></strong> <em>You should be careful with the command above because it creates a new container every time you run it.</em></p><p>If you want to restart exist container without creating a new one, you should stop it:</p><pre>docker compose down</pre><p>and then use this command:</p><pre>docker compose up --force-recreate --build -d</pre><h4>Monitoring:</h4><p>Let’s check what we have already done. The next command provides information about running docker instances:</p><pre>docker system df</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*iO9bKAECKDQUD9eRBF9G2g.png" /><figcaption>result of docker system df</figcaption></figure><p><strong><em>Note:</em></strong><em> You should monitor your created instances regularly. Increasing numbers of images, containers, caches, etc. could be the reason for filling your memory unexpectedly.</em></p><p>To see information of your active containers use:</p><pre>docker ps</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*sJksUlLTtYlv8fvvz1eKNw.png" /><figcaption>the result of docker ps</figcaption></figure><p>To see the container&#39;s real-time logs use the container ID:</p><pre>docker logs --follow ec479385f3fd</pre><p>To execute commands inside the container use:</p><pre>docker exec -it ec479385f3fd sh -c &quot;echo a &amp;&amp; echo b&quot;</pre><p>To remove all your <strong>inactive</strong> docker instances use:</p><pre>docker system prune --volumes -af</pre><h4>Caching:</h4><p>Caching processes are really important in using dockers, but it is really a huge topic for other research. I recommend reading this <a href="https://depot.dev/blog/docker-clear-cache">material</a> that covers most of the scenarios working with a cache.</p><h4>Documentation:</h4><ol><li><a href="https://docs.docker.com/manuals/">Official Docker</a>.</li><li><a href="https://nodejs.org/en">Official Node.js</a>.</li><li><a href="https://depot.dev/blog/docker-clear-cache">How to clear Docker cache and free up space on your system</a>.</li></ol><h4>Conclusion:</h4><p>In summary, understanding Docker basics is crucial for getting the most out of Node.js apps. This article breaks down key Docker concepts like creating images, managing containers, and dealing with files. By carefully following Dockerfile and docker-compose setups, developers can create strong and scalable environments. Think of Docker images as the building blocks, perfectly wrapping up Node.js apps. Docker-compose is like the conductor, managing the complexity of your app setup. Just be cautious when using Docker commands to avoid using up too much memory. With insights on building images, creating containers, and keeping an eye on things, developers can smoothly use Docker to integrate Node.js apps into the world of containers.</p><p>Thank you for reading. If you have any questions you can find me on <a href="https://www.linkedin.com/in/oleh-teslenko-720443161/">LinkedIn</a>. Feel free to leave comments. You can also read my previous articles:</p><ul><li><a href="https://levelup.gitconnected.com/handling-100gb-data-in-real-time-with-node-js-and-bloom-filters-in-redis-bdb80746cd96">Handling 100GB Data in Real Time with Node.js and Bloom Filters in Redis</a></li><li><a href="https://levelup.gitconnected.com/automatic-application-deployment-using-bitbucket-pipelines-9fa8680e4371">Automatic Application Deployment Using Bitbucket Pipelines</a></li><li><a href="https://towardsdev.com/encrypt-decrypt-password-node-js-d8be1ccb42cf">Encrypt &amp; decrypt password | Node.js</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b661233d3905" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/avoid-common-mistakes-using-a-docker-container-b661233d3905">Avoid Common Mistakes Using a Docker Container</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>