<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Faraz Faraji on Medium]]></title>
        <description><![CDATA[Stories by Faraz Faraji on Medium]]></description>
        <link>https://medium.com/@faraz-faraji?source=rss-d5a8fe51f700------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*H5vv5cBqprCDeD1xU_9R7w.jpeg</url>
            <title>Stories by Faraz Faraji on Medium</title>
            <link>https://medium.com/@faraz-faraji?source=rss-d5a8fe51f700------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 23:32:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@faraz-faraji/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Design a reliable IoT architecture]]></title>
            <link>https://faraz-faraji.medium.com/design-a-reliable-iot-architecture-0536d532859f?source=rss-d5a8fe51f700------2</link>
            <guid isPermaLink="false">https://medium.com/p/0536d532859f</guid>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[Faraz Faraji]]></dc:creator>
            <pubDate>Thu, 15 Aug 2024 21:10:34 GMT</pubDate>
            <atom:updated>2024-08-25T02:57:07.777Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bCIhuA_l80NxIHytSJ0xlg.png" /></figure><p>When designing IoT architectures, it’s crucial to thoroughly explore various factors and think more about them before implementing the system. This comprehensive approach helps us identify and implement the most effective solutions.</p><p>IoT involves handling massive amounts of data quickly and securely. Building a system to receive, process, and respond to this amount of information is challenging. There are many ways to approach this, but I want to share our approach at <a href="http://fasterbase.com/"><strong>Fasterbase</strong></a><strong> </strong>our latest startup project.</p><h3>Connectivity</h3><p>This is a critical component of our system. Devices send us messages, and our service must receive, process, distribute, and store them as quickly as possible.</p><h4>MQTT</h4><p>We chose <strong>RabbitMQ</strong> to handle incoming messages, but we needed a way for all type of devices to send data to our servers. Since our customers use different hardware, we decided on MQTT to support many IoT devices with different structures and OS.</p><p><strong>Mosquitto</strong> is known as a lightweight, fast, and secure MQTT broker. However, we decided to use <a href="https://www.hivemq.com/">HiveMQ</a> to handle this part of our system so we could focus on developing our core features.</p><p>We carefully configured Mosquitto with a <strong>QoS level of 2</strong> to prioritize reliability over speed.</p><p>To strengthen security, we enabled <strong>TLS</strong> and added <strong>JWT</strong> token verification within the payload to identify devices for each message.</p><p>Choosing effective <strong>topic names </strong>can be challenging. The best approach depends on your specific needs. In our system, with multiple clients, customers, and devices, we use the /client_id/device_id/customer_id/event_id pattern to <strong>route</strong> MQTT messages efficiently.</p><h4>RabbitMQ</h4><p>While MQTT handles device-to-server communication, we needed a way to distribute messages between our internal services (processor, log, data provider) and client instances. To improve MQTT performance and distribute the load, we integrated <strong>RabbitMQ</strong> as a message broker.</p><p>Incoming MQTT messages are directly pushed into <strong>RabbitMQ</strong>, allowing for efficient message routing and processing.</p><h3>Data Storage</h3><p>We use a multi-storage approach using three different services. Our tenant-based strategy assigns each client an independent database, so each client&#39;s information is in a separate database with different types of databases.</p><h4>Cold Storage</h4><p>We set up Cassandra for direct log storage. Due to Cassandra’s query limitations, we index essential data for filtering while storing the remaining information in a non-indexed column. Cold storage is read-only, serving as an archive for historical data.</p><h4>Warm Storage</h4><p>Warm storage holds recently accessed data for rapid retrieval. For instance, consider a temperature sensor that triggers AC activation when the last 5 minutes’ average exceeds 25 degrees.</p><p>We store the past 5 minutes of temperature data in a time-series <strong>MongoDB</strong> database (our warm storage) for quick access. To further optimize performance, we cache calculated data in Redis.</p><h4>Devices Configuration</h4><p>We employ a <strong>tenant-based</strong> storage strategy, providing each client with an isolated database. For other data, we use <strong>MongoDB</strong> due to its flexibility in handling diverse device schemas defined by our customers.</p><h3>Conclusion</h3><p>There are numerous ways to design IoT architecture. While cloud giants like Google, Amazon, and Microsoft offer comprehensive IoT platforms, there are also low-code solutions like our <a href="http://fasterbase.com/">Fasterbase</a> that accelerate backend development. Ultimately, the best approach depends on specific project needs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0536d532859f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Innovation in IoT With “Fasterbase”]]></title>
            <link>https://faraz-faraji.medium.com/innovation-in-iot-with-fasterbase-005e3bf5c517?source=rss-d5a8fe51f700------2</link>
            <guid isPermaLink="false">https://medium.com/p/005e3bf5c517</guid>
            <category><![CDATA[investment]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[smart-home]]></category>
            <category><![CDATA[platform]]></category>
            <dc:creator><![CDATA[Faraz Faraji]]></dc:creator>
            <pubDate>Mon, 22 Jul 2024 21:23:56 GMT</pubDate>
            <atom:updated>2024-07-22T21:23:56.333Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*9kpb6FseNeNiT6xYk27b6A.png" /></figure><h3>The Story</h3><p>Let me start by telling a story behind Fasterbase.</p><p>A few years ago, I <strong>founded a company</strong> to produce a smart IoT device. After securing funding from an investor, I began creating the first MVP version. However, after six months, I realized I was <strong>running out of fund</strong>s. Despite this, we struggled for another four months with fundamental challenges like<strong> device communication</strong>, <strong>application integration</strong>, and <strong>security</strong>. Unfortunately, these obstacles <strong>prevented</strong> us from progressing on the core <strong>IoT innovation</strong>.</p><p>In discussions with other CEOs, I discovered a common challenge: They were spending a significant<strong> amount of time</strong> on the <strong>foundational</strong> aspects of IoT devices, potentially losing sight of their core innovation. This, I believe, is a contributing factor to the <strong>struggles of many IoT companies.</strong></p><p>When I discovered this problem, I identified a more significant challenge within the IoT development space. I decided to focus my efforts on <strong>developing a solution</strong>: a service that <strong>helps IoT entrepreneurs</strong>. This service should s<strong>peed up the implementation</strong> of fundamental functionalities, allowing them to dedicate their resources to their <strong>core innovations.</strong></p><h3>Fasterbase</h3><p>Fasterbase is a <strong>SaaS platform</strong> for IoT devices. It helps device producers create your logic by dragging and dropping blocks. You can customize it as much as you want and add steps to make it work.</p><p>When your logic is ready, you can download the <strong>SDK</strong> and upload it to devices. Fasterbase provides many tools such as <strong>Data Storage</strong>, <strong>Cloud functions, Notification System </strong>and <strong>integrations</strong> to be the most flexible low-code platform in this section.</p><p>Additionally, we offer a <strong>web</strong> and <strong>mobile application</strong> <strong>builder</strong> to let administrators create amazing tools for their customers to manage their devices. With our user-friendly tools, startups can accelerate growth and focus on their core innovations.</p><h3>Why Fasterbase?</h3><p>Our service lets you create <strong>any number</strong> of devices under your brand and design their logic easily. Once finished, publish your design and get an <strong>SDK </strong>for popular languages like<strong> C++</strong> or <strong>Python</strong>. Load the SDK onto your device, let it to connect the internet, and voilaaa! Now, your device will automatically show up in your pre-designed app, ready for customer.</p><p>Skip <strong>2500</strong> hours of development! Fasterbase is here to help. We provide the <strong>infrastructure</strong>, <strong>security</strong>, <strong>Data storage</strong>,<strong> IntgrA </strong>and application generation for your IoT device — all with a single click. Focus on your innovation, let Fasterbase handle the rest!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vCe2kAabmYYyWxVKK-oIJw.png" /></figure><h3><strong>Data Storage</strong></h3><p>No more worrying about device logs and data <strong>storage</strong>! Fasterbase offers <strong>unlimited</strong> storage to keep everything you need: <strong>logs</strong>, <strong>customer data</strong>, <strong>settings</strong>, <strong>device data</strong>, and more. Just open your dashboard and define the schma using our easy-to-use tools. Don’t worry about the <strong>security</strong> because we <strong>encode</strong> all data with a <strong>pre-defined key</strong>.</p><h3>Cloud functions</h3><p>We give you the freedom to design and code anything you imagine.Write your own <strong>JavaScript</strong> functions to handle anything beyond our pre-built blocks.</p><p>We also provide an SDK for developers to easily connect to our APIs and manage data storage and built-in apps and notification service</p><h3>Notification Service</h3><p><strong>Communication</strong> is the most crucial thing in the IoT section. Devices need to talk to each other, even outside of the Fasterbase environment, so we created a <strong>push notification system</strong> in the most <strong>secure</strong> way to let devices connect and share their data regardless of whether Fasterbase powers them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_fs3m7eDpBs_jadoPlk3zA.png" /></figure><h3>App Store</h3><p>Once to sign up in Fasterbase you will get access to many<strong> pre-built apps</strong> that connect your devices to other systems easily.</p><p>Imagine turning on a lamp and automatically sending an SMS — with Fasterbase, you just pick the SMS app, enter your details, and drag its block into your designed logic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H_JMq6MB6SruwNEscGM_7A.png" /></figure><h3>Web / Application Builder</h3><p>Our team has been working hard to develop a user-friendly platform that lets you create a great customer experience with your products. It’s a single app where customers can easily control and manage all their devices. Plus, we provide full access to all APIs, so your developers can build custom apps tailored to your needs.</p><h3>It’s Interesting, Isn’t it?</h3><p>We’re still building this exciting project, but if you’d like to be an early user of our service, invest in it, or just chat about it, <a href="https://adplist.org/mentors/faraz-faraji">feel free to book an appointment</a> or <a href="/service/mailto:faraz.faraji@gmail.com">send me an email</a>. Let’s talk now!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=005e3bf5c517" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Generate SDK from endpoints to connect the Frontend to the Backend in the Standard way and share …]]></title>
            <link>https://faraz-faraji.medium.com/generate-sdk-from-endpoints-to-connect-the-frontend-to-the-backend-in-the-standard-way-and-share-922b549cc5bd?source=rss-d5a8fe51f700------2</link>
            <guid isPermaLink="false">https://medium.com/p/922b549cc5bd</guid>
            <category><![CDATA[sdk]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[typescript]]></category>
            <category><![CDATA[nestjs]]></category>
            <category><![CDATA[react]]></category>
            <dc:creator><![CDATA[Faraz Faraji]]></dc:creator>
            <pubDate>Mon, 28 Nov 2022 19:53:37 GMT</pubDate>
            <atom:updated>2022-11-29T12:45:34.497Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HbLPkkzz3chgq0YhPWb0Lg.jpeg" /></figure><h3>Generate SDK from endpoints to connect the Frontend to the Backend in the Standard way and share types and DTOs in Typescript ( NestJS and ReactJS )</h3><h3>What We Are Going To Do 😎?</h3><p>The goal of this article is to generate an SDK from endpoints that we can share with the front end. the <strong>advantages </strong>of this SDK would be:<br>1- Sharing <strong>DTOs</strong> with the front end and keeping them in sync.<br>2- Save time by using prepared functions to call endpoints.</p><p>To reach this purpose, we are going to use <strong>OpenAPI</strong>, <strong>Swagger</strong> and <strong>NestJS.</strong></p><h3>Steps 🧐:</h3><p>1- Create a DTO and use Swagger API.<br>2- Create Controller.<br>3- Set up Swagger in NestJS.<br>4- Generate SDK<br>5- Fix Pagination Issue<br>6- Create a custom hook in ReactJS to call the API</p><p>Also at the end of the article, There is a video about how to implement it in NestJs.</p><h3>Create DTO</h3><p>Let’s assume that we want to create an API to create a new user and each user can have some customers. First of all, we have to create its <a href="https://en.wikipedia.org/wiki/Data_transfer_object"><strong>DTO</strong></a><strong>. </strong>Already, I’ve written another blog for validation which will be usable in this article also(<a href="https://faraz-faraji.medium.com/validate-income-and-outgoing-parameters-in-nestjs-50f5d1e9555c">Validate Income And Outgoing Parameters In NestJS</a>)🤨. It this important to add the correct decorator to define your variable to Swagger.</p><pre>import { ApiProperty } from &#39;@nestjs/swagger&#39;;<br><br>export class UserDto {<br>  @ApiProperty({required: true, type: String })<br>  username: string;<br><br>  @ApiProperty({required: true, type: String })<br>  password: string;<br><br>  @ApiProperty({required: true, enum: Role })<br>  role: Role;<br><br>  @ApiProperty({ required: false, isArray: true, type: () =&gt; CustomerDto })<br>  customers?: CustomerDto[];<br>}<br><br>export class CustomerDto {<br>  @ApiProperty({required: true, type: String })<br>  name: string;<br>  @ApiProperty({required: true, type: Number })<br>  phoneNumber: number;<br>}</pre><p>When you use @ApiProperty() it means that you are telling swagger that, please document this variable in the document.</p><h3>Create Controller</h3><p>In this step, to handle controllers, I’m using NestJS, but according to your framework, you can customize this part by yourself.</p><p>Here, after defining Controller, we need to define our controller to Swagger and to make it better documentation we can start by adding @ApiTags at the top of the class, and also define the functions to Swagger by adding @ApiResponse at the top of the controller name. In advanced mode, you can use <a href="https://docs.nestjs.com/openapi/operations">other OpenAPI decorators</a> to define your functions:</p><pre>import { ApiResponse, ApiTags } from &#39;@nestjs/swagger&#39;;<br><br>@ApiTags(&#39;Users&#39;) //add a tag to your controller<br>@Controller(&#39;user&#39;)<br>export class UserController {<br><br>  @ApiResponse({ type: UserDto }) // define controller to Swagger<br>  @Post()<br>  async addNewUser(@Body() userDto: UserDto): Promise&lt;UserDto&gt; {<br>    return userDto;<br>  }<br><br>}</pre><h3>Set up Swagger In NestJS</h3><p>It’s time to test and open the swagger document. It’s happening just by adding some lines to the main.ts in NestJs.</p><pre>import { NestFactory } from &#39;@nestjs/core&#39;;<br>import { DocumentBuilder, SwaggerModule } from &#39;@nestjs/swagger&#39;;<br>import { AppModule } from &#39;./app.module&#39;;<br><br>async function bootstrap() {<br>  const app = await NestFactory.create(AppModule);<br><br>// setup Swagger<br>  const config = new DocumentBuilder()<br>    .setTitle(&#39;Fasterbase&#39;)<br>    .setDescription(&#39;Fasterbase API description&#39;)<br>    .setVersion(&#39;1.0&#39;)<br>    .build();<br>  const document = SwaggerModule.createDocument(app, config);<br>  SwaggerModule.setup(&#39;api&#39;, app, document); // here you can define the swagger route<br><br>  await app.listen(3000);<br>}<br>bootstrap();</pre><p>After making the changes, your API will be accessible by adding /api it to the end of your application: http://localhost:3000/api . Also, you should have an access to http://localhost:3000/api-json see the JSON format of the document 😇.</p><h3>Generate SDK</h3><p>Till we can have access to the document, we should be able to generate the client SDK. Let’s install CodeGen library in the project directory 😇.</p><pre>yarn -D add swagger-codegen<br>#or using npm<br>npm install swagger-codegen --save-dev</pre><p>And then by running the following line you can generate the SDK:</p><pre>swagger-codegen generate -i http://localhost:3000/api-json -l typescript-axios -o client-sdk </pre><p>-i is a variable given the JSON source? <br>-l is responsible for the export language and its library<br>-o represents the output folder</p><p>After generating the SDK, you should able to see the client-SDK folder. 🤓</p><h3>Fix Pagination Issue 🧐:</h3><p>Imagine that, our controller should respond the data in a pagination format:</p><pre>import { ApiProperty } from &#39;@nestjs/swagger&#39;;<br><br>export class PaginationDto&lt;T&gt; {<br>  data: T[];<br><br>  @ApiProperty({ type: String })<br>  token?: string;<br><br>  @ApiProperty({ type: Boolean })<br>  more: boolean;<br>}</pre><p>And add it to your controller:</p><pre>@ApiResponse({ type: UserDto }) // define controller to Swagger<br>  @Get()<br>  async usersList(): Promise&lt;PaginationDto&lt;UserDto&gt;&gt; {<br>    const users = await userService.getUsers();<br>    return {<br>      data:users,<br>      more: false<br>    };<br>  }<br></pre><p>But Codegen can not support Generic Types, So we need to add a custom decorator to fix this issue:</p><pre>import { PaginationDto } from &#39;./dtos/pagination.dto&#39;;<br>import { applyDecorators, Type } from &#39;@nestjs/common&#39;;<br>import { ApiOkResponse, getSchemaPath } from &#39;@nestjs/swagger&#39;;<br><br>export const ApiPaginationResponse = &lt;TModel extends Type&lt;any&gt;&gt;(<br>  model: TModel,<br>  title: string,<br>) =&gt; {<br>  return applyDecorators(<br>    ApiOkResponse({<br>      schema: {<br>        title,<br>        allOf: [<br>          { $ref: getSchemaPath(PaginationDto) },<br>          {<br>            properties: {<br>              data: {<br>                type: &#39;array&#39;,<br>                items: { $ref: getSchemaPath(model) },<br>              },<br>              more: {<br>                type: &#39;boolean&#39;,<br>              },<br>            },<br>          },<br>        ],<br>      },<br>    }),<br>  );<br>};</pre><p>And then easily we can add this decorator to the controller function. But still, we should be sure that we are adding the DTO class to the function by adding @ApiExtraModels(className)</p><pre>@ApiExtraModels(UserDto) // include UserDto to the client-sdk<br>@ApiPaginationResponse(UserDto, &#39;getUsers&#39;)<br>@Get()<br>  async usersList(): Promise&lt;PaginationDto&lt;UserDto&gt;&gt; {<br>    const users = await userService.getUsers();<br>    return {<br>      data:users,<br>      more: false<br>    };<br>  }</pre><p>Now the client-SDK is generated and we can create an instance by importing index.js file.</p><h3>Use It In React By Creating Custom Hook 😎</h3><p>First of all, let’s create the instance of the endpoint.</p><pre>import { Configuration, UserApi } from &quot;./client-sdk&quot;;<br><br>const configuration = new Configuration({ basePath: &quot;http://localhost:3000&quot; });<br>export const user = new UserApi(configuration);</pre><p>And for example, you can create a file and call it useNewDevice.ts to implement the hook by react-query to insert a new user.</p><pre>import { useMutation, useQueryClient } from &quot;@tanstack/react-query&quot;;<br>import { UserDto } from &quot;src/client-sdk/models&quot;;<br><br>import { device } from &quot;src/client-sdk&quot;;<br><br>export const useNewUser = () =&gt; {<br>  const queryClient = useQueryClient();<br><br>  return useMutation(<br>    [],<br>    (userDto: UserDto) =&gt; {<br>      return user.userControllerAddNewUser(userDto);<br>    },<br>    {<br>      onSuccess: () =&gt; {<br>        queryClient.invalidateQueries([&quot;users&quot;]);<br>      },<br>    }<br>  );<br>};</pre><p>That’s It. Now you have can publish your SDK to the NPM repository to share it with your team easily.</p><h3>Video</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FaOMSSxh8w9A%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DaOMSSxh8w9A&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FaOMSSxh8w9A%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/1139e81303f96bb63f9494e24d98922b/href">https://medium.com/media/1139e81303f96bb63f9494e24d98922b/href</a></iframe><p>I hope you can find the article useful. Feel free to ask your questions and don’t forget to subscribe to me if you want to see more useful articles.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=922b549cc5bd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Validate Income And Outgoing Parameters In NestJS]]></title>
            <link>https://faraz-faraji.medium.com/validate-income-and-outgoing-parameters-in-nestjs-50f5d1e9555c?source=rss-d5a8fe51f700------2</link>
            <guid isPermaLink="false">https://medium.com/p/50f5d1e9555c</guid>
            <category><![CDATA[developer]]></category>
            <category><![CDATA[typescript]]></category>
            <category><![CDATA[nestjs]]></category>
            <category><![CDATA[professional-development]]></category>
            <category><![CDATA[clean-code]]></category>
            <dc:creator><![CDATA[Faraz Faraji]]></dc:creator>
            <pubDate>Thu, 03 Nov 2022 09:40:18 GMT</pubDate>
            <atom:updated>2022-12-07T04:11:02.281Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8TZjlMNJE1YZx8vPqoNaYg.png" /></figure><h3>Introduction:</h3><p>We always should <strong>care</strong> about what kind of <strong>data is coming</strong> to our <strong>REST API.</strong> sometimes we need to <strong>remove extra parameters</strong>, and sometimes we have to <strong>check types</strong>. some variables should send as a <strong>number</strong> some of them <strong>string</strong> or maybe <strong>time </strong>or <strong>URL.</strong></p><p>The problem is if even we do validate for incoming data, most of the time we forgot to validate our response. but:<br><strong>Why Validate Outgoing parameters? </strong>As some consumers are consuming our response, we should be careful about what we are going to change in some existing code. probably by our changes response will change. Sometimes we remove or add a new parameter which can break our consumer.</p><h3>Prepare NestJS:</h3><p>Before starting to write validations, we should add a pipe to our project to catch the incoming errors from validation.</p><p>To that purpose, we need to just the following line to the main.ts</p><pre>const app = await NestFactory.create(AppModule);</pre><pre>app.useGlobalPipes(new ValidationPipe()); &lt;------ Add this line</pre><pre>await app.listen(3000);</pre><h3>Incoming Parameters:</h3><p>First of all, let’s assume that we have a form which is responsible to get the data from the user to <strong>register</strong> him/her in our database.<br>Fields which we need are like the following code (you can find the completed code at the end of the section) :</p><pre>string name;<br>string birthday;<br>string imagePath;<br>number age; // age can be optional</pre><p>For the incoming data, we have to create a <strong>DTO</strong> file for each endpoint, for example, here we have a register.dto.ts file.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e638d7e8fc8f0ace04a5b59b9d2a139f/href">https://medium.com/media/e638d7e8fc8f0ace04a5b59b9d2a139f/href</a></iframe><p>The next step is installing the related packages to verify the DTO</p><pre>npm install class-validator class-transformer</pre><p>Typescript by Itself is handling the <strong>decorators</strong>. so like the following code we easily add the validation for each parameter:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4508cbf11b3228acf9470e59373f7476/href">https://medium.com/media/4508cbf11b3228acf9470e59373f7476/href</a></iframe><p>class-validator supports many decorators and for each of them, it has many options. So, I recommend you have a look at their documentation <a href="https://github.com/typestack/class-validator">https://github.com/typestack/class-validator</a>.</p><h3>Outgoing Parameters:</h3><p>The issue with outgoing is we have to <strong>call</strong> the <strong>verification function</strong> <strong>manually</strong>, before sending it as a response.</p><p>So, the first step of the process is to <strong>convert</strong> the <strong>object</strong> to a <strong>class</strong> by the class-transformerpackage. The following code can convert your object to a class and throw an error if there is any.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a59214031df5a54dcaf2de9e8c3cf79a/href">https://medium.com/media/a59214031df5a54dcaf2de9e8c3cf79a/href</a></iframe><p>The next step is extending our current DTOs by our verification class and then easily calling the verification function before passing it as a response.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e77c0efbbbd138769c7fd924133e5d90/href">https://medium.com/media/e77c0efbbbd138769c7fd924133e5d90/href</a></iframe><p>Here, It’s how we are going to use what we have in a sample controller</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5f9dd6593753cf2681fd4ded95cde902/href">https://medium.com/media/5f9dd6593753cf2681fd4ded95cde902/href</a></iframe><p>I hope you find out this article useful, <strong>ask</strong> me if you have any questions and <strong>follow</strong> if you want to be like a pro ✌️</p><h3>Training</h3><p>Here, I provided a video to create a verification function for incoming and outgoing parameters in Nest.JS, also we will test more decorators of class-validator there</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F_9qCvD5XDMI%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D_9qCvD5XDMI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F_9qCvD5XDMI%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/eeb80c1b36068695abd9a980034f965e/href">https://medium.com/media/eeb80c1b36068695abd9a980034f965e/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=50f5d1e9555c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is Monorepo? and how to Implement it in NestJS?]]></title>
            <link>https://faraz-faraji.medium.com/what-is-monorepo-and-how-to-implement-it-in-nestjs-c88eac6fe1a?source=rss-d5a8fe51f700------2</link>
            <guid isPermaLink="false">https://medium.com/p/c88eac6fe1a</guid>
            <category><![CDATA[nestjs]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[typescript]]></category>
            <category><![CDATA[monorepo]]></category>
            <dc:creator><![CDATA[Faraz Faraji]]></dc:creator>
            <pubDate>Mon, 31 Oct 2022 10:12:24 GMT</pubDate>
            <atom:updated>2022-12-07T04:11:42.412Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cHe-sXdujtYsSDHh" /><figcaption>Photo by <a href="https://unsplash.com/@pankajpatel?utm_source=medium&amp;utm_medium=referral">Pankaj Patel</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>What Is Monorepo?</h3><p>Monorepo is the opposite of multi-repo which is sometimes called <strong>monolithic repositories. </strong>We should aware that it’s completely different from Monolotics and microservices architectures.<br>To <strong>summarize</strong>, Monorepo is included multiple projects ( which are related or unrelated to each other) in the same repository. In <strong>multi-repo</strong> for each service, we use a separate repository.</p><p>Monorepo is used by many big companies, including <strong>Google</strong>, <strong>Facebook</strong>, <strong>Microsoft</strong>, and <strong>Twitter</strong>.</p><p>First of All, I’m very <strong>happy</strong> with <strong>Monorepo</strong> but just let’s be honest about the pros and cons of using Monorepo and according to your situation you should decide about using Monorepo or not.</p><h3><strong>How To Use In Microservices?</strong></h3><p>Monorepo have been designed specially for microservices . After Passing CI/CD you can create a separated docker container for each app which means also, scaling up would be easy.</p><h3>What Are Benefits?</h3><p><strong>Visibility</strong>: According to my experience, the most useful thing about Monorepo is visibility. it means that the developer can reach all the codes in one place. Also, they can connect apps to each other easily and use shared libraries such as DTOs, Types, and Communication libraries.</p><p><strong>Dependencies And Avoid Repeating: </strong>As I said, all the codes are in the same place, so easily we can connect apps together. Your framework can solve the dependency issue depending on what you’re using. For an instance, if you create a library and don’t use it in your code, it won’t be included in your project.</p><p><strong>Code Style</strong>: Another thing that I like is we can keep the Code style the same in all apps. You have one Eslint in whole the project, and whenever you change it, it will affect all the codes in the project.</p><p><strong>Tests and CI/CD: </strong>As you have all the code, writing the E2E test is much easier if you don’t want to use mock. for each push, All tests in the project will run and check if everything is working. So you will be sure that the changes won’t have any harmful effect on whole the project.</p><h3>What are Disadvantages?</h3><p><strong>Too much sensitive: </strong>If you push a bad commit, it will affect the whole project. it means no one can run any apps on their local machine.</p><p><strong>Large Volume: </strong>Since you pull the code, you will receive all the apps and codes. sometimes it can reach some gigabytes even.</p><p><strong>Longer CI/CD</strong>: For example, you have forgot to add a semicolon at the end of the line. if you do it and then push the code, CI/CD will build all the apps from scratch and takes too much time.</p><p><strong>Create A New Monorepo Project In NestJS</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FuWwGbLdGPr8%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DuWwGbLdGPr8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FuWwGbLdGPr8%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/290bb0a4e53d0a9ea7d4abc79c1a0ec6/href">https://medium.com/media/290bb0a4e53d0a9ea7d4abc79c1a0ec6/href</a></iframe><p>Don’t forget to subscribe me and my youtube challenge if you want to get new updates.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c88eac6fe1a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How We Designed Our Architecture In NodeJS [Part-2]]]></title>
            <link>https://faraz-faraji.medium.com/how-we-designed-our-architecture-in-nodejs-part-2-c7815cb85fe5?source=rss-d5a8fe51f700------2</link>
            <guid isPermaLink="false">https://medium.com/p/c7815cb85fe5</guid>
            <category><![CDATA[development]]></category>
            <category><![CDATA[bullmq]]></category>
            <category><![CDATA[system-architecture]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[rabbitmq]]></category>
            <dc:creator><![CDATA[Faraz Faraji]]></dc:creator>
            <pubDate>Tue, 08 Feb 2022 07:13:07 GMT</pubDate>
            <atom:updated>2022-02-08T07:13:07.513Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lW7beBiGPsxyPaFd.jpg" /></figure><p>First I have to say, this article is a good example of any system architecture, <strong>not </strong>just an SMS system.</p><p>In the<strong> </strong><a href="https://faraz-faraji.medium.com/how-to-make-a-good-team-while-we-are-creating-a-perfect-product-part-1-a5cba7a9b33c"><strong>previous article</strong></a>, We talked about How we created our team, how we were following the rules and this kind of stuff. Here were are going to deep into how we designed our architecture.</p><p>First, I need to explain what was our project. <strong>SMS Panel</strong>. we won’t have too many customers, but our customer has a lot of customers. It means maybe they want to send 20 Million SMS in a second to their contacts.</p><p>We listed our features lists, like sending SMS with delay, send by the scheduled time and etc. then we understood which kind of technologies we have to use.</p><p>Let’s deep into how SMS panels are working:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/631/1*5_NjK0omz_qBDuGlzoNMhg.png" /></figure><p>But we couldn&#39;t continue like this, we had a lot of requirements that by this architecture we couldn’t continue. Let’s list them:<br>1- Send too many SMS concurrently <br>2- We need to have a scheduler SMS<br>3- Getting reports from huge data</p><p>So We started to draw a new system architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*Bsin1bFRs42z_AP0o0rWmw.png" /></figure><p>Here I’m going to explain step by step the process.</p><h3>How SMS Sending is Working:</h3><p>There is the <strong>SMPP </strong>protocol for making communication to send SMS.<br>In this project, we designed an SMPP engine to make a connection with the operator’s servers. <br>First, we need to be <strong>authenticated </strong>and verified, then the process will be like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*tLAoZKQjBCipZU4Wa0bAZA.png" /></figure><p>So after sending each SMS, we are <strong>receiving </strong>an <strong>ID </strong>from the operator to <strong>specify the SMS</strong>.</p><h3><strong>Job Management:</strong></h3><p>We have <strong>two </strong>kinds of send requests:<br>1- Sometimes our customers wants to set a<strong> scheduler SMS</strong>, which means they want to send the SMS at a specific date, or with a delay, for example, <strong>2 hours later.</strong></p><p>2- Send immediately after the customer clicks on the send button.</p><p>In these two methods, we will add the request into the <strong>queue</strong>, because maybe <strong>another customer</strong> will start their campaign <strong>simultaneously</strong>, and it will have a huge load on our servers.</p><p>We used <a href="https://github.com/taskforcesh/bullmq"><strong>BullMQ</strong></a><strong>, </strong>which can support <strong>queue</strong>, and <strong>scheduler jobs</strong> simultaneous. requests will add to the BullMQ, and when the server is free, requests will start to process.<br>Also, it’s scalable, easily we can add more workers to speed up the process.</p><p>It’s important that our <strong>contacts </strong>for sending SMS are too much, maybe a <strong>million</strong>, So it’s hard and impossible to get all of them and add them into the Queue, Here we just get data by streaming data from the database and bunch by bunch add them into the BullMQ.</p><h3>RabbitMQ:</h3><p>Now it’s time to publish messages into our rabbitMQ when it’s time to send an SMS. <br>Why do we use rabbitMQ?</p><p>When it’s time to send an SMS, many processes should be started.</p><p><strong>Provision: </strong>we are sending a message from rabbitMQ to the Billing part of the application. each<strong> SMS has a cost,</strong> but we should be aware that if a message is flagged as a<strong> un delivery</strong>, we should <strong>pay back</strong> the money to the user. so we are<strong> not deducting</strong> costs directly from the user’s account. for the last decision about cost, we should <strong>wait for the operator’s response</strong> from the SMPP system. Because of the huge transactions and different schema, we are using MongoDB for this part to store the finance data.</p><p><strong>Archive: </strong>We should get reports from each campaign, so we are saving a record for each SMS that we are sending and when we receive any delivery from the SMPP system, we have to update the delivery flag. We are using <strong>PostgreSQL</strong> to save the archives.</p><p><strong>SMPP: </strong>As I Explained, SMPP is responsible to<strong> send our request to the operator’s server</strong>. So data through <strong>rabbiMQ </strong>will send to the SMPP services, and SMPP will redirect them to the operators. Also, the SMPP service is <strong>publishing the result </strong>into the <strong>RabbitMQ </strong>service and the specified consumer is using the received data.</p><p>In the next article, we will continue on how to get huge reports by background jobs. So follow me to don’t miss the next article</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c7815cb85fe5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How To Make A Good Team While We Are Creating A Perfect Product? [ Part-1 ]]]></title>
            <link>https://faraz-faraji.medium.com/how-to-make-a-good-team-while-we-are-creating-a-perfect-product-part-1-a5cba7a9b33c?source=rss-d5a8fe51f700------2</link>
            <guid isPermaLink="false">https://medium.com/p/a5cba7a9b33c</guid>
            <category><![CDATA[team-building]]></category>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[git]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[scrum]]></category>
            <dc:creator><![CDATA[Faraz Faraji]]></dc:creator>
            <pubDate>Mon, 24 Jan 2022 10:34:17 GMT</pubDate>
            <atom:updated>2022-02-08T07:14:57.188Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*viqTVjrPxicY14-vHDnpLg.jpeg" /></figure><p>In this article, I’m going to share my experience while I was a <strong>Team leader</strong> in a software company, and at that moment, I tried to make a good infra-structure for the software team while we were developing a good project. Here we will continue how to <strong>manage</strong> <strong>team</strong>, <strong>developers</strong> and <strong>project</strong> together. Then I will deep on the <strong>technical side</strong> and the <strong>architecture</strong> and <strong>deployment</strong> part.</p><p>First, let me explain our team and our tech stack because it will be helpful to understanding.</p><p>We were 6 developers and a UI/UX designer in our team. we were using <strong>Scrum</strong> methodology and <strong>Zoho Sprint</strong> for our task management. We use <strong>NodeJS</strong>, <strong>Nest.JS, MongoDB</strong> and <strong>Postgre</strong> to develop our project.</p><p>Let’s start, On the first day of the project, most of our team was junior. So the first thing is:</p><h3><strong>How to act with Junior developers</strong></h3><p>Most of the times juniors doesn’t know exactly what they want. they <strong>don’t know</strong> they want to continue on the <strong>front-end or backend</strong> side. So they need to be in a team to find themselves. Also, they need to be familiar with both sides, then they can decide. As I understood, we can not force them to work where the product needs (if you are hiring for a specific part, don’t hire junior). In my opinion, juniors should be start as a full-stack developer first.</p><p>Let’s assume we hired a junior, We can assign him/her a <strong>simple backend task</strong>, when it is finished we can tell him/her to start to write the<strong> front part</strong> too. The important part for junior is we shouldn’t <strong>change his/her stack immediately.</strong> also, they need a teammate. We <strong>shouldn’t join </strong>them in a team <strong>without any members</strong>.</p><h3>How We Are Planning?</h3><p>We are doing <strong>scrum</strong> because our goal is to finish the project as <strong>soon </strong>as possible <strong>without bugs</strong>. Also every week our <strong>stakeholder </strong>wants to <strong>see </strong>and <strong>use </strong>what we did.</p><p>Our sprint planning is happening every<strong> two weeks</strong> on Monday. According to the product owner sight (which part should be done earlier) we choose <strong>Epics </strong>and conquer it to the <strong>stories</strong>, and stories to the <strong>tasks</strong>.<br>Epics are already written before by the product owner. Also, we know the <strong>roadmap</strong>, so it depends on the scrum team to decide what they want to do in this sprint.</p><p>In a day, we are writing the code of hard parts of the projects in the early sprints and at the end of the project, we won&#39;t have any amazing and challenging tasks in our Sprint, Then the team will be <strong>demotivated</strong>.<br>The Important thing while we are choosing Epics, it’s we have to <strong>balance </strong>the tasks. normally day by day developer’s energy is decreasing in a project. But if we can balance the tasks it can be helpful for keeping the team motivated.</p><p>It means for each sprint, we should use just <strong>one </strong>new <strong>technology </strong>or <strong>challenge</strong>, Then developers are always busy with new challenges to solve them.</p><p><strong>Timing?</strong></p><p>We are playing <strong>Poker-game</strong> to assign time to the tasks. The important this for the poker game is, The skills of our development teams must be the same, Then times tend towards each other. Anyway we just choose the most time provided</p><p><strong>Assigning?</strong></p><p>We <strong>do not assign</strong> the tasks to anyone, we just write them and developers can take them and start them. Because:<br>- Dependency will be decrease<br>- All the developers will know about other parts of the code<br>- It will create a good and fair competitor to choose good tasks.<br>- It will increase the team conversation</p><p>But we should respect some rules like a developer can just have <strong>one task</strong> in the In-progress section.</p><h3>How We Used Version Control (Git)?</h3><p><strong>Bitbucket</strong> is our git code management. We used it because we were using <strong>Jira</strong> before,<strong> I didn’t have good experience</strong> with using bitbucket, sometimes It <strong>was down</strong>, sometimes<strong> pipelines were not working and slow </strong>and many other small problems.</p><p><strong>What is git-flow?</strong></p><p>It’s a method for managing git, in summary, in git-flow, we will have 3 main branches, development, staging, production. for each feature, developers should add a new branch and when it is finished, they should merge it to the root branch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/0*BaKR1n_wIYLVLdZa.png" /></figure><p>git-flow has a hotfix branch too, it’s coming from Master branch to fix bugs immediately.</p><p>We are using <a href="https://danielkummer.github.io/git-flow-cheatsheet/"><strong>git-flow</strong></a> for our git structure. for each story, we create a new branch and named it as the story title, and after finishing the task, we push the code into Git. After the sprint has been finished, developers should merge their code to the Staging branch.</p><p>Also, we have another optional meeting in the afternoons ( when all developers are ready and their code is pushed) we called it daily review which we are showing what we did since the last daily review.</p><p><strong>Finished</strong></p><p>Thank you for reading this article, in the <a href="https://faraz-faraji.medium.com/how-we-designed-our-architecture-in-nodejs-part-2-c7815cb85fe5"><strong>Next Article</strong></a>, I will deep into the technology part, CI/CD and test functions. Please write down your comment to make me better.</p><p>For receiving notification for the new articles, don’t forget to follow me :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a5cba7a9b33c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Faster CSV File Reader and Processor in Node.js]]></title>
            <link>https://javascript.plainenglish.io/faster-csv-file-reader-and-processor-in-nodejs-e69bbc19f465?source=rss-d5a8fe51f700------2</link>
            <guid isPermaLink="false">https://medium.com/p/e69bbc19f465</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[csv]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Faraz Faraji]]></dc:creator>
            <pubDate>Thu, 17 Jun 2021 15:35:37 GMT</pubDate>
            <atom:updated>2021-06-18T13:31:39.452Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*I4CnfQFMpJcN4v0yTU56Ew.jpeg" /></figure><h3>What is the problem?</h3><p>First, I want to explain what my problem was and how I solved it. One day, one of my colleagues sent me a <strong>1GB CSV</strong> ( something about <strong>60M</strong> line) file and said there are many customer’s phone numbers some of them are duplicated and we don’t need them. Please insert them <strong>into the database</strong> and give me the table name.</p><h3>What did I do at first?</h3><p>At first, I thought it’s easy and I started to read the file line by line.<br>for each data I checked the database by <strong>SQL Query</strong> If I added this number before, skip the inserting process, and if not insert the number into the table.</p><p>In this way, I had three problems:</p><ol><li><strong>I couldn’t read a big file</strong> with normal file readers like fs library because of the <strong>memory limitation</strong></li><li>It takes<strong> too much time</strong> to run a query for each number.</li><li><strong>I couldn’t run a child process</strong> by this way, so it will take more than 10 days.</li></ol><h3>How I solved the problem?</h3><p>To summarize:</p><ul><li>The<strong> </strong><a href="https://www.npmjs.com/package/line-reader"><strong>line-reader</strong></a> for reading a file.</li><li><strong>MySQL</strong> as a <strong>database</strong></li><li><a href="https://www.rabbitmq.com/"><strong>RabbitMQ</strong></a><strong> </strong>for<strong> messaging control</strong></li><li><a href="https://www.npmjs.com/package/pm2"><strong>PM2</strong></a> for running the services in the background</li></ul><p>At first, I decided to change my database table <strong>structure</strong> to <strong>avoid</strong> using <strong>SQL queries</strong> to find if this number is <strong>duplicate</strong> or not. So I changed the number column structure to unique.</p><pre>ALTER TABLE numbers<br>ADD CONSTRAINT constraint_name UNIQUE (number);</pre><p>After this change, I won’t check number is duplicate or not, I just insert the number, <strong>database will decide</strong> is a new number or not. Even for more efficiency, I added the index to my column by the following command:</p><pre>CREATE UNIQUE INDEX number_idx ON numbers (number);</pre><p>For reading the file, I started to <strong>read the file</strong> by <a href="https://www.npmjs.com/package/line-reader"><strong>line-reader</strong></a> package because it helps me to read the file <strong>buffered</strong> and <strong>line by line</strong>. But the problem is, it can not be synchronous, so if I want to insert the data while I’m reading the file I will get an error because I’m sending too much data<strong> in parallel</strong> to the database.</p><p>At first, I wrote this code:</p><pre><em>const </em>lineReader = require(&#39;line-reader&#39;);<br><em>async function </em>start() {<br>    <em>const </em>csvFile = __dirname + &quot;/assets/data.csv&quot;;<br>    lineReader.eachLine(csvFile, <em>async function</em>(line, last, cb) {<br>        <em>// My insertion Process</em><br>        cb();<br>    });<br>}</pre><p>After this, I decided to use <strong>RabbitMQ</strong> to solve this problem.</p><p>I created two services, one<strong> reader service</strong>, and one <strong>insertion service</strong> and I connect them by RabbitMQ.</p><p>I publish the data from the reader service, and from the insertion service, I read and inserted them into the database. for improving speed, I clone three insertion services.</p><p>Here I’m sharing my code of services:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3dcf42dfaa202fc1c34aa9fe45b28e51/href">https://medium.com/media/3dcf42dfaa202fc1c34aa9fe45b28e51/href</a></iframe><p>Let me explain some <strong>important</strong> things:</p><ul><li>In line <strong>23</strong>, I’m using <strong>persistent</strong> because when the service will close, all data will be lost. It can <strong>reduce the speed</strong> but it’s worth it.</li><li>In line <strong>19</strong>, because I’m using persistent data, I need to have an option to whenever I want, I can <strong>clear the queue</strong>.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/30ceb957c47cd687380e5561fe1bffbe/href">https://medium.com/media/30ceb957c47cd687380e5561fe1bffbe/href</a></iframe><p>Let me explain some <strong>important</strong> things:</p><ul><li>In line <strong>39</strong>, <strong>ack</strong> means our service received the message, and the process is done, so the message broker can <strong>remove</strong> it from the <strong>queue</strong>.</li><li>In line <strong>37</strong>, prefetch means the Message broker won’t send text messages till we call ack. It means our service will insert data one by one.</li></ul><p>Because of <strong>prefetch</strong>, our second service is reading data<strong> one by one</strong>, so we can open <strong>many</strong> from second services as we want and our database can support to <strong>increase the speed.</strong></p><p>After deployment, I want to close my <strong>SSH</strong> from our server, so I wanna use <a href="https://www.npmjs.com/package/pm2">PM2</a>. to manage it, I created a file and named it <strong>run.json</strong></p><pre>{<br>  &quot;apps&quot;: [<br>    {<br>      &quot;name&quot;: &quot;reader-service&quot;,<br>      &quot;script&quot;: &quot;./reader.service.js&quot;,<br>      &quot;autorestart&quot;: true<br>    },<br>    {<br>      &quot;name&quot;: &quot;insertion-service-1&quot;,<br>      &quot;script&quot;: &quot;./insertion-service.js&quot;,<br>      &quot;autorestart&quot;: true<br>    },<br>    {<br>      &quot;name&quot;: &quot;insertion-service-2&quot;,<br>      &quot;script&quot;: &quot;./insertion-service.js&quot;,<br>      &quot;autorestart&quot;: true<br>    }<br>  ]<br>}</pre><p><em>More content at </em><a href="http://plainenglish.io/"><em>plainenglish.io</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e69bbc19f465" width="1" height="1" alt=""><hr><p><a href="https://javascript.plainenglish.io/faster-csv-file-reader-and-processor-in-nodejs-e69bbc19f465">Faster CSV File Reader and Processor in Node.js</a> was originally published in <a href="https://javascript.plainenglish.io/">JavaScript in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Everything about databases — How to implement Sequelize? [Part-2]]]></title>
            <link>https://faraz-faraji.medium.com/everything-about-databases-how-to-implement-sequelize-part-2-23f2df3eaf47?source=rss-d5a8fe51f700------2</link>
            <guid isPermaLink="false">https://medium.com/p/23f2df3eaf47</guid>
            <dc:creator><![CDATA[Faraz Faraji]]></dc:creator>
            <pubDate>Tue, 15 Jun 2021 09:06:53 GMT</pubDate>
            <atom:updated>2021-06-16T00:22:48.066Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*CXkW09ru573y6YNYl3OiIg.png" /></figure><h3>Everything about databases — How to implement Sequelize? [Part-2]</h3><h3>Summerize of the previous post</h3><p>Thanks to <a href="https://faraz-faraji.medium.com/everything-about-databases-how-to-choose-part-1-35a7d1a0c9d1"><strong>Part-1</strong></a> we could decide about which database is better to use, after that we had talked about running the Postgres by docker in our Operator system.</p><h3><strong>What is ORM</strong></h3><p>As I mentioned in <a href="https://www.instagram.com/p/CQGpSEeD_j6/?utm_source=ig_web_button_share_sheet">my Instagram post</a> ORM is a tool that you can easily connect to your database and fetch the data as you want. Getting easier data is not just the point of using ORM, Also, increase security (SQL Injections), clean code, avoid hard code, and many other valuable things. Because of these reasons, we will continue using the database by ORM.</p><p>We have many ORM tools for node js, such as <a href="https://typeorm.io/"><strong>typeORM </strong></a>and <a href="http://sequelize.org/"><strong>sequelize</strong></a>.</p><h3>Familiarity with a few terms</h3><h4>1- Migration: (schema migration)</h4><p>as <a href="https://en.wikipedia.org/wiki/Schema_migration">Wikipedia</a> explained:</p><blockquote>In software engineering, schema migration refers to the management of incremental, reversible changes and version control to relational database schemas. A schema migration is performed on a database whenever it is necessary to update or revert that database’s schema to some newer or older version.</blockquote><p>It means migration is something like version controllers (for example Git) for your table. If you want to add a new column, you can easily add it to your migration file, and it will automatically affect your table.</p><p>Using migration is not necessary, but it’s better to use in your project.</p><h4>2- Seed:</h4><p>Assume you created a table to insert your user&#39;s data, but you need data on it for writing test functions. One way is adding data manually to your table. But another way is using seed data to fill your table.</p><h4>3- Models:</h4><p>A <strong>model</strong> is an abstraction that represents a table in your database.</p><h3>Let’s try Sequelize</h3><p>We have many steps to finalize the database with sequelize in our project.<br>In this example, we will create migration, seed, and model files to implement a user&#39;s table.</p><p>1- Install sequelize and sequelize-cli and dependencies of your database (here is Postgres).<br>2- Initialize Postgres structure in your project.<br>3- Generate migrations and models and seeds data<br>4- Use it in your project</p><h3>1: Install Sequelize and sequelize-cli.</h3><p>According to the <a href="https://faraz-faraji.medium.com/everything-about-databases-how-to-choose-part-1-35a7d1a0c9d1"><strong>previous post</strong></a>, we had run Postgres on our machine by docker, Here to install all we need, run the below codes:</p><pre>mkdir sequelize-test<br>cd sequelize-test<br>npm init<br>npm i sequelize sequelize-cli pg pg-hstore faker<br>npx sequelize-cli init</pre><p>For more information and to get familiar with CLI commands, have a look at their <a href="https://www.npmjs.com/package/sequelize-cli"><strong>Documentation</strong></a>.</p><h3>2: Using the .env file in your Sequelize project (optional)</h3><p>in normal mode, sequelize can not accept .env files for an auto-generating so we have to edit some files in our project.</p><p>First, go to the <strong>database</strong> folder generated by Sequelize and open <strong>config </strong>folder. then change the file name to config.js and put these contents on it:</p><pre>require(&#39;dotenv&#39;).config(); <em>// this is important!<br></em>module.exports = {<br>    &quot;development&quot;: {<br>        &quot;username&quot;: process.env.DB_USERNAME_DEV,<br>        &quot;password&quot;: process.env.DB_PASSWORD_DEV,<br>        &quot;database&quot;: process.env.DB_DATABASE_DEV,<br>        &quot;host&quot;: process.env.DB_HOST_DEV,<br>        &quot;dialect&quot;: &quot;postgres&quot;<br>    },<br>    &quot;test&quot;: {<br>        &quot;username&quot;: process.env.DB_USERNAME_TEST,<br>        &quot;password&quot;: process.env.DB_PASSWORD_TEST,<br>        &quot;database&quot;: process.env.DB_DATABASE_TEST,<br>        &quot;host&quot;: process.env.DB_HOST_TEST,<br>        &quot;dialect&quot;: &quot;postgres&quot;<br>    },<br>    &quot;production&quot;: {<br>        &quot;username&quot;: process.env.DB_USERNAME,<br>        &quot;password&quot;: process.env.DB_PASSWORD,<br>        &quot;database&quot;: process.env.DB_DATABASE,<br>        &quot;host&quot;: process.env.DB_HOST,<br>        &quot;dialect&quot;: &quot;postgres&quot;<br>    }<br>};</pre><p>Then create a file and name it <strong>.sequelizerc </strong>and put these contents on it:</p><pre>const path = require(&#39;path&#39;);<br><br>module.exports = {<br>  &#39;config&#39;: &#39;database/config/config.js&#39;, //config file path<br>  &#39;models-path&#39;: &#39;database/models&#39;, //model path<br>  &#39;migrations-path&#39;: &#39;database/migrations&#39;, //migration path<br>  &#39;seeders-path&#39;: &#39;database/seeders&#39; //seeders path<br>};</pre><p>Now your sequelize CLI will use your .env file to apply your changes.</p><h3>3: Create a migration file</h3><p>Generate a migration file by followed command:</p><pre>npx sequelize migration:create --name name_of_your_migration</pre><p>As you can see in your migrations folder, there is a file that started with a timestamp. It means when. you want to run migrations, CLI will run from older files to newer files.</p><p>Put the following code in the generated file:</p><pre>&#39;use strict&#39;;<br><br>module.exports = {<br>    up: <em>async </em>(queryInterface, Sequelize) =&gt; {<br>        <em>return </em>queryInterface.createTable(&#39;users&#39;,<br>            {<br>                user_id: {<br>                    type: Sequelize.INTEGER,<br>                    autoIncrement: <em>true</em>,<br>                    primaryKey: <em>true</em>,<br>                },<br>                username: {<br>                    type: Sequelize.STRING<br>                },<br>                email: {<br>                    type: Sequelize.STRING<br>                },<br>                password: {<br>                    type: Sequelize.STRING<br>                },                <br>                createdAt: {<br>                    type: Sequelize.DATE<br>                },<br>                updatedAt: {<br>                    type: Sequelize.DATE<br>                },<br>            }<br>        );<br>    },<br><br>    down: <em>async </em>(queryInterface, Sequelize) =&gt; {<br>        <em>await </em>queryInterface.dropTable(&#39;users&#39;);<br>    }<br>};</pre><p>Now you have a file that contents a migration script. it means whenever you run the migration command, your table will create automatically. let’s do it by this command:</p><pre>npx sequelize db:migrate</pre><h4>For more information: (update your table schema)</h4><p>Assume after a while, when you have many users, you decide to add the nickname of each customer.</p><p>Let’s <strong>create</strong> another <strong>migration</strong> file by this command:</p><pre>npx sequelize migration:create --name name_of_your_migration</pre><p>Then change the content to:</p><pre>module.exports = {<br>  up: function(queryInterface, Sequelize) {<br>    return queryInterface.addColumn(<br>      &#39;users&#39;,<br>      &#39;nickname&#39;,<br>     Sequelize.STRING<br>    );<br><br>  },<br><br>  down: function(queryInterface, Sequelize) {<br>    // logic for reverting the changes<br>    return queryInterface.removeColumn(<br>      &#39;users&#39;,<br>      &#39;nickname&#39;<br>    );<br>  }<br>}</pre><p>and again run the generation command:</p><pre>npx sequelize-cli db:migrate</pre><p>You will see, nicknames column is <strong>added</strong> to your user&#39;s table.<br>And whenever you want you can <strong>undo</strong> your changes version by version by this command:</p><pre>npx sequelize db:migrate:undo</pre><h3>4: Create a seed file</h3><p>Let’s start this process by creating a seed file by:</p><pre>npx sequelize seed:create --name name_of_your_seeds</pre><p>and then put the following code into the file</p><pre>&#39;use strict&#39;;<br><br>var faker = require(&quot;faker&quot;);<br><br>module.exports = {<br><br>    up: (queryInterface, Sequelize) =&gt; {<br><br>        var newData = [];<br><br>        for (let i = 0; i &lt; 10; i++) {<br>            const seedData = {<br>                nickname: faker.name.findName(),<br>                username: faker.internet.username(),<br>                email: faker.internet.email(),<br>                password: faker.internet.password(),<br>                createdAt: new Date(),<br>                updatedAt: new Date()<br>            };<br>            newData.push(seedData);<br>        }<br><br>        return queryInterface.bulkInsert(&#39;users&#39;, newData);<br>    },<br><br>    down: (queryInterface, Sequelize) =&gt; {<br>        return queryInterface.bulkDelete(&#39;users&#39;, null, {});<br>    }<br>};</pre><p>right now you have every basic thing about the Database and sequelize</p><p>just run this command to fill your data by seeder:</p><pre>npx sequelize-cli db:seed:all</pre><p>In the next post, I will explain deeply about Sequelize and how to fetch and insert data into your table.</p><p>Don’t forget to follow my Instagram page to get updates and notices.</p><p><a href="https://www.instagram.com/p/CQGpSEeD_j6/?utm_source=ig_web_copy_link"><strong>https://www.instagram.com/p/CQGpSEeD_j6/?utm_source=ig_web_button_share_sheet</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23f2df3eaf47" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Everything about Databases- Part 1: How to Choose a Database?]]></title>
            <link>https://javascript.plainenglish.io/everything-about-databases-how-to-choose-part-1-35a7d1a0c9d1?source=rss-d5a8fe51f700------2</link>
            <guid isPermaLink="false">https://medium.com/p/35a7d1a0c9d1</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Faraz Faraji]]></dc:creator>
            <pubDate>Mon, 10 May 2021 13:23:40 GMT</pubDate>
            <atom:updated>2021-06-21T05:16:00.947Z</atom:updated>
            <content:encoded><![CDATA[<h3>Everything about Databases— Part 1: How to Choose a Database?</h3><h4>Why do we need a database?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8dmEVT8rS2rXCGMK" /><figcaption>Photo by <a href="https://unsplash.com/@jankolar?utm_source=medium&amp;utm_medium=referral">Jan Antonin Kolar</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Why shouldn’t we? Throughout <strong>most projects, </strong>we make use of databases. Essentially, for saving any data, we use a database. For example, for <strong>signing </strong>up a user, we have to store the data (the user’s credentials) somewhere.</p><p>Actually, whether it’s for storing data, <strong>processing data</strong>, <strong>artificial intelligence</strong>, <strong>IoT</strong>, anywhere can you think of, developers are using databases.</p><h4>Finally, which kind of database?</h4><p>You have many choices for selecting a database. It depends on what you want to store and what your data structure looks like. We have two types of databases, <strong>SQL </strong>and <strong>NoSQL </strong>(not only SQL). Before choosing a database, know your data, extract what you want as a result, and be aware of your operators.</p><h4>When to use SQL, When to use NoSQL?</h4><p>If your data is very structured, use <strong>SQL</strong>, if not, use <strong>NoSQL</strong>. Let me explain more.</p><p>If you need to have an <strong>ACID</strong> (atomicity, consistency, isolation, durability) in your data or have <strong>relational </strong>data, it’s better to choose SQL. For example, imagine you are going to transfer money to someone else in the ATM. In this kind of data, the transaction is the most important rule, because you should be ensured if the other side received money, then the system can deduct money from the first side.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tEOr_cWlNgGNoXnE8GTlqw.jpeg" /></figure><p>As I said when you have relational data like products and categories, it’s better to use SQL. For example, you are creating an <strong>online shop</strong>, you have two different categories like man and woman, and each of your products is related to one of those categories. Look at the below table.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0415fd997cacf82078c3c9c65ea101d1/href">https://medium.com/media/0415fd997cacf82078c3c9c65ea101d1/href</a></iframe><p>As you can see, they are connected to each other by category_id, They call it Foreign Key, and you can create a relation by “join” commands in SQL.</p><p>It’s a good example for learning how to create joins, I recommend you have a look at this link:<a href="https://www.w3schools.com/sql/sql_join.asp"> <strong>https://www.w3schools.com/sql/sql_join.asp</strong></a></p><h4>How many kinds of SQL databases are there?</h4><p>We have many databases such as <strong>Mysql</strong>, <strong>Postgres</strong>, <strong>MSSQL</strong>, <strong>Oracle</strong>, <strong>MariaDB, </strong>etc. We can not say which one is better than the others. In my opinion, bad development can change and affect the best technology and turn it into the worst technology. So don’t spend too much time choosing the database. There are some advantages and disadvantages of each of them. I always prefer MySQL or Postgres (to be honest, without any specific reason as such). Because of this, I will explain these two. Here, we will have a look at some points:</p><ul><li><strong>MySQL </strong>is <strong>faster </strong>than <strong>Postgres </strong>on reading.</li><li><strong>MySQL </strong>is the most popular SQL Database.</li><li><strong>Postgres </strong>is an <strong>Object-Oriented</strong> database.</li><li><strong>Postgres </strong>is faster than <strong>MySQL </strong>on writing.</li><li><strong>Postgres </strong>is a single-threaded database but my MySQL is not.</li><li><strong>Postgres </strong>online management panel is <strong>PgAdmin </strong>and for <strong>MySQL </strong>is <strong>PHPMyAdmin</strong></li></ul><p>In this article, I will continue on Postgres.</p><h4>How to install and use Postgres?</h4><p>There many ways to install them according to your OS. but I recommend using docker for running them.</p><p>Also, you can use docker-compose to run Postgres and PgAdmin together</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/41357c16f5c3407e348239f86e68a550/href">https://medium.com/media/41357c16f5c3407e348239f86e68a550/href</a></iframe><p>Name it as docker-compose.yml and then run it by docker-compose</p><pre>docker-compose up -d</pre><p>Great! Now you’ve used Postgres and pgAdmin to edit your data in your machine. <a href="https://faraz-faraji.medium.com/everything-about-databases-how-to-implement-sequelize-part-2-23f2df3eaf47">In the next part</a>, I will explain how to connect to Postgres via the <a href="http://sequelize.org/">Sequelize</a> ORM library in Node.js.</p><p><em>More content at </em><a href="http://plainenglish.io/"><em>plainenglish.io</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=35a7d1a0c9d1" width="1" height="1" alt=""><hr><p><a href="https://javascript.plainenglish.io/everything-about-databases-how-to-choose-part-1-35a7d1a0c9d1">Everything about Databases- Part 1: How to Choose a Database?</a> was originally published in <a href="https://javascript.plainenglish.io/">JavaScript in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>