<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pavlo Kolodka on Medium]]></title>
        <description><![CDATA[Stories by Pavlo Kolodka on Medium]]></description>
        <link>https://medium.com/@pavlokolodka?source=rss-c87ee5e3bb6f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*GLfsgYQnHKB_9X9lCmnCvw.jpeg</url>
            <title>Stories by Pavlo Kolodka on Medium</title>
            <link>https://medium.com/@pavlokolodka?source=rss-c87ee5e3bb6f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 14:05:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pavlokolodka/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Writing your own JSON serializer and deserializer]]></title>
            <link>https://levelup.gitconnected.com/writing-your-own-json-serializer-and-deserializer-1669f9a4686e?source=rss-c87ee5e3bb6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/1669f9a4686e</guid>
            <category><![CDATA[json]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[from-scratch]]></category>
            <category><![CDATA[serialization]]></category>
            <category><![CDATA[deserialization]]></category>
            <dc:creator><![CDATA[Pavlo Kolodka]]></dc:creator>
            <pubDate>Fri, 20 Jun 2025 02:08:24 GMT</pubDate>
            <atom:updated>2025-06-23T10:16:42.797Z</atom:updated>
            <content:encoded><![CDATA[<h4>A practical guide to understanding JSON’s core principles by re-creating it in JavaScript.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kRTO5Qbkb7h-pmKpino5MA.png" /><figcaption>Thumbnail created by <a href="https://medium.com/@pavlokolodka"><strong>Author</strong></a></figcaption></figure><h3>Introduction</h3><p>I believe you truly understand something only when you can reimplement it. That’s when you encounter edge cases, hidden complexities, and unexpected design decisions — things you might never notice by just using the tool.</p><p>Previously, we’ve discovered the inner workings of JavaScript’s Promise by implementing it ourselves. If you haven’t read this one, you can check it out here:</p><p><a href="https://javascript.plainenglish.io/i-wrote-my-own-promise-implementation-5175231b229d">I wrote my own Promise implementation</a></p><p>Today, we’re going to write our own JSON serializer/deserializer to better understand how things work under the hood.</p><h3>History of JSON</h3><p><a href="https://www.json.org/json-en.html">JSON</a> (JavaScript Object Notation) is a lightweight data interchange format that was discovered and popularized by Douglas Crockford in the early 2000s. It is based on a subset of the JavaScript programming language, although JSON is language-independent and has been adopted across virtually all programming environments.</p><p>JSON emerged as a need for simple data exchange between web browsers and servers. Later, when AJAX was presented, JSON became even more popular than XML (the “X” in AJAX acronym) because of its core feature of simplicity, and nowadays it is de facto a standard for data interchange format in the web realm.</p><p>As it was mentioned before, JSON is a strict subset of JavaScript. That means all data expressed with its syntax is automatically valid JavaScript types.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/0*LROYboQgbyleLvga" /><figcaption>Example of using a JSON object without additional parsing.</figcaption></figure><h3>Syntax</h3><p>JSON supports a limited set of data types:</p><ol><li>Strings: “text”</li><li>Numbers: 42, 3.14</li><li>Booleans: true, false</li><li>Null: null</li><li>Objects: { “key”: “value” }</li><li>Arrays: [1, 2, 3]</li></ol><p>All strings and object keys must be in double quotes. Numbers are signed decimal numbers that could contain a fractional part and use the E-notation. Arrays in JSON are heterogeneous arrays (mixed-type arrays), and should start with “[“ and end with “]”. Objects are associative arrays delimited with curly brackets “{}”; they use commas to separate each pair, and the colon ‘:’ character to separate the key from its value.</p><h3>Serializer</h3><p>Serialization is the process of transforming data into a format that can be stored or transmitted, with the ability to restore it to its original form. Examples of such formats include CSV, JSON, XML, YAML, protobuf, and more.</p><p>Serialization in JavaScript land is called stringification. It’s done using <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/stringify">stringify</a> method and essentially boils down to the following string manipulations:</p><ol><li>All datatypes are converted to their string representation, for example, using String() casting.</li><li>Strings and object keys should be additionally enclosed with double quotes.</li><li>Undefined, functions, and symbols should be ignored as plain values or values in objects and replaced with null in arrays.</li><li>Infinity and NaN are considered null. Converting BigInt throws an error.</li><li>Only enumerable properties are taken into account. So, Map, Set, WeakMap, WeakSet, objects with explicit properties set as non-enumerable, etc., will result in an empty object “{}“.</li></ol><p>And here is the implementation:</p><pre>/**<br> *<br> * @param {any} value<br> * @returns {string | undefined}<br> */<br>function stringify(value) {<br>  switch (typeof value) {<br>    case &quot;number&quot;:<br>      if (!Number.isFinite(value)) {<br>        return &quot;null&quot;;<br>      }<br>      return String(value);<br>    case &quot;bigint&quot;:<br>      throw new Error(&quot;Cannot serialize BigInt value&quot;);<br>    case &quot;string&quot;:<br>      return `&quot;${value}&quot;`;<br>    case &quot;boolean&quot;:<br>      return value ? &quot;true&quot; : &quot;false&quot;;<br>    case &quot;object&quot;:<br>      if (value === null) {<br>        return &quot;null&quot;;<br>      }<br>      if (value instanceof Date) {<br>        return `&quot;${value.toISOString()}&quot;`;<br>      }<br>      if (Array.isArray(value)) {<br>        return stringifyArray(value);<br>      }<br>      return stringifyObject(value);<br>  }<br>}<br><br>function stringifyArray(value) {<br>  const values = value<br>    .map((el) =&gt; (isInvalidJSONValue(el) ? stringify(null) : stringify(el)))<br>    .join(&quot;,&quot;);<br>  return `[${values}]`;<br>}<br><br>function stringifyObject(value) {<br>  const entries = Object.entries(value)<br>    .map(([key, val]) =&gt;<br>      isInvalidJSONValue(val) ? undefined : `&quot;${key}&quot;:${stringify(val)}`<br>    )<br>    .filter((el) =&gt; el !== undefined)<br>    .join(&quot;,&quot;);<br>  return `{${entries}}`;<br>}<br><br>function isInvalidJSONValue(value) {<br>  return (<br>    typeof value === &quot;undefined&quot; ||<br>    typeof value === &quot;function&quot; ||<br>    typeof value === &quot;symbol&quot;<br>  );<br>}<br><br></pre><p>Let’s test it:</p><pre>const user = {<br>  name: &quot;John Doe&quot;,<br>  age: 30,<br>  isActive: true,<br>  hobbies: [&quot;reading&quot;, &quot;traveling&quot;, &quot;gaming&quot;],<br>  address: {<br>    street: &quot;123 Main St&quot;,<br>    city: &quot;New York&quot;,<br>    postalCode: &quot;10001&quot;,<br>  },<br>  spouse: null,<br>  middleName: undefined,<br>  greet: function () {<br>    return `Hello, my name is ${this.name}.`;<br>  },<br>};<br><br>console.log(stringify(user));<br>// Outputs: <br>{<br>  &quot;name&quot;: &quot;John Doe&quot;,<br>  &quot;age&quot;: 30,<br>  &quot;isActive&quot;: true,<br>  &quot;hobbies&quot;: [<br>    &quot;reading&quot;,<br>    &quot;traveling&quot;,<br>    &quot;gaming&quot;<br>  ],<br>  &quot;address&quot;: {<br>    &quot;street&quot;: &quot;123Main St&quot;,<br>    &quot;city&quot;: &quot;New York&quot;,<br>    &quot;postalCode&quot;: &quot;10001&quot;<br>  },<br>  &quot;spouse&quot;: null<br>}</pre><h3>Deserializer</h3><p>Deserialization is the opposite process of serialization.</p><p>To implement a deserializer, we basically need to write a parser that would take an arbitrary JSON-compatible string and return a valid JavaScript value or throw an error.</p><p>Parsing often contains two main stages: lexical analysis and syntactic analysis. During lexical analysis, the input is broken down into the smallest units of language, called tokens. At syntactic analysis, previously constructed tokens are output in a structured way, compared to a given input, for subsequent processing.</p><h3>Lexer</h3><p>Implementing a lexer may be the most difficult part of this project. First, we need to define a set of tokens for further searching in the input text.</p><pre>const TokenType = {<br>  OPEN_OBJECT: &quot;OPEN_OBJECT&quot;,<br>  CLOSE_OBJECT: &quot;CLOSE_OBJECT&quot;,<br>  COLON: &quot;COLON&quot;,<br>  COMMA: &quot;COMMA&quot;,<br>  OPEN_ARRAY: &quot;OPEN_ARRAY&quot;,<br>  CLOSE_ARRAY: &quot;CLOSE_ARRAY&quot;,<br>  NUMBER: &quot;NUMBER&quot;,<br>  STRING: &quot;STRING&quot;,<br>  NULL: &quot;NULL&quot;,<br>  BOOLEAN: &quot;BOOLEAN&quot;,<br>};</pre><p>A token itself would look like this:</p><pre>{ type: TokenType, value: any }</pre><p>Then we would go character by character and try to map defined tokens with their value.</p><pre>let char = text[charIndex];<br><br>if (char === Char.WHITESPACE) {<br>  charIndex++;<br>  continue;<br>}<br><br>if (char === Char.DOUBLE_QUOTE) {<br>  ... <br>}<br>if (RegEx.NUMBER_START.test(char)) {<br>  ... <br>}<br>if (RegEx.TRUE.test(text.slice(charIndex))) {<br>  ...   <br>}<br>if (RegEx.FALSE.test(text.slice(charIndex))) {   <br>  ... <br>}<br>if (RegEx.NULL.test(text.slice(charIndex))) {  <br>  ...   <br>}<br>...</pre><p>One of the challenges was to parse a number. The JSON spec allows signed numbers, which could be integers, floating point numbers, or expressed using scientific notation.</p><p>With that in mind, my approach was to look at a current character; if it’s a digit or a minus sign, then we presumably found the start of a number. After that, we would buffer the next characters until we encounter something that’s not a digit, the “.” or “e”/”E” character. Then we would concatenate our buffer and try to cast it to a number.</p><pre> if (RegEx.NUMBER_START.test(char)) {<br>    checkBoolOrNull(tokens, char);<br>    const value = [char];<br>    charIndex++;<br>    char = text[charIndex];<br>    while (RegEx.NUMBER.test(char)) {<br>      value.push(char);<br>      charIndex++;<br>      char = text[charIndex];<br>    }<br><br>    if (<br>      Number.isNaN(Number(value.join(&quot;&quot;))) ||<br>      (value.length &gt; 2 &amp;&amp;<br>      ((value[0] === &quot;0&quot; &amp;&amp; value[1] !== &quot;.&quot;) ||<br>       value[value.length - 1] === &quot;.&quot;))<br>    ) {<br>      throw new Error(&quot;Invalid number literal&quot;);<br>    }<br><br>    tokens.push({ type: TokenType.NUMBER, value: Number(value.join(&quot;&quot;)) });<br>    RegEx.NUMBER.test(char) ? charIndex++ : null;<br>    continue;<br>}</pre><p>For objects and arrays, we would parse critical tokens like “{”, “}”, “:”, “[“, “]”, “,” and then ensure that they appear in the right sequence.</p><pre>{<br>...<br>  // OBJECT<br>  if (char === Char.OPEN_OBJECT) {<br>    charIndex++;<br>    const nextChar = text<br>      .slice(charIndex)<br>      .replace(RegEx.ALL_WHITESPACES, &quot;&quot;)[0];<br>    if (nextChar !== Char.DOUBLE_QUOTE &amp;&amp; nextChar !== Char.CLOSE_OBJECT) {<br>      throw new Error(`Unexpected token after &#39;{&#39;: ${nextChar}`);<br>    }<br><br>    tokens.push({ type: TokenType.OPEN_OBJECT, value: char });<br>    numberOfLeftBrace++;<br>    continue;<br>  }<br>  if (char === Char.CLOSE_OBJECT) {<br>    if (!numberOfLeftBrace || numberOfLeftBrace &lt; numberOfRightBrace + 1) {<br>      throw new Error(`Unexpected close object: ${char}`);<br>    }<br><br>    tokens.push({ type: TokenType.CLOSE_OBJECT, value: char });<br>    numberOfRightBrace++;<br>    charIndex++;<br>    continue;<br>  }<br>  if (char === Char.COLON) {<br>    const previousToken = tokens[tokens.length - 1];<br>    if (previousToken &amp;&amp; previousToken.type !== TokenType.STRING) {<br>      throw new Error(`Unexpected colon: ${char}`);<br>    }<br><br>    tokens.push({ type: TokenType.COLON, value: char });<br>    charIndex++;<br>    continue;<br>  }<br>  if (char === Char.COMMA) {<br>    const openBrace = tokens.find((el) =&gt; el.type === TokenType.OPEN_OBJECT);<br>    const openBracket = tokens.find((el) =&gt; el.type === TokenType.OPEN_ARRAY);<br>    if (!openBrace &amp;&amp; !openBracket) {<br>      throw new Error(`Unexpected non-whitespace character: ${char}`);<br>    }<br><br>    charIndex++;<br>    const nextChar = text<br>      .slice(charIndex)<br>      .replace(RegEx.ALL_WHITESPACES, &quot;&quot;)[0];<br>    if (<br>      nextChar == Char.CLOSE_OBJECT ||<br>      nextChar == Char.CLOSE_ARRAY ||<br>      nextChar == Char.COMMA ||<br>      nextChar == Char.COLON<br>    ) {<br>      throw new Error(`Unexpected token: ${nextChar}`);<br>    }<br><br>    tokens.push({ type: TokenType.COMMA, value: char });<br>    continue;<br>  }<br>  // ARRAY<br>  if (char === Char.OPEN_ARRAY) {<br>    charIndex++;<br>    const nextChar = text[charIndex];<br>    if (!nextChar) {<br>      throw new Error(`Unexpected token after &#39;[&#39;: ${nextChar}`);<br>    }<br><br>    tokens.push({ type: TokenType.OPEN_ARRAY, value: char });<br>    numberOfLeftBracket++;<br>    continue;<br>  }<br>  if (char === Char.CLOSE_ARRAY) {<br>    if (<br>      !numberOfLeftBracket ||<br>      numberOfLeftBracket &lt; numberOfRightBracket + 1<br>    ) {<br>      throw new Error(`Unexpected close array: ${char}`);<br>    }<br><br>    tokens.push({ type: TokenType.CLOSE_ARRAY, value: char });<br>    numberOfRightBracket++;<br>    charIndex++;<br>    continue;<br>  }<br><br>  throw new Error(`Unexpected token: ${char}`);<br>}<br><br>if (numberOfLeftBrace !== numberOfRightBrace) {<br>  throw new Error(`Expected &#39;,&#39; or &#39;}&#39; after property value`);<br>}<br>if (numberOfLeftBracket !== numberOfRightBracket) {<br>  throw new Error(`Expected &#39;,&#39; or &#39;]&#39; after array element`);<br>}<br><br>}</pre><p>Now we have tokenized our input source, so we can move to the second part: parsing.</p><h3>Parser</h3><p>One of the main advantages of splitting deserialization into two parts is that we don’t need to check the correctness of the input data during the parsing stage. Instead, we can focus purely on constructing the required output — JavaScript objects.</p><p>To parse strings, numbers, booleans, and null values, we just need to return their already prepared values.</p><pre>/**<br> *<br> * @param {Array&lt;{type: TokenType, value: any}} tokens<br> * @returns {any}<br> */<br>function parseValue(tokens) {<br>  const token = tokens[tokenIndex];<br><br>  switch (token.type) {<br>    case TokenType.STRING:<br>    case TokenType.NUMBER:<br>    case TokenType.BOOLEAN:<br>    case TokenType.NULL:<br>      return token.value;<br>    case TokenType.OPEN_OBJECT:<br>      return parseObject(tokens, tokenIndex);<br>    case TokenType.OPEN_ARRAY:<br>      return parseArray(tokens, tokenIndex);<br>    default:<br>      throw new Error(`Unexpected token type: ${token.type}`);<br>  }<br>}</pre><p>To parse objects and arrays, we would use recursion, as data types can be easily nested into multiple levels.</p><pre>/**<br> *<br> * @param {Array&lt;{type: TokenType, value: any}} tokens<br> * @returns {Object}<br> */<br>function parseObject(tokens) {<br>  const obj = {};<br>  tokenIndex++;<br>  let token = tokens[tokenIndex];<br>  while (token.type !== TokenType.CLOSE_OBJECT) {<br>    if (token.type === TokenType.STRING) {<br>      tokenIndex++;<br>    }<br>    if (token.type === TokenType.COMMA) {<br>      tokenIndex++;<br>    }<br>    if (token.type === TokenType.COLON) {<br>      const key = tokens[tokenIndex - 1];<br>      const value = parseValue(tokens, ++tokenIndex);<br>      obj[key.value] = value;<br>      tokenIndex++;<br>    }<br><br>    token = tokens[tokenIndex];<br>  }<br>  return obj;<br>}<br><br>/**<br> *<br> * @param {Array&lt;{type: TokenType, value: any}} tokens<br> * @returns {Object}<br> */<br>function parseArray(tokens) {<br>  const arr = [];<br>  tokenIndex++;<br>  let token = tokens[tokenIndex];<br>  while (token.type !== TokenType.CLOSE_ARRAY) {<br>    if (token.type === TokenType.COMMA) {<br>      tokenIndex++;<br>    } else {<br>      const value = parseValue(tokens, tokenIndex);<br>      arr.push(value);<br>      tokenIndex++;<br>    }<br><br>    token = tokens[tokenIndex];<br>  }<br>  return arr;<br>}</pre><h3>Conclusion</h3><p>That’s it! You’ve now built a basic, functioning JSON serializer/deserializer for JavaScript. While it’s far from perfect, you could enhance it with custom formatting options, support for string escaping, and more precise error reporting. Still, I hope you’ve learned something valuable from this article and that it inspires you to explore everyday tools and create something cool.</p><p>The full source code for this article is available on <a href="https://github.com/pavlokolodka/from-scratch/tree/main/javascript/json">GitHub</a>.</p><p>Thank you for reading this article!</p><p>Any questions or suggestions? Feel free to write a comment.</p><p>I’d also be happy if you followed me and gave this article a few claps! 😊</p><p>Check out some of my latest articles here:</p><ul><li><a href="https://medium.com/@pavlokolodka/list/98fb3445020a">Software Engineering</a></li><li><a href="https://medium.com/@pavlokolodka/list/4b0f9817d17d">Node.js</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1669f9a4686e" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/writing-your-own-json-serializer-and-deserializer-1669f9a4686e">Writing your own JSON serializer and deserializer</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[In Search of Improving Database Performance: A Comprehensive Guide with 8 Key Strategies]]></title>
            <link>https://levelup.gitconnected.com/in-search-of-improving-database-performance-a-comprehensive-guide-with-8-key-strategies-3496f2262cdb?source=rss-c87ee5e3bb6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/3496f2262cdb</guid>
            <category><![CDATA[database-performance]]></category>
            <category><![CDATA[database-design]]></category>
            <category><![CDATA[partitioning]]></category>
            <category><![CDATA[database-indexing]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Pavlo Kolodka]]></dc:creator>
            <pubDate>Mon, 02 Sep 2024 01:17:17 GMT</pubDate>
            <atom:updated>2024-09-04T09:26:01.174Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learn how to boost database efficiency and handle growth.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7rwzlTaxImSEVBQmfi_Aqg.png" /><figcaption>Thumbnail created by <a href="https://medium.com/@pavlokolodka"><strong>Author</strong></a></figcaption></figure><h3>Introduction</h3><p>As software grows, there comes a time when you need to improve the performance of data storage. This could manifest as a slowdown in search speed, a degradation in write operations, or an overall performance decrease. While understanding these challenges is critical, knowing the common approaches to address them is equally important.</p><p>In this article, I’ll try to gather all available techniques that I consider when dealing with database performance. I’ll start with simple tips and gradually introduce more complex methods as the cost of implementation increases.</p><p>This guide will be helpful for anyone who wants to expand their knowledge about databases. Grasping all of this may be challenging, but is definitely worth exploring.</p><h3>Query optimization</h3><p>Let’s start with the query itself. In this section, you will gain an understanding of execution order, the danger of a correlated subquery, recognizing when to apply batching and materialize view, and analyzing the execution plan.</p><h4>Leverage execution order</h4><p>You can tell what approximate query complexity is by looking at it. The trick here is to know the order in which SQL statements are executed.</p><p>Here is the bottom-up order of a typical SQL query execution:</p><ol><li>FROM</li><li>JOIN</li><li>WHERE</li><li>GROUP BY</li><li>HAVING</li><li>SELECT</li><li>ORDER BY</li><li>OFFSET</li><li>LIMIT</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_zbhp80vxHHJgOdo" /><figcaption>Source: <a href="https://www.youtube.com/watch?v=BHwzDmr6d7s">Secret To Optimizing SQL Queries — Understand The SQL Execution Order</a></figcaption></figure><p>When you know the order of query execution, you understand that the earlier you apply a constraint, the more benefit you will get in terms of processing performance.</p><p>As an example, let’s consider that we have a product table with different categories, and we need to find the average price of the ‘Electronics’ category. One of the possible queries could be this:</p><pre>SELECT category, AVG(price) AS avg_price<br>FROM products<br>HAVING category = &#39;Electronics&#39; AND AVG(price) &gt; 100<br>GROUP BY category;</pre><p>But as you might have noticed, category filtration happens in the HAVING block. The downside of this is that the filtration by category will always take place after aggregation, which, with a large amount of data, may result in a significant query time increase, especially if we have an index on the category column. That’s because the HAVING clause is applied for a temporary aggregated table that, without the database optimizer, cannot use indexes for filtration (you can reveal that with the EXPLAIN command).</p><p>It’s better to apply initial filtering as early as possible to reduce processing power on aggregation and further limiting and sorting.</p><pre>SELECT category, AVG(price) AS avg_price<br>FROM products<br>WHERE category = &#39;Electronics&#39;<br>GROUP BY category<br>HAVING AVG(price) &gt; 100;</pre><h4>Correlated subquery</h4><p>As we discuss improving query performance, there is a type of subquery you need to pay attention to: a correlated subquery.</p><p>A correlated subquery is a subquery that uses values from the outer query. The drawback of such a nested query is that it will be executed for every outer row. That’s a high computation effort for a large dataset.</p><p>One of the alternatives to correlated subquery is to use JOIN. Consider the following example, where we want to find employees whose salary is above the average salary in their department:</p><pre><br>                             Table &quot;public.Employee&quot;<br>    Column     |  Type   | Collation | Nullable |             Default             <br>---------------+---------+-----------+----------+---------------------------------<br> id            | integer |           | not null | nextval(&#39;employee_id_seq&#39;::regclass)<br> department_id | integer |           |          | <br> name          | text    |           |          | <br> salary        | numeric |           |          | </pre><pre>SELECT e1.employee_id, e1.employee_name, e1.salary<br>FROM employee e1<br>WHERE e1.salary &gt; (<br>  SELECT AVG(e2.salary)<br>  FROM employee e2<br>  WHERE e2.department_id = e1.department_id<br>);</pre><p>Let’s rewrite it using JOIN:</p><pre>SELECT e1.employee_id, e1.employee_name, e1.salary<br>FROM employee e1<br>JOIN (<br>  SELECT department_id, AVG(salary) AS avg_salary<br>  FROM employee<br>  GROUP BY department_id<br>) e2 ON e1.department_id = e2.department_id<br>WHERE e1.salary &gt; e2.avg_salary;</pre><p>The main difference is that in the case of the correlated subquery, the average will be executed for every row from the employee table, and in the case of using JOIN, the average calculation will be executed only once. You will see the detailed difference in the query analyzing chapter.</p><h4>Apply batching</h4><p>One technique that can noticeably boost database execution is to apply batching when possible. When you deal with bulk operations, instead of executing individual insert, update, or delete statements, you group them together into a single batch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a6zap7e28wygQ-LIvJZjyw.png" /><figcaption>Standalone request vs. batch request</figcaption></figure><p>This includes two major benefits:</p><ul><li>Reduced network latency: By sending multiple operations in a single request, you minimize the network latency that occurs with each round-trip between the client and the database server.</li><li>Improved throughput: A database server can process a batch more efficiently than many individual operations, often leading to higher throughput. That’s because, under the hood, many databases wrap operations in the transactions, and batch processing helps to remove the overhead of starting and committing them at high volumes.</li></ul><h4>Utilize materialized view</h4><p>When you have a case of heavy computation, such as aggregates involving the combination of several tables, and that computation occurs rarely, you might consider using a <a href="https://en.wikipedia.org/wiki/Materialized_view">materialized view</a>.</p><p>A materialized view is very similar to a basic VIEW, except it stores its values on disk, which is the opposite of fetching data every time when you query a basic VIEW.</p><p>This is a good mechanism to implement pre-computation with a custom refresh interval. Instead of scanning thousands of rows and then processing them, you execute one SELECT query and get all the results.</p><h4>Analyze query execution plan</h4><p>To apply the above tips properly for a particular DBMS with a special environment, you need to understand how exactly the database will execute your query. This is because when you write a query, essentially, what you’re doing is describing what data you want to get, and the implementation details go up to the database.</p><p>One possible way to examine underlying magic is to use the EXPLAIN command. Let’s analyze the performance between previously discussed correlated and JOIN queries:</p><p>The first query using a correlated subquery:</p><pre>EXPLAIN ANALYZE SELECT e1.employee_id, e1.employee_name, e1.salary<br>FROM employee e1<br>WHERE e1.salary &gt; (<br>    SELECT AVG(e2.salary)<br>    FROM employee e2<br>    WHERE e2.department_id = e1.department_id<br>);</pre><p>The output:</p><pre>QUERY PLAN                                                                                                          |<br>--------------------------------------------------------------------------------------------------------------------+<br>Seq Scan on employee e1  (cost=0.00..16339.60 rows=270 width=68) (actual time=0.045..0.063 rows=4 loops=1)          |<br>  Filter: (salary &gt; (SubPlan 1))                                                                                    |<br>  Rows Removed by Filter: 7                                                                                         |<br>  SubPlan 1                                                                                                         |<br>    -&gt;  Aggregate  (cost=20.14..20.15 rows=1 width=32) (actual time=0.004..0.004 rows=1 loops=11)                   |<br>          -&gt;  Seq Scan on employee e2  (cost=0.00..20.12 rows=4 width=32) (actual time=0.001..0.001 rows=6 loops=11)|<br>                Filter: (department_id = e1.department_id)                                                          |<br>                Rows Removed by Filter: 5                                                                           |<br>Planning Time: 0.153 ms                                                                                             |<br>Execution Time: 0.090 ms                                                                                            |                                                                                  |</pre><p>The second query using the JOIN operator:</p><pre>EXPLAIN ANALYZE SELECT e1.employee_id, e1.employee_name, e1.salary<br>FROM employee e1<br>JOIN (<br>    SELECT department_id, AVG(salary) AS avg_salary<br>    FROM employee<br>    GROUP BY department_id<br>) e2 ON e1.department_id = e2.department_id<br>WHERE e1.salary &gt; e2.avg_salary;</pre><p>The output:</p><pre>QUERY PLAN                                                                                                             |<br>-----------------------------------------------------------------------------------------------------------------------+<br>Hash Join  (cost=29.15..49.39 rows=270 width=68) (actual time=0.038..0.041 rows=4 loops=1)                             |<br>  Hash Cond: (e1.department_id = employee.department_id)                                                               |<br>  Join Filter: (e1.salary &gt; (avg(employee.salary)))                                                                    |<br>  Rows Removed by Join Filter: 7                                                                                       |<br>  -&gt;  Seq Scan on employee e1  (cost=0.00..18.10 rows=810 width=72) (actual time=0.004..0.005 rows=11 loops=1)         |<br>  -&gt;  Hash  (cost=26.65..26.65 rows=200 width=36) (actual time=0.024..0.025 rows=3 loops=1)                            |<br>        Buckets: 1024  Batches: 1  Memory Usage: 9kB                                                                   |<br>        -&gt;  HashAggregate  (cost=22.15..24.65 rows=200 width=36) (actual time=0.015..0.017 rows=3 loops=1)             |  <br>              Group Key: employee.department_id                                                                        |<br>              Batches: 1  Memory Usage: 40kB                                                                           |<br>              -&gt;  Seq Scan on employee  (cost=0.00..18.10 rows=810 width=36) (actual time=0.001..0.002 rows=11 loops=1)|<br>Planning Time: 0.163 ms                                                                                                |<br>Execution Time: 0.100 ms                                                                                               |                                                                                      |</pre><p>Summary of operations:</p><p>1. First query (correlated subquery):</p><ul><li>Sequential Scan on employee e1: 1 time</li><li>Sequential Scan on employee e2: 11 times (once for each row in employee e1)</li><li>Aggregate Calculation: 11 times (once for each row in employee e1)</li></ul><p>2. Second query (JOIN with aggregated subquery):</p><ul><li>Sequential Scan on employee e1: 1 time</li><li>Sequential Scan on employee e2 (for subquery): 1 time</li><li>Hash Aggregate: 1 time</li><li>Hash Join: 1 time</li></ul><p>What you can see here is that in the case of the JOIN query, the average aggregation is executed once for the entire table. In contrast, the AVG function was executed 11 times for the first query — the same number of times as there are rows in the table (loops=11). Since the number of rows is very small, the time difference is minimal, but in a large dataset, the difference can be significant.</p><h3>Indexes:</h3><p>When facing the problem of speeding up read operations, one potential solution is to use <a href="https://en.wikipedia.org/wiki/Database_index">indexing</a>. Creating indexes is a useful choice as it does not require modifying an existing schema and can greatly improve search performance.</p><p>There are two types of indexes: clustered and non-clustered. A clustered index is mostly automatically created for a primary key, as it indicates the physical order of rows. In contrast, a non-clustered index does not necessarily reflect the actual order of data; it’s an additional list that speeds up a lookup by a certain attribute. Next, I’ll talk about non-clustered indexes.</p><p>One of the important things when working with indexes is to know when to apply them. Generally, you prioritize indexing for columns that are frequently used in query filters. In particular, they are useful with these 3 main clauses:</p><ul><li>WHERE</li><li>JOIN</li><li>ORDER BY</li></ul><p>What’s the downside of indexes? It’s a space and write cost. Indexes reduce the number of disk I/O operations, allowing the database not to perform a full table scan by given criteria.</p><p>It’s done by creating an additional storage that holds an indexed column(s) and the pointer to the actual row. You can think about it as key-value storage, where a key is an indexed column, and the value is a reference to an actual row. In fact, the two most popular implementations of indexes include creating a B-tree (B+ tree) or a hash table. That’s the storage overhead.</p><p>The write cost means operations like INSERT, UPDATE, DELETE will take more time because, in addition to the operation itself, you need to keep the index order up-to-date.</p><p>Let’s compare a search for the employee table for a given name.</p><p>Without an index for the department column, you can see the next query plan:</p><pre>EXPLAIN ANALYZE SELECT * FROM employee WHERE name = &#39;John&#39;;<br><br>QUERY PLAN                                                                                           |<br>-----------------------------------------------------------------------------------------------------+<br>Seq Scan on employee  (cost=0.00..18.50 rows=200 width=4) (actual time=0.010..0.097 rows=200 loops=1)|<br>  Filter: (name = &#39;John&#39;::text)                                                                        |<br>  Rows Removed by Filter: 800                                                                        |<br>Planning Time: 0.142 ms                                                                              |<br>Execution Time: 0.130 ms                                                                             |</pre><p>Let’s add a new index for the name attribute:</p><pre>CREATE INDEX idx_employee_name<br>ON employee (name);</pre><p>And reanalyze the query plan:</p><pre>EXPLAIN ANALYZE SELECT * FROM employee WHERE name = &#39;John&#39;;<br><br>QUERY PLAN                                                                                                                           |<br>-------------------------------------------------------------------------------------------------------------------------------------+<br>Index Scan using idx_employee_name on employee  (cost=0.15..10.76 rows=178 width=4) (actual time=0.084..0.237 rows=178 loops=1)|<br>  Index Cond: (name = &#39;John&#39;::text)                                                                                              |<br>Planning Time: 0.148 ms                                                                                                              |<br>Execution Time: 0.097 ms                                                                                                             |</pre><p>In addition to decreased searing time, you can see the employment of the created index¹.</p><blockquote>Note 1: In the case when multiple rows satisfy the condition, a database is likely to use a <a href="https://en.wikipedia.org/wiki/Bitmap_index">bitmap index</a>.</blockquote><h4>Covering indexes</h4><p>What if I told you that you can speed up indexes themselves if you create them properly? The way to do it is to use covering indexes.</p><p>Suppose we want to find the salary of an employee by a given name. In the typical index search, there are two phases: index scan and table scan. When the database finds a needed index element, it’ll grab the internal ID of the row that’s stored next to the index element and perform the additional lookup on the main table using that ID.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zW571qUte7FuRwUFmSqu9A.png" /><figcaption>Example of an index structure on the name attribute</figcaption></figure><p>But when you know the queried values beforehand, you can eliminate the second scan, resulting in the index-only scan. The way to do it is to create a covering index that covers all the needed columns.</p><p>Let’s conduct some tests. First of all, we will analyze a plain index search:</p><pre>EXPLAIN ANALYZE SELECT salary FROM employee WHERE name = &#39;John&#39;;<br><br>QUERY PLAN                                                                                                                           |<br>-------------------------------------------------------------------------------------------------------------------------------------+<br>Index Scan using idx_employee_name on employee  (cost=0.15..10.72 rows=178 width=4) (actual time=0.084..0.237 rows=178 loops=1)|<br>  Index Cond: (name = &#39;John&#39;::text)                                                                                              |<br>Planning Time: 0.132 ms                                                                                                              |<br>Execution Time: 0.095 ms</pre><p>To utilize a covering index, we need to recreate an index to cover additional fields (in our case, it’s a salary attribute). There are two options for doing it: add covered attributes directly to the index or include needed columns into the index payload.</p><pre>DROP INDEX idx_employee_name<br>CREATE INDEX idx_employee_name<br>ON Employee (name, salary);</pre><p>And analyze the query:</p><pre>EXPLAIN ANALYZE SELECT salary FROM employee WHERE name = &#39;John&#39;;<br><br>QUERY PLAN                                                                                                                               |<br>-----------------------------------------------------------------------------------------------------------------------------------------+<br>Index Only Scan using idx_employee_name on employee  (cost=0.28..5.59 rows=178 width=4) (actual time=0.080..0.101 rows=178 loops=1)|<br>  Index Cond: (name = &#39;John&#39;::text)                                                                                                  |<br>  Heap Fetches: 0                                                                                                                        |<br>Planning Time: 0.136 ms                                                                                                                  |<br>Execution Time: 0.086 ms                                                                                                                 |</pre><p>Here, we can see that Postgres successfully applied the index, resulting in the index-only scan. Also, you can observe Heap Fetches: 0, meaning Postgres did not conduct a search in the main table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5l1QZ2fFktKAXE6scKR1uQ.png" /><figcaption>Example of an index structure on the name attribute with a covered salary attribute (using direct indexing of additional attribute)</figcaption></figure><p>Postgres allows for the<a href="https://www.postgresql.org/docs/current/indexes-index-only-scans.html"> addition of covered columns into the index payload</a> instead of indexing them directly. So, while a payload will not be used in conditional like WHERE, it will be helpful in eliminating the main table lookup.</p><pre>CREATE INDEX idx_employee_name<br>ON Employee (name) INCLUDE (salary);<br><br>EXPLAIN ANALYZE SELECT salary FROM employee WHERE name = &#39;John&#39;;<br><br>QUERY PLAN                                                                                                                               |<br>-----------------------------------------------------------------------------------------------------------------------------------------+<br>Index Only Scan using idx_employee_name on employee  (cost=0.28..5.59 rows=178 width=4) (actual time=0.068..0.080 rows=178 loops=1)|<br>  Index Cond: (name = &#39;John&#39;::text)                                                                                                  |<br>  Heap Fetches: 0                                                                                                                        |<br>Planning Time: 0.153 ms                                                                                                                  |<br>Execution Time: 0.075 ms                                                                                                                 |</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t1jtiDOQQP-SjNFKQAExgQ.png" /><figcaption>Example of an index structure on the name attribute with a covered salary attribute (using INCLUDE keyword)</figcaption></figure><h3>Denormalization</h3><p>There are situations when you cannot improve performance by not touching the current database scheme. It may involve adding new columns for frequently accessed or precomputed values, removing a relation table to avoid joins, or, on the contrary, creating duplicating tables to store denormalized data. Such a process is called <a href="https://en.wikipedia.org/wiki/Denormalization">denormalization</a>.</p><p>Denormalization is a technique used to improve database read performance by intentionally adding data redundancy. Mainly, denormalization is utilized to remove heavy JOIN operations. By duplicating data to leverage fast access, it brings a downside in the form of increased complexity of write operations and maintaining data integrity.</p><p>To give you a concrete example, let’s imagine we have a customer table and an order table.</p><pre>TABLE Order (<br>    id INT PRIMARY KEY,<br>    customer_id INT,<br>    date DATE<br>);<br><br>TABLE Customer (<br>    id INT PRIMARY KEY,<br>    name VARCHAR(100)<br>);</pre><p>You have a simple task: to output all the orders with their related customer names. This can be done using a single JOIN by referring to the customer_id from the ‘Order’ table and matching it with the primary key of the ‘Customer’ table. It’s not too complex at this stage, but as other fields or tables are added, the computation time for fetching data from the disk and building a temporary table will increase, potentially slowing down the search query.</p><p>In the context of denormalization, the solution is to store the customer’s name directly in the ‘Order’ table so that we can query the needed information without joining tables.</p><pre>TABLE Order (<br>    id INT PRIMARY KEY,<br>    customer_id INT,<br>    customer_name VARCHAR(100), -- Redundant column<br>    date DATE<br>);<br><br>TABLE Customer (<br>    id INT PRIMARY KEY,<br>    name VARCHAR(100)<br>);</pre><h3>Configuration adjustment</h3><p>Assume you have already applied query optimization strategies, but the performance still doesn’t meet the requirements. One potential solution is to adjust the current configuration settings. Keeping the default values is not always a good idea, as your working environment may be either resource-limited or resource-wasting. Customizing settings, such as the maximum number of connections, buffer pool size, transient memory, etc., can provide the fine-tuning options you need.</p><h4>Connection pooling</h4><p>One thing that should definitely be considered is the use of <a href="https://en.wikipedia.org/wiki/Connection_pool">connection pooling</a>. A connection pool is a component that acts as a cache for open database connections, allowing them to be reused. Opening a new connection for every request can lead to a significant increase in time for operation execution. That’s because there is a cost associated with the underlying opening/closing of the connection.</p><p>When using a connection pool, you have two options: implement it at the application level or as a standalone infrastructure component. The first option is usually chosen when you have only one application connecting to a dedicated database. The second option is appropriate when multiple clients cannot share their connections from a local connection pool, requiring you to manage connection pooling separately.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zzk5d4Cy57aBGI3aS1XuUA.png" /><figcaption>Local connection pool vs. global connection pool</figcaption></figure><h4>Compression</h4><p>As data grows, it requires more and more storage. Not only that, but a database system can become I/O-bound, potentially resulting in a bottleneck.</p><p>Many databases support lossless compression algorithms out of the box. The expected benefit is a reduced size of data that needs to be written and read, allowing faster access. The downside is the additional CPU cycles required to compress and decompress the data. However, in most cases, the CPU is fast enough that we still notice a performance speedup.</p><h3>Caching</h3><p>Caching is the process of utilizing pre-saved data in fast-access storage to reduce the computation and time needed to fetch data from the main storage. A fast-access storage component is called a <a href="https://en.wikipedia.org/wiki/Cache_(computing)">cache</a>, and at the software level, it usually takes the form of in-memory storage, such as Redis, Memcached, KeyDB, etc.</p><p>There are several strategies to implement caching. As you will see, each strategy has a trade-off between write speed, data inconsistency, and architecture resilience.</p><h4>Cache-aside</h4><p>A cache-aside strategy is the most simple pattern to use for caching. In this setup, a cache provider is located next to the primary data storage. Any time an application needs to fetch the data, it will go to the cache and read it from there. If there is a cache miss, then the application will read from the database and rehydrate the cache.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*616YIKHqWGjpOIdC7PZeGw.png" /><figcaption>Cache-aside strategy</figcaption></figure><p>The problem with this strategy it’s a cache invalidation: once a record is stored in the cache, it’s no longer reupdated. Since all orchestration is done by the application, the responsibility for cache invalidation is on the developer’s shoulders.</p><h4>Read-through</h4><p>In a read-through cache strategy, a cache is sitting in the middle between the application and the database. It may be a part of the database or as a standalone provider, such as Redis. In any case, a read request will always go through the cache level: if there is a cache hit, then data is immediately returned; otherwise, the cache component will go to the database, replicate the record in cache storage, and return the requested resource.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pcv24_-2pAPd7GAPE2xCaA.png" /><figcaption>Read-through strategy</figcaption></figure><p>The drawback of this strategy is the same as that of a read-aside strategy: having to update the cache. The advantage, compared to the previous strategy, is that reading operations go to the single entry point in the form of a cache component, and it takes care of itself for cache miss.</p><h4>Write-around</h4><p>A write-around strategy is a primary strategy on how you add new data with configured cache-aside or read-through caching setups. In this strategy, new information is written directly to the database and read from the cache. The pros and cons are the same as the chosen cache-read strategy. This design is better suited for frequent data reads and rare data updates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-K0hhNrThRg9pMBc6ucSrg.png" /><figcaption>Write-around strategy</figcaption></figure><h4>Write-through</h4><p>Instead of writing and reading separately, a write-through strategy combines these two operations. When new data arrives to the application, the application issues a new write request directly to the cache component. A cache then stores this copy of data and immediately requests a write operation to the database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nJ96-8Uh63mkutl1W2j55w.png" /><figcaption>Write-through strategy</figcaption></figure><p>Direct writing to the cache can introduce more latency compared to direct writing to the database.</p><p>Another downside of this strategy is the single point of failure. When the cache is down, the application needs to be reconfigured to write to the database. As a potential solution to overcome this problem is to spin multiple cache nodes.</p><p>On the contrary, the benefit of a write-through setup is the removed data inconsistency and lack of the need to warm up the cache to ensure all up-to-date data will be present in that component.</p><h4>Write-back</h4><p>A write-back caching strategy is similar to the previous one except for one difference: it writes to the database once in a while instead of immediately writing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p7Hnwohj6pmIAtfl7eI1tg.png" /><figcaption>Write-back strategy</figcaption></figure><p>This results in improved time for write delay as the number of network roundtrips between the cache and the database can be dramatically decreased. It will be the best choice for applications with a write-heavy workload.</p><p>But it also means that the data will have an eventual consistency. And as with write-through there is a chance for data loss in case of cache system failure.</p><p>As was shown, there is no silver bullet approach to implementing caching; every strategy has its own drawbacks. You choose what’s suitable for your concrete use case. However, you can also tune these caching strategies if needed. For example, you may add a custom algorithm to the write-around strategy to improve the data consistency characteristic. Or, with cache-aside or read-through, you can warm up the cache by issuing common queries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*efWNn3Ltop7zNCk_QW0kOw.png" /><figcaption>Comparison of different caching strategies</figcaption></figure><h3>Partitioning</h3><p><a href="https://en.wikipedia.org/wiki/Partition_(database)">Database partitioning</a> is a technique that splits data into chunks so that all operations are performed directly on the fragments rather than on the entire data set, which speeds up the overall performance of the system. Generally, it falls into two main categories: vertical and horizontal partitioning.</p><p>You may have already heard about vertical and horizontal scalability. Vertical scaling involves adding more resources to a single machine, such as increasing RAM, CPU cores, or storage space. Whereas, horizontal scaling involves using multiple machines to improve overall computation ability. In terms of partitioning, the concept is similar.</p><p>As a rule of thumb, you don’t need partitioning when you have a small table, and your data ingesting pace isn’t going to be high.</p><h4>Vertical partitioning</h4><p>Vertical partitioning is a type of partitioning that involves splitting a table by columns into several tables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JazH-p93efCW2jtpOYYyKg.png" /><figcaption>Example of applying vertical partitioning</figcaption></figure><p>In the classical table, you store all data together, but in the vertical partitioned one, you can store ‘hot’ columns in the dedicated table and the rest of the columns in the main table.</p><p>The benefit of such a partition is that it reduces the amount of data per row that needs to be scanned. When you need to fetch only the columns stored in some partition, you query it directly, which can give a noticeable performance gain over querying specific columns in a table with many attributes.</p><p>Vertical partitioning also gives the ability to store different partitions on different machines: partitions with frequently updated columns could be stored in more performant, high-throughput database servers, and partitions with infrequent update rates or fragments with static fields can be kept on less performant instances.</p><p>The disadvantage of vertical partitioning, however, is that in the case of a full data request, it requires additional joins across partitions in comparison with a single table request that stores all the data together. It’s worth noting that overall maintenance also increases, such as dealing with indexes and constraints over partitions.</p><h4>Horizontal partitioning</h4><p>Horizontal partitioning is a type of partitioning that involves splitting a table by rows into several smaller tables. These subtables have an identical structure to the original table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NWcOCqEzKsMG6CqIlcuYuQ.png" /><figcaption>Example of applying horizontal partitioning</figcaption></figure><p>Horizontal partitioning is useful for several reasons. One of them is to improve query performance. If you have a large data table or expect it to be so, is wise to consider utilizing partitioning benefits. Instead of going row by row to scan a table, you only need to find a needed partition range that, as in result, can dramatically decrease query time.</p><p>A prime example of horizontal partitioning is when working with time-bound (time series) data. By partitioning data based on specific time ranges, you can quickly locate the required time bucket. Another example is partitioning rows by columns with specific value ranges, such as enumerations, allowing you to query only the subset of data that matches the value you’re interested in.</p><p>You may ask why you should not just use an index. This is because with a large amount of data, an index scan becomes less beneficial, and there is a higher maintenance overhead, for example, when you need to defragment deleted values.</p><p>Another benefit of horizontal partitioning is the ease of data management. If we want to control partitioned data independently, partitioning facilitates that. For instance, if we want to delete records for a specific month and year, we can partition the data by time and treat the resulting chunks as standalone sets. This approach ensures that when we delete a particular chunk, all associated data is removed as well. It is also much faster and more resource-efficient than deleting a set of rows individually.</p><p>Managing partitions must done carefully. Picking the wrong partition attributes can lead to uneven data distribution, causing performance bottlenecks. Or the opposite, if you over-partition your dataset, it takes away all the advantages due to query planning overhead.</p><h3>Sharding</h3><p>Sharding is a technique of storing a large dataset on multiple machines called <a href="https://en.wikipedia.org/wiki/Shard_(database_architecture)">shards</a>. The principle of splitting the data remains the same as in the previous chapter. Typically, sharding is done using horizontal partitioning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y3MoUYVgxVynLbSe-4R1YQ.png" /><figcaption>Sharding</figcaption></figure><p>What benefits does this bring? One of the most obvious is that you can now store data closer to your customers. For example, if we shard a database table by location, we can store different shards in different regions, thereby improving latency. Another advantage of sharding is that the database load can be distributed across different hardware, enhancing overall system performance.</p><p>That may sound like a universal solution to dealing with database performance, doesn’t it? However, as you may know, everything comes with a cost, and sharding is no exception. The challenge lies in choosing a shard key that will uniquely identify a single partition. If you fail with this task, then you will have to scan several shards to find the needed information, thus removing the main point of sharding. Data backup is also starting to get complicated: instead of backing up a single table, you now have to consider several shards. The same goes for replication, synchronization, and overall operation complexity. So, the decision to implement sharding should be made cautiously, with all benefits and drawbacks in mind.</p><h3>Replication</h3><p><a href="https://en.wikipedia.org/wiki/Replication_(computing)">Database replication</a> is a technique that involves creating copies of databases across different servers. You can think about it as a file backup process that is used to ensure data will not be lost when the main source is damaged or cannot be accessed.</p><p>The main benefits of database replication include reducing the workload on the data storage layer, providing fault tolerance in case of a crash, enabling scaling by allowing reading from and writing to multiple database instances, and improving user experience by locating a database replica closer to the end user.</p><p>When it comes to implementing replication, there are two main strategies: master-slave and multi-master.</p><h4>Master-slave replication</h4><p><a href="https://en.wikipedia.org/wiki/Master%E2%80%93slave_(technology)">Master-slave replication</a>, sometimes called single-leader, primary-secondary, or active-passive, is a type of replication where there is only one primary node that is in charge of modifying data and multiple secondary nodes that serve read requests. Every node in this setup holds the same copy of data so that the read commands can be delegated to one of them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WqaUZ1hDCb5g3NgS9dpdbg.png" /><figcaption>Master-slave replication</figcaption></figure><p>As we have more instances of database we can easily scale read operations. But what about the write operations? Commands like INSERT, UPDATE, DELETE become tricky because there is only one entry that accepts it. Also, there is additional time for data synchronization between the primary and secondary nodes.</p><p>And what if the main node goes down? Well, there are several mechanisms to deal with it, but the common one is that the system tries to take one of the active replicas and make it a master replica.</p><p>As you can see, this architectural strategy is more suitable when there is no expectation of high write throughput, as the master node becomes the bottleneck of such a setup.</p><h4>Multi-master replication</h4><p><a href="https://en.wikipedia.org/wiki/Multi-master_replication">Multi-master replication</a>, sometimes called multi-leader, is a type of replication where there are multiple primary nodes that handle read and write requests. It is typically used when shortcomings of the previous strategy are critical.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8u361Fa1c6Q_83pIbqnnhw.png" /><figcaption>Multi-master replication</figcaption></figure><p>Each node responsible for write and read operations can optionally have a set of nodes handling only read operations, thus forming multiple local subclusters and combining master-slave and multi-master strategies together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*McuOeondMKOwa1ZwbjxO1A.png" /><figcaption>Multi-master replication with read-only replicas</figcaption></figure><p>By providing multiple nodes that can handle write operations, this setup eliminates the potential single point of failure in case the master node goes down. In addition to it, it distributes write operations, so scaling those can now be possible.</p><p>The drawback, however, is that it is hard to synchronize modified data. Imagine a case where the same piece of data is modified on multiple nodes. It’s also worth noting that the more nodes you have, the more time it takes to synchronize them. So, this setup is definitely complex to configure.</p><h3>Conclusion</h3><p>In the final part, I want to remind you of the two-step process for enhancing database performance.</p><p>First of all, you need to understand what exactly causes a performance decrease. It may be a particular operation such as a heavy join, a set of operations on an increasingly large dataset, network latency, or a redundant big payload.</p><p>Once you’ve pinpointed the issue, the next step is selecting the appropriate solution. As we’ve discussed, different strategies address different aspects of database performance, whether it’s query optimization, indexing, denormalization, configuration tuning, caching, partitioning, sharding, or replication.</p><p>While choosing a solution, consider not only the benefits but also the potential drawbacks of each strategy. Some methods might improve speed at the cost of increased complexity or resource consumption. Evaluate your chosen approach’s impact by measuring performance before and after implementation. Having the right balance is key to achieving sustainable, long-term improvements for your database.</p><p>Thank you for reading this article!</p><p>Any questions or suggestions? Feel free to write a comment.</p><p>I’d also be happy if you followed me and gave this article a few claps! 😊</p><p>Check out some of my latest articles here:</p><ul><li><a href="https://medium.com/@pavlokolodka/list/98fb3445020a">Software Engineering</a></li><li><a href="https://medium.com/@pavlokolodka/list/4b0f9817d17d">Node.js</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3496f2262cdb" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/in-search-of-improving-database-performance-a-comprehensive-guide-with-8-key-strategies-3496f2262cdb">In Search of Improving Database Performance: A Comprehensive Guide with 8 Key Strategies</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The 20 Essential Principles of Software Development: LoD, SoC, SOLID, and Beyond.]]></title>
            <link>https://levelup.gitconnected.com/the-20-essential-principles-of-software-development-lod-soc-solid-and-beyond-7a39a98b685d?source=rss-c87ee5e3bb6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a39a98b685d</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[grasp]]></category>
            <category><![CDATA[software-principles]]></category>
            <category><![CDATA[solid]]></category>
            <category><![CDATA[design-principles]]></category>
            <dc:creator><![CDATA[Pavlo Kolodka]]></dc:creator>
            <pubDate>Sun, 14 Apr 2024 17:36:26 GMT</pubDate>
            <atom:updated>2024-08-27T08:58:55.071Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zKJ9V43EwT1PybhNIcbTPg.png" /><figcaption>Thumbnail created by <a href="https://medium.com/@pavlokolodka"><strong>Author</strong></a></figcaption></figure><h3>Introduction</h3><p>Software design principles are the foundation of software development. As a software engineer, you can find them in your work tools, languages, frameworks, paradigms, and patterns. They are the core pillars of “good” and “readable” code. Once you understand them, you can see them everywhere.</p><p>The skill of seeing and applying them is what distinguishes a good engineer from a bad one. No one framework or tool can improve your quality of writing good code without understanding the fundamentals; moreover, without them, you become a hostage of that tool.</p><p>This article isn’t a reference guide but rather my try to systemize a list of core principles that need to be refreshed from time to time.</p><h3>Abstraction</h3><p><a href="https://en.wikipedia.org/wiki/Abstraction_(computer_science)">Abstraction</a> is one of the most significant principles in general. To abstract something means to focus on the important part while neglecting other details. Abstraction can be interpreted in two main ways: as a process of generalization and as the result of this generalization itself.</p><p>In software development, it often comes paired with encapsulation, a method used to hide the implementation of the abstracted parts. You can observe abstraction in various forms. For example, when you define a type, you abstract yourself from the memory representation of a variable. Similarly, when you abstract an interface or the signature of a function, you focus on what’s important: the contract to work with. When designing a class, you select only the relevant attributes for your domain and the specific business use cases. There are tons of other examples, but the main purpose of abstraction is that you don’t need to know the details of implementation to work with something; therefore, you can better focus on what is essential for you.</p><p>This principle is not exclusive to application development. You, as a programmer, abstracted through language syntax from underlying actions with an operation system. The OS, in turn, abstract your language translater from underlying operations with a CPU, memory, NIC, and so on. The more you go deeper, the more you understand that this is just a matter of abstraction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*2f9JU3ZErdL3tJkh" /><figcaption>Source: <a href="https://www.reddit.com/r/ProgrammerHumor/comments/orerw4/abstraction/">Reddit</a></figcaption></figure><h3>Encapsulate what varies</h3><p>As you can see, abstraction can manifest itself in different forms — from data (implementation) abstraction to hierarchical. A general rule of thumb for using abstraction is the principle: “Encapsulate what varies.” Identify the potentially changeable part and declare a concrete interface for it. This way, even if the internal logic changes, the client will still have the same interaction.</p><p>Suppose you need to calculate a currency conversion. At the moment, you only have two currencies. You can come up with something like this:</p><pre>if (baseCurrency == &quot;USD&quot; and targetCurrency == &quot;EUR&quot;) return amount * 0.90;<br>if (baseCurrency == &quot;EUR&quot; and targetCurrency == &quot;USD&quot;) return amount * 1.90;</pre><p>But another type of currency may be added in the future, which would require changes to the client code. Instead, it is better to abstract and encapsulate all the logic in a separate method and call that method from the client side when needed.</p><pre>function convertCurrency(amount, baseCurrency, targetCurrency) {<br>  if (baseCurrency == &quot;USD&quot; and targetCurrency == &quot;EUR&quot;) return amount * 0.90;<br>  if (baseCurrency == &quot;EUR&quot; and targetCurrency == &quot;USD&quot;) return amount * 1.90;<br>  if (baseCurrency == &quot;USD&quot; and targetCurrency == &quot;UAH&quot;) return amount * 38.24;<br>  …<br>}</pre><h3>DRY</h3><p><a href="https://en.wikipedia.org/wiki/Don%27t_repeat_yourself">DRY</a> (don’t repeat yourself), also known as DIE (duplication is evil), states that you shouldn’t duplicate information or knowledge across your code base.</p><blockquote>“Every piece of knowledge must have a single, unambiguous, authoritative representation within a system” — Andy Hunt and Dave Thomas, The Pragmatic Programmer.</blockquote><p>The benefit of reducing code repetition is the simplicity of changing and maintaining. If you duplicate your logic in several places and then find a bug, you’re likely to forget to change it in one of the places, which will lead to different behavior for seemingly identical functionality. Instead, find a repetitive functionality, abstract it in the form of a procedure, class, etc., give it a meaningful name and use it where needed. This advocates a single point of change and minimizes the breaking of unrelated functionality.</p><h3>KISS</h3><p>The <a href="https://en.wikipedia.org/wiki/KISS_principle">KISS</a> (keep it simple, stupid) phrase was coined by aircraft engineer Kelly Johnson, who challenged his engineering team that the jet aircraft they were designing must be repairable by an average mechanic in the field under combat conditions with only specific tools.</p><p>The main idea behind it is to focus on the simplicity of a system, which increases understanding and reduces overengineering while using only the tools you really need.</p><h3>YAGNI</h3><p>When you design a solution to the problem, you are thinking about two things: how to better adapt it to the current system and how to make it extensible for possible future requirements. In the second case, the desire to build a premature feature for the sake of better extensibility is usually wrong: even if you now think that this will reduce the cost of integration, the maintenance and debugging of such code may not be obvious and unnecessarily complicated. Thus, you violate the previous principle by increasing the redundant complexity of the solution to the current problem. Also, don’t forget here’s a good chance that your presumed functionality may not be needed in the future, and then you’re just wasting resources.</p><p>That’s is what <a href="https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it">YAGNI</a> or “You aren’t gonna need it” all about. Don’t get it wrong; you should think about what will be with your solution in the future, but only add code when you actually need it.</p><h3>LoD</h3><p>The <a href="https://en.wikipedia.org/wiki/Law_of_Demeter">Law of Demeter</a> (LoD), sometimes referred to as the principle of least knowledge, advises against talking to “strangers”. Because LoD is usually considered with OOP, a “stranger” in that context means any object not directly associated with the current one.</p><p>The benefit of using Demeter’s Law is maintainability, expressed by avoiding immediate contact between unrelated objects.</p><p>As a result, when you interact with an object and one of the following scenarios is not met, you violate this principle:</p><ul><li>When the object is the current instance of a class (accessed via this)</li><li>When the object is a part of a class</li><li>When the object passed to a method through the parameters</li><li>When the object instantiated inside a method</li><li>When the object is globally available.</li></ul><p>To give an example, let’s consider a situation when a customer wants to make a deposit into a bank account. We might end up here having three classes — Wallet, Customer and Bank.</p><pre>class Wallet {<br> private decimal balance;<br><br> public decimal getBalance() {<br>   return balance;<br> }<br><br> public void addMoney(decimal amount) {<br>   balance += amount<br> }<br><br> public void withdrawMoney(decimal amount) {<br>   balance -= amount<br> }<br>}<br><br>class Customer {<br> public Wallet wallet;<br><br> Customer() {<br>   wallet = new Wallet();<br> }<br>}<br><br>class Bank {<br> public void makeDeposit(Customer customer, decimal amount) {<br>   Wallet customerWallet = customer.wallet;<br><br>   if (customerWallet.getBalance() &gt;= amount) {<br>     customerWallet.withdrawMoney(amount);<br>     //...<br>   } else {<br>     //...<br>   } <br> }<br>}</pre><p>You can see the violation of the Demeter law in the makeDeposit method. Accessing a customer wallet in terms of LoD is right (although it’s strange behavior from the logic perspective). But here, a bank object invokes the getBalance and withdrawMoney from the customerWallet object, thus talking to a stranger (wallet), instead of a friend (customer).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2HD2riIJ7620CzX-hFdXEA.png" /><figcaption>Before applying the LoD principle</figcaption></figure><p>Here’s how to fix it:</p><pre>class Wallet {<br> private decimal balance;<br><br> public decimal getBalance() {<br>   return balance;<br> }<br><br> public boolean canWithdraw(decimal amount) {<br>   return balance &gt;= amount;<br> }<br><br> public boolean addMoney(decimal amount) {<br>   balance += amount<br> }<br><br>  public boolean withdrawMoney(decimal amount) {<br>   if (canWithdraw(amount)) {<br>     balance -= amount;<br>   }<br> }<br>}<br><br>class Customer {<br> private Wallet wallet;<br><br> Customer() {<br>   wallet = new Wallet();<br> }<br><br> public boolean makePayment(decimal amount) {<br>   return wallet.withdrawMoney(amount);<br> }<br>}<br><br>class Bank {<br> public void makeDeposit(Customer customer, decimal amount) {<br>   boolean paymentSuccessful = customer.makePayment(amount);<br><br>   if (paymentSuccessful) {<br>     //...<br>   } else {<br>     //...<br>   }   <br> }<br>}</pre><p>Now all interaction with a customer wallet is going through the customer object. This abstraction favors loose coupling, easy changing of the logic inside the Wallet and Customer classes (a bank object shouldn’t worry about the customer’s internal representation), and testing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xx7e7SMLZF50vlWCgHzUrg.png" /><figcaption>After adopting the LoD principle</figcaption></figure><p>Generally, you can say that LoD fails when there are more than two dots applied to one object, like object.friend.stranger instead of object.friend.</p><h3>SoC</h3><p>The <a href="https://en.wikipedia.org/wiki/Separation_of_concerns">Separation of Concerns</a> (SoC) principle suggests breaking a system into smaller parts depending on its concerns. A “concern” in that meaning implies a distinctive feature of a system.</p><p>For example, if you are modeling a domain then each object can be treated as a special concern. In a layered system, each layer has its own care. In a microservice architecture, each service has its own purpose. This list can continue indefinitely.</p><p>The main thing to take out about the SoC is:</p><ol><li>Identify the system’s concerns;</li><li>Divide the system into separate parts that solve these concerns independently of each other;</li><li>Connect these parts through a well-defined interface.</li></ol><p>In this fashion, the separation of concerts is very similar to the abstraction principle. The result of adhering to SoC is easy-to-understand, modular, reusable, built on stable interfaces and testable code.</p><h3>SOLID</h3><p>The <a href="https://en.wikipedia.org/wiki/SOLID">SOLID</a> principles are a set of five design principles introduced by Robert Martin aimed to clarify the initial constraint of object-oriented programming and make programs more flexible and adaptable.</p><h4>Single responsibility principle</h4><blockquote>“A class should have one, and only one, reason to change.”</blockquote><p>In other words:</p><blockquote>“Gather together the things that change for the same reasons. Separate those things that change for different reasons.”</blockquote><p>This is very similar to SoC, isn’t it? The difference between these two principles is that SRP aims at class-level separation, while SoC is a general approach that works both on high (e.g., layers, systems, services) and low-level (classes, functions, etc.) abstraction.</p><p>The single responsibility principle has all the advantages of SoC, in particular, it promotes high cohesion and low coupling and avoids the <a href="https://en.wikipedia.org/wiki/God_object">god object</a> anti-pattern.</p><h4>Open-closed principle</h4><blockquote>“Software entities should be open for extension but closed for modification.”</blockquote><p>When you implement a new feature, you should keep existing code from breaking changes.</p><p>A class is considered <em>open</em> when you can extend it and add required modifications. A class is considered <em>closed</em> when it has clearly defined interfaces and won’t be changing in the future, i.e., it is available to use for another piece of code.</p><p>Imagine a classic OOP inheritance: you created a parent class and then later extended it with a child class with addition functionality. Then, for some reason, you decided to change an internal structure for the parent class (e.g. add a new field or remove some method), that is also accessible or directly impacts the derived class. By doing it, you violate this principle, because now you not only have to change the parent class, you also should adapt the child class for new changes. It’s happening because information hiding is not being properly applied. Instead, if you give a child class a stable contract through a public property or method, you are free to change your internal structure as long as it does not affect that contract.</p><p>This encourages client dependency on abstraction (e.g., interface or abstract class) rather than implementation (a concrete class). Acting in this way, a client that depended on abstraction is considered closed, but at the same time, it’s open for extension because all new modifications that comply with that abstraction can be seamlessly integrated for the client.</p><p>Let me give you another example. Let’s imagine we are developing discount calculation logic. So far, we only have two types of discount. Before applying the open-closed principle:</p><pre>class DiscountCalculator {<br> public double calculateDiscountedPrice(double amount, DiscountType discount) {<br>   double discountAmount = 15.6;<br>   double percentage = 4.0;<br>   double appliedDiscount;<br><br>   if (discount == &#39;fixed&#39;) {<br>     appliedDiscount = amount - discountAmount;<br>   }<br><br>   if (discount == &#39;percentage&#39;) {<br>     appliedDiscount = amount * (1 - (percentage / 100))  ;<br>   }<br><br>   // logic<br> }<br>}</pre><p>Now, the client (DiscountCalculator) is dependent on the external DiscountType. If we add a new type of discount, we will need to go to this client logic and extend it. That’s undesirable behavior.</p><p>After applying the open-closed principle:</p><pre>interface Discount {<br> double applyDiscount(double amount);<br>}<br><br>class FixedDiscount implements Discount {<br> private double discountAmount;<br><br> public FixedDiscount(double discountAmount) {<br>     this.discountAmount = discountAmount;<br> }<br><br> public double applyDiscount(double amount) {<br>     return amount - discountAmount;<br> }<br>}<br><br>class PercentageDiscount implements Discount {<br> private double percentage;<br><br> public PercentageDiscount(double percentage) {<br>     this.percentage = percentage;<br> }<br><br> public double applyDiscount(double amount) {<br>     return amount * (1 - (percentage / 100));<br> }<br>}<br><br>class DiscountCalculator {<br> public double calculateDiscountedPrice(double amount, Discount discount) {<br>   double appliedDiscount = discount.applyDiscount(amount);<br>   // logic<br> }<br>}</pre><p>Here, you take advantage of the open-closed principle and polymorphism instead of adding multiple if statements to determine the type and future behavior of some entity. All classes that implement the Discount interface are closed with respect to the public applyDiscount method, but at the same time, they are open for modification of their internal data.</p><h4>Liskov substitution</h4><blockquote>“Derived classes must be substitutable for their base classes.”</blockquote><p>Or, more formally:</p><blockquote>“Let φ(x) be a property provable about objects x of type T. Then φ(y) should be true for objects y of type S where S is a subtype of T” (Barbara Liskov &amp; Jeannette Wing, 1994)</blockquote><p>In simple words, when you extend a class, you shouldn’t break the contract established in it. By “breaking a contract”, it means a failure to fulfill one of the next requirements:</p><ol><li>Don’t change the parameters in derived classes: child classes should conform parent’s method signatures, i.e., accept the same parameters as the parent does or accept more abstract parameters.</li><li>Don’t change the return type in derived classes: child classes should return the same type as the parent does or return more concrete (subtype) parameters.</li><li>Don’t throw an exception in derived classes: child classes shouldn’t throw an exception in their methods unless the parent class does. In that case, a type of exception should be the same or be a subtype of the parent’s exception.</li><li>Don’t strengthen preconditions in derived classes: child classes shouldn’t change the expecting client’s behavior by restricting their work to some condition, e.g., in a parent class, you accept a string, but in a child class, you accept a string not more than 100 characters.</li><li>Don’t weaken postconditions in derived classes: child classes shouldn’t change the expecting client’s behavior by allowing turn down some work, e.g., don’t clean up a state after an operation, don’t close a socket, etc.</li><li>Don’t weaken invariants in derived classes: child classes shouldn’t change conditions defined in the parent class, e.g., don’t reassign the field of the parent class because you may not realize the whole logic around it.</li></ol><h4>Interface segregation</h4><blockquote>“Make fine-grained interfaces that are client-specific.”</blockquote><p>Any code shouldn’t depend on methods that it doesn’t need. If a client doesn’t use some behavior of an object, why should it be forced to depend on it? Similarly, if a client doesn’t use some methods, why implementer should be forced to provide this functionality?</p><p>Break “fat” interfaces into more specific ones. If you change a concrete interface, these changes won’t affect unrelated clients.</p><h4>Dependency inversion</h4><blockquote>“Depend on abstractions, not on concretions.”</blockquote><p>Uncle Bob <a href="https://web.archive.org/web/20110714224327/http://www.objectmentor.com/resources/articles/dip.pdf">described</a> this principle as a strict following of OCP and LSP:</p><blockquote>“In this column, we discuss the structural implications of the OCP and the LSP. The structure that results from rigorous use of these principles can be generalized into a principle all by itself. I call it “The Dependency Inversion Principle” (DIP).” — Robert Martin.</blockquote><p>Dependency inversion consists of two main statements:</p><ol><li>High-level modules should not depend upon low-level modules. Both should depend on abstractions</li><li>Abstractions should not depend on details. Details should depend on abstractions.</li></ol><p>For illustration, let’s say we are developing a user service responsible for user management. To persist changes, we decided to use PostgreSQL.</p><pre>class UserService {<br> private PostgresDriver postgresDriver;<br><br> public UserService(PostgresDriver postgresDriver) {<br>     this.postgresDriver = postgresDriver;<br> }<br><br> public void saveUser(User user) {<br>   postgresDriver.query(&quot;INSERT INTO USER (id, username, email) VALUES (&quot; + user.getId() + &quot;, &#39;&quot; + user.getUsername() + &quot;&#39;, &#39;&quot; + user.getEmail() + &quot;&#39;)&quot;);<br> }<br><br> public User getUserById(int id) {<br>   ResultSet resultSet = postgresDriver.query(&quot;SELECT * FROM USER WHERE id = &quot; + id);<br>   User user = null;<br>   try {<br>       if (resultSet.next()) {<br>           user = new User(resultSet.getInt(&quot;id&quot;), resultSet.getString(&quot;username&quot;), resultSet.getString(&quot;email&quot;));<br>       }<br>   } catch (SQLException e) {<br>       e.printStackTrace();<br>   }<br>   return user;<br> }<br><br> // ...<br>}</pre><p>Right now, UserService is tightly coupled to its dependency (PostgresDriver). But later on, we decided to migrate to the MongoDB database. Because MongoDB differs from PostgreSQL, we need to rewrite every method in our UserService class.</p><p>The solution for it is to introduce an interface:</p><pre>interface UserRepository {<br> void saveUser(User user);<br> User getUserById(int id);<br> // ...<br>}<br><br>class UserPGRepository implements UserRepository {<br> private PostgresDriver driver;<br><br> public UserPGRepository(PostgresDriver driver) {<br>   this.driver = driver;<br> }<br><br> public void saveUser(User user) {<br>   // ...<br> }<br><br> public User getUserById(int id) {<br>   // ...<br> }<br> // ...<br>}<br><br>class UserMongoRepository implements UserRepository {<br> private MongoDriver driver;<br><br> public UserMongoRepository(MongoDriver driver) {<br>   this.driver = driver;<br> }<br><br> public void saveUser(User user) {<br>   // ...<br> }<br><br> public User getUserById(int id) {<br>   // ...<br> }<br> // ...<br>}<br><br>class UserService {<br> private UserRepository repository;<br><br> public UserService(UserRepository repository) {<br>     this.repository = repository;<br> }<br><br> public void saveUser(User user) {<br>   repository.saveUser(user);<br> }<br><br> public User getUserById(int id) {<br>   return repository.getUserById(id);<br> }<br> // ...<br>}</pre><p>Now high-level module (UserService) is dependent on abstraction (UserRepository), and abstraction does not depend on details (SQL API for PostgreSQL and Query API for MongoDB); it depends on the interface constructed for the client.</p><p>In particular, to achieve dependency inversion, the dependency injection technique can be used, which you can read about at the link below:</p><p><a href="https://javascript.plainenglish.io/demystifying-dependency-injection-an-essential-guide-for-software-developers-364949c28a5f">Demystifying Dependency Injection: An Essential Guide for Software Developers</a></p><h3>GRASP</h3><p>The <a href="https://en.wikipedia.org/wiki/GRASP_(object-oriented_design)">General Responsibility Assignment Principles</a> (GRASP) is a set of nine principles used in object-oriented design present by Craig Larman in his book Applying UML and Patterns.</p><p>Similar to SOLID, these principles aren’t built from scratch but rather composed of time-tested programming guidelines in the context of OOP.</p><h4>High cohesion</h4><blockquote>“Keep related functionalities and responsibilities together.”</blockquote><p>The high cohesion principle focused on keeping complexity manageable. In this context, the cohesion is a degree of how close the responsibilities of an object are. If a class has low cohesion, it means it’s doing work unrelated to its primary purpose or doing work that can be delegated to another subsystem.</p><p>Generally, a class designed with high cohesion has a small number of methods, and all these methods are highly related by their functionality. Doing that increases the maintainability, understanding, and reuse of code.</p><h4>Low coupling</h4><blockquote>“Reduce relations between unstable elements.”</blockquote><p>This principle aims to provide low dependency between elements, therefore preventing side effects arising from a code change. Here, coupling means a measure of how strongly one entity is dependent (has knowledge of or relies on) on another entity.</p><p>Program elements with high coupling very strongly rely on each other. If you have a class with high coupling, changes in it will lead to local changes in other parts of a system or vice versa. Such a design limits code reuse and takes more time to understand. On the other hand, low coupling supports the design of classes that are more independent, which reduces the impact of change.</p><p>The principles of coupling and cohesion go hand in hand. If you have two classes with high cohesion, the connection between them is usually weak. Similarly, if those classes have a low coupling between each other, by definition, they have a high cohesion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aiWLgpfgWSu5Ibu76bqvbA.png" /><figcaption>The relation between coupling and cohesion</figcaption></figure><h4>Information expert</h4><blockquote>“Place responsibilities with data.”</blockquote><p>Information expert pattern answers the question of how we should assign responsibility for knowing a piece of information or doing some work. By following this pattern, an object who has direct access to needed information is considered to be an information expert on it.</p><p>Remember the example of applying Demeter’s Law between a customer and a bank? It’s essentially the same thing:</p><ul><li>The Wallet class is an information expert on knowing balance and managing it.</li><li>The Customer class is an information expert regarding its internal structure and behavior.</li><li>The Bank class is an information expert in a bank area.</li></ul><p>The fulfillment of a responsibility often requires gathering information across different parts of a system. Because of that, there should be intermediate information experts. With them, objects keep their internal information, which increases encapsulation and lowers coupling.</p><h4>Creator</h4><blockquote>“Assign the responsibility for object creation to a closely related class.”</blockquote><p>Who should be responsible for creating a new object instance? According to the Creator pattern, to create a new instance of x class, a creator class should have one of the following properties:</p><ul><li>Aggregate x;</li><li>Contain x;</li><li>Records x;</li><li>Closely use x;</li><li>Have required initialization data for x.</li></ul><p>This principle promotes low coupling because if you find the right creator for an object, i.e., a class that is already somehow related to that object, you won’t increase their connectivity.</p><h3>Controller</h3><blockquote>“Assing the responsibility for handling system messages to a specific class.”</blockquote><p>A controller is a system object responsible for receiving and delegating user events to the domain layer. It’s the first element that receives a service request from UI. Usually, one controller is used for handling similar use cases, such as UserController for managing user entity interaction.</p><p>Keep in mind that the controller should not do any business work. It should be as thin as possible. It should delegate work to the appropriate classes, not be responsible for it.</p><p>As an example, we can find the controller in MVC-like design patterns. Instead of direct communication between the model and the view, MVC introduces the controller — an intermediate part responsible for handling interaction between the view and model. With this component, the model becomes independent of external interaction with it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*627DDHv0O-W9xhbT" /><figcaption>Source: <a href="https://developer.mozilla.org/en-US/docs/Glossary/MVC">MDN Web Docs</a></figcaption></figure><h4>Indirection</h4><blockquote>“To support a low coupling, assign the responsibility to an intermediate class.”</blockquote><p>There is a famous aphorism of Butler Lampson: “All problems in computer science can be solved by another level of indirection.”</p><p>The indirection principle has the same idea as the dependency inversion principle: to introduce an intermediate between two elements so they become <em>indirect</em>. This is done to support weak coupling and has all the advantages that come with it.</p><h4>Polymorphism</h4><blockquote>“When related alternative behaviors vary by type, use polymorphism to assign responsibility to the types for which the behavior varies.”</blockquote><p>If you saw the code that checks object types using if/switch statements, it probably has a lack of polymorphism application. When required to extend this code with new functionality, you have to go to the place where the condition is checked and add a new if statement. That’s a poor design.</p><p>Using the polymorphism principle for different classes with related behavior, you unify different types, which leads to interchangeable software components, each responsible for specific functionality. Depending on the language, it can be done in numerous ways, but the common ones are to implement the same interface or use inheritance, in particular, giving the same name to methods for different objects. In the end, you get pluggable, easy-to-extend elements that don’t require changing unrelated code.</p><p>Like with the similar open-closed principle, it’s important to use polymorphism properly. It’s needed to apply only when you are sure that some components will or may vary. You don’t need to create an abstraction for implementing polymorphism, for example, on top of language-internal classes or a framework. They are already stable and you are just doing unnecessary work.</p><h4>Pure fabrication</h4><blockquote>“To support a high cohesion, assign the responsibility to the convenient class.”</blockquote><p>Sometimes, to follow high cohesion/low coupling principles, there is no corresponding real-world entity. From this point of view, you are creating a <em>fabrication</em>, something that does not exist in a domain. It’s <em>pure</em> because the responsibilities of this entity are clearly designed. Craig Larman suggests using this principle when an information expert applies logically wrong.</p><p>For example, a controller pattern is a pure fabrication. A DAO or a repository is also a fabrication. These classes are not exclusive to some domains but are convenient for developers. Although, we could put data access logic directly in the domain classes (since there are expert in their area), this would lead to a violation of high cohesion because data management logic is not directly related to how the domain object behaves. Coupling is also increased because we need to depend on the database interface. And code duplication is likely to be high because managing the data for different domain entities has similar logic. In other words, this leads to mixing different abstractions in one place.</p><p>The benefit of using pure fabrication classes is to group related behavior in objects, for which there is no alternative in the real world. This results in a good design with code reuse and low dependency on different responsibilities.</p><h4>Protected variations</h4><blockquote>“Protect predicted variations by introducing a stable contract.”</blockquote><p>To provide future changes without breaking other parts, you need to introduce a stable contract that stops undetermined impacts. This principle emphasizes the importance of earlier discussed principles for separating responsibilities between different objects: you need to apply the indirection to easily switch between different implementations, you need to use the information expert to decide who should be responsible for the fulfillment of a requirement, you need to design your system with the polymorphism in mind to introduce varying pluggable solutions, etc.</p><p>The protected variations principle is a core concept that drives other design patterns and principles.</p><blockquote>“At one level, the maturation of a developer or architect can be seen in their growing knowledge of ever-wider mechanisms to achieve PV, to pick the appropriate PV battles worth fighting, and their ability to choose a suitable PV solution. In the early stages, one learns about data encapsulation, interfaces, and polymorphism — all core mechanisms to achieve PV. Later, one learns techniques such as rule-based languages, rule interpreters, reflective and metadata designs, virtual machines, and so forth — all of which can be applied to protect against some variation.” — Craig Larman, Applying UML and Patterns.</blockquote><h3>Conclusion</h3><p>I am sure that before reading this article you have already unconsciously used some of the principles discussed. Now, you have a name for them, making it easier to communicate on the language-agnostic level.</p><p>You may have noticed that some of the principles have the same underlying idea. Essentially, it’s how it is. For example, information expert, SoC, high cohesion &amp; low coupling, SRP, interface segregation, etc., all have the same point of view to separate concerns between different software elements. And they do it to fulfill protected variations, dependency inversion, and indirection to get maintainable, extensible, understandable, and testable code.</p><p>As with any tool, this is a guideline, not a strict rule. The key is to understand the trade-offs and make informed decisions. I intentionally did not touch upon the topic of the misapplication of software principles so that you could think for yourself and find answers. Not knowing the other side of the coin is as bad as not knowing these principles at all.</p><p>Thank you for reading this article!</p><p>Any questions or suggestions? Feel free to write a comment.</p><p>I’d also be happy if you followed me and gave this article a few claps! 😊</p><p>Check out some of my latest articles here:</p><ul><li><a href="https://medium.com/@pavlokolodka/list/98fb3445020a">Software Engineering</a></li><li><a href="https://medium.com/@pavlokolodka/list/4b0f9817d17d">Node.js</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a39a98b685d" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/the-20-essential-principles-of-software-development-lod-soc-solid-and-beyond-7a39a98b685d">The 20 Essential Principles of Software Development: LoD, SoC, SOLID, and Beyond.</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Delving into Message Brokers Internals]]></title>
            <link>https://levelup.gitconnected.com/delving-into-message-brokers-internals-deeb23a20ff0?source=rss-c87ee5e3bb6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/deeb23a20ff0</guid>
            <category><![CDATA[point-to-point]]></category>
            <category><![CDATA[message-queue]]></category>
            <category><![CDATA[pub-sub]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[message-broker]]></category>
            <dc:creator><![CDATA[Pavlo Kolodka]]></dc:creator>
            <pubDate>Thu, 29 Feb 2024 02:52:32 GMT</pubDate>
            <atom:updated>2024-04-12T06:49:17.173Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understanding a Message Queue, Communication Models, Pros and Cons.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2H6PJuZB_rKOjQaPCoV-iw.png" /><figcaption>Thumbnail created by <a href="https://medium.com/@pavlokolodka"><strong>Author</strong></a></figcaption></figure><h3>Introduction</h3><p>Message brokers are one of the most important components of modern software architecture. Understanding them is key to designing reliable services that can handle high loads, multiple long-running tasks, and support asynchronous communication.</p><p>In this article, I will focus on the concept of a queue-based message broker, one of the most popular broker implementations.</p><h3>Message queue</h3><p>Before we delve into message broker details, we must understand its fundamental part — the message queue.</p><p>A <a href="https://en.wikipedia.org/wiki/Message_queue">message queue</a> (MQ) is a component used in distributed systems to enable asynchronous communication. In essence, it’s a <a href="https://en.wikipedia.org/wiki/Queue_(abstract_data_type)">queue data structure</a> (<a href="https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)">FIFO</a> order) that accepts and stores messages until they are processed by an application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eOewcI3cLUdih6hCUy28Aw.png" /><figcaption>Example of a message queue</figcaption></figure><p>A message represents a unit of data that is sent from one application to another through a message queue.</p><p>It may be sent with various data types (text, binary, JSON, etc.) and includes a message body, headers, and other configuration properties such as delivery mode, acknowledgment, designation queue, policies, etc.</p><p>Messages fall into several categories depending on their purpose:</p><ul><li><a href="https://www.enterpriseintegrationpatterns.com/patterns/messaging/CommandMessage.html">Commands</a>: These messages are intended to transfer a specific action to the consumer. They often carry action names such as create_order, and additional details to process requests. A command message implies one consumer.</li></ul><pre>{<br>  &quot;command&quot;: &quot;create_order&quot;,<br>  &quot;order_id&quot;: &quot;1573b634-b1fc-4e72–8f57–86ec461391d7&quot;,<br>  &quot;customer_id&quot;: &quot;a84abe57-e7f9–4619-a7e9–4f08652849f2&quot;,<br>  &quot;products&quot;: [<br>    {<br>      &quot;product_id&quot;: &quot;2f5f32f2–3818–4876-aa77-a87d0f309498&quot;,<br>      &quot;quantity&quot;: 2<br>    },<br>    {<br>      &quot;product_id&quot;: &quot;f50f8aff-0f2d-4a30–8bbb-1040c1b723a5&quot;,<br>      &quot;quantity&quot;: 1<br>    }<br>  ],<br>  &quot;total_amount&quot;: 49.99,<br>  &quot;timestamp&quot;: &quot;2024–02–27T12:00:00Z&quot;<br>}</pre><ul><li><a href="https://www.enterpriseintegrationpatterns.com/patterns/messaging/EventMessage.html">Events</a>: This is a special type of message to represent a state change. Oftentimes, events don’t need additional context because the event itself is sufficient and more valuable than the data contained within it.</li></ul><pre>{<br>  &quot;event_type&quot;: &quot;package_status_changed&quot;,<br>  &quot;package_id&quot;: &quot;PKG123456789&quot;,<br>  &quot;status&quot;: &quot;delivered&quot;,<br>  &quot;location&quot;: &quot;123 Main St, Anytown, USA&quot;,<br>  &quot;timestamp&quot;: &quot;2024–02–27T15:30:00Z&quot;<br>}</pre><ul><li><a href="https://www.enterpriseintegrationpatterns.com/patterns/messaging/DocumentMessage.html">Documents</a>: A document is a message intended to pass data and let the receiver decide what to do. A document message itself can be any kind of message, e.g. serializable domain object. This type of message implies one zero or n consumers.</li></ul><p>The main difference between a document message and an event message is that in a document message, the content is more important than the sending/receiving time.</p><pre>{<br>  &quot;id&quot;: &quot;1573b634-b1fc-4e72–8f57–86ec461391d7&quot;,<br>  &quot;product_name&quot;: &quot;Smartphone X&quot;,<br>  &quot;description&quot;: &quot;A high-performance smartphone with advanced features.&quot;,<br>  &quot;price&quot;: 599.99,<br>  &quot;category&quot;: &quot;Electronics&quot;,<br>  &quot;brand&quot;: &quot;TechCo&quot;,<br>  &quot;availability&quot;: true,<br>  &quot;timestamp&quot;: &quot;2024–02–27T10:00:00Z&quot;<br>}</pre><ul><li>Queries: It’s a special type of command to retrieve information. A query command implies one consumer.</li></ul><pre>{<br>  &quot;type&quot;: &quot;get_order&quot;,<br>  &quot;order_id&quot;: &quot;1573b634-b1fc-4e72–8f57–86ec461391d7&quot;,<br>}</pre><ul><li>Replies: Reply messages are the response to a command or query message. The response address is typically sent along with the request message, and when the consumer has processed the message, it sends it back to its destination. A reply message implies one consumer.</li></ul><p>Command message:</p><pre>{<br>  &quot;message_type&quot;: &quot;command&quot;,<br>  &quot;command_type&quot;: &quot;fetch_user_data&quot;,<br>  &quot;user_id&quot;: &quot;123456&quot;,<br>  &quot;response_address&quot;: &quot;response_queue&quot;,<br>  &quot;timestamp&quot;: &quot;2024–02–27T13:00:00Z&quot;<br>}</pre><p>Reply message:</p><pre>{<br>  &quot;message_type&quot;: &quot;response&quot;,<br>  &quot;request_id&quot;: &quot;14892cf3-656b-462c-98d0-6df5124bf915&quot;,<br>  &quot;user_id&quot;: &quot;123456&quot;,<br>  &quot;name&quot;: &quot;John Doe&quot;,<br>  &quot;email&quot;: &quot;john.doe@example.com&quot;,<br>  &quot;address&quot;: &quot;123 Main St, Anytown, USA&quot;,<br>  &quot;timestamp&quot;: &quot;2024–02–27T13:05:00Z&quot;<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-mDq8IYpW9DTj3Dmi8evgw.png" /><figcaption>The five main message types</figcaption></figure><p>To standardize communication between systems using a message queue, one of these protocols is often used:</p><ul><li><a href="https://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol">Advanced Message Queuing Protocol</a> (AMQP): AMQP is an open standard messaging protocol designed for message-oriented middleware environments.</li><li><a href="https://en.wikipedia.org/wiki/Streaming_Text_Oriented_Messaging_Protocol">Streaming Text Oriented Messaging Protocol</a> (STOMP): STOMP is a text-based messaging protocol that defines a format for sending and receiving messages between clients and message brokers.</li><li><a href="https://en.wikipedia.org/wiki/MQTT">MQ Telemetry Transport</a> (MQTT): MQTT is a lightweight publish-subscribe messaging protocol designed for IoT (Internet of Things) applications and low-bandwidth, high-latency, or unreliable networks.</li></ul><p>MQ systems ensure reliable and ordered delivery of messages, typically providing features like persistence, message acknowledgment, and guaranteed delivery.</p><h3>Message broker</h3><p>A message broker is an intermediary software component responsible for facilitating communication between different applications or components by routing, transforming, and usually storing messages.</p><p>It acts as a middleware (<a href="https://en.wikipedia.org/wiki/Message-oriented_middleware">MOM</a>) where producers (applications that send messages) and consumers (applications that receive messages) can connect without being aware of each other.</p><p>Most message brokers are based on a message queue, although they may support a different structure, e.g., log-based, or other ordering using stacks, priority or delay features, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wUY3xt5u_xEjXKPhtlTJuA.png" /><figcaption>The relationship between message broker and message queue</figcaption></figure><p>On top of that, they provide additional features such as message filtering, routing based on message content, protocol translation, and message transformation.</p><p>To receive a message from a broker, there are two options: pull and push. With pull clients manually request the broker to receive the message, while with push clients subscribe to the broker, and the broker sends the message automatically.</p><p>Popular examples of queue-based message brokers include RabbitMQ, ActiveMQ, IBM MQ, Amazon SQS, Redis, EMQX.</p><h3>Message sending patterns:</h3><p>When employing a broker, there are two main messaging patterns: P2P and Pub/Sub. These patterns describe a relation between the sender and receiver.</p><p>Each of them typically supports asynchronous fire-and-forget and asynchronous request-response communication methods which are focused on the overall message flow.</p><p>You can see the difference between them in each messaging pattern below, but in short, with the fire-and-forget strategy message sender does not expect to receive a response from the consumer.</p><h4>P2P (work/task queue)</h4><p><a href="https://en.wikipedia.org/wiki/Point-to-point_(telecommunications)">Point-to-point</a> (P2P) is a messaging pattern used to enable a one-to-one communication strategy.</p><p>In that type of communication, a producer must know a destination (queue).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r7J4sOF9z02Iiaxbn3D_0w.png" /><figcaption>Example of point-to-point communication with the fire-and-forget approach</figcaption></figure><p>Because the P2P messaging pattern guarantees that only one client will receive a message from a queue, typical load balancing between consumers is done using the <a href="https://en.wikipedia.org/wiki/Round-robin_scheduling">round-robin algorithm</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zLqbJaWnrdHDvoxIC3Fl3g.png" /><figcaption>Point-to-point communication with multiple consumers</figcaption></figure><blockquote>It is worth noting that when there are multiple consumers, messages do not always go in FIFO order — one consumer may take longer to process messages than another and thus disrupt the order.</blockquote><p>In the request-response type of messaging, a message publisher sends a message to one queue (request queue) and then waits, usually blocking the dedicated thread, for a response in another queue (response queue).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J3fR7xs1HeIJhZvvOCSclw.png" /><figcaption>Example of point-to-point communication with the request/response approach</figcaption></figure><h4>Pub/Sub</h4><p><a href="https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern">Publish/Subscribe</a> (Pub/Sub) is a messaging pattern used to enable a one-to-many communication strategy.</p><p>In the typical Pub/Sub communication, there is a producer (also called a publisher) and a consumer (subscriber). The producer sends a message to a topic, and then the broker broadcasts it to all interested subscribers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ve6EiDNPrtX3dbivPxAoIg.png" /><figcaption>Example of typical Pub/Sub communication</figcaption></figure><p>When we use the Pub/Sub model with message queueing, it creates one or several queues in front of subscribers. In such a way, the broker combines broadcast transmission with a delivery guarantee.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jREg2aJFBrynTr_GmM4Ngg.png" /><figcaption>Pub/Sub communication with message retention</figcaption></figure><p>Here, you can see the difference between the P2P and the Pub/Sub patterns:</p><ol><li>In the P2P a producer should know the exact queue to send for, whereas in the Pub/Sub a producer should know only the topic.</li><li>In a Pub/Sub strategy, all consumers subscribed to the same topic receive their own copy of the message, whereas in P2P, even if multiple consumers subscribe to the same queue, only one of them will receive the message at a single point in time.</li></ol><h3>Pros and cons</h3><p>The message broker plays an important role in various distributed system architectures, but like any other technology, it has its trade-offs.</p><p><strong>Pros:</strong></p><ul><li>Long-running task and pick load surviving: When a large volume of messages is received from producers, the message broker can preserve them, ensuring that downstream services are not overloaded.</li><li>Decoupling of applications: Message brokers enable decoupling between applications or services by acting as intermediaries for communication. This decoupling allows applications to interact without needing to know the specifics of each other’s implementations, leading to more modular and scalable architectures.</li><li>Asynchronous communication: Because message brokers act as middleware, they facilitate asynchronous communication between components, where messages are sent and received without requiring both parties to be actively available at the same time.</li><li>Load balancing and scaling: Message brokers can distribute messages across multiple consumers, enabling load balancing and horizontal scaling of processing tasks. This capability is especially valuable in scenarios with high message throughput or variable processing demands.</li><li>Microservices communication: In microservices architectures, message brokers are commonly used for inter-service communication, allowing microservices to communicate asynchronously without direct dependencies on each other.</li></ul><p><strong>Cons:</strong></p><ul><li>Steep learning curve: While the total concept of message brokers may be easy to understand, actual adoption is not that simple. There are many nuances to consider: what type of message broker will be more appropriate, what configuration will be used, etc.</li><li>Overall system complexity: Integrating a message broker into a system leads to increased architectural complexity. It includes refactoring existing communication, adding security options, complicated debugging, etc. Also, introducing a message broker may slow down existing message communication because of the additional component in the middle. So now you rely on eventual consistency, instead of immediate data processing and access.</li></ul><h3>Summing up</h3><p>In this article, we have looked at the message queue, which is the key mechanism of most brokers, different types of messages, the message broker, types of communication between the broker and clients, and the pros and cons of using it.</p><p>Thank you for reading this article!</p><p>Any questions or suggestions? Feel free to write a comment.</p><p>I’d also be happy if you followed me and gave this article a few claps! 😊</p><p>Check out some of my latest articles here:</p><ul><li><a href="https://medium.com/@pavlokolodka/list/98fb3445020a">Software Engineering</a></li><li><a href="https://medium.com/@pavlokolodka/list/4b0f9817d17d">Node.js</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=deeb23a20ff0" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/delving-into-message-brokers-internals-deeb23a20ff0">Delving into Message Brokers Internals</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I wrote my own Promise implementation]]></title>
            <link>https://javascript.plainenglish.io/i-wrote-my-own-promise-implementation-5175231b229d?source=rss-c87ee5e3bb6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/5175231b229d</guid>
            <category><![CDATA[promises-in-javascript]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Pavlo Kolodka]]></dc:creator>
            <pubDate>Thu, 25 Jan 2024 12:17:36 GMT</pubDate>
            <atom:updated>2024-01-25T12:17:36.985Z</atom:updated>
            <content:encoded><![CDATA[<h4>Here’s how you can write your own Promises/A+ compatible implementation of the JavaScript Promise object</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*asPRW6PWYfzY0s5SnUv9Hw.png" /><figcaption>Thumbnail created by <a href="https://medium.com/@pavlokolodka"><strong>Author</strong></a></figcaption></figure><p>When it comes to learning a new concept or technology, the best way to truly understand it is to try to implement it by yourself.</p><p>You will not only get the idea of how your tool works, but also you will use it more confidently instead of thinking about it as a “<a href="https://en.wikipedia.org/wiki/Black_box">black box</a>”.</p><p>I love how Richard Feynman once famously said:</p><blockquote>“What I cannot create, I do not understand. Know how to solve every problem that has been solved.”</blockquote><p>So, I decided to deepen my understanding of how the JavaScript Promise works by implementing it myself.</p><h3>What is a promise?</h3><p>The <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise">Promise</a> is an object that represents a value at some time in the future.</p><p>Historically, the Promise object came to JavaScript in 2015 with the <a href="https://www.w3schools.com/js/js_es6.asp">ES6</a> (ECMAScript 2015) version to provide an abstraction for asynchronous operations on top of callbacks.</p><p>Let’s take a look at a simple example using a callback to handle an asynchronous operation:</p><pre>function fetchData(callback) {<br>  // Simulating an asynchronous operation (e.g., API call)<br>  setTimeout(function () {<br>    const data = { message: &quot;Data fetched successfully!&quot; };<br>    // Transferring data for further processing<br>    callback(null, data);<br>  }, 1000);<br>}<br><br>// Using callback to handle async operation<br>fetchData(function (error, result) {<br>  if (error) {<br>    console.error(&quot;Error:&quot;, error);<br>  } else {<br>    console.log(result);<br>  }<br>});</pre><p>Here is the equivalent version using the Promise object:</p><pre>function fetchData() {<br>  return new Promise(function (resolve, reject) {<br>    // Simulating an asynchronous operation (e.g., API call)<br>    setTimeout(function () {<br>      const data = { message: &quot;Data fetched successfully!&quot; };<br>      // Transferring data for further processing<br>      resolve(data);<br>    }, 1000);<br>  });<br>}<br>// Using Promise to handle async operation<br>fetchDataPromise()<br>.then(function (result) {<br>  console.log(result);<br>})<br>.catch(function (error) {<br>  console.error(&quot;Error:&quot;, error);<br>});</pre><p>Now, with the Promise as a contract for asynchronous operations, developers can write more clean code, in particular, to avoid the <a href="http://callbackhell.com/">callback hell</a>.</p><h4>Terminology</h4><p>The Promise object has 3 states:</p><ol><li><em>pending</em> — this is the initial state, e.g. new Promise()</li><li><em>fulfilled</em> — an indication of the successful promise resolution, e.g. the onFulfilled callback in promise.then(onFulfilled) was called.</li><li><em>rejected</em> — an indication of the unsuccessful promise resolution, e.g. the onRejected callback in promise.then(undefined, onRejected) was called.</li></ol><p>These 3 states are mutually exclusive, meaning if a promise has moved to one of them, like pending =&gt; fulfilled or pending =&gt; rejected, it cannot move into the other, like fulfilled =&gt; pending. If a promise is <em>fulfilled</em> or <em>rejected</em>, it is called <em>settled</em>.</p><p>Here is a detailed picture of the states discussed above:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/0*zZRhLUW7g1nk8o7R" /><figcaption>Source: <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise">MDN Promise object</a></figcaption></figure><p>You cannot directly access the internal state of a promise or its value/reason. To do it, you should use the then() method of the promise instance.</p><h4>The then() method</h4><p>The <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/then">then()</a> method is a significant method in the Promise object. It allows you to access a promise value via two prepared callbacks: then(onFulfilled, onRejected)</p><ul><li>onFulfilled(value) — will be asynchronously executed when a promise that has a called then() method is fulfilled. If omitted, the fulfillment value will be passed on.</li><li>onRejected(reason) — will be asynchronously executed when a promise that has a called then() method is rejected. If omitted, the rejection reason will be passed on.</li></ul><p>When the then() is called, it returns a new Promise instance, allowing you to construct a <a href="https://medium.com/@pavlokolodka/i-wrote-my-own-promise-implementation-5175231b229d">promise chaining</a>, where for the promise returned by then() method, another then() method is called, and so on.</p><p>The returned value inside onFulfilled or onRejected function will be the fulfillment value of this promise.</p><h3>Implementation</h3><p>When I was developing my version of the Promise, I used two things:</p><ol><li>Specification: I used <a href="https://developer.mozilla.org/en-US/">MDN docs</a> and Promises/A+ standard as a reference.</li><li>Testing: I used Promises/A+ tests to make sure I didn’t break existing functionality while adding new code.</li></ol><h4>Promises/A+ standard</h4><p>As stated on their <a href="https://promisesaplus.com/">website</a>:</p><blockquote>“An open standard for sound, interoperable JavaScript promises — by implementers, for implementers.”</blockquote><p>It’s the specification that mainly focused on then() method. They have a detailed algorithm for running then() method as well as for <a href="https://promisesaplus.com/#the-promise-resolution-procedure">resolution procedure</a>.</p><p>I already covered the essentials for then() method, so let’s look at the resolution procedure. In simple words, if a promise is an instance of the Promise object and x is a value to resolve that promise, then:</p><ol><li>If x is a <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise#thenables">thenable</a>, then the thenable’s then() method is called with the same resolve and reject callback function that was passed into the Promise constructor to access the thenable’s value (it may also be a thenable, so step 1 will be done again). If calling then() method throws an exception, the current promise is rejected with this exception as a reason.</li><li>If x is a non-thenable, then the promise is immediately fulfilled with x as a value.</li></ol><h4>The code</h4><p>The implementation process was pretty straightforward. <a href="https://github.com/pavlokolodka/async/blob/master/promise.js">Here</a> is the final code with comments pointing to concrete steps in the specification.</p><pre>module.exports = class MyPromise {<br>    // 2.1 the states of MyPromise: pending, fulfilled, or rejected<br>    #state;<br>    // a MyPromise’s eventual value/reason<br>    #value;<br>    // an array for fulfilled handler functions<br>    #onFulfilledHandlers;<br>    // an array for rejected handler functions<br>    #onRejectedHandlers;<br>  <br>    /**<br>     * @param {Function} fn - A handler function with 2 prepared callbacks: `resolve` and `reject`  <br>     * <br>     * `resolve`- resolves with a given value<br>     * <br>     * `reject`- rejects with a given reason<br>     * <br>     * @example<br>     * <br>     * const promise = new MyPromise((resolve, reject) =&gt; {<br>     *    setTimeout(() =&gt; resolve(&#39;resolved!&#39;), 1000);<br>     *    // setTimeout(() =&gt; reject(&#39;rejected!&#39;), 1000);<br>     * })<br>     * @return {MyPromise}<br>     */<br>    constructor(fn) {<br>      this.#state = &#39;pending&#39;;<br>      this.#onFulfilledHandlers = [];<br>      this.#onRejectedHandlers = [];<br>  <br>      try {<br>        this.#resolve(fn, this.#fulfill.bind(this), this.#reject.bind(this));<br>      } catch (error) {<br>        this.#reject(error);<br>      }<br>    }<br>  <br>    // 2.3<br>    #handleResolution(promise, x, resolve, reject) { <br>      // 2.3.1<br>      if (promise === x) {<br>        reject(new TypeError(&#39;Chaining cycle detected for promise&#39;));<br>        return;<br>      }<br>  <br>      if (x &amp;&amp; (typeof x === &#39;object&#39; || typeof x === &#39;function&#39;)) {<br>        let thenCalled = false;<br>  <br>        try {<br>          // 2.3.3.1<br>          const then = x.then;<br>  <br>          // 2.3.3.3<br>          if (typeof then === &#39;function&#39;) {<br>            then.call(<br>              x,<br>              (y) =&gt; {<br>                // 2.3.3.3.3<br>                if (thenCalled) return;<br>                thenCalled = true;<br>                // 2.3.3.3.1<br>                this.#handleResolution(promise, y, resolve, reject);<br>              },<br>              (r) =&gt; {<br>                // 2.3.3.3.3<br>                if (thenCalled) return;<br>                thenCalled = true;<br>                // 2.3.3.3.2<br>                reject(r);<br>              }<br>            );<br>            return;<br>          }<br>         // 2.3.3.2, 2.3.3.3.4<br>        } catch (error) {<br>          // 2.3.3.3.4.1<br>          if (thenCalled) return;<br>          // for an async resolvePromise/rejectPromise invocation<br>          thenCalled = true;<br>          // 2.3.3.3.4.2<br>          reject(error);<br><br>          return;<br>        }<br>      }<br>  <br>      resolve(x);<br>    }<br>  <br>    #fulfill(value) {<br>      // 2.1.2, 2.1.3<br>      if (this.#state !== &#39;pending&#39;) return;<br><br>      this.#value = value;<br>      this.#state = &#39;fulfilled&#39;;<br>      <br>      // 2.2.4<br>      setTimeout(() =&gt; {<br>        // 2.2.6.1<br>        this.#onFulfilledHandlers.forEach((fn) =&gt; fn());<br>      }, 0);<br>    }<br>  <br>    #reject(reason) {<br>      // 2.1.2, 2.1.3<br>      if (this.#state !== &#39;pending&#39;) return;<br>      <br>      this.#value = reason;<br>      this.#state = &#39;rejected&#39;;<br>      <br>      // 2.2.4<br>      setTimeout(() =&gt; {<br>        // 2.2.6.2<br>        this.#onRejectedHandlers.forEach((fn) =&gt; fn());<br>      }, 0);<br>    }<br>   <br>    #resolve(fn, onFulfilled, onRejected) {<br>      fn(<br>        (value) =&gt; {<br>          this.#handleResolution(this, value, onFulfilled, onRejected)<br>        },<br>        (reason) =&gt; {<br>          onRejected(reason);<br>        }<br>    );<br>  }<br>  <br>    #callCallback(value, callback, resolve, reject) {<br>      try {     <br>        // 2.2.5           <br>        const result = callback(value);<br>        // 2.2.7.1<br>        resolve(result);<br>      } catch (error) {<br>        // 2.2.7.2<br>        reject(error);<br>      }<br>    }<br>  <br>    /**<br>     * If the passed argument is a function, it will be called asynchronously when the current `MyPromise` is settled.<br>     * <br>     * Subscribe to given `MyPromise` instance<br>     * and register 2 callbacks:<br>     * `onFulfilled` and `onRejected`<br>     * <br>     * `onFulfilled(value)` takes the fulfilled value as a parameter from subscribed `MyPromise`.  Its return value becomes the fulfillment value of the promise returned by `then()`.<br>     * <br>     * `onRejected(reason)` takes the rejected reason as a parameter from subscribed `MyPromise`.  Its return value becomes the fulfillment value of the promise returned by `then()`.<br>     * <br>     * It immediately returns an equivalent `MyPromise` object, allowing you to chain<br>     * calls to other `MyPromise` methods. This new promise is always pending when returned, regardless of the current promise&#39;s status.<br>     * <br>     * @example<br>     * <br>     * then(onFulfilled)<br>     * then(onFulfilled, onRejected)<br>     * then(undefined, onRejected)<br>     * then(undefined, undefined)<br>     *<br>     * then(<br>     *  (value) =&gt; { fulfillment handler },<br>     *  (reason) =&gt; { rejection handler },<br>     * )<br>     * @param {Function | Any} onFulfilled<br>     * @param {Function | Any} onRejected<br>     * @return {MyPromise}<br>     */<br>    // 2.2<br>    then(onFulfilled, onRejected) {<br>      // 2.2.1.1, 2.2.7.3<br>      onFulfilled = typeof onFulfilled === &#39;function&#39; ? onFulfilled : (x) =&gt; x;<br>      // 2.2.1.2, 2.2.7.4<br>      onRejected = typeof onRejected === &#39;function&#39; ? onRejected : (r) =&gt; {throw r};<br>      // 2.2.7<br>      return new MyPromise((resolve, reject) =&gt; {<br>        if (this.#state === &#39;pending&#39;) {<br>          // 2.2.2<br>          this.#onFulfilledHandlers.push(() =&gt; {<br>              this.#callCallback(this.#value, onFulfilled, resolve, reject);<br>          });<br>          // 2.2.3<br>          this.#onRejectedHandlers.push(() =&gt; {<br>            this.#callCallback(this.#value, onRejected, resolve, reject);<br>          });<br>        }<br>  <br>        if (this.#state === &#39;fulfilled&#39;) {<br>          // 2.2.4<br>          setTimeout(() =&gt; {<br>            this.#callCallback(this.#value, onFulfilled, resolve, reject);<br>          }, 0);<br>        }<br>        <br>        if (this.#state === &#39;rejected&#39;) {<br>          // 2.2.4<br>          setTimeout(() =&gt; {<br>            this.#callCallback(this.#value, onRejected, resolve, reject);<br>          }, 0);<br>        }<br>      });<br>    }<br>  <br>    /**<br>     * Syntactic sugar for .then(undefined, reject)<br>     * <br>     * @param {Function} onRejected<br>     * @returns {MyPromise} <br>     */<br>    catch(onRejected) {<br>      return this.then(undefined, onRejected);<br>    }<br>  };</pre><h4>Testing</h4><p>Testing is a crucial part. To ensure that my code remains Promise/A+ compatible, I used <a href="https://github.com/promises-aplus/promises-tests">their tests</a>.</p><p>All implementations, <a href="https://promisesaplus.com/implementations#the-ecmascript-specification">including</a> ECMAScript specification, passed this test suit can called Promises/A+ compliant.</p><p>To run the tests, you need to install the promises-aplus-tests package and prepare the <a href="https://github.com/promises-aplus/promises-tests?tab=readme-ov-file#adapters">adapter interface</a>:</p><pre>const MyPromise = require(&#39;../promise&#39;);<br><br>module.exports = {<br>    deferred: function() {<br>        let resolve, reject;<br>        return {<br>            promise: new MyPromise(function (res, rej) {<br>                resolve = res;<br>                reject = rej;<br>            }),<br>            resolve: resolve,<br>            reject: reject<br>        };<br>    }<br>};<br><br>// Backward compatibility with unhandledRejection changes<br>// Source: https://github.com/promises-aplus/promises-tests/issues/93<br>require(&#39;process&#39;).on(&#39;unhandledRejection&#39;, (err) =&gt; console.debug)</pre><p>Result of the running tests:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8BJkpxPPg0sMG9it" /><figcaption>Screenshot of the passed tests</figcaption></figure><h3>Conclusion</h3><p>Understanding the internals of the language will help you write cleaner, more reliable code. Plus, you will feel more confident with the tool you rely on.</p><p>I had a lot of fun doing this! I can recommend writing your own version to truly understand the internal workings of the Promise object.</p><p>If you find a bug, please post it in the comments below or <a href="https://github.com/pavlokolodka/async/issues">submit</a> the issue on GitHub.</p><p>If you’ve made it through the entire article, congratulations! You’re awesome! ️😎</p><p>Any questions or suggestions? Feel free to write a comment.</p><p>I’d also be happy if you followed me and gave this article a few claps! 😊</p><p>Check out some of my latest articles here:</p><ul><li><a href="https://medium.com/@pavlokolodka/list/4b0f9817d17d">Node.js</a></li><li><a href="https://medium.com/@pavlokolodka/list/98fb3445020a">Software Engineering</a></li></ul><h3>In Plain English 🚀</h3><p><em>Thank you for being a part of the </em><a href="https://plainenglish.io/"><strong><em>In Plain English</em></strong></a><em> community! Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://twitter.com/inPlainEngHQ"><strong>X</strong></a><strong> | </strong><a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a><strong> | </strong><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a><strong> | </strong><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a><strong> | </strong><a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a></li><li>Visit our other platforms: <a href="https://stackademic.com/"><strong>Stackademic</strong></a><strong> | </strong><a href="https://cofeed.app/"><strong>CoFeed</strong></a><strong> | </strong><a href="https://venturemagazine.net/"><strong>Venture</strong></a></li><li>More content at <a href="https://plainenglish.io/"><strong>PlainEnglish.io</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5175231b229d" width="1" height="1" alt=""><hr><p><a href="https://javascript.plainenglish.io/i-wrote-my-own-promise-implementation-5175231b229d">I wrote my own Promise implementation</a> was originally published in <a href="https://javascript.plainenglish.io/">JavaScript in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Free hosting services for your projects in 2024]]></title>
            <link>https://levelup.gitconnected.com/free-hosting-services-for-your-projects-in-2024-50e76918a3f2?source=rss-c87ee5e3bb6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/50e76918a3f2</guid>
            <category><![CDATA[hosting-service]]></category>
            <category><![CDATA[free-hosting]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[hosting]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Pavlo Kolodka]]></dc:creator>
            <pubDate>Thu, 04 Jan 2024 15:44:50 GMT</pubDate>
            <atom:updated>2024-01-04T15:44:50.108Z</atom:updated>
            <content:encoded><![CDATA[<h4>Discover 7 free hosting services for static and dynamic websites.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Koddpd1VUCWPhzDP2fAWuQ.png" /><figcaption>Thumbnail created by <a href="https://medium.com/@pavlokolodka"><strong>Author</strong></a></figcaption></figure><h3>Introduction</h3><p>For developers who want to showcase their projects, experiment with new technologies, or simply share their creations with the world, choosing the right hosting can be crucial.</p><p>However, the cost of hosting can be an obstacle for many, especially for those who are just starting their coding journey. This is where free hosting comes to the rescue.</p><p>In this article, I’ll show you 7 free hosting alternatives for both static and dynamic websites.</p><h3>Static site hosting</h3><p>A static website is a type of website created using only HTML, CSS and JavaScript. The pages of such a site are stored and requested as standalone files.</p><p>The main difference between a static website and a dynamic website is the content of a static website updates manually and it’s the same for every user.</p><h4>GitHub pages</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PuVF1teQ_-sq-RwOl_U1MQ.png" /><figcaption><a href="https://pages.github.com/">GitHub pages</a></figcaption></figure><p><a href="https://pages.github.com/">GitHub pages</a> is a service that allows publishing static webpages through the GitHub repository.</p><p>With a free plan, you can create your site with built-in HTTPS encryption, a rate limiter, a custom domain name, and more than 750 MIME types.</p><p><a href="https://docs.github.com/en/pages/getting-started-with-github-pages/about-github-pages#usage-limits">Limitations</a>:</p><ul><li>Up to 1GB for site size</li><li>Up to 100GB of site bandwidth</li><li>Up to 10 minutes for site deployment</li></ul><h4>GitLab pages</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AxnipyxdSr8_wJzJL2H3eA.png" /><figcaption><a href="https://about.gitlab.com/stages-devops-lifecycle/pages/">GitLab pages</a></figcaption></figure><p>Similar to GitHub pages, <a href="https://about.gitlab.com/stages-devops-lifecycle/pages/">GitLab pages</a> also lets you publish your static website via the GitLab repository.</p><p>GitLab automatically protects default domains .gitlab.io with HTTPS protocol, allows you to use a custom domain, as well as create custom error pages.</p><p><a href="https://docs.gitlab.com/ee/user/gitlab_com/">Limitations</a>:</p><ul><li>Up to 1GB for site size</li><li>Up to 200000 site files</li></ul><h4>Cloudflare pages</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sU3vYtZtF2SQuvY5ujHjdQ.png" /><figcaption><a href="https://pages.cloudflare.com/">Cloudflare pages</a></figcaption></figure><p><a href="https://pages.cloudflare.com/">Cloudflare pages</a> is a hosting service for any static HTML site. It provides integration with many frameworks and static site generators like React, Vue, Gatsby.</p><p><a href="https://developers.cloudflare.com/pages/platform/limits/">Limitations</a>:</p><ul><li>Up to 25MIB for site size</li><li>Up to 20000 site files</li><li>Up to 500 times/month builds</li></ul><h3>Dynamic site hosting</h3><h4>Vercel</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RhN4XH-ksshmPQwXndyREA.png" /><figcaption><a href="https://vercel.com/">Vercel</a></figcaption></figure><p><a href="https://vercel.com/">Vercel</a> is a cloud platform created primarily for deploying frontend applications.</p><p>But you can also use its Serverless Function infrastructure to deploy your backend APIs.</p><p><a href="https://vercel.com/docs/limits/overview">General limitations</a>:</p><ul><li>Up to 13GB of disk size</li><li>Up to 100GB of site bandwidth</li><li>Up to 45 minutes for site deployment</li><li>Up to 8192MB of RAM</li><li>Up to 2 CPU cores</li></ul><p><a href="https://vercel.com/docs/functions/serverless-functions/runtimes#size-limits">Serverless Function limitations</a>:</p><ul><li>Up to 250MB bundle size</li><li>Up to 100GB of site bandwidth</li><li>Up to 10s of max duration</li><li>Up to 4.5MB of the request body size</li><li>Up to 1024MB of RAM</li></ul><h4>Firebase</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1LlQKQbPXA7YWYLVYB6u1g.png" /><figcaption><a href="https://firebase.google.com/">Firebase</a></figcaption></figure><p><a href="https://firebase.google.com/">Firebase</a> is a development platform that offers several services, including building, monitoring, and hosting software.</p><p>With Firebase hosting, you can publish both static and dynamic content, Express.js APIs with many built-in features, such as changing a domain name, configuring an SSL, and integration with popular frameworks.</p><p><a href="https://firebase.google.com/pricing">Limitations</a>:</p><ul><li>Up to 10GB of disk size</li><li>Up to 360MB/day, 10GB/month of site bandwidth</li></ul><h4>Render</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WFFoM_XHDYWfODVCn2xLCA.png" /><figcaption><a href="https://render.com/">Render</a></figcaption></figure><p><a href="https://render.com/">Render</a> is a platform providing cloud infrastructure.</p><p>It comes with many built-in runtimes like Node.js, Python, Go, Docker. You can also use this service for static sites.</p><p>Render gives you web services functionality with HTTP/2 and TLS, the capability to change a domain name, automatic continuous deploys from Git.</p><p><a href="https://render.com/pricing">Limitations</a>:</p><ul><li>Up to 100GB of site bandwidth</li><li>Up to 512MB of RAM</li><li>Up to 0.15 CPU cores</li><li>Up to 500min/month of pipeline minutes</li></ul><h4>Back4App</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4ETtIuvHmx7MR2R9oeX1-g.png" /><figcaption><a href="https://www.back4app.com/">Back4App</a></figcaption></figure><p><a href="https://www.back4app.com/">Back4App</a> offers built-in APIs, databases, hosting, and more.</p><p>You can run your code inside Docker containers, configure seamless GitHub deployment integration and monitor it.</p><p><a href="https://www.back4app.com/pricing/container-as-a-service">Limitations</a>:</p><ul><li>Up to 100GB of site bandwidth</li><li>Up to 256MB of RAM</li><li>Up to 0.25 CPU cores</li></ul><h3>Wrapping up</h3><p>The choice of hosting depends on the unique requirements of your project, your skill level, and the community and support you are looking for.</p><p>By navigating the variety of offerings available, you will not only save on costs, but also gain valuable experience managing and deploying applications in a real-world environment.</p><p>Thank you for reading this article! If you have any questions or suggestions, feel free to write a comment.</p><p>I’d also be happy if you clapped this article a few times! 😊</p><p>Check out some of my latest articles:</p><ul><li><a href="https://javascript.plainenglish.io/how-i-choose-the-right-packages-for-my-projects-b0e3efc075b6">How I Choose the Right Packages for My Projects</a></li><li><a href="https://levelup.gitconnected.com/from-roles-to-attributes-comparing-authorization-models-94e670016e4e">From Roles to Attributes — Comparing Authorization Models</a></li><li><a href="https://levelup.gitconnected.com/deep-dive-into-node-js-asynchronous-architecture-cd61c643135e">Deep Dive into Node.js Asynchronous Architecture</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=50e76918a3f2" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/free-hosting-services-for-your-projects-in-2024-50e76918a3f2">Free hosting services for your projects in 2024</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Choose the Right Packages for My Projects]]></title>
            <link>https://javascript.plainenglish.io/how-i-choose-the-right-packages-for-my-projects-b0e3efc075b6?source=rss-c87ee5e3bb6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/b0e3efc075b6</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[npm]]></category>
            <category><![CDATA[npm-package]]></category>
            <category><![CDATA[project-dependencies]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Pavlo Kolodka]]></dc:creator>
            <pubDate>Thu, 28 Dec 2023 04:16:56 GMT</pubDate>
            <atom:updated>2023-12-28T10:23:55.604Z</atom:updated>
            <content:encoded><![CDATA[<h4>Here are the three steps I apply when picking a dependency for a project.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LQT79ae0NOXWs8MwpKBzgw.png" /><figcaption>Thumbnail created by <a href="https://medium.com/@pavlokolodka">Author</a></figcaption></figure><h3>Introduction</h3><p>Choosing the right package for a project can be a challenging task, especially when there are a large number of options to choose from.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*l0a_pIcRJXC24Vhv.jpg" /><figcaption>Source: <a href="https://programmerhumor.io/javascript-memes/npm-install-wolololo-2/">programmerhumor.io</a></figcaption></figure><p>No matter what programming language you use, here are three tips you can apply when picking a dependency for your project to save hours of debugging.</p><h3>The first thing that you need to check is a project’s license.</h3><p>The license defines restrictions of package use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8J-21QWL-FKS9dGs" /><figcaption>Source: <a href="https://snyk.io/learn/what-is-gpl-license-gplv3-explained/">snyk.io</a></figcaption></figure><p>In many cases, you will see that the package uses a permissive type license like BSD, MIT or Apache.</p><p>But it may also use other licenses like GPL, which is a copyleft license type. So, be sure you don’t skip this step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SP0b3ojassoaosg6uFbhmA.png" /><figcaption>Example of using BSD and MIT licenses</figcaption></figure><h3>After that, you need to inspect the project’s maintenance.</h3><p>You can do the basic check directly on a GitHub page.</p><p>If a project doesn’t have a description, README, license or recent contributions, you should start worrying about it.</p><p>Also, review security policies, the number of pull requests and issues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KzqGR-SMArTbIeOk-C4GHQ.png" /><figcaption>Example of the <a href="https://github.com/auth0/node-jsonwebtoken">jsonwebtoken GitHub repository</a></figcaption></figure><p>If the project is large, it can contain up to several hundred issues. But if such issues are too many, consider using an alternative to it.</p><h3>And the third tip is to check the project’s popularity.</h3><p>Look at the number of weekly downloads, the number of “Used By” and the number of dependents. Typically, big numbers are a sign of project trust.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-q4Trfx-5xzt1B6EQJmycg.png" /><figcaption>Comparison of two similar packages designed to test whether a number is odd or not</figcaption></figure><h3>Conclusion</h3><p>These tips are the minimum I use when I find several alternative solutions for my problem. And what is your approach? Share in the comments!</p><p>Thank you for reading this article! If you have any questions or suggestions, feel free to write a comment.</p><p>I’d also be happy if you clapped this article a few times! 😊</p><p>Check out some of my latest articles:</p><ul><li><a href="https://levelup.gitconnected.com/from-roles-to-attributes-comparing-authorization-models-94e670016e4e">From Roles to Attributes — Comparing Authorization Models</a></li><li><a href="https://levelup.gitconnected.com/deep-dive-into-node-js-asynchronous-architecture-cd61c643135e">Deep Dive into Node.js Asynchronous Architecture</a></li><li><a href="https://levelup.gitconnected.com/understanding-abstraction-levels-in-database-interactions-dal-dao-raw-queries-query-builder-4819d607b0d6">Understanding Abstraction Levels in Database Interactions: DAL, DAO, Raw Queries, Query Builder…</a></li></ul><h3><a href="https://plainenglish.io/">PlainEnglish.io</a> 🚀</h3><p><em>Thank you for being a part of the In Plain English community! Before you go:</em></p><ul><li><em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> the writer</em><strong>️</strong></li><li><em>Learn how you can also </em><a href="https://plainenglish.io/blog/how-to-write-for-in-plain-english"><strong><em>write for In Plain English</em></strong></a><em>️</em></li><li><em>Follow us: </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>X</em></strong></a><strong><em> | </em></strong><a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><strong><em> | </em></strong><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a><strong><em> | </em></strong><a href="https://discord.gg/in-plain-english-709094664682340443"><strong><em>Discord</em></strong></a><strong><em> | </em></strong><a href="https://newsletter.plainenglish.io/"><strong><em>Newsletter</em></strong></a></li><li><em>Visit our other platforms: </em><a href="https://stackademic.com/"><strong><em>Stackademic</em></strong></a><strong><em> | </em></strong><a href="https://cofeed.app/"><strong><em>CoFeed</em></strong></a><strong><em> | </em></strong><a href="https://venturemagazine.net/"><strong><em>Venture</em></strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b0e3efc075b6" width="1" height="1" alt=""><hr><p><a href="https://javascript.plainenglish.io/how-i-choose-the-right-packages-for-my-projects-b0e3efc075b6">How I Choose the Right Packages for My Projects</a> was originally published in <a href="https://javascript.plainenglish.io/">JavaScript in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Roles to Attributes — Comparing Authorization Models]]></title>
            <link>https://levelup.gitconnected.com/from-roles-to-attributes-comparing-authorization-models-94e670016e4e?source=rss-c87ee5e3bb6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/94e670016e4e</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[rbac]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[authorization]]></category>
            <category><![CDATA[access-control]]></category>
            <dc:creator><![CDATA[Pavlo Kolodka]]></dc:creator>
            <pubDate>Thu, 16 Nov 2023 16:12:46 GMT</pubDate>
            <atom:updated>2023-11-16T16:12:46.936Z</atom:updated>
            <content:encoded><![CDATA[<h3>From Roles to Attributes — Comparing Authorization Models</h3><h4>Exploring the differences between RBAC, CBAC, PBAC, and ABAC authorization modes.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IzNWAG_LWjEC5WGTU8Kmvg.png" /><figcaption>Thumbnail created by <a href="https://medium.com/@pavlokolodka"><strong>Author</strong></a></figcaption></figure><h3>Introduction</h3><p>Keeping digital information safe is a top priority. One crucial aspect of this is controlling who has access to what, and that’s where authorization models come in.</p><p>Choosing the suitable authorization model is like picking the best tool for the job, especially as organizations deal with different types of users and ever-changing environments. Each model has its own strengths, perks, and challenges. In this journey, we’ll explore RBAC, PBAC, CBAC, and ABAC to understand what makes each of them tick.</p><h3>Role-based access control</h3><p>In <a href="https://csrc.nist.gov/projects/role-based-access-control">role-based access control</a> (RBAC), each user is assigned a specific role(s). For instance, roles can be User, Admin, Developer, etc.</p><p>Each role means some predefined set of permissions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*C_D7-paUa9zObE5k" /><figcaption>Potential RBAC database design</figcaption></figure><p>Back to our example, the User role could imply that a customer has the ability only to add, delete, update their own resources, and view other user’s resources, while the Admin role can mean all described before plus managing other users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XCU9zvdPvXd2IbjM" /><figcaption>RBAC evaluation flow</figcaption></figure><p>Difficulties with RBAC start when your app grows.</p><p>Imagine a situation when your security policy changes and you must remove or add roles for certain request paths.</p><p>Or what would you do if some subset of users in a role have unique requirements that do not apply to other users with the same role?</p><h3>Claims-based access control</h3><p>In contrast to RBAC, in <a href="https://learn.microsoft.com/en-us/aspnet/core/security/authorization/claims?view=aspnetcore-7.0">claims-based access control</a> (CBAC), each user has one or more claims.</p><p>A claim is a name-value pair representing what the subject is, not what the subject can do.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tG71Rsro-w9TH-KB" /><figcaption>Potential CBAC database design</figcaption></figure><p>A name of the claim is the declarative string describing the type of claim. If we want to check a user’s age, the claim name may be dateOfBirth.</p><p>A value of the claim is an optional value that the system can refer to if the claim name is insufficient. Back to the user’s age example, the value of the age claim could be 09/23/1983.</p><p>User role can also be considered as a claim.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tgrDHAqX6J1Om4lf" /><figcaption>CBAC evaluation flow</figcaption></figure><p>Claims can be issued by anyone. It may be your authorization server, Google authentication, or whatever. It’s up to the application to understand and accept claims.</p><p>CBAC also has the disadvantages described above. If your app needs more flexibility in the context of access control, binding broadly defined authorization rights in the application logic is not a good idea.</p><h3>Permission-based access contol</h3><p>In permission-based access control (PBAC), instead of assigning a role or a claim to a user, we grant them specific permission.</p><p>A permission itself is one small indivisible action on a resource.</p><p>Depending on the app’s requirements, you may split permissions to reading/writing or even break them down to CRUD operations.</p><p>As an illustration, if you have a Post entity representing posts in the blog system, you can create 4 main permissions: createPost, readPost, updatePost, and deletePost.</p><p>If a user has the readPost permission, they can access a resource but cannot modify it. Similarly, if the user has the deletePost permission, they can perform a delete operation.</p><p>You may ask, “How to manage all these permissions?”</p><p>The trick is to not assign permission directly to a user. Instead, assign it to a role or a claim. Thus, a role or a claim would be a group of permissions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZfDs2tJmzmTIT1Kz" /><figcaption>PBAC evaluation flow</figcaption></figure><p>If you want to change a user’s permissions for a role/claim, <strong>you don’t need to make code changes and redeploy the entire application.</strong></p><p>Or, if you come up with the idea to add a new role to the application structure, you don’t need to audit roles across the entire code base.</p><p>Instead, you can configure the desired behavior from outside. In such a way, you separate your business logic from the security logic.</p><h3>Attribute-based access control</h3><p>In <a href="https://csrc.nist.gov/projects/attribute-based-access-control">attribute-based access control</a> (ABAC), a user’s privileges are determined by attributes.</p><blockquote>Sometimes, ABAC is referred to as CBAC or the policy-based access control. CBAC is more Microsoft term.</blockquote><p>An attribute can be anything. In general, attributes fall into the following categories:</p><ol><li>Subject attributes — attributes that specify the user who requested the resource. Examples: role, age, name, etc.</li><li>Object attributes — attributes that specify the requested object. Examples: isPublished, createdBy, ID, etc.</li><li>Action attributes — attributes that specify an action on the requested object. Examples: read, write, invite.</li><li>Environment attributes — attributes that are specified the context where the operation is done. Examples: time, environment — development, production; location, etc.</li></ol><p>ABAC can combine all access control strategies together because a user’s role, claim, and permission can be thought of as an attribute.</p><p>Let’s imagine an auth policy for a blog system.</p><p>A user can do CRUD actions on their own articles and read other articles. An admin can manage all.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*32FV9Q4cCqhdetbB" /><figcaption>Potential ABAC requirements for a blog system</figcaption></figure><p>As you can see, ABAC requirements are more flexible and, therefore, complex to implement.</p><p>Utilizing ABAC often involves using a policy engine (aka decision point) that evaluates attributes and makes access control decisions. Typical request in the system with attribute-based access control is described using <a href="https://en.wikipedia.org/wiki/Abbreviated_Language_for_Authorization">ALFA</a> or <a href="https://en.wikipedia.org/wiki/XACML">XACML</a> language.</p><h3>Conclusion</h3><p>The choice between RBAC, PBAC, CBAC, and ABAC is a delicate balance between simplicity and flexibility, structure and adaptability.</p><p>RBAC takes the stage in the simplicity of authorization models, offering convenience through predefined roles. It brings order and clarity to access control, making it an excellent choice for straightforward scenarios. However, this simplicity comes at a cost — flexibility. While RBAC excels in structured environments, adapting to evolving access requirements might feel like fitting a square peg into a round hole.</p><p>On the other hand, CBAC steps onto the scene, embracing flexibility by basing access on claims. This model accommodates dynamic user scenarios, allowing for more adaptability to changing conditions. Yet, the trade-off lies in potential complexity, as managing and interpreting claims can add additional complexity to the authorization process.</p><p>PBAC steps in, providing a balanced approach with its permission-based model. It bridges the simplicity of RBAC with the adaptability of CBAC, offering a middle ground that caters to a wide array of use cases.</p><p>Finally, ABAC takes the spotlight, providing a granular approach by considering attributes in access decisions. This model embraces complexity to deliver precision, making it suitable for scenarios where detailed control over access is paramount. However, the drawback lies in potential administrative overhead and complexity of implementation due to managing many attributes.</p><p>Thank you for reading this article! If you have any questions or suggestions, feel free to write a comment.</p><p>I’d also be happy if you clapped this article a few times! 😊</p><p>Check out some of my latest articles:</p><ul><li><a href="https://levelup.gitconnected.com/deep-dive-into-node-js-asynchronous-architecture-cd61c643135e">Deep Dive into Node.js Asynchronous Architecture</a></li><li><a href="https://levelup.gitconnected.com/understanding-abstraction-levels-in-database-interactions-dal-dao-raw-queries-query-builder-4819d607b0d6">Understanding Abstraction Levels in Database Interactions: DAL, DAO, Raw Queries, Query Builder…</a></li><li><a href="https://javascript.plainenglish.io/demystifying-dependency-injection-an-essential-guide-for-software-developers-364949c28a5f">Demystifying Dependency Injection: An Essential Guide for Software Developers</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=94e670016e4e" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/from-roles-to-attributes-comparing-authorization-models-94e670016e4e">From Roles to Attributes — Comparing Authorization Models</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Developer’s Reflections: 8 Things I’d Tell Myself Before Starting Out]]></title>
            <link>https://medium.com/@pavlokolodka/developers-reflections-8-things-i-d-tell-myself-before-starting-out-75dfac146bf5?source=rss-c87ee5e3bb6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/75dfac146bf5</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[storytelling]]></category>
            <category><![CDATA[advice]]></category>
            <category><![CDATA[i-wish-id-known-before]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Pavlo Kolodka]]></dc:creator>
            <pubDate>Mon, 28 Aug 2023 06:01:01 GMT</pubDate>
            <atom:updated>2023-08-28T06:01:01.996Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kEnW-q1yzQpyBnwv" /><figcaption>Photo by <a href="https://unsplash.com/@karlp?utm_source=medium&amp;utm_medium=referral">Karl Pawlowicz</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>When you start something from scratch, you will make mistakes that you can only realize after you have grown in the endeavor.</p><p>In this article, I want to share my top common mistakes that I wish I had known about when I started out as a software engineer.</p><p>P.S. This article was originally planned as a continuation of my <a href="https://www.linkedin.com/posts/pavlokolodka_learntocode-programming-softwaredevelopment-activity-7096732202071171073-fHGE?utm_source=share&amp;utm_medium=member_desktop">LinkedIn post</a>, so <a href="https://medium.com/linkedin.com/in/pavlokolodka/">follow me</a> to make sure you don’t miss any similar content.</p><h3>1. Practice trumps pure theory:</h3><p>While theory is essential, practical application is key.</p><p>The Pareto principle holds true — 20% theory lays the foundation for 80% practical use. Dive into a hands-on experience to truly grasp concepts. The synergy between theory and practice is where true mastery takes root, empowering you to confidently navigate the dynamic landscape of software development.</p><h3>2. Progressive problem-solving:</h3><p>Don’t get stuck on a single problem. If you hit a roadblock, move on to the next challenge. Returning later with fresh eyes can often lead to breakthroughs without excessive time investment.</p><h3>3. Try to understand how things work under the hood:</h3><p>In a world focused on integration, grasp the inner workings of your tools. As Richard Feynman wisely said,</p><blockquote>“What I cannot create, I don’t understand.”</blockquote><p>Creating something from scratch is not only fun, but it helps you understand your tool more deeply and use it more reasonably.</p><h3>4. Learn computer science.</h3><p>In the above tip, I said that you don’t need to focus on theory so much.</p><p>While this is true, understanding the basis of CS will make you a well-rounded developer. Technologies may vary, but the basic principle will be the same. Learn algorithms, data structures, asymptotic analysis, discrete mathematics, different programming paradigms, patterns, etc.</p><h3>5. Contribute to open source.</h3><p>If it sounds scary, trust me it’s not.</p><p>You don’t have to contribute to large, complicated projects. Start with a simple one. Find developers on the same level as you or a little higher and ask how you can help them. You can start your own project and write about it on social platforms to gain the look of future contributors.</p><h3>6. Document your journey.</h3><p>This is probably one of the most important pieces of advice.</p><p>I started my blog this year and even though I don’t have many followers I enjoy leading it. Documenting your journey allows you to track your achievements and see the progress. This is one of the main parts that helps you stay motivated.</p><p>As a software developer, you can use a simple document where you will write all your local achievements, blogging platforms like Medium or Hashnode, do videos on YouTube, or write posts on social media like Linkedin.</p><h3>7. Don’t compare yourself with others.</h3><p>I love the 4th rule from Jordan Peterson’s book “12 Rules for Life”:</p><blockquote>“Compare Yourself to who you Were Yesterday, not to who Someone Else is Today”.</blockquote><p>Comparing yourself with others can be detrimental to your growth and well-being. From bad moods to impostor syndrome and negative motivation, it can all be the result of comparing yourself to others.</p><p>Everyone has a different background, some requirements can vary from company to company. In one company you may be Senior Engineer but at the same time, you may be Middle Engineer in another.</p><p>Focus on self-improvement and embrace the opportunity to learn, grow, and evolve on your own terms. Remember, your journey as a software developer is uniquely yours.</p><h3>8. Software engineering is a marathon, not a sprint.</h3><p>Just as in a marathon, success in software engineering comes not from rushing through the course, but from pacing oneself, maintaining consistency, and continually pushing forward.</p><p>You can’t learn software engineering in 6 months, you can’t learn it in 1 year and you will probably never learn it completely. There will always be an opportunity to grow and explore something new.</p><p>Software development demands continuous learning and adaptation to evolving technologies, methodologies, and user needs. Set long-term goals, celebrate milestones, and enjoy your journey.</p><h3>Conclusion:</h3><p>That’s all!</p><p>Although this list could continue, these are the top 8 mistakes I wish I had been aware of when I started out.</p><p>And what were your biggest mistakes when you first started your developer journey? Share your opinions in the comments!</p><p>Thank you for reading this article! If you have any questions or suggestions, feel free to write a comment.</p><p>Check out my last articles:</p><ul><li><a href="https://levelup.gitconnected.com/deep-dive-into-node-js-asynchronous-architecture-cd61c643135e">Deep Dive into Node.js Asynchronous Architecture</a></li><li><a href="https://levelup.gitconnected.com/understanding-abstraction-levels-in-database-interactions-dal-dao-raw-queries-query-builder-4819d607b0d6">Understanding Abstraction Levels in Database Interactions: DAL, DAO, Raw Queries, Query Builder…</a></li><li><a href="https://javascript.plainenglish.io/demystifying-dependency-injection-an-essential-guide-for-software-developers-364949c28a5f">Demystifying Dependency Injection: An Essential Guide for Software Developers</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75dfac146bf5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deep Dive into Node.js Asynchronous Architecture]]></title>
            <link>https://levelup.gitconnected.com/deep-dive-into-node-js-asynchronous-architecture-cd61c643135e?source=rss-c87ee5e3bb6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd61c643135e</guid>
            <category><![CDATA[event-loop]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <category><![CDATA[libuv]]></category>
            <category><![CDATA[asynchronous]]></category>
            <category><![CDATA[nodejs]]></category>
            <dc:creator><![CDATA[Pavlo Kolodka]]></dc:creator>
            <pubDate>Thu, 27 Jul 2023 14:03:19 GMT</pubDate>
            <atom:updated>2024-07-12T13:24:51.962Z</atom:updated>
            <content:encoded><![CDATA[<h3>Deep Dive into Node.js Asynchronous Architecture: Understanding libuv, the Event Loop, and the Thread Pool”</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VRYzIWkEeNnCco3UDdlo-w.png" /><figcaption>Thumbnail created by <a href="https://medium.com/@pavlokolodka">Author</a></figcaption></figure><h3>Introduction</h3><p>Unless you’re a Node.js developer or a framework creator, you don’t necessarily need to know how everything works under the hood.</p><p>But understanding the underlying architecture, the pros and cons, and the bottlenecks is key to being a good user of your tool.</p><p>In this article, we will delve into the topic of asynchronous execution in Node.js, specifically focusing on two crucial components: the event loop and the thread pool.</p><p>By exploring these concepts, you will gain a comprehensive understanding of how Node.js adeptly manages asynchronous operations, empowering you to develop highly scalable and performant applications.</p><h3>What is Node js?</h3><p><a href="https://nodejs.org/en">Node.js</a> is a JavaScript platform that supports nonblocking single-threaded event-driven execution.</p><p>In other words, it is an environment where JavaScript is executed (similar to the browser).</p><p>Node.js takes JavaScript from a domain-specific to a general-purpose programming language by giving it APIs to work with the operating system.</p><p>Due to its architecture, it is more suitable for I/O-intensive tasks and loses out to other programming languages in CPU-intensive tasks¹.</p><p>Node.js consists of several components. Most important is the engine (v8), abstraction for asynchronous working with OS (libuv), and Node.js API that provides functions and utilities on top of C++.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CpVhRdAI4FzP7p5l.png" /><figcaption>Source: <a href="https://chathuranga94.medium.com/nodejs-architecture-concurrency-model-f71da5f53d1d">https://chathuranga94.medium.com/nodejs-architecture-concurrency-model-f71da5f53d1d</a></figcaption></figure><blockquote>Note 1: You can overcome this problem by spawning threads/processes or by using C/C++ addons/bindings.</blockquote><h3>Event-driven</h3><p>In the context of Node.js, being “event-driven” refers to the fundamental architectural design and programming paradigm that Node.js follows. Node.js applications are structured around the concept of events and event handling.</p><p>In an event-driven architecture, the flow of the program is determined by events that occur asynchronously and trigger corresponding event handlers or callbacks. Instead of following a traditional sequential execution flow, where one operation is performed after another, an event-driven program responds to events as they happen.</p><p>In Node.js, events can be generated by various sources, such as user input, network requests, file system operations, timers, or external services. When an event occurs, Node.js triggers the associated event handler, allowing the program to respond to that event.</p><p>To implement the concept of event-driven programming, Node.js offers the event emitter pattern via <a href="https://nodejs.org/api/events.html#class-eventemitter">EventEmitter</a> class.</p><h4>Event emitter</h4><p>The event emitter pattern (aka Observer pattern) is a widely used software design pattern that facilitates communication and interaction between different components or objects in an event-driven system. It provides a mechanism for one object (the event emitter) to notify other objects (event listeners or subscribers) when a specific event or action occurs.</p><p>The event emitter pattern consists of the following key components:</p><ul><li><strong>Event Emitter</strong>: An event emitter is an object that can emit events. It provides methods to register event listeners, emit events, and manage subscriptions. The event emitter keeps track of the registered event listeners and dispatches events to them synchronously when appropriate.</li><li><strong>Event</strong>: An event represents a specific action or occurrence that can trigger notifications. It can be predefined (e.g., ‘click’, ‘change’, ‘request’) or custom-defined based on the application’s needs.</li><li><strong>Event Listener/Subscriber</strong>: Event listeners or subscribers are objects or functions that are interested in receiving notifications about specific events. They register themselves with the event emitter and provide callback functions or event handlers that are executed when the corresponding event occurs.</li></ul><p>Most Node.js core modules are built with the EventEmitter interface. For example, an instance of <a href="https://nodejs.org/api/net.html#class-netserver">net.Server</a> emits the connection event when a new connection is made, in <a href="https://nodejs.org/api/stream.html#class-streamreadable">stream.Readable</a> event data is emitted whenever data is available for processing.</p><h3>What is asynchronous programming?</h3><p><a href="https://en.wikipedia.org/wiki/Asynchrony_(computer_programming)">Asynchronous execution</a> (async) refers to a programming paradigm that allows tasks or operations to be performed independently of the main program thread. That means that the main thread wouldn’t be blocked.</p><p>JavaScript, by itself, is a blocking and synchronous programming language. What makes it asynchronous is an environment such as a browser or Node.js.</p><p>When we talk about asynchronous execution in Node.js, we mostly mean non-blocking I/O, which, unlike synchronous execution, where operations are performed sequentially (blocking the main thread), does not require waiting for the main thread.</p><p>This allows programs to process multiple operations at the same time without waiting for each one to complete.</p><p>To better understand asynchronous execution, let’s take a look at the following diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5RMpzpUSY1HvhQEFyQ05JA.png" /><figcaption>Single-threaded application (synchronous)</figcaption></figure><p>In the single-threaded blocked app, we receive a request, blocking the main thread to process it and then return a response. Only after that we can process the next request.</p><p>This approach has its disadvantages because we can only process and execute 1 request at a time, while other client requests have to wait for their turn.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UeYAPGq6LeFl-dZ9vRu04A.png" /><figcaption>Multi-threaded application (synchronous)</figcaption></figure><p>To eliminate this disadvantage, we can use the multi-threaded blocking concept. In this approach, for every request the application will assign a new thread and execution will be done concurrently. Incoming requests will wait only if the maximum number of threads is spawned and all threads are busy processing earlier received requests.</p><p>This model has its disadvantages, such as thread <a href="https://en.wikipedia.org/wiki/Context_switch">context switching</a> overhead, or the situation when all threads are only waiting for each other to finish (<a href="https://en.wikipedia.org/wiki/Deadlock">deadlock</a>).</p><p>To achieve the same aim, we can use asynchronous execution to make our application non-blocking and, at the same time, process multiple requests in a single-threaded application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9MIEYRsa5kEQHe50cT9gPA.png" /><figcaption>Single-threaded application (asynchronous)</figcaption></figure><p>In this example, req 1 arrives at the server first, then at some point in the execution, it schedules an async operation and returns the control flow back to Node.js. While Node.js is waiting for the completion of the first request, it is free to process another one.</p><p>This can be a benefit because actually, to read/write to an I/O device, such as a storage disk or a network interface, we only need to issue a command, and the rest of the time, the CPU is free (asynchronously waiting for the result). So why, while the CPU is just waiting for the result, can’t we use it for something useful?</p><p>But to do this, we need some sort of polling mechanism that will check the availability of resources to process and notify us when they are ready. It would also be nice to have a cross-platform solution because different operating systems provide different mechanisms for working with asynchronous operations.</p><p>In Node.js, abstraction for asynchronous execution is provided by libuv.</p><h3>Libuv</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*YB1XSkpR07mJ2KXt" /><figcaption>Source: <a href="https://docs.libuv.org/en/latest/index.html">https://docs.libuv.org/en/latest/index.html</a></figcaption></figure><blockquote>libuv is cross-platform support library which was originally written for <a href="https://nodejs.org/">Node.js</a>. It’s designed around the event-driven asynchronous I/O model.</blockquote><p><a href="https://docs.libuv.org/en/latest/index.html">Libuv</a> provides asynchronous² API for common I/O operations (even if they are synchronous by nature) through the event loop and the thread pool. This is how Node.js can be single-threaded and process multiple requests simultaneously.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/0*wabfqHGvLI-wABS3" /><figcaption>Source: <a href="https://docs.libuv.org/en/latest/index.html">https://docs.libuv.org/en/latest/index.html</a></figcaption></figure><blockquote>Note 2: Libuv also doing synchronous implementation if you did not specify a callback.</blockquote><h4>Libuv overview</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/0*JEMY70F4mBO-M2pC.png" /><figcaption>Source: <a href="https://medium.com/preezma/node-js-event-loop-architecture-go-deeper-node-core-c96b4cec7aa4">https://medium.com/preezma/node-js-event-loop-architecture-go-deeper-node-core-c96b4cec7aa4</a></figcaption></figure><p>Libuv was designed over a pattern called <a href="https://en.wikipedia.org/wiki/Reactor_pattern">Reactor</a>. This design pattern describes how you can build single-threaded applications with concurrent event-driven architecture.</p><p>The main parts of this pattern are a request of resource, resource handler, event demultiplexer, and event dispatcher.</p><p>From a high point of view, when an application is invoking asynchronous I/O operation, the control flow will be returned back immediately because of the non-blocking³ nature provided by libuv.</p><p>Once a request has been registered to the event demultiplexer, it will be executed within a pre-calculated time. In libuv, the event loop is responsible for <a href="https://docs.libuv.org/en/latest/design.html#file-i-o">calculating the timeout</a> for an asynchronous operation.</p><p>Event demultiplexer provides an abstraction on top of OS to handle I/O requests. For example, Lunix will use epoll to check the availability of requested operations, in MacOS analog is kqueue, and for Windows is IOCP.</p><p>After a resource becomes available (such as data being ready to read from a socket), the event demultiplexer will send it as an event to the event dispatcher, more concrete to the event queue with the corresponding event handler. The event handler is a callback that a user is specified with an async operation.</p><p>The dispatcher (aka event loop) will process events in a specific order and call their callbacks. The callback will be processed by the application in a synchronous manner, and it can also initiate an asynchronous operation.</p><p>If there are no pending asynchronous operations or unprocessed callbacks, the event loop will stop, and Node.js will exit automatically.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Efq6aBuPikny5YKrqlFMSg.png" /><figcaption>The reactor pattern</figcaption></figure><blockquote>Note 3: event demultiplexer provides a non-blocking, asynchronous way to process events from the perspective of Node.js. But it will block the event loop (if the event queue is empty) while waiting for the completion of requests and after that the event loop process these requests in a synchronous way. So, for example, if the demultiplexer uses a call like select(), it will block the main thread until the requested resource is available. This way the event loop will only process ready events, there will be no <a href="https://en.wikipedia.org/wiki/Busy_waiting">busy waiting</a>.</blockquote><blockquote>Note 4: event demultiplexer is not a real thing, but rather an abstract concept. In <a href="https://github.com/libuv/libuv/tree/v1.x">libuv</a>, the event loop itself is responsible for receiving I/O requests and managing the overall event-driven flow of the application. It utilizes event demultiplexing mechanisms provided by the operating system or I/O subsystems, but the event loop is the core component that handles the reception, processing, and coordination of I/O requests within the library.</blockquote><h3>Event loop</h3><p>The event loop is the heart of Node.js. It is responsible for managing and executing asynchronous operations, event handling, and ensuring the non-blocking nature of I/O operations.</p><p>You can think about it as an infinite loop that constantly checks for events and executes the corresponding callbacks or event handlers. It will only stop if there are no active asynchronous operations, such as pending callbacks, timers, or operations from a previous iteration.</p><blockquote>The event loop is not executed while the synchronous code is running. The event loop will only start executing callbacks (event handlers) if the call stack is empty.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/0*aKj_4MebjmYDYfGE.png" /><figcaption>Source: <a href="https://docs.libuv.org/en/latest/design.html#file-i-o">https://docs.libuv.org/en/latest/design.html#file-i-o</a></figcaption></figure><p>If we take a look at the <a href="https://docs.libuv.org/en/latest/design.html#file-i-o">design</a> of the event loop, we can see that it has some algorithm that is executed at each iteration. It consists of several stages, which are important to understand when working with asynchronous operations. The function responsible for running that algorithm is called <a href="https://github.com/libuv/libuv/blob/2f87d5c114dd628d611766a3347a11f77799f06c/src/unix/core.c#L414C1-L479C2">uv_run</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eddab2b47901f019477211d9030f7446/href">https://medium.com/media/eddab2b47901f019477211d9030f7446/href</a></iframe><h4>Overview of the event loop stages:</h4><pre> <br>   ┌───────────────────────────┐<br>┌─&gt;│           timers          │<br>│  └─────────────┬─────────────┘<br>│  ┌─────────────┴─────────────┐<br>│  │     pending callbacks     │<br>│  └─────────────┬─────────────┘<br>│  ┌─────────────┴─────────────┐<br>│  │       idle, prepare       │<br>│  └─────────────┬─────────────┘      ┌───────────────┐<br>│  ┌─────────────┴─────────────┐      │   incoming:   │<br>│  │           poll            │&lt;─────┤  connections, │<br>│  └─────────────┬─────────────┘      │   data, etc.  │<br>│  ┌─────────────┴─────────────┐      └───────────────┘<br>│  │           check           │<br>│  └─────────────┬─────────────┘<br>│  ┌─────────────┴─────────────┐<br>└──┤      close callbacks      │<br>   └───────────────────────────┘</pre><ol><li><strong>timers</strong>: this phase executes callbacks scheduled by setTimeout() and setInterval().</li><li><strong>pending callbacks</strong>: executes I/O callbacks deferred to the next loop iteration (but for most cases, I/O callbacks are executed right after the poll stage).</li><li><strong>idle, prepare</strong>: Node.js takes control of these stages. Here’s where some internal libuv operations and preparations for the poll stage take place.</li><li><strong>poll</strong>: 2 things are done here: 1) libuv will calculate the time needed to block for I/O operations. 2) blocking for waiting for operations and executing I/O-related callbacks (except for close callbacks and timers callback);</li><li><strong>check</strong>: executing callbacks for setImmediate().</li><li><strong>close callbacks</strong>: some close callbacks, e.g. socket.on(‘close’, …).</li></ol><p>Also, there are 2 additional stages — for process.nextTick() and Promise callbacks. Technically, they are not a part of libuv, but they are still important in Node.js.</p><blockquote><a href="https://nodejs.org/en/docs/guides/event-loop-timers-and-nexttick">Here’s</a> a good article that covers every event loop phase in depth.</blockquote><p>To track all associated handlers to asynchronous operations, the event loop used a queue for each phase. There are <a href="https://www.youtube.com/watch?v=PNa9OMajw9w">6 types</a> of main queues that interest us as Node.js developers. We can divide into 2 categories — macrotask and microtask queue:</p><ol><li><strong>Macrotask</strong>: Timer queue (setTimeout() and setInterval() callbacks), I/O queue (for network/disk/child processes callbacks), check queue (setImmediate() callbacks), close queue (close events callbacks).</li><li><strong>Microtask</strong>: Contain nextTickQueue (process.nextTick() callbacks) and promises queue (callbacks for Promise). It has the highest priority over macrotask queues and is executed between each of them. <a href="https://nodejs.org/en/blog/release/v11.0.0">Starting from Node.js v11 and above</a> nextTickQueue will be run after each immediate and timer callback.</li></ol><p>The order of execution of these queues: nextTick -&gt; Promises -&gt; timers -&gt; I/O -&gt; check -&gt; close.⁵</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mv3VNDifZ0U2UeSB" /><figcaption>Source: <a href="https://www.builder.io/blog/visual-guide-to-nodejs-event-loop">https://www.builder.io/blog/visual-guide-to-nodejs-event-loop</a></figcaption></figure><blockquote>Note 5: It’s assumed that the script is being executed in the CommonJS environment. There is a difference between ESM and CommonJS in the way they are executed. In the ESM module system, the order will be the following: <em>Promises</em>-&gt; nextTick -&gt; timers -&gt; I/O -&gt; check -&gt; close. This is due to the fact that the ESM is started in async environment. The detailed explanation can be found <a href="https://stackoverflow.com/a/76378403">here</a>.</blockquote><p>By the way, if you are interested in writing your own implementation of Promise, you can check out my article:</p><p><a href="https://javascript.plainenglish.io/i-wrote-my-own-promise-implementation-5175231b229d">I wrote my own Promise implementation</a></p><h3><strong>Thread pool</strong></h3><p>In libuv, the thread pool is a component that allows for concurrent execution of certain tasks, typically those that are blocking or CPU-intensive, outside of the event loop. It provides a way to offload these tasks to separate threads, freeing up the event loop to continue processing other events and maintaining the overall responsiveness of the system.</p><p>The default size of the thread pool is 4, but it can be changed by setting the UV_THREADPOOL_SIZE environment variable to any value (the absolute maximum is 1024).</p><h4><strong>A little background on why we need a thread pool</strong></h4><p>When it comes to working with I/O there are several ways to do it.</p><p>For example, our server can be synchronous and single-threaded. In Linux, you can use accept() call to get a file descriptor of incoming requests. Then, we can use read() to read the data from the socket. This approach is slow because the server is executing in a synchronous and blocking way.</p><p>As an alternative, you can use accept() and pass the parameter to make it non-blocking. If you try to read from such a socket and there is no data, you just get an error.</p><p>One way to resolve such a problem is to create a loop and polling socket while data is not available. However, this is not an efficient method as you waste your CPU for busy-waiting.</p><p>Another approach is to use select() like commands. Instead of checking availability through the loop, you delegate poll functionality to the OS system. Once the file descriptor is ready to process, OS returns control flow back to the thread that invoked the select() command. You are still blocking your thread, but not the CPU. The CPU is available to do something useful.</p><p>While this is true for network sockets, for real files this strategy is different, because calling select() for a file will always return that it is ready for reading/writing.</p><p>Not every OS provides an asynchronous implementation for reading files. And even if it does, there are certain problems that will not be addressed in this article.</p><p>That’s why we need a thread pool in which we can delegate file I/O to the separate thread.</p><h4><strong>A brief overview of how the thread pool works:</strong></h4><ol><li><strong>Request submission</strong>: When a file I/O operation is needed, instead of directly performing the operation within the event loop, you submit a request to the thread pool, typically to the request queue. The queue in the thread pool algorithm ensures that multiple requests are processed sequentially, and threads are efficiently utilized to execute the operations concurrently.</li><li><strong>Thread pool managemen</strong>t: The thread pool consists of a group of pre-allocated threads. These threads are responsible for executing the requested file I/O operations concurrently.</li><li><strong>Task execution</strong>: The thread pool assigns one of its available threads to handle the file I/O request. The assigned thread performs the blocking file I/O operation without blocking the event loop.</li><li><strong>Event notification</strong>: Once the file I/O operation is completed in a separate thread, it can notify the event loop through a callback or event mechanism, indicating that the operation is finished and providing the necessary result or data.</li><li><strong>Event handling</strong>: The event loop receives the notification from the file I/O thread and can continue executing the corresponding event handler, utilizing the results from the file operation.</li></ol><h4><strong>Tasks that are executed with the thread pool</strong></h4><ul><li><strong>I/O-intensive.</strong></li></ul><ol><li><a href="https://nodejs.org/api/dns.html">DNS</a>: dns.lookup(), dns.lookupService().</li><li><a href="https://nodejs.org/api/fs.html#fs_threadpool_usage">File System</a>: All file system APIs except fs.FSWatcher() and those that are explicitly synchronous use libuv’s threadpool.</li></ol><p><strong>CPU-intensive:</strong></p><ol><li><a href="https://nodejs.org/api/crypto.html">Crypto</a>: crypto.pbkdf2(), crypto.scrypt(), crypto.randomBytes(), crypto.randomFill(), crypto.generateKeyPair().</li><li><a href="https://nodejs.org/api/zlib.html#zlib_threadpool_usage">Zlib</a>: All zlib APIs, except those that are explicitly synchronous, use libuv’s threadpool.</li></ol><p>By the way, you can see that libuv not only cares about async I/O operations but also for CPU intensive like crypto and zlib.</p><h3>Conclusion</h3><p>In conclusion, understanding the concepts of asynchronous programming, Node.js, libuv, the event loop, and the thread pool is crucial for developing efficient and scalable applications.</p><p>Node.js, built on the V8 JavaScript engine, embraces an asynchronous, non-blocking programming model. This allows applications to handle multiple concurrent operations without blocking the execution thread, leading to better performance and scalability.</p><p>At the heart of Node.js lies libuv, a powerful library that provides an event-driven, cross-platform I/O framework. libuv abstracts the underlying I/O operations and handles event management, timers, and network operations. It plays a significant role in enabling Node.js to be asynchronous and efficient.</p><p>The event loop is the core of libuv and serves as the orchestrator for handling events and callbacks. It manages the flow of events, calculates waiting times, and dispatches callbacks when events occur. The event loop works in harmony with the event demultiplexer, which efficiently waits for events on multiple file descriptors or sockets.</p><p>The thread pool in Node.js, separate from the event loop, provides a way to execute computationally intensive or blocking operations in separate worker threads. This offloads heavy tasks from the event loop, preventing it from being blocked and ensuring responsiveness for I/O operations.</p><p>But it should also be noted that the implementation of some aspects may differ. For example, the way libuv provides the event loop algorithm may be different from the way Node uses it. This can lead to misunderstandings, and I recommend checking the documentation and source code of the actual tool of interest.</p><p>Thank you for reading this article! If you have any questions or suggestions, feel free to write a comment.</p><p>Check out my other articles:</p><ul><li><a href="https://levelup.gitconnected.com/understanding-abstraction-levels-in-database-interactions-dal-dao-raw-queries-query-builder-4819d607b0d6">Understanding Abstraction Levels in Database Interactions: DAL, DAO, Raw Queries, Query Builder…</a></li><li><a href="https://javascript.plainenglish.io/demystifying-dependency-injection-an-essential-guide-for-software-developers-364949c28a5f">Demystifying Dependency Injection: An Essential Guide for Software Developers</a></li><li><a href="https://medium.com/@pavlokolodka/from-code-to-deployment-building-a-free-ci-cd-workflow-for-node-js-application-3568da8325bd">From Code to Deployment: Building a Free CI/CD Workflow for Node.js Application</a></li><li><a href="https://medium.com/@pavlokolodka/efficient-client-server-communication-an-overview-of-protocols-and-techniques-f73c1c1016d">Efficient Client-Server Communication: An Overview of Protocols and Techniques</a></li></ul><h3>Level Up Coding</h3><p>Thanks for being a part of our community! Before you go:</p><ul><li>👏 Clap for the story and follow the author 👉</li><li>📰 View more content in the <a href="https://levelup.gitconnected.com/?utm_source=pub&amp;utm_medium=post">Level Up Coding publication</a></li><li>💰 Free coding interview course ⇒ <a href="https://skilled.dev/?utm_source=luc&amp;utm_medium=article">View Course</a></li><li>🧠 AI Tools ⇒ <a href="https://www.aimind.so/">View Now</a></li></ul><p>🔔 Follow us: <a href="https://twitter.com/gitconnected">Twitter</a> | <a href="https://www.linkedin.com/company/gitconnected">LinkedIn</a> | <a href="https://newsletter.levelup.dev/">Newsletter</a></p><p>🚀👉 <a href="https://jobs.levelup.dev/talent/welcome?referral=true"><strong>Join the Level Up talent collective and find an amazing job</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd61c643135e" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/deep-dive-into-node-js-asynchronous-architecture-cd61c643135e">Deep Dive into Node.js Asynchronous Architecture</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>