<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alexey Samoshkin on Medium]]></title>
        <description><![CDATA[Stories by Alexey Samoshkin on Medium]]></description>
        <link>https://medium.com/@alexeysamoshkin?source=rss-c22efb938ed8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*W6wuqntGDCUkLToOkq56ZQ.png</url>
            <title>Stories by Alexey Samoshkin on Medium</title>
            <link>https://medium.com/@alexeysamoshkin?source=rss-c22efb938ed8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 23:50:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@alexeysamoshkin/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Relational and NoSQL databases comparison]]></title>
            <link>https://medium.com/hackernoon/relational-and-nosql-databases-comparison-1d374a828ea9?source=rss-c22efb938ed8------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d374a828ea9</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[information-technology]]></category>
            <dc:creator><![CDATA[Alexey Samoshkin]]></dc:creator>
            <pubDate>Sat, 30 Sep 2023 21:23:06 GMT</pubDate>
            <atom:updated>2023-09-30T21:23:06.688Z</atom:updated>
            <content:encoded><![CDATA[<p>In this blog post, I want to explore key differences between RDBMS and NoSQL data stores. I will touch on differences in design philosophy and also talk about their distinct features and nuances.</p><p>The post consists of two major sections: first explains everything related to relational databases and second covers NoSQL approach. Each section starts with a summary section (aka TLDR).</p><h3>Relational databases</h3><h4>Overview</h4><ul><li>Based on flat relational two-dimensional model. This model is built according to normalization rules in order to reduce data duplication.</li><li>Relational model does not require you to be aware of data access patterns and queries upfront when designing database schema. It allows you to make new unforeseen queries without changing schema design.</li><li>Relational model trade-offs read performance for the ease of writes</li><li>Strict schema (also called schema-on-write) means that the schema must be predefined upfront before you can store any data.</li><li>Developer has to bridge a gap and translate between relational and object model in order to retrieve and manipulate data</li><li>Puts an emphasize on data integrity rules and strong consistency.</li><li>Provides flexible and powerful querying capabilities based on SQL standard. JOINs, complex aggregations, subqueries, recursive queries are supported. Query shape and size is limited only by the developer’s imagination.</li><li>Performance of queries is a black box and depends on multiple factors: table size, concurrent queries, presence of indexes, index size.</li><li>Not designed to scale out due to interconnected data nature and ultra flexible query language. Suitable for vertical scaling only.</li><li>Supports ACID-compliant transactions that can span multiple tables and rows.</li><li>Favors strong consistency over availability and latency in replicated scenarios.</li><li>Can hold a part of business logic on a database side represented by stored procedures and triggers.</li><li>Has connection-oriented model, that relies on long-lived connections between client and a database; relies on connection pooling mechanism to efficiently manage connections.</li></ul><h4>Use cases</h4><p><strong>When to use:</strong></p><ul><li>For structured relational data with a fixed schema.</li><li>When data integrity and strong consistency are crucial. This is achieved through strict schema, constraints, and ACID transactions.</li><li>When your data relies heavily on interconnections and relationships.</li><li>When powerful query capabilities are needed, particularly if your application often uses joins and aggregations.</li><li>When you don’t know data usage scenarios upfront, or don’t have primary structure for manipulating data</li></ul><p><strong>When not to use:</strong></p><ul><li>If the relational model isn’t suitable for your data, such as with unstructured or semi-structured data.</li><li>When you prioritize high write throughput and have low latency requirements.</li><li>When high and predictable read performance is essential.</li><li>When dealing with large data volumes, and you need to scale horizontally.</li><li>If you anticipate regular changes to data structures. Continually altering database schema can be cumbersome and can impact runtime performance.</li><li>If your data consists of small records with complex many-to-many relationships or if you’re representing data as nodes and edges (graph databases are better suited for this).</li><li>For large-scale data analytics and processing.</li></ul><h4>Relational model</h4><p>The relational data model structures data into tables (relations) and rows (tuples). Each tuple represents a collection of name-value pairs. Columns can reference rows from the same table or another, establishing relationships between data points.</p><p>Every SQL operation takes and produces relations, adhering to the principles of relational algebra. This uniformity puts some constraints on the data model. Specifically, table structure has to be flat, meaning tuples don’t accommodate nested structures like records or lists. In a nutshell, relational model can be viewed as a set of flat two-dimensional tables.</p><p>Relational model focuses on data integrity rules. It ensures that relationships between tables remain consistent (called referential integrity): if a foreign key in one table matches a primary key in another, the corresponding record must exist. This prevents orphaned records and ensures that interlinked data accurately represents real-world associations and remains reliable throughout its lifecycle.</p><p>It strives to reduce data duplication and data anomalies through a process called normalization. Normalization is a set of rules on how to break down larger tables into smaller eliminating any repetitive or duplicated data. This process conserves storage; but what is more important, it makes easier to update data, as any data changes need to be made in just one location. While it’s easier to update data, reading the data which is scattered across tables will require extensive use of JOINs and will impact query performance. In other words, relational model trade-offs read performance for ease of writes.</p><p>The relational model differs from the in-memory data structures used in applications. Developers have to translate data between object and relational representations to address the mismatch between these two worlds. ORM frameworks often handle much of this heavy lifting, but they can introduce extra complexity on their own and sometimes abstract the developer too much from the underlying database. This can result in complex, large queries being generated by the framework without the developer even realizing this.</p><h4>Strict schema</h4><p>SQL databases have strict schema, also called as schema-on-write. Before you can store some data, the schema for it must be predefined. The schema dictates the structure of the database, detailing the tables, their respective columns, and the data types each column can contain.</p><p>The goal of strict schema is to enforce data consistency and integrity rules. Also it acts as sort of documentation, so one can quickly grasp the data model from the table definitions without having to extract implicit knowledge of database structure from application code. Finally, since database knows about the structure of the data upfront it can store it more efficiently.</p><p>There’re several drawbacks though. Single predefined schema demand data uniformity (all records within a table must have the same structure), and it becomes challenging to accommodate non-uniform or semi-structured data. The strict schema is inflexible schema, which means less agility. As data requirements evolve, making frequent changes to schema can be cumbersome, slowing down the development time and having negative runtime effects such as keeping table-wide locks for the duration of the “ALTER TABLE” operation.</p><h4>Querying</h4><p>SQL isn’t just another database language; it has become an international standard and gained a wide adoption. Different vendors have their SQL dialects, but the core essence remains consistent across them.</p><p>SQL comes with ultra powerful querying capabilities like complex joins, aggregations, subqueries, recursive queries. With JOINs you can easily fetch multiple, heterogeneous items from a database within a single network round-trip.</p><p>You can go ahead and create a normalized data model without thinking about access patterns upfront. Then you can fetch and analyze data in many different unforeseen ways without needing to modify the database schema design to adopt a new query pattern. This makes relational databases the right choice for applications or startups when you don’t know data usage scenarios upfront. Also, the flexibility of querying makes SQL databases suitable for OLAP operations, where multi-dimensional analytical queries are used to provide answers to complex business questions.</p><h4>Performance</h4><p>The downside of flexible querying is the fact that queries are relatively expensive and don’t scale well in high-traffic and high data volume situations.</p><p>Operations like JOINs inherently have a poor time complexity because JOINs are computed by a database at a query time. Aggregations are heavy as well and often need to traverse through vast amounts of data. As the size of your tables grow, these operations will get slower and slower.</p><p>Also, there’s no inherent limit to the amount of data you can scan in a single request to your database, which also means there’s a risk that single misconfigured or inefficient query can consume disproportionate resources and stall other concurrent operations.</p><p>Indexes are commonly used to speed up queries. The most common type of index in a relational database is the B-tree index. When you create an index on a column, the database builds a B-tree data structure on that column’s data. The database then uses this tree structure to quickly locate rows containing specific column values. Other types of indexes are usually supported: composite indexes, unique indexes, bitmap, hash indexes. Relational databases come with query planner that automatically chooses the right index for the given query, so developer does not have to specify index explicitly at a query time.</p><p>To conclude, with relational databases, performance is like a black box, and is a subject to multiple factors: table size, index size, concurrent queries, presence and design of indexes.</p><h4><strong>Scaling</strong></h4><p>Relational database is not designed to scale out and is not designed to be run on clusters due to highly interconnected data.</p><p>Horizontal scaling works best when you can shard the data in a way that a single request can be handled by a single machine. The flexibility of the query syntax makes it impossible for the database to know which pieces of data will be fetched until the query is executed. It’s unclear how to partition the data so that executing a single JOIN operation doesn’t necessitate the database to fetch data from multiple servers, triggering numerous network requests. As such, all data needs to be kept local, on the same node, to avoid cross-machine network calls when executing a query.</p><p>Also, partitioning the data will break other guarantees that are inherent to relational databases, such as referential integrity and ACID transactions.</p><p>Relational databases are suitable for vertical scaling. Nevertheless, there are usually some limits on a maximum number and a size of data files used under the hood, which imposes upper limits on scalability. And when talking about high volumes of data, it will be impossible to host all data on a single machine anyway.</p><h4><strong>Strong consistency</strong></h4><p>Ensuring data consistency and integrity is one of the key features of relational databases. Data integrity is enforced by strict schema, explicit data types, constraints and other mechanisms like triggers. Referential integrity is enforced via foreign keys: this ensures that relationships between tables make logical sense and prevents any anomalies due to orphaned records.</p><p>Transactions abstracts the complexity of concurrency from application developer. All concurrent access to data is controlled through transactions.</p><p>Transactions are defined by the ACID properties. <strong>Atomicity</strong> ensures that operations within a transaction are all-or-nothing. A transaction will either fully complete, with all its operations successfully executed, or if any part of it fails, none of the operations within the transaction are executed, preventing any half-done operations. <strong>Consistency</strong> guarantees that the database remains in a valid state after a transaction, with all relevant consistency checks (outlined in a paragraph above) being validated upon the transaction’s completion. <strong>Isolation</strong> addresses the behavior of the database when concurrent transactions execute at the same time. <strong>Durability</strong> assures that once a transaction has been committed, the changes are permanent. Even in the face of system failures, the database will ensure that all committed transactions remain alive.</p><p>Replication technologies are limited and typically choose consistency over availability in face of network partition (referring to CAP theorem). They traditionally perform synchronous replication in order to provide a strongly consistent data view.</p><h4>Connection model</h4><p>Relational databases are connection-oriented, meaning that a dedicated connection is established between the client and the server for the duration of a session. If a wire protocol used to talk to a database does not support pipelining, then each connection is exclusively locked for the duration of a query. This means that a database client has to wait for a reply to a previous query before it can reuse the same connection to send a next query. Moreover each connection consumes server resources and establishing new connection comes with an overhead (TCP and TLS handshakes, database authentication, etc).</p><p>To mitigate these issues, the concept of connection pooling is often employed. Connection pooling involves maintaining a cache of database connections that can be reused by multiple clients. Instead of establishing a new connection each time, clients retrieve an existing connection from the pool, use it, and then return it to the pool. This approach ensures reduced latency and better scalability, especially in applications with high number of concurrent online users.</p><p>Note, if you have a highly concurrent environment with long queries and a non-pipelined protocol, then each connection from a pool will be exclusively locked to serve a particular query, and each new incoming request will spin up a new connection to the database. After some time you will reach the limit of maximum allowed connections. In this situation connection pool mechanism is not going to help you much.</p><p>Connection-oriented model works fine for long-running compute instances, but fails to meet the nature of server-less compute environment, which consists of ephemeral short-lived execution environments.</p><h4><strong>Integration solution</strong></h4><p>Relational database can act as an integration/interoperability mechanism to connect multiple applications: shared integration database.</p><p>Relational databases remain the backbone of enterprise IT systems. They are often used as an integration point between various applications and platforms. Standard SQL interface ensures that multiple systems, regardless of their vendor or underlying architecture, can interact with a relational database seamlessly.</p><p>Keep in mind though, that shared integration database is viewed as an anti-pattern according to modern standards.</p><h3>NoSQL databases</h3><h4>Overview</h4><ul><li>Designed with scalability and large data volumes in mind, influencing many other design decisions, including data model, query capabilities, and consistency guarantees.</li><li>Doesn’t restrict data to a one-size-fits-all relational model. Instead, NoSQL database provide various specialized data models optimizing for specific use cases, including key-value, document, column family, graph, and time-series structures.</li><li>Favors aggregate-oriented data models, which consist of large records with simple connections between them. Referential integrity is not supported.</li><li>NoSQL modeling mindset requires you to map out your query patterns up front and then design your NoSQL schema around it. Changes in query patterns might necessitate a schema redesign, making it challenging to accommodate unforeseen queries in the future.</li><li>Data is often denormalized in order to retrieve related datasets with a single request from a single database node. The responsibility of managing duplicated data is usually pushed to the application side.</li><li>The responsibility of enforcing data integrity rules is shifted to the application side.</li><li>For the sake of scalability and performance, querying capabilities are intentionally limited. Operations like JOINs and aggregations, and any other operations that span multiple aggregate boundaries may be unsupported or severely restricted.</li><li>Loose schema. Does not require you to define schema upfront. Allows records within same table to have varying non-uniform structure.</li><li>Transactions, if present, typically support atomic operations only against a single document. Otherwise transactions are not fully ACID compliant.</li><li>NoSQL databases usually relax strong consistency guarantees in scenarios where data is replicated across multiple nodes.</li></ul><h4>Use cases</h4><p><strong>When to use:</strong></p><ul><li>When the data schema is rapidly evolving or the data structure isn’t predefined, a schema-less database offers enhanced agility and flexibility.</li><li>When dealing with nonuniform, semi-structured, or unstructured data.</li><li>For data models of records with simple or limited connections, so it’s easier to draw boundaries between aggregates.</li><li>When working with data models that don’t fit into a relational approach.</li><li>For sparse data. Column-family stores are well-suited for datasets where many attributes have null or missing values, as they don’t use up space for null entries.</li><li>When you anticipate large amount of traffic or large data volumes that would be best distributed across servers. When horizontal scaling is the must.</li><li>In scenarios demanding a high volume of write operations.</li></ul><p><strong>When not to use:</strong></p><ul><li>When you need strong consistency and ACID-compliant transactions.</li><li>When complex querying is needed.</li><li>When data access patterns are unknown upfront or when you need to support unforeseen queries of varying shape in future (e.g OLAP systems).</li><li>When your data fits best in a simple table format or if you have many relationships between your data records.</li></ul><h4>Non-relational data models</h4><p>NoSQL databases don’t constrain data into two-dimensional tables. They permit complex nested tree-like objects. This eliminates the need to translate between in-memory and database representations and simplifies database access for developers.</p><p>There is no universal data model across all NoSQL databases. Instead, you can select one that best aligns with your needs: document-based, key-value, column-family, graph-like, time-series. The strength and weakness of these models lie in their specialization. If used outside of their intended context, they can become overly complex and inefficient. In contrast, with the relational model, you can model a variety of problem domains: key-value data storage, hierarchical relationships, highly interconnected data with prevalent many-to-many relationships, time-series data. It might be more complex and not as efficient solution, but it will nevertheless work.</p><h4><strong>Aggregate-oriented data model</strong></h4><p>Most NoSQL databases were designed with horizontal scalability in mind, inspired by the necessity to operate on clusters. This led to the development of aggregate-oriented data models consisting of large records with simple connections as opposed to highly interconnected set of small records that relational model promotes.</p><p>“Aggregate” term comes from domain-driven design and represents a collection of related objects that we wish to treat as a unit. Data is read and written as a single unit, which is more efficient for distributed systems. So aggregate is a unit of data retrieval and unit of consistency: we like to update aggregates with atomic operations and communicate with our data storage in terms of aggregates.</p><p>NoSQL databases are optimized for accessing or updating single document/aggregate at a time. On the other hand, SQL databases can efficiently manage operations across multiple rows and tables.</p><p>What else does aggregate-orientation mean in practice?</p><ul><li>Lack of JOIN Operation. In relational databases, the JOIN operation allows you to combine rows from two or more tables based on related columns. In NoSQL databases, there&#39;s no equivalent to this operation (or very limited surrogate). This means you may have to denormalize data, embedding relevant data directly within a single document or record, leading to potential data redundancy.</li><li>NoSQL databases have limited support for aggregations. While some basic operations like counting or sum might be supported, complex aggregations will require additional work or may not be feasible at all.</li><li>Inter-document relationships are more difficult to handle than intra-document relationships. Establishing relationships between documents is tricky. You might use reference IDs, much like foreign keys in SQL, but the database does not enforce referential integrity checks.</li><li>When transactions are supported, they are usually constrained to a single document. Atomic updates within a single document are possible, but operations that span multiple documents are not atomic. It might be a tough time when you have a business transaction that spans multiple documents and you need strong consistency guarantees.</li></ul><p>Choosing appropriate aggregate boundaries in NoSQL databases depends entirely on your anticipated data access patterns. The advice is to group data that is frequently accessed together to optimize retrieval speeds. While a particular decomposition schema can help with specific interactions, it might hinder others. If your querying requirements evolve, you could find yourself having to redesign your schema to accommodate them. In contrast, the relational model is very flexible and allows you to easily look at the data in different ways without changing the schema design. The relational schema is primarily dictated by normalization rules and is not tied to specific access patterns.</p><p>Denormalization is more common with aggregate-based approach, because we want to minimize the number of aggregates we access during a data interaction. Denormalized store allows you to retrieve all information about a specific data item in a single request.</p><p>However, denormalization comes with its own challenges, notably data duplication. NoSQL databases sidestep this issue by pushing the data integrity responsibilities onto the application side. Developers may find themselves having to update the same piece of data in multiple locations when change occurs.</p><p>Data is designed mostly for read purposes, rather than for ease of storing and write flexibility.</p><h4>Loose schema</h4><p>NoSQL databases allow you to store records without first defining any schema. Multiple records within a same table might have different structure: there’s no requirement for all records of a table to have uniform structure.</p><p>However, the absence of schema is just an illusion. Even when you don’t have to define explicit schema, there’s typically an implicit schema around. This implicit schema is a set of assumptions about the data’s structure present in the application code that manages this data. Essentially, a schema-less database moves the schema definition to the application code, and often this knowledge is scattered all over the codebase (or is at least confined to data access-related modules).</p><p>Implicit schema gives you more agility. As application requirements evolve, the table or collection can adapt records with new data structure without the need for time-consuming database refactoring. This reduces the challenges associated with schema migration and the lock-step rollout of new app and database versions. On the flip side, now the application logic has to deal with multiple versions of the same data entity being stored at the same time.</p><p>Another advantage is the ease of handling non-uniform data, where each record can have a distinct set of fields.</p><p>The disadvantage is that without the schema defined on the database side, the database can no longer enforce consistency checks and validate data, raising the risk of accepting inconsistent data. This concern becomes more important when multiple applications, possibly developed by different teams, access the same database (a scenario known as the shared integration database).</p><p>Also note, that the flexibility of implicit schema only applies within an aggregate — if you need to change your aggregate boundaries, the migration is as complex as it is in the relational case.</p><h4>Querying</h4><p>Query capabilities in NoSQL databases are typically more restricted than in RDBMS. Data can be efficiently queried in a limited set of ways; outside of these, queries can become either very inefficient or are not supported at all.</p><p>Inter-aggregate JOINs and aggregations are not supported deliberately. This is to minimize the number of aggregates (documents) query can access for the purpose of horizontal scaling. When JOIN operations are necessary, they’re usually handled in the application layer.</p><p>There’s no common standard language for querying like SQL. Each NoSQL database comes with their unique query language.</p><p>Querying is not as flexible as in RDMBS. With RDBMS, you can create a normalized data model without thinking about access patterns upfront. By contrast, in NoSQL, you shouldn’t start designing your schema until you know the questions that it needs to answer. Understanding the business problems and the application use cases up front is absolutely essential. You won’t design your tables first and then see what questions you need to answer. You map out our query patterns up front, then design your NoSQL schema around it. You have to do the hard work up front to really think about your data access patterns and how you will satisfy them.</p><p>Rather than reassembling your data at read time using JOINs, NoSQL database require you to store data in a such way that “pre-joins” and keeps related data together. NoSQL database organizes data so that its shape in the database corresponds with what will be queried.</p><p>You design your schema specifically to make the most common and important queries as fast and as inexpensive as possible. Your data structures are tailored to the specific requirements of your business use <br>cases.</p><p>NoSQL database trade-offs write simplicity for read performance. While data might be structured in a way that makes it hard to update and manage it, this schema will give you good performance for a selected subset of queries.</p><h4>Scaling</h4><p>NoSQL databases were designed to handle horizontal scaling and manage large datasets with high ingestion throughput from the day zero. This design goal has a strong influence on other aspects, such as data model, querying capabilities, consistency and durability guarantees.</p><p>To achieve this, they remove features traditionally found in relational databases that can hinder performance, such as joins and aggregations. Query capabilities and flexibility are deliberately sacrificed in favor of scalability and performance.</p><p>Additionally, these databases implement explicit partitioning to distribute data across multiple storage nodes effectively. It forces you to segment your data (with explicit partition/shard key), allowing for easier horizontal scaling. All queries are then confined to this segment. This partitioning mechanism is what allows to scale almost infinitely without performance degradation. As your data volume increases, you can just add additional nodes as needed.</p><h4>Consistency</h4><p>With scalability in mind, most NoSQL stores lack true ACID transactions and favor relaxed consistency model.</p><p>Usually, NoSQL stores don’t have transactions that span multiple aggregates. Instead, they provide atomic manipulation of a single aggregate at a time. The aggregate acts as the atomic unit for retrieval and updates. This provides somewhat limited amount of transactional control.</p><p>In scenarios when data is replicated across multiple nodes, application developer has to deal with a new class of anomalies. In a strongly consistent system (one that has linearizability property to be more correct) reads will always return the most recent version of the data, and writes will be visible to all nodes immediately after they occur. In distributed systems this is no longer true by default and reads might not always reflect the most recent write. Various consistency models (causal consistency, read-your-writes, monotonic reads, etc) describe the guarantees provided by the system about the order in which write operations appear to occur to clients. And the developer has to be aware which consistency levels are provided by the particular database, understand which level is needed for the particular use case or feature, and therefore make an informed decision.</p><p>In face of network partition, NoSQL stores allow you to trade between consistency and availability. In normal operation mode, they allow you to trade between consistency and latency. Some database are opinionated enough and stick to a particular side of spectrum, while others give you more control of the desired consistency level. This can be a database-wide setting or can be tweaked on a per-query basis.</p><h3>Conclusion</h3><p>Recently, there has been a convergence between relational databases and NoSQL solutions. Relational databases are adopting aspects previously exclusive to NoSQL solutions only, such as supporting complex nested data types like XML and JSON, focusing on specialized data models and use cases like time-series data, introducing geospatial indexes. On the other hand, NoSQL solutions are now incorporating multi-document transactions, advanced aggregation operations, and offering the capability to define validation schemas within the database.</p><p>It’s essential to view those two categories described in this post not as a binary choice but as endpoints on a spectrum. Real-world database solutions will often lie somewhere in between these points, leaning more towards one end or the other.</p><p>While the line that distinguish relational from NoSQL databases seems to be blurring, it’s crucial to understand that this apparent similarity can be deceptive. It’s important to be aware and remember the original design goals of each solution, the purpose for which each database was created, and the specific problems they are meant to address, and employ them accordingly.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d374a828ea9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/relational-and-nosql-databases-comparison-1d374a828ea9">Relational and NoSQL databases comparison</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reclaim disk space by removing stale and unused Docker data]]></title>
            <link>https://medium.com/@alexeysamoshkin/reclaim-disk-space-by-removing-stale-and-unused-docker-data-a4c3bd1e4001?source=rss-c22efb938ed8------2</link>
            <guid isPermaLink="false">https://medium.com/p/a4c3bd1e4001</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[tutorial]]></category>
            <dc:creator><![CDATA[Alexey Samoshkin]]></dc:creator>
            <pubDate>Fri, 17 Sep 2021 21:33:19 GMT</pubDate>
            <atom:updated>2021-09-18T12:48:03.090Z</atom:updated>
            <content:encoded><![CDATA[<h4>including stopped containers, orphan images, unused volumes, Docker builder cache, and more</h4><p>Recently I faced an issue with Docker Desktop for Mac. I was writing a Dockerfile in an attempt to containerize existing NodeJS apps. It was an endless loop — tweak a Dockerfile a bit, build a new version of an image, run the container, figure out something is wrong, tweak a Dockerfile once again, build an image once again, and so on and so on. Suddenly, at some point, the Docker engine refused to build a new image once again, complaining it has run out of disk space.</p><pre>write /var/lib/docker/overlay2/r6i5qjvzgo679wk9lyvu0qgzi/diff/yarn.lock: no space left on device</pre><p>Such an issue is not something you’re likely to face on an everyday basis. So in this post, I’m going to shed some light on the cause and share the recipe for how to remedy this problem.</p><p>But first, let’s explore the difference between how the Docker engine works on macOS and on Linux.</p><h4>Docker client-server architecture</h4><p>Let’s quickly recall the high-level client-server Docker architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UiTwpu8kZLVncsVYMKtqdw.png" /><figcaption>Docker Engine client-server architecture</figcaption></figure><p>On a Linux, both Docker client and daemon are running on the same machine, and talk to each other via REST API called <a href="https://docs.docker.com/engine/api/">Docker engine API</a> through the Unix domain socket located at /var/run/docker.sock</p><p>We can even connect to this socket using the netcat utility and manually feed it the raw HTTP request text. Append two trailing newlines to send the request (as HTTP protocol prescribes). For example, let&#39;s query the Docker engine for all images (similar to what docker imagesCLI command does for you under the hood). Here is the <a href="https://docs.docker.com/engine/api/v1.40/#operation/ImageList">API documentation</a> for the “List images” endpoint.</p><pre>$ nc -U /var/run/docker.sock<br>GET /images/json HTTP/1.1<br>Host: docker.com</pre><pre>HTTP/1.1 200 OK<br>Api-Version: 1.40<br>Content-Type: application/json<br>Date: Thu, 16 Sep 2021 19:02:21 GMT<br>Docker-Experimental: false<br>Ostype: linux<br>Server: Docker/19.03.13 (linux)<br>Transfer-Encoding: chunked</pre><pre>...the remaining response body</pre><p>Here is the screenshot of the full JSON response from the Docker engine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hiyEm1vndbrlJw8S6YOQKA.png" /><figcaption>“/images/json” response from the Docker Engine API</figcaption></figure><h4>On macOS, Docker daemon runs inside a Virtual Machine</h4><p>On macOS, things are a bit complicated, since docker daemon and underlying container runtime require a Linux-based environment. Thus, the Docker engine does not run directly on your host machine but runs inside a dedicated minimal Linux OS virtual machine specially built for running containers. For example, with “MacOS 10.15” and “Docker Desktop For Mac 2.4.0.0” being in use, the virtual machine has the Linux docker-desktop 4.19.76-linuxkit distro running. <a href="https://blog.docker.com/2017/04/introducing-linuxkit-container-os-toolkit/">LinuxKit</a> is a lightweight, container-optimized Linux distribution.</p><p>Docker client (Docker CLI) on the other hand still runs on your host machine as a regular binary and talks to the Docker daemon running inside a Linux virtual machine through the Unix domain socket in the same way, as I explained before.</p><pre># Docker client is a part of &quot;Docker Desktop for Mac&quot; app<br>$ readlink `which docker`<br>/Applications/Docker.app/Contents/Resources/bin/docker</pre><h4>Out of space disk error and “Docker.raw” file</h4><p>Getting back to the “out of disk space” error.</p><pre>write /var/lib/docker/overlay2/r6i5qjvzgo679wk9lyvu0qgzi/diff/yarn.lock: no space left on device</pre><p>/var/lib/docker/... directory is where the Docker daemon persists all its data: images, containers, volumes. On macOS, it exists inside an aforementioned Linux virtual machine, so you cannot directly explore its contents from the host machine.</p><p>All you can find on the host machine is the single huge Docker.raw file which is reported to be 60GB (allocated size of the file, tells the maximum potential disk size which can be used), but actually taking 22GB in terms of how many bytes are actually consumed by the file on the file system right now.</p><pre>$ cd ~/Library/Containers/com.docker.docker/Data/vms/0/data</pre><pre>$ ls -lh Docker.raw<br>-rw-r--r-- 1 oleksii.samoshkin 1846670294 <strong>60G</strong> 2021-09-16 21:48 Docker.raw</pre><pre>$ ls -skh Docker.raw<br><strong>22G</strong> Docker.raw</pre><p>Similar information can be found on the Dashboard of the “Docker Desktop for Mac” app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gr-RLe7UZFjTDRYUiEUlaQ.png" /></figure><p>Sooner or later you will run out of space allocated for the Docker Engine to store all its data (is 60GB in my case). It could happen even sooner than you expect, for example when creating a new Dockerfile that assumes a lot of image build/rebuild cycles.</p><h4>Getting into the shell of the Docker VM on Mac</h4><p>The simplest approach as of 2021 is to use <a href="https://man7.org/linux/man-pages/man1/nsenter.1.html">nsenter</a> program running in a privileged container. It will get you into the shell of the Docker VM:</p><pre>$ docker run -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh</pre><p>Whoa, what’s going on here.</p><p>We are running a container using the Debian image, which already has nsenter binary installed. <a href="https://docs.docker.com/engine/reference/run/#pid-settings---pid">pid=host</a> means you get access to the process space of the VM running Linux docker-desktop 4.19.76-linuxkit . Basically it removes the process isolation that containers normally promote. Then nsenter says “whatever is PID 1, use that as context, and enter all the namespaces of that, and run a shell there”.</p><p>Alternatively, instead of using a full-blown Debian image (124 MB), you can use the <a href="https://github.com/justincormack/nsenter1">justincormack/nsenter1</a> image, which does not use any base image at all (FROM scratch directive), but builds nsenter binary from sources using C compiler. This is an ultra-lightweight image, that takes only 101Kb.</p><pre>$ docker run -it --rm --privileged --pid=host justincormack/nsenter1</pre><p>So it will bring you into the /bin/sh shell inside the Docker VM, where you can find the famous /var/lib/docker directory. You can explore how the docker daemon manages its data, or see the disk usage on a per-directory basis.</p><pre>docker run -it --rm --privileged --pid=host justincormack/nsenter1</pre><pre>/ # cd /var/lib/docker</pre><pre># Explore the space distribution<br>/var/lib/docker # du -csh *<br>20.0K   builder<br>5.5M    buildkit<br>404.0K  containerd<br>396.0K  containers<br>89.1M   image<br>88.0K   network<br>16.4G   overlay2<br>20.0K   plugins<br>4.0K    runtimes<br>4.0K    swarm<br>4.0K    tmp<br>4.0K    trust<br>1.4G    volumes<br>17.9G   total</pre><pre>/var/lib/docker # du -csh volumes/*<br>31.9M   volumes/0a7a66280fbb815de.....f63776ac<br>12.0K   volumes/564162944bf44c50f.....d0d34ce7<br>48.2M   volumes/8faf86bb916d717e7.....07954193<br>16.0K   volumes/app_certs<br>1.2G    volumes/app_node_modules<br>8.0K    volumes/c6043fa4e5dc31603w....w803fa08a<br>48.2M   volumes/cd58df7a53c7df1340....782281915<br>48.2M   volumes/ffbd465f433b7eebb3....1195c5d4c<br>48.0K   volumes/metadata.db<br>1.4G    total</pre><p>Or you can drill down inside one of your volumes and check its contents. For example, I have an “app_node_modules” volume with many NPM packages installed.</p><pre>/var/lib/docker # ls -1 volumes/app_node_modules/_data | head -n 10<br><a href="http://twitter.com/babel">@babel</a><br><a href="http://twitter.com/bcoe">@bcoe</a><br><a href="http://twitter.com/cnakazawa">@cnakazawa</a><br><a href="http://twitter.com/danmarshall">@danmarshall</a><br><a href="http://twitter.com/deck">@deck</a>.gl<br><a href="http://twitter.com/grpc">@grpc</a><br><a href="http://twitter.com/hapi">@hapi</a><br><a href="http://twitter.com/istanbuljs">@istanbuljs</a><br><a href="http://twitter.com/jest">@jest</a><br><a href="http://twitter.com/loaders">@loaders</a>.gl</pre><p>Still, the approach of jumping inside the Docker VM is limited, because it uses a read-only file system, and it’s not intended to be tinkered with. For example, it’s not possible to install vimto edit some files inside the volume.</p><pre>/var/lib/docker # which apk<br>/sbin/apk</pre><pre>/var/lib/docker # apk add --no-cache vim<br>ERROR: Unable to lock database: Read-only file system<br>ERROR: Failed to open apk database: Read-only file system</pre><p>Anyway, at least it demystifies how the Docker Engine works on the macOS, and now you know where the /var/lib/docker directory comes from and what is its purpose.</p><h4>How to clean Docker data and reclaim space</h4><p>The Docker objects that stay around may be of various types: containers, images, volumes, networks.</p><p>First, you might be interested in the high-level Docker disk usage information.</p><pre>$ docker system df<br>TYPE                SIZE                RECLAIMABLE<br>Images              10.17GB             7.155GB (70%)<br>Containers          354B                354B (100%)<br>Local Volumes       1.105GB             151.1MB (13%)<br>Build Cache         3.409GB             3.409GB</pre><p>Next, <strong>remove all stopped containers.</strong></p><pre>$ docker ps --filter “status=exited” -q | xargs -r docker rm --force</pre><pre># or through a simpler built-in command<br>$ docker container prune --force</pre><p>Or you might want to remove all the containers, no matter whether they’re running or not. If I were cleaning up stuff on my local machine, I’d rather just drop all of them.</p><pre>$ docker ps -a -q | xargs -r docker rm — force</pre><p><strong>Remove dangling images.</strong> Docker images consist of multiple layers. Dangling images are layers that have no relationship to any tagged images. They are no longer used and just consume disk space.</p><pre>$ docker images -f “dangling=true” -q | xargs -r docker rmi -f</pre><pre># or through a simpler built-in command<br>$ docker image prune</pre><p>Also, you might want to remove all images, that have no running containers associated with them. Beware that if all containers were removed at the previous step, this command will just remove all the images. This might be not something you would like to do.</p><pre># NOTE: you cannot query all unused images through &quot;docker images&quot; command<br>$ docker image prune --all --force</pre><p><strong>Remove dangling volumes.</strong> The “dangling volume” semantic is a bit different from what the “dangling image” means. A dangling volume is one that exists and is no longer connected to any containers — just an unused volume.</p><p>You might end up with quite many Docker volumes over time, especially when new anonymous volumes are created at each container spin up. It’s a good practice to check and clean your volumes regularly.</p><pre>$ docker volume ls -qf dangling=true | xargs -r docker volume rm</pre><pre># or through a simpler built-in command<br>$ docker volume prune --force</pre><p><strong>Clean the Docker builder cache.</strong> Normally, Docker caches the results of the first build of a Dockerfile, allowing subsequent builds to be fast.</p><pre>DOCKER_BUILDKIT=1 docker builder prune --all --force</pre><p><strong>Remove networks</strong>, which are not used by at least one container. Networks are usually created and removed by tools like Docker Compose, so probably, you don’t need to remove them manually. But anyway I will put this command here for the sake of completeness.</p><pre># will remove all networks not used by at least one container<br>docker network prune --force</pre><h4>Use the “docker system prune” shortcut command</h4><p>Instead of removing all those objects individually one by one, Docker provides you with a single “kill-em-all” command — docker system prune . This command takes care of all those object types:</p><ul><li>all stopped containers</li><li>unused networks</li><li>dangling images and/or unused images</li><li>dangling volumes</li><li>builder cache.</li></ul><pre>$ docker system prune --force --volumes</pre><h4>Shrink the “Docker.raw” file on macOS</h4><p>According to the <a href="https://docs.docker.com/desktop/mac/space/">Docker docs</a>, once you removed those objects from inside the Docker VM, it might take a few minutes to reclaim space on the host machine depending on the format of the disk image file.</p><ul><li>if the file is named Docker.raw space on the host should be reclaimed within a few seconds.</li><li>if the file is named Docker.qcow2 space will be freed by a background process after a few minutes.</li></ul><p>However, space is only freed when images are deleted. Space is not freed automatically when files are deleted inside running containers.</p><p>You might want to trigger a space reclamation at any point manually by running the official docker/desktop-reclaim-space image.</p><pre>$ cd ~/Library/Containers/com.docker.docker/Data/vms/0/data/</pre><pre># Given that block-size = 1KB. <br># Space used = 22135MB<br>$ ls -sk Docker.raw<br>22666720 Docker.raw</pre><pre># Discard the unused blocks on the file system<br>$ docker run --privileged --pid=host docker/desktop-reclaim-space</pre><pre># Savings are (22666720 - 22110948)/1024 = ~542MB<br>$ ls -sk Docker.raw<br>22110948 Docker.raw</pre><p>docker/desktop-reclaim-space image uses the aforementioned nsenter1 binary to get access to the shell of Docker VM, and then it discards the unused blocks on the file system using the <a href="https://man7.org/linux/man-pages/man8/fstrim.8.html">fstrim</a> program.</p><blockquote>fstrim<strong> </strong>is used on a mounted filesystem to discard (or “trim”) blocks which are not in use by the filesystem. This is useful for solid-state drives (SSDs) and thinly-provisioned storage.</blockquote><pre>$ docker inspect docker/desktop-reclaim-space</pre><pre>...<br>“Entrypoint”: [<br> “/usr/bin/nsenter1”,<br> “/sbin/fstrim”,<br> “/var/lib/docker”<br> ],<br>...</pre><p><strong>PITFALL</strong>: Apple M1 users will not be able to run <a href="https://hub.docker.com/r/docker/desktop-reclaim-space/tags?page=1&amp;ordering=last_updated">docker/desktop-reclaim-space</a> image, since it was built only for “linux/amd64&quot; architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5j01OJQkK95U2MrBv_wJAQ.png" /></figure><p>See <a href="https://github.com/samoshkin/docker-reclaim-disk-space/issues/1">this issue</a> for how the error looks like, so you’re not get confused.</p><pre>$ docker run --privileged --pid=host docker/desktop-reclaim-space<br>WARNING: The requested image&#39;s platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested<br>setns:mnt: Invalid argument</pre><h4>Restart the Docker engine</h4><p>Finally, if you have some strange issue, or want to make sure you have the pristine Docker state ever, restart the Docker engine. If you’re on Mac, it’s easier to restart through the “Docker Desktop” app manually. However, it’s also possible to do using uselaunchd and launchctl if you need to automate things.</p><pre># see if Docker service is running<br>local docker_service=$(launchctl list | grep &quot;com.docker.docker&quot; | awk &#39;$0 != &quot;-&quot; { print $3 }&#39;)</pre><pre># stop it<br>if [ -n &quot;$docker_service&quot; ]; then<br>  launchctl stop &quot;$docker_service&quot; || true;<br>fi</pre><pre># and then start it<br># not that we&#39;re starting it indirectly through the &quot;com.docker.helper&quot; Launch agent<br>launchctl start com.docker.helper</pre><p>Things are much easier with systemd on Linux.</p><pre>sudo systemctl stop docker.service || true<br>sudo systemctl start docker.service</pre><pre># or just<br>sudo systemctl restart docker.service</pre><h4>How much disk space I reclaimed?</h4><p>Finally, let’s check out the results of the clean-up. Run the Docker disk usage report once again. As you see, ~5GB space is still occupied, because I didn’t remove all the images.</p><pre>$ docker system df</pre><pre>TYPE                SIZE                RECLAIMABLE<br>Images              4.924GB             4.924GB (100%)<br>Containers          0B                  0B<br>Local Volumes       0B                  0B<br>Build Cache         0B                  0B</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ixTQjbxeWuQyP42bqMxPNQ.png" /></figure><pre># 9203256KB/2^20 = ~8.77GB is used<br>$ ls -ks  Docker.raw<br>9203256 Docker.raw</pre><pre># WARN: the reported file size is still 60GB<br># don&#39;t be confused<br>$ ls -lh  Docker.raw<br>-rw-r--r-- 1 oleksii.samoshkin 1846670294 60G 2021-09-17 00:27 Docker.raw</pre><p>Roughly, I reclaimed 12.8GB = (21.6GB − 8.8GB). It’s the garbage that accumulated during the period of approx. 1 week working with Docker on my machine</p><p>Before that, it ate all the 60GB of space available after a month of usage or so. Now I’m running the clean-up script twice per month or so, and I’m fine.</p><h4>Automate “clean up” activities through a Bash script</h4><p>I’ve prepared a single script, that contains all the commands and tricks I described in this blog post. Check it out at <a href="https://github.com/samoshkin/docker-reclaim-disk-space">https://github.com/samoshkin/docker-reclaim-disk-space</a></p><p>The usage is simple. You event don’t need to clone the repo. Just use wget or curl</p><pre>$ bash -c &quot;$(curl -fsSL https://raw.githubusercontent.com/samoshkin/docker-reclaim-disk-space/master/script.sh)&quot;</pre><p>Or with wget:</p><pre>$ bash -c &quot;$(wget -qO - https://raw.githubusercontent.com/samoshkin/docker-reclaim-disk-space/master/script.sh)&quot;</pre><p>Basically, the script does the following:</p><ul><li>prints the Docker disk usage information</li><li>interactively prompts you for confirmation</li><li>removes stopped containers</li><li>removes orphan (dangling) images layers</li><li>removes unused volumes</li><li>removes Docker build cache</li><li>shrinks the Docker.raw file, if you’re on the macOS</li><li>restarts the Docker engine (through launchctl on macOS or systemctl on Linux). Waits until the Docker is up and running after the restart.</li><li>prints the Docker disk usage once again</li></ul><p>If you want to suppress interactive prompts, pass -y flag. If you don’t want to restart the Docker engine, pass the --no-restart flag.</p><p>Here is the short screencast of this script in action.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F8iqihcaU398%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D8iqihcaU398&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F8iqihcaU398%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/133c0df31a35f9e23248124c7342a8ce/href">https://medium.com/media/133c0df31a35f9e23248124c7342a8ce/href</a></iframe><h3>Resources</h3><p>Disk utilization in Docker for Mac | Docker Documentation <a href="https://docs.docker.com/desktop/mac/space/">https://docs.docker.com/desktop/mac/space/</a></p><p>Docker prune explained — usage and examples <a href="https://takacsmark.com/docker-prune/">https://takacsmark.com/docker-prune/</a></p><p>justincormack/nsenter1: simple nsenter to namespaces of pid 1 <a href="https://github.com/justincormack/nsenter1">https://github.com/justincormack/nsenter1</a></p><p>Where is /var/lib/docker on Mac/OS X — <a href="https://stackoverflow.com/a/65645462">https://stackoverflow.com/a/65645462</a></p><p>Getting a Shell in the Docker Desktop Mac VM <a href="https://gist.github.com/BretFisher/5e1a0c7bcca4c735e716abf62afad389">https://gist.github.com/BretFisher/5e1a0c7bcca4c735e716abf62afad389</a></p><p>Docker overview | Docker Documentation <a href="https://docs.docker.com/get-started/overview/#docker-architecture">https://docs.docker.com/get-started/overview/#docker-architecture</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a4c3bd1e4001" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Working with AWS and Sentry through a CLI by running it in a Docker container]]></title>
            <link>https://medium.com/@alexeysamoshkin/working-with-aws-and-sentry-through-a-cli-by-running-it-in-a-docker-container-8b86b2c12916?source=rss-c22efb938ed8------2</link>
            <guid isPermaLink="false">https://medium.com/p/8b86b2c12916</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Alexey Samoshkin]]></dc:creator>
            <pubDate>Mon, 23 Aug 2021 20:00:35 GMT</pubDate>
            <atom:updated>2021-08-23T20:00:35.344Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JUOITpaBdlrMP9D__-K5Fw.png" /><figcaption>Docker official logo image</figcaption></figure><p>Cloud providers like AWS or GCP typically provide you with an HTTP API, which can be used to interact with those services programmatically. This is not limited to cloud providers only, for example, the popular error tracking service <a href="https://sentry.io/">Sentry</a>, which I was working with recently, comes with an HTTP API as well.</p><p>API is useful for different kinds of integrations and automation. For example, within your CI/CD pipeline, you might want to notify Sentry about an upcoming release version and upload source map files for minified and optimized JS bundles prepared by the Webpack.</p><p>Services like Sentry or AWS go even further and provide CLI tools, that abstract developers away from learning and understanding nuances of HTTP interactions, e.g. how to craft the request properly, what is the URL to send the request, how to parse and read the response.</p><p>All you need to do as a developer is to install a particular tool on your local machine and authenticate yourself by acquiring some kind of access token.</p><p><strong>The question is</strong>: what’s the point of installing some binaries on my local machine and littering in a couple of directories if I can just use the Docker image and run the same command by spawning it within a Docker container.</p><h3>Working with S3 through AWS CLI by running it in a Docker container</h3><p>First, let’s pull the <a href="https://hub.docker.com/r/amazon/aws-cli">official </a><a href="https://hub.docker.com/r/amazon/aws-cli">amazon/aws-cli Docker image</a>.</p><pre>$ docker pull amazon/aws-cli</pre><pre>Using default tag: latest<br>latest: Pulling from amazon/aws-cli<br>d36ac8072fa2: Already exists<br>c189463b6c08: Pull complete<br>cd55a4a954a6: Pull complete<br>afcddbfdb53e: Pull complete<br>16858f14c6a1: Pull complete<br>Digest: sha256:488f27c19e1acb098a3eef27818102ebe510ec44f454fbfde9b7887a448b763d<br>Status: Downloaded newer image for amazon/aws-cli:latest<br>docker.io/amazon/aws-cli:latest</pre><h4>One-time configuration</h4><p>You would need to configure few settings, including the access key ID and the secret associated with a particular IAM user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bCFfLEn8gJvzMiEcsvKtmw.png" /><figcaption>Obtain access key ID and secret from the IAM dashboard</figcaption></figure><p>Configuration settings are stored in the $HOME directory at the ~/.aws path. There’re two files: config and credentials.</p><pre>$ ls -1 ~/.aws<br>config<br>credentials</pre><p>For persistency purposes we will keep those files on our local machine and pass them through a volume mount inside a container: ~/.aws =&gt; /root/.aws . Optionally, it’s useful to add an extra mount point to share files between the local file system and the docker container: $(pwd) =&gt; /aws</p><pre>$ docker run --rm -it -v ~/.aws:/root/.aws -v $(pwd):/aws amazon/aws-cli --help<br>usage: aws [-h] [--profile PROFILE] [--debug]</pre><pre>optional arguments:<br>  -h, --help         show this help message and exit<br>  --profile PROFILE<br>  --debug</pre><p>When you work on different projects (e.g. home, work) each having different IAM users and access credentials, AWS provides you with a concept of a PROFILE to manage and use multiple identities at the same time. It’s a very convenient feature. For this blog post, I will use the pet_project for the PROFILE name.</p><p>Since we haven’t run through the initial configuration, let’s do it right away. Use aws configure subcommand, which will interactively prompt you several questions.</p><pre>$ docker run --rm -it -v ~/.aws:/root/.aws -v $(pwd):/aws amazon/aws-cli configure --profile pet_project</pre><pre>AWS Access Key ID [None]: AKIAIH627HJ6TNNEGPIQ<br>AWS Secret Access Key [None]: .....<br>Default region name [None]: us-east-2<br>Default output format [None]: json</pre><p>After the command has run to completion, the Docker container is stopped and is automatically removed. That is, each command invocation spawns a new short-lived container.</p><p>Let’s explore the contents of the ~/.aws/config file, which is in the <a href="https://en.wikipedia.org/wiki/INI_file">INI file format</a>. Notice how settings from different profiles are kept under different sections. The same holds for ~/.aws/credentials file, that stores access keys ID and secrets. This is how AWS CLI allows working with multiple identities simultaneously.</p><pre>[profile work_local_env]<br>region = eu-central-1<br>output = json</pre><pre>[profile pet_project]<br>region = us-east-2<br>output = json</pre><p>Now, let’s explore available S3 buckets. The profile is specified through a command-line argument.</p><pre>$ docker run --rm -it -v ~/.aws:/root/.aws -v $(pwd):/aws amazon/aws-cli --profile pet_project s3 ls</pre><pre>2021-08-23 16:35:34 pet-project-full-resolution-images</pre><p>An alternative and more convenient way of selecting the profile would be exporting the AWS_PROFILE environment variable.</p><pre>$ export AWS_PROFILE=pet_project</pre><pre>$ docker run --rm -it -e AWS_PROFILE -v ~/.aws:/root/.aws -v $(pwd):/aws amazon/aws-cli s3 ls s3://pet-project-full-resolution-images</pre><pre>2021-08-23 17:01:37    1931986 2020-11-05_17-33-02.png<br>2021-08-23 17:01:38    2117010 2020-11-06_10-57-17.png<br>2021-08-23 17:01:38     242779 2020-11-12_12-09-37.png<br>2021-08-23 17:01:38    2450599 2020-11-14_12-31-54.png<br>2021-08-23 17:01:38    2779765 2020-11-25_18-50-09.png<br>2021-08-23 17:01:38     103076 2020-11-25_19-10-15.png</pre><h4>Manage environment variables on a per-directory basis using the <a href="https://direnv.net/">direnv</a> tool</h4><p>How and when you configure the environment is up to you. You might set variables in your SHELL configuration files (e.g ~/.bash_profile or ~/.zshenv). Personally, I prefer using the tool called <a href="https://direnv.net/">direnv</a>. It allows me to maintain different sets of environment variables per directory. It requires creating a special file .envrc , which is automatically loaded by the tool when I cd into the directory.</p><pre>$ pwd<br>/Volumes/DATA/projects/personal/blog_on_medium</pre><pre>$ ls -1 ./using_cli_through_docker/.*<br>./using_cli_through_docker/.envrc</pre><pre>$ cat ./using_cli_through_docker/.envrc<br>export AWS_PROFILE=pet_project</pre><p>Now, when I cd into the directory for the first time, direnv will refuse to source the file, since I haven’t approved it yet. This is a security measure since an automatically loaded .envrc file is a great place for code injection attacks.</p><pre>$ cd using_cli_through_docker</pre><pre>direnv: error .envrc is blocked. Run `direnv allow` to approve its content.</pre><p>To approve the file, run the direnv allow command. Note that any future changes to the file will require you to re-approve it once again for security purposes.</p><pre>$ direnv allow</pre><pre>direnv: loading .envrc</pre><pre># check if the AWS_PROFILE is set<br>$ echo $AWS_PROFILE<br>pet_project </pre><p>Usually, I have a dedicated .envrc file per each project I’m working on. It’s git ignored and contains various local-only settings and secrets, which are not supposed to be shared with a team. When I jump between different projects, direnv automatically unloads old and loads the new file. It works behind the scenes, so sometimes I even forgot that I have such a great tool.</p><h4>Create SHELL alias for ease of use</h4><p>So far so good. But it’s still tough to type the long docker command while trying not to forget all the details: how to correctly mount volumes, pass the right environment variables. The better solution would be extracting the command and creating a SHELL alias. Since I’m using the <a href="https://www.zsh.org/">zsh</a>, I’ll declare an alias in the ~/.zshrc file.</p><pre># Alias to work with AWS CLI through the docker image<br>alias aws=&#39;docker run --rm -it -e AWS_PROFILE -v ~/.aws:/root/.aws -v $(pwd):/aws amazon/aws-cli&#39;</pre><p>Now the usage of AWS CLI would look like this, which is indistinguishable from the <a href="https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-mac.html">regular installation</a> approach.</p><pre>$ aws --version</pre><pre>aws-cli/2.2.31 Python/3.8.8 Linux/4.19.76-linuxkit docker/x86_64.amzn.2 prompt/off</pre><pre>$ aws s3 ls s3://pet-project-full-resolution-images</pre><pre>2021-08-23 17:01:37    1931986 2020-11-05_17-33-02.png<br>2021-08-23 17:01:38    2117010 2020-11-06_10-57-17.png<br>2021-08-23 17:01:38     242779 2020-11-12_12-09-37.png<br>2021-08-23 17:01:38    2450599 2020-11-14_12-31-54.png<br>2021-08-23 17:01:38    2779765 2020-11-25_18-50-09.png<br>2021-08-23 17:01:38     103076 2020-11-25_19-10-15.png</pre><h3>Using the Sentry CLI by running it in a Docker container</h3><p>The same approach could be applied to whatever CLI tool, given that the corresponding docker image exists.</p><p>Let’s take a look at <a href="https://docs.sentry.io/product/cli/">Sentry CLI</a>. On the current project, I’m going to use the Sentry CLI within a CI/CD pipeline <a href="https://docs.sentry.io/product/releases/setup/release-automation/jenkins/">to notify Sentry about the new release version and to upload JS source map files</a> of React app for better stack traces.</p><p>Fortunately, the Sentry team <a href="https://docs.sentry.io/product/cli/installation/#docker-image">distributes their CLI tool as a docker image</a>.</p><pre>$ docker pull getsentry/sentry-cli</pre><pre>Using default tag: latest<br>latest: Pulling from getsentry/sentry-cli<br>339de151aab4: Pull complete<br>2e65fd2d37d5: Pull complete<br>4518c0069cf2: Pull complete<br>6543a499611e: Pull complete<br>2ade6052dc88: Pull complete<br>Digest: sha256:31b8d58091557b8266a143cd8b1ce2e683cd734f65ba145251abc31c71d4e304<br>Status: Downloaded newer image for getsentry/sentry-cli:latest<br>docker.io/getsentry/sentry-cli:latest</pre><p>Configuration is kept in the ~/.sentryclirc file. Read through the c<a href="https://docs.sentry.io/product/cli/configuration/">onfiguration and authentication guide</a>. If you’re running a <a href="https://develop.sentry.dev/self-hosted/">self-hosted Sentry </a>specify its URL. Other non-secret data include your organization name and project name. The project name is optional since you can specify it later via a command-line argument.</p><pre>[defaults]<br>url=${YOUR_<a href="https://dev.sentry.espressoresearch.com:9000/">S</a>ENTRY_URL}<br>org=${YOUR_<a href="https://dev.sentry.espressoresearch.com:9000/">S</a>ENTRY_ORGANIZATION}<br>project=${YOUR_<a href="https://dev.sentry.espressoresearch.com:9000/">S</a>ENTRY_PROJECT_NAME}</pre><pre>[auth]<br>token=${YOUR_SECRET_ACCESS_TOKEN}</pre><p>The secret part is the auth token, which you can generate from Sentry Web UI at the “Settings -&gt; Account -&gt; API -&gt; Auth tokens” page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rXqrD3GcUNwqjx2iWZ2LQw.png" /><figcaption>Obtain an access token from Sentry Web UI</figcaption></figure><p>Unlike AWS, .sentryclirc file does not have a concept of PROFILE to manage multiple identities. So I chose to keep the .sentryclirc file inside the project directory, rather than in the $HOME directory. Thus, each project directory would have an individual .sentryclirc file.</p><p>Here is a very simple command, that lists all configured projects.</p><pre>$ docker run --rm -v $(pwd)/sentry/.sentryclirc:/root/.sentryclirc getsentry/sentry-cli projects list</pre><pre>+----+----------------------+------+----------------------+<br>| ID | Slug                 | Team | Name                 |<br>+----+----------------------+------+----------------------+<br>| 4  | app_project_1        | dev  | project_1            |<br>| 2  | app_project_2        | dev  | project_2            |<br>| 3  | app_project_3        | dev  | project_3            |<br>+----+----------------------+------+----------------------+</pre><p>Given I stick with the same convention of having the .sentryclirc file at the ./sentry/.sentryclirc path on each project I’m working on, I can create a reusable SHELL alias as well.</p><pre># Alias to work with Sentry CLI through the docker image<br>alias sentry-cli=&#39;docker run --rm -v $(pwd)/sentry/.sentryclirc:/root/.sentryclirc getsentry/sentry-cli&#39;</pre><p>Then, the usage would be nothing different from the <a href="https://docs.sentry.io/product/cli/installation/">regular Sentry CLI installation</a>.</p><pre>$ sentry-cli -V<br>sentry-cli 1.68.0</pre><pre>$ which sentry-cli<br>sentry-cli: aliased to docker run --rm -v $(pwd)/sentry/.sentryclirc:/root/.sentryclirc getsentry/sentry-cli</pre><h3>Conclusion</h3><p>Docker is an amazing tool, which proves to be useful even in such unexpected scenarios.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8b86b2c12916" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What does the Tech Lead role means?]]></title>
            <link>https://medium.com/@alexeysamoshkin/what-does-the-tech-lead-role-means-17a69f509ea0?source=rss-c22efb938ed8------2</link>
            <guid isPermaLink="false">https://medium.com/p/17a69f509ea0</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Alexey Samoshkin]]></dc:creator>
            <pubDate>Sat, 21 Aug 2021 14:51:38 GMT</pubDate>
            <atom:updated>2021-08-21T14:51:38.727Z</atom:updated>
            <content:encoded><![CDATA[<p>When it comes to software engineer grades, like junior, middle or senior developer, I always find it hard to tell where the boundary between those roles lie. Sure there’re no clear and strict boundaries. There’s no place where the white color suddenly turns into the black color, rather the color palette of software developers community is full of gray colors of different shades, saturation and brightness.</p><p>Software engineer grades might look something like this: associate, junior, intermediate level 1, intermediate level 2, senior level 1, senior level 2, principal software engineer. IT companies usually classify employees based based on their hard skills, soft skills and experience.</p><p>However, in this post, I would like to talk about the “Tech Lead” role — what responsibilities and tasks the person is supposed to take. First of all, it’s an attempt to put my own thoughts into some structure and answer this question to myself. I see many people using terms like Tech Lead, Team Lead, Solutions Architect, but seems like everyone has its own understanding of that role. , and it leads to confusion.</p><p>Below is my own understanding of the Tech Lead role.</p><h3>#1</h3><p>Tech Lead has a high-level vision of the project architecture, but also aware of low-level details and nuances, by being involved in everyday development activities.</p><p>Tech lead spends ~75% of his time in development activities. Otherwise, he puts himself at a risk of losing the feeling of the technical side of the project. As a result he turns into an “abstract architect”, who can give a lot of smart advices, except that those advices are not tailored and don’t take into account the current project and it’s specifics. There’s no value in such advices then.</p><p>In the same way, Tech Lead is not limited only to technical, tech debt or infrastructure-related tasks, he works with everyday product-related feature tasks, same to any other Team member. Otherwise puts himself at a risk of losing the feeling of the business/product side of the project. Losing either of those aspects — technical or business — is unacceptable for the Tech Lead.</p><h3>#2</h3><p>Tech Lead contributes to a high-level or a medium-level software design. It means splitting the feature into software modules with each module having its own responsibility, well-defined interface (aka API), dependencies to other modules/components and clear communication path and data flows between components. Term “Software modules” is deliberately made too abstract , because the exact name varies and depends on the chosen “zoom level”: packages, classes, services, functions, whatever you name it.</p><p>Tech Lead splits the feature into developer-oriented tasks, and passes them over to team members. Implementation details of each module/component are less relevant than the high-level decomposition and modularization. Given Tech Lead contributed to a clear modularization solution, and established interfaces, further work can be passed over and managed by other team members.</p><h3>#3</h3><p>Tech Lead understands the product from the business viewpoint. Speaks with the business the same language. Translates business problems into technical opportunities. It can to some extent serve as a bridge between business and technical riversides.</p><h3>#4</h3><p>Tech Lead is the person who not only talks to developer or tech-savvy people within his team, but can effectively collaborate with multiple non-technical teams. For example, has some awareness of how QA specialists work, their approaches, practices, tools. Can setup a healthy collaboration between DEV and QA team.</p><h3><strong>#5</strong></h3><p>Tech Lead understands the the overarching principle, that any other software design principles and practices take their root from, is the “<strong>COST OF CHANGE</strong>”. We don’t apply modularization, SOLID, testing practices, abstraction and encapsulation, readability principles, layered architecture, all those GoF design patterns just for the sake of applying them.</p><p>Tech Lead strives to keep the technical quality of the project at a high level in order to <strong>REDUCE THE COST OF CHANGE</strong> (the cost of new feature development, or the cost of change request to an existing feature), so the business can quickly adopt to ever-changing environment and circumstances, or quickly materialize the idea and test on the market.</p><h3>#6</h3><p>Tech Lead is the person who is responsible for the technical quality of the project. Tech lead establishes and the Team agrees to adhere to a particular level of quality and policies, for instance, no linter errors, no failing tests, naming conventions, Git branching model, release cycles, modular structure, testability, following various software design practices and principles.</p><p>Then the Tech Lead person is the last line of defense, who detects violations to policies or detects the poor level of quality of new contributions, communicates it back to team members, explains what, why and how should be improved.</p><h3>#7</h3><p>Tech Lead is the person who owns the Tech Debt.</p><p>Tech Leads is aware of pain points, technical issues, tech debt (e.g lack of Continuous Integration, lack of Testing, lack of modularity, entangled project structure, high cost of adding new changes/feature in the particular subsystem, etc). He has a vision of what is the next issue to address, what is the level of urgency of each issue, and can come up with a technical solution/vision on how to address and fix those issues.</p><h3>#8</h3><p>Tech Lead along with C-level management, System Architect and Tech/Team Leads from other teams contributes to and takes critical system-level technical decisions (e.g. pick a particular technology in favor or another, choose a type and vendor of a database shared across teams, choose a messaging solution, choose a cloud provider, agree on cross-team components and their interfaces, e.g GRPC user identity service).</p><p>Tech Lead takes technical decisions or pick technologies on its own for those things, which are limited to a scope of his Team only. Examples are choose the testing library and testing approach, choose the approach of how to run local environment in Docker Compose or MiniKube, choose logging and error tracking solution, decide how to split and decouple UI and backend parts of code, what naming convention to choose, what project directory structure to use, etc.</p><h3>#9</h3><p>Tech Lead is the person who can choose the right tool for the job/task.</p><h3>#10</h3><p>Tech Leads is the referee to choose between one or another alternative technical solution, in case when two or more team members cannot reach a consensus on their own, due to each having different vision, different arguments, pros and cons.</p><h3>#11</h3><p>Tech Lead has wide and extensive knowledge in areas not directly related to the current tech stack and has a fundamental knowledge base of programming topics in general.</p><p>For example, let’s suppose you current tech stack is MERN. Tech Lead is a person, who is also aware of:</p><ul><li>software design principles and practices</li><li>programming paradigms (OOP, FP)</li><li>SDLC approaches</li><li>Testing</li><li>Maybe has background or proficiency in a completely different language or stack (e.g. Python, .NET)</li><li>CI/CD</li><li>Infrastructure</li><li>versioning strategies</li><li>UI/UX aspects</li><li>has wide horizons of interests: Operating systems, Networking, Cloud solutions and virtualization, containerization, concurrency and parallelism, data structures and algorithms, compilers and interpreters, security aspects, computer hardware, processor architectures, etc.</li><li>has an organized and fined tuned toolset</li></ul><h3>#12</h3><p>Tech Lead mentors, helps teammates and skills them up. He shares his knowledge, and motivates the team to grow. Tech Lead teaches by his own personal example.</p><h3>Wrapping up</h3><p>This is how I see the Tech Lead role as of 2021.</p><p>Initially, it was just an attempt to answer to myself who the heck is the Tech Lead. Maybe there’re a lot of intersection with “Solutions Architect” or “Principal Software Engineer” role in terms or tasks and responsibilities. To be frank, it’s still not completely clear to me who the “Solutions Architect” is and how is it different from my vision of who is the Tech Lead. Maybe you can give me some input in the comments.</p><p>And the last thing I would like to stress.</p><p>“Tech Lead” does not equal to “React developer”. “Tech Lead” does not equal to “Node developer”. “Tech Lead” does not even equal to “Fullstack React + Node developer”.</p><p><strong>“Tech Lead” must be a “Software Engineer” in the first place.</strong> The particular technology or tech stack is the secondary aspect. If tomorrow on a project, we decide to switch from MongoDB to PostgreSQL, or even to move from Node to Python altogether — 60–70% of Tech Lead’s knowledge is still relevant and could be applied, he just needs to catch up with new technologies. Sure, the example with switching between different tech stacks, like Python an JS, is exaggerated. But nevertheless, for the Tech Lead it’s just a matter of time available to spend learning and practicing the new stuff.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=17a69f509ea0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Conditional spread operator in object and array literals in JavaScript]]></title>
            <link>https://medium.com/@alexeysamoshkin/conditional-spread-operator-in-object-and-array-literals-in-javascript-d05342f1406d?source=rss-c22efb938ed8------2</link>
            <guid isPermaLink="false">https://medium.com/p/d05342f1406d</guid>
            <category><![CDATA[it]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[react]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Alexey Samoshkin]]></dc:creator>
            <pubDate>Fri, 20 Aug 2021 19:59:32 GMT</pubDate>
            <atom:updated>2021-08-20T19:59:32.210Z</atom:updated>
            <content:encoded><![CDATA[<h3>Conditional rendering in JSX using logical “&amp;&amp;” operator</h3><p>If you’re a React developer, chances are you’re familiar with the common idiom of rendering component conditionally.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/488cd527d90d51acc1c2eb6478240acc/href">https://medium.com/media/488cd527d90d51acc1c2eb6478240acc/href</a></iframe><p>In the example above, if props.eventId is missing, the expression below will yield the undefined.</p><pre>{eventId &amp;&amp; renderEventId(eventId)}</pre><p>JSX is <a href="https://reactjs.org/docs/conditional-rendering.html#inline-if-with-logical--operator">designed with such cases in mind</a>, and if an expression returns false or undefined , React will treat it as a noop and not render anything.</p><p>Using logical &amp;&amp; operator in place of if..else statement is somewhat a controversial topic in JS community. But in case of JSX, the expression looks concise and improves the readability. From a pragmatic programmer viewpoint, I would favor readability in this particular case.</p><p>Sometimes I’m faced with a task when I need to compose an object or an array, where some properties or items should be added depending on the condition evaluation. Let’s see if the&amp;&amp; logical operator can help us with this.</p><h3>Add items to an array based on the condition</h3><p>For example, I need to create an array of items, where each item represents an Integration object from <a href="http://sentry/react%20-%20npm%20https://www.npmjs.com/package/@sentry/react">@sentry/react</a> or <a href="https://www.npmjs.com/package/@sentry/browser">@sentry/browser</a> npm packages.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b0a833b6ace4f8e3f10e2f49d2c49bc0/href">https://medium.com/media/b0a833b6ace4f8e3f10e2f49d2c49bc0/href</a></iframe><p>The trick here is to add the trailing .filter(Boolean) method, which will remove all those falsy fantom items.</p><p>When you need to add a collection of items to an array all of which are based on the single condition (e.g deployment.debug), you can use this variation of the above example.</p><pre>return [<br>  new ExtraErrorDataIntegration(),</pre><pre>  ...deployment.debug &amp;&amp; [<br>    new DebugIntegration(),<br>    new CaptureConsoleIntegration(),<br>    new OfflineIntegration()<br>  ],<br>].filter(Boolean);</pre><p>This approach might look strange for those who are unfamiliar with <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_syntax">the “rest/spread” syntax</a>. Compare it with the following conventional approach based on if/else blocks usage. Here, you get a mix of declarative and imperative approaches, where array literal declaration followed by push() mutation statements. Personally, I don’t like it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a44be529f19546480c78ca19a0a7bfb7/href">https://medium.com/media/a44be529f19546480c78ca19a0a7bfb7/href</a></iframe><p>Someone may argue that the latter example is more readable, and it does not rely on those strange syntactic constructs. I would agree that too much syntax sugar added to the language forces the developer to write too smart code which harmful, but this is not that case.</p><h3>Augment object with extra properties based on the condition</h3><p>The same approach could be applied when composing an object. Consider the simple example first:</p><pre>const { user, language } = appConfig;</pre><pre>const sentryUserFeedbackWidgetOptions = {<br>  lang: language.locale,<br>  ...user &amp;&amp; { <br>     email: user.email,<br>     name: user.name,<br>  },<br>};</pre><p>We augment object with two extra properties email and name, but only when the user expression is not falsy. Otherwise, object will contain only the lang property.</p><p>More complex example would be composing an object, that represents a MongoDB query.</p><pre>let filters = {<br>  ...ids &amp;&amp; { _id: { $in: ids } },<br>  ...tagId &amp;&amp; { tagId },<br>  ...experiences &amp;&amp; { experiences: { $in: experiences } },<br>  ...name &amp;&amp; { name },<br>  ...accountId &amp;&amp; { accountId }, <br>  ...createdBy &amp;&amp; { &#39;createdBy.id&#39;: createdBy },<br>};</pre><p>Notice, that it’s not the same with the example below. In case of a falsy condition, you’ll end up with a new property added to the object, but having null or undefined value. It’s not the same to the object without a property altogether. Finally, you might compose an invalid MongoDB query object, which will yield completely different results.</p><pre>let filters = {};<br>filters.ids = ids ? { _id: { $in: ids } } : null;<br>filters.tagId = tagId ? { tagId } : null;<br>filters.experiences = experiences ? { experiences: { $in: experiences } } : null;<br>filters.name = name ? { name } : null;<br>filters.accountId = accountId ? { accountId } : null;<br>filters.createdBy = createdBy ? { &#39;createdBy.id&#39;: createdBy } : null;</pre><p>The only equivalent solution is to use the conventional approach with if..else blocks. But, IMO, it looks too verbose.</p><pre>let filters = {};<br>if (filters.id) { <br>  filters.ids = { _id: { $in: ids } };<br>}<br>if (filters.tagId) { <br>  filters.tagId = { tagId };<br>}<br>if (filters.experiences) {<br>  filters.experiences = { experiences: { $in: experiences } };<br>}<br>if (filters.name) {<br>  filters.name = { name };<br>}<br>if (filters.accountId) {<br>  filters.accountId = { accountId };<br>}<br>if (filters.createdBy) {<br>  filters.createdBy = { &#39;createdBy.id&#39;: createdBy };<br>}</pre><h3>Wrapping up</h3><p>Choose whatever solution you like more. It’s jus at matter of preference.</p><p>But, please, don’t blindly follow advices, rules, guides on the Internet. Yes, usually it’s better to use if..else statements in favor of logical &amp;&amp; operator. But it’s not a hard rule, and final choice is always up to you, depending on the project needs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d05342f1406d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Secure and sane setup of new virtual instance on any cloud provider]]></title>
            <link>https://medium.com/@alexeysamoshkin/secure-and-sane-setup-of-new-virtual-instance-on-any-cloud-provider-c9c78342ad24?source=rss-c22efb938ed8------2</link>
            <guid isPermaLink="false">https://medium.com/p/c9c78342ad24</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[it]]></category>
            <dc:creator><![CDATA[Alexey Samoshkin]]></dc:creator>
            <pubDate>Mon, 09 Aug 2021 21:11:38 GMT</pubDate>
            <atom:updated>2021-08-09T21:14:41.964Z</atom:updated>
            <content:encoded><![CDATA[<p>We’ll set up a new virtual instance on a cloud provider, change the default configuration towards more secure and reasonable settings, create a new user, change sshd daemon configuration, harden firewall settings, and more…</p><p><strong>Here is the agenda for this blog post.</strong></p><ul><li>How to spin up a new instance on the Linode cloud provider.</li><li>Connecting to the remote host under root user via SSH</li><li>Creating a new dedicated user asamoshkin and adding it to the sudo group, so it can run commands on behalf of root user using sudo command</li><li>Generating new SSH key pair for asamoshkin user and configure the remote host to authorize connections by uploading the public key</li><li>Changing sshd daemon configuration to make it more secure. Disable password-based authentication. Disable login for the root user. Listen on IPv4 interface only. Allow TCP port forwarding (optional).</li><li>Learning a bit of theory about how iptables firewall work under the hood.</li><li>Changing firewall default policy from ALLOW ALL to DENY ALL, and enable only inbound traffic to port 22, where the sshd daemon is listening on.</li><li>Using Linode Lish console to connect directly to the host on behalf of the root user, in case you messed up with your configuration and cannot connect via SSH any more.</li><li>Save iptables rules to a file, in order to restore them back using iptables-save and iptables-restore commands</li><li>Create an image from the running Linode instance. Delete an instance to avoid extra charges being applied. Create a new Linode instance by restoring it from an image.</li></ul><h3>Spin up a new instance on Linode</h3><p>I will use <a href="https://www.linode.com/">Linode</a>, just because I’d already signed up with it some time ago and have an account there.</p><p>For the instance configuration, I will use the “Debian 10” distribution, opt to run an instance in Frankfurt region and choose the cheapest instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pt0xVHfKTA0f7LuePs0RvA.png" /><figcaption>Creating a new Linode isntance</figcaption></figure><p>Come up with some strong password for the root user. Note, that it is often advisable to completely disable root login through SSH after you have set up a dedicated user account that has sudo privileges. We’ll do it a bit later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*de9CFXZbsqkCwpgbwLKzOQ.png" /><figcaption>Choose a strong password for the root user</figcaption></figure><p>Now go ahead, create an instance, and within a minute it will be up and running. The new host comes with a public IP address.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CYVNNpIF8kEW4EYuK27GRA.png" /><figcaption>New instance is up and running</figcaption></figure><h3>Connecting to the remote host via SSH</h3><p>Connect to the instance through the SSH on behalf of the root user to ensure everything is up and running:</p><pre>ssh root@172.104.141.119<br>The authenticity of host &#39;172.104.141.119 (172.104.141.119)&#39; can&#39;t be established.<br>ECDSA key fingerprint is SHA256:qLLYD/xHE6SuVtp6ZxfT5BqY410u4PZm7vIMv5R9QNs.<br>Are you sure you want to continue connecting (yes/no/[fingerprint])? yes<br>Warning: Permanently added &#39;172.104.141.119&#39; (ECDSA) to the list of known hosts.<br>root@172.104.141.119&#39;s password:<br>Linux localhost 4.19.0-17-amd64 #1 SMP Debian 4.19.194-3 (2021-07-18) x86_64</pre><pre>The programs included with the Debian GNU/Linux system are free software;<br>the exact distribution terms for each program are described in the<br>individual files in /usr/share/doc/*/copyright.</pre><pre>Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent<br>permitted by applicable law.<br>root@localhost:~#</pre><h3>Creating a dedicated user</h3><p>Create a brand new user, so we don’t run any further activities on behalf of the root user. Regarding options used, -m means create home directory, -G sudo will add a new user to the sudo as his secondary group, and -s will use the /bin/bash as a primary shell.</p><pre>root@localhost:~# useradd -d /home/asamoshkin -m -G sudo -s /bin/bash asamoshkin</pre><p>Set a password for the newly created user using passwd command:</p><pre>root@localhost:~# passwd asamoshkin<br>New password:<br>Retype new password:<br>passwd: password updated successfully</pre><p>Now let’s verify the user is created and we can run other commands through the sudo with OS asking user’s own password first.</p><pre># change user to &quot;asamoshkin&quot; using &quot;su&quot; command<br>root@localhost:~# su asamoshkin</pre><pre># change to the home directory of the user<br>asamoshkin@localhost:/root$ cd<br>asamoshkin@localhost:~$ pwd<br>/home/asamoshkin</pre><pre># list the $HOME directory contents<br>asamoshkin@localhost:~$ ls -la<br>total 20<br>drwxr-xr-x 2 asamoshkin asamoshkin 4096 Aug  9 17:24 .<br>drwxr-xr-x 3 root       root       4096 Aug  9 17:24 ..<br>-rw-r--r-- 1 asamoshkin asamoshkin  220 Apr 18  2019 .bash_logout<br>-rw-r--r-- 1 asamoshkin asamoshkin 3526 Apr 18  2019 .bashrc<br>-rw-r--r-- 1 asamoshkin asamoshkin  807 Apr 18  2019 .profile</pre><pre># ensure user is a member of the &quot;sudo&quot; group<br>asamoshkin@localhost:~$ groups asamoshkin<br>asamoshkin : asamoshkin sudo</pre><pre># ensure user can execute commands through sudo, which will ask user&#39;s own password first<br>asamoshkin@localhost:~$ sudo su</pre><pre>We trust you have received the usual lecture from the local System<br>Administrator. It usually boils down to these three things:</pre><pre>#1) Respect the privacy of others.<br>    #2) Think before you type.<br>    #3) With great power comes great responsibility.</pre><pre>[sudo] password for asamoshkin:<br>root@localhost:/home/asamoshkin#</pre><p>You might be wondering about why adding user to the sudo group allows him to run commands on behalf of the root user. Where’s the magic comes from?</p><p>Actually, there’s no magic. There’s a special files called /etc/sudoers that defines which users or groups can utilize sudo , which commands can be run through it, and on behalf of which users. If you’re going to modify this file, please use the special utility visudo , rather than opening the file through a regular text editor (nano or vim). visudo runs some extra safety checks upon save to ensure whether your changes conform to the expected file’s structure.</p><pre>$ sudo cat /etc/sudoers</pre><pre># User privilege specification<br>root    ALL=(ALL:ALL) ALL</pre><pre># Allow members of group sudo to execute any command<br>%sudo   ALL=(ALL:ALL) ALL</pre><h3>Setting up SSH keys for the new user</h3><p>We can connect to the remote host using asamoshkin user by typing our password, but the best practice is to use public key authentication.</p><pre>$ ssh asamoshkin@172.104.141.119<br>asamoshkin@172.104.141.119&#39;s password:</pre><p>Let’s generate new SSH public-private key pair. We will use RSA algorithm and the key length of 4096 bits (which gives us a security strength of ~140 bits).</p><pre>ssh-keygen -t rsa -b 4096 -C &quot;<a href="/service/mailto:alexey.samoshkin@gmail.com">al.samoshkin@gmail.com</a>&quot; -f ./asamoshkin_at_linode_network_diagnostic_tools</pre><p>Note, <a href="https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-57pt1r5.pdf">that key length is not the same as a security strength</a> (measures in bits).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zOhzqd9ramdx0wYhhn4Xdg.png" /><figcaption>Security strength in bits and the corresponding key lengths of RSA and ECC algorithms</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yd4PGh440Ey3-cHz04lxFQ.png" /></figure><p>Public and private key pair is stored in the current working directory:</p><pre>$ ls -1 asamoshkin*<br>asamoshkin_at_linode_network_diagnostic_tools<br>asamoshkin_at_linode_network_diagnostic_tools.pub</pre><p>Next, append the public key to the the ~/.ssh/authorized_keys file on the remote host. First, we create ~/.ssh directory if it’s not yet there, and upload public key from the local machine to the ~/.ssh/authorized_keys file.</p><pre>$ cat asamoshkin_at_linode_network_diagnostic_tools.pub | ssh asamoshkin@172.104.141.119 &quot;umask 077 &amp;&amp; mkdir -p ~/.ssh &amp;&amp; cat &gt;&gt; ~/.ssh/authorized_keys &amp;&amp; chmod 700 ~/.ssh &amp;&amp; chmod 600 ~/.ssh/authorized_keys&quot;<br>asamoshkin@172.104.141.119&#39;s password:</pre><p>The pitfall is that the directory and the file should not have group and other ownership, so we change mode to rwx------ (700) and rw-------(600) using chmod command. Let’s make sure these changes are applied.</p><pre>$ ssh asamoshkin@172.104.141.119 &quot;ls -la ~/.ssh&quot;<br>asamoshkin@172.104.141.119&#39;s password:<br>total 12<br>drwx------ 2 asamoshkin asamoshkin 4096 Aug  9 17:38 .<br>drwxr-xr-x 3 asamoshkin asamoshkin 4096 Aug  9 17:38 ..<br>-rw------- 1 asamoshkin asamoshkin  748 Aug  9 17:38 authorized_keys</pre><p>Now we can connect to the instance through the SSH on the behalf of the newly created user, rather than the root user. -i points to the private key, which should be kept in secret.</p><pre>$ ssh -i asamoshkin_at_linode_network_diagnostic_tools asamoshkin@172.104.141.119</pre><pre>Linux localhost 4.19.0-17-amd64 #1 SMP Debian 4.19.194-3 (2021-07-18) x86_64</pre><pre>The programs included with the Debian GNU/Linux system are free software;<br>the exact distribution terms for each program are described in the<br>individual files in /usr/share/doc/*/copyright.</pre><pre>Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent<br>permitted by applicable law.<br>Last login: Mon Aug  9 17:37:38 2021 from 45.67.215.37<br>asamoshkin@localhost:~$</pre><h3>Changing sshd daemon configuration to make it more secure.</h3><p>Let’s go to the sshd config and tweak some settings to enforce the security aspects.</p><pre>$ sudo vim /etc/sshd/sshd_config</pre><pre># Disable password-based authentication altogether, and allow only connections with SSH keys<br>PasswordAuthentication no<br>PubkeyAuthentication yes</pre><pre># allow TCP tunneling (aka port forwarding)<br>AllowTcpForwarding yes</pre><pre># disable login for the root user (neither password nor pubkey auth)<br># note, that it&#39;s still possible to login as the root user from Linode Web UI (Lish Console)<br>PermitRootLogin no</pre><pre># by default, listen on all network interfaces, but IPv4 only<br>ListenAddress 0.0.0.0<br>AddressFamily inet</pre><pre># you might also change default port, if you want to add some obfuscation<br># Port 22</pre><p>Save the file with :wq command in Vim, and let’s restart the sshd daemon. Note, that you will be disconnected during the restart, but we’ll reconnect back shortly.</p><pre>$ sudo systemctl restart sshd</pre><p>An alternative would be to just reload the daemon. systemctl will send the SIGHUP signal to the sshd process, that serves as a request to reload configuration file only.</p><pre>$ sudo systemctl reload sshd</pre><p>Let’s ensure it listens only on the IPv4 interfaces:</p><pre>$ ss -tnl<br>State       Recv-Q      Send-Q           Local Address:Port           Peer Address:Port<br>LISTEN      0           128                    0.0.0.0:22                  0.0.0.0:*</pre><p>Now, let’s disconnect, and try to connect to the host on behalf of the root user, as we did at the beginning of the post. It will fail and that’s how it’s supposed to be.</p><pre>$ ssh -o &quot;PasswordAuthentication=yes&quot; asamoshkin@172.104.141.119<br>asamoshkin@172.104.141.119: Permission denied (publickey).</pre><pre>ssh -o &quot;PasswordAuthentication=yes&quot; root@172.104.141.119<br>root@172.104.141.119: Permission denied (publickey).</pre><p>Note, that it’s still possible to connect with asamoshkin user via SSH using pubkey authentication.</p><pre>ssh -i asamoshkin_at_linode_network_diagnostic_tools asamoshkin@172.104.141.119</pre><h3>Firewall and iptables. A bit of theory</h3><p>The next goal is to restrict our firewall rules using iptables to allow only inbound and outbound traffic to SSH daemon running on port 22.</p><p>In Debian 10 (Buster) the iptables framework is replaced by the <a href="https://wiki.nftables.org/wiki-nftables/index.php/Main_Page">nftables</a> framework. However, iptables is still there, except that it works in nftables compatibility mode.</p><pre># check out the OS release <br>$ lsb_release -d<br>Description:    Debian GNU/Linux 10 (buster)</pre><pre># iptables is still there<br>asamoshkin@localhost:~$ sudo which iptables<br>/usr/sbin/iptables</pre><pre># but the binary is a symlink<br>asamoshkin@localhost:~$ ls -la /usr/sbin/iptables<br>lrwxrwxrwx 1 root root 26 Jul 29 04:11 /usr/sbin/iptables -&gt; /etc/alternatives/iptables</pre><pre># that points to nftables binary<br>$ realpath /usr/sbin/iptables<br>/usr/sbin/xtables-nft-multi</pre><p>The moral is that we should use nftables instead, but since I don’t have enough expertise with nftables (you know, it’s not my everyday work to setup firewall rules as a software developer), I will be using iptables , which I’m familiar with.</p><p>Now a bit of theory.</p><p>In iptables rules are organized in <strong>tables</strong> and <strong>chains</strong>. <strong>Chain</strong> is essentially a collection of rules within a specific table. <strong>Rules</strong> are placed within a specific chain of a specific table.</p><p>Built-in chains are: PREROUTING, INPUT, FORWARD, OUTPUT, POSTROUTING . Each chain is called by <a href="https://www.netfilter.org/">netfilter</a> hook, and the network packet is matched against each rule in order. The first rule that matches the packet is applied, and no further rules are probed. Besides built-in chains, you can create your custom chains for organisational purposes.</p><p>Tables classify rules according to the type of decision/action they are used to make:</p><ul><li>raw; marking packets in order to implement connection state tracking</li><li>filter; used to filter network traffic based on allow or deny rules.</li><li>nat; used to modify the packet’s source or destination addresses in order to impact the way that the packets are further routed. Useful for configuring your Linux system as a router or for port forwarding.</li><li>mangle; used for special altering of IP headers of packets: change TOS or TTL. Can also put an internal kernel “mark” on the packet for further processing in other tables and/or by other networking tools.</li></ul><p>Here are two nice diagrams that summarises the above theory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/514/1*b_mGiw6ppu8VjNTIhiVzSw.png" /><figcaption><a href="https://crosp.net/wp-content/uploads/2017/07/packet_flow10-514x1024.png">https://crosp.net/wp-content/uploads/2017/07/packet_flow10-514x1024.png</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/971/1*Vs4XnYTCI4fXYuGl2V3xfw.png" /><figcaption><a href="https://www.booleanworld.com/wp-content/uploads/2017/06/Untitled-Diagram.png">https://www.booleanworld.com/wp-content/uploads/2017/06/Untitled-Diagram.png</a></figcaption></figure><p>View all rules (by default it lists rules from the “filter” table). As you see, the default firewall rules is to enable any inbound and outbound traffic.</p><pre>$ sudo iptables -L</pre><pre>Chain INPUT (policy ACCEPT)<br>target     prot opt source               destination</pre><pre>Chain FORWARD (policy ACCEPT)<br>target     prot opt source               destination</pre><pre>Chain OUTPUT (policy ACCEPT)<br>target     prot opt source               destination</pre><h3>Restricting firewall rules to open particular ports only</h3><p>Let get back to our terminal. We will allow any outgoing traffic from the host, but disable incoming traffic to the host, except for the already established connections (e.g you make an HTTP request from the host and would like to receive a HTTP response).</p><pre># allow any inbound traffic related to already established connections<br>$ sudo iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT</pre><pre># allow inbound SSH connections <br>$ sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT</pre><p>Finally, let’s change the default policy from ALLOW ALL to DENY ALL for the incoming traffic.</p><pre># change the default policy: allow any outbound traffic, deny any inbound traffic<br>$ sudo iptables -P OUTPUT ACCEPT<br>$ sudo iptables -P INPUT DROP</pre><p><strong>BE CAREFUL</strong>, because it’s easy to mess up when setting up firewall rules, and rules are applied immediately (personally, I assumed that it requires an explicit restart through systemd, but that’s not true).</p><p>For example, if I started my configuration with changing the default policy, like this, I will be kicked off from the server, as the firewall will step in immediately and drop my current SSH connection.</p><pre>$ sudo iptables -P OUTPUT ACCEPT<br>$ sudo iptables -P INPUT DROP</pre><pre># now the SSH connection freezes and I&#39;m kicked off the server# asamoshkin@localhost:~$ client_loop: send disconnect: Broken pipe</pre><p>If it happens to you, don’t worry. You can still get the root access through the Linode Lish Console, and revert back your firewall rules.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZdHWRAzkStWpw3QLi5GMFw.png" /><figcaption>Retain root access to the instance through the Linode Lish Console</figcaption></figure><p>Now the host will accept only the inbound traffic on port 22 (where our SSH daemon is listening on).</p><pre>$  sudo iptables -L -n<br>Chain INPUT (policy DROP)<br>target     prot opt source               destination<br>ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED<br>ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:22</pre><pre>Chain FORWARD (policy ACCEPT)<br>target     prot opt source               destination</pre><pre>Chain OUTPUT (policy ACCEPT)<br>target     prot opt source               destination</pre><h3>Double checking our firewall configuration</h3><p>Let’s double check this by running simple TCP server using netcat command on arbitrary port, let’s say 25 (e.g pretend to be an SMTP server).</p><pre>$ sudo nc -l -p 25</pre><p>Then open another SSH connection, and make sure that the socket on port 25 is in a LISTEN state and listens on all network interfaces (i.e. 0.0.0.0). Use ss command, which is a modern alternative to netstat that allows exploring opened sockets. Notice that the port 22 is also in a LISTEN state — that’s our sshd daemon running in the background.</p><pre>sudo ss -tnl4<br>State       Recv-Q      Send-Q           Local Address:Port           Peer Address:Port<br>LISTEN      0           128                    0.0.0.0:22                  0.0.0.0:*<br>LISTEN      0           1                      0.0.0.0:25                  0.0.0.0:*</pre><p>Now, from the local machine, try to connect to the port 22 and then to the port 25 using the same netcat utility, but this time running as a raw TCP client.</p><pre># from your local machine<br># connection to port 25 fails<br>$ nc -v -G 5 172.104.141.119 25<br>nc: connectx to 172.104.141.119 port 25 (tcp) failed: Operation timed out</pre><pre># but connection to port 22 works fine<br>$ nc -v -G 5 172.104.141.119 22<br>Connection to 172.104.141.119 port 22 [tcp/ssh] succeeded!<br>SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XES2TLmHa7VwaNBD0b0mFQ.png" /><figcaption>Using “netcat” and “ss” utilities to double check firewall configuration</figcaption></figure><h3>Create an image from your Linode instance</h3><p>You might want to shut down the instance, if you’re not going to use it right now. Note, that despite being powered off, it will still accrue charges.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DU4pPLkd9GegOpj1lYefbQ.png" /><figcaption>Charges are still applied even for powered off instances</figcaption></figure><p>The alternative would be creating an image from your running instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KIc-jvZINKWujeHZW7gFMA.png" /><figcaption>Create an image from the running Linode instance</figcaption></figure><p>It takes some time to create an image. So, just relax and wait.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*01LRxQc241Mqk0CTN6so9w.png" /><figcaption>Capturing an image</figcaption></figure><p>Custom images cost <strong>$0.10/GB per month</strong>, whereas a running Linode instance costs <strong>$5.00 per month</strong>. As soon as I got an image of around ~1.3GB in size, the obvious solution is to delete the instance altogether, and then spin up a new instance from the image when I need it.</p><p>Let’s validate this idea and spin up yet another instance from the image to make sure the solution works, and our setup we did so far is not lost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pLDkK1LjeV05OFBKnwPObQ.png" /><figcaption>Create a new Linode instance by restoring it from the image</figcaption></figure><p>Wait until it’s up and running. You will see a new public IP address allocated for the new instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N5rU4sV4iwXopCemhXJ1jA.png" /><figcaption>Second instance is up and running</figcaption></figure><p>To make sure everything works fine, all you need is to connect to the new instance through SSH using existing asamoshkin user and his SSH keys. If instance was correctly restored from the image, SSH public key and asamoshkin user should be in place. Use new IP address, which is 139.162.201.180 in my case.</p><pre>$ ssh -i asamoshkin_at_linode_network_diagnostic_tools asamoshkin@139.162.201.180<br>The authenticity of host &#39;139.162.201.180 (139.162.201.180)&#39; can&#39;t be established.<br>ECDSA key fingerprint is SHA256:qLLYD/xHE6SuVtp6ZxfT5BqY410u4PZm7vIMv5R9QNs.<br>Are you sure you want to continue connecting (yes/no/[fingerprint])? yes<br>Warning: Permanently added &#39;139.162.201.180&#39; (ECDSA) to the list of known hosts.<br>Linux localhost 5.13.4-x86_64-linode146 #1 SMP Wed Jul 21 14:45:45 EDT 2021 x86_64</pre><pre>The programs included with the Debian GNU/Linux system are free software;<br>the exact distribution terms for each program are described in the<br>individual files in /usr/share/doc/*/copyright.</pre><pre>Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent<br>permitted by applicable law.<br>Last login: Mon Aug  9 19:15:30 2021 from 45.67.215.37</pre><p>Looks like everything went smooth. <strong>However, there is a pitfall.</strong> iptables configuration is not preserved, and got reset to ALLOW ALL default policy.</p><pre>$ sudo iptables -L<br>Chain INPUT (policy ACCEPT)<br>target     prot opt source               destination</pre><pre>Chain FORWARD (policy ACCEPT)<br>target     prot opt source               destination</pre><pre>Chain OUTPUT (policy ACCEPT)<br>target     prot opt source               destination<br># Warning: iptables-legacy tables present, use iptables-legacy to see them</pre><p>To overcome this issue, you might save your iptables configuration to some file which will be kept in the image, and restore rules from the file after the initial boot of your new instance.</p><pre>$ mkdir -p ~/firewall<br>$ sudo iptables-save &gt; ~/firewall/iptables.rules<br>$ iptables-restore -c &lt; ~/firewall/iptables.rules</pre><p>So, go ahead and reconnect to the first instance. Save iptables rules to a file, as described above. Then drop the old image and create it from scratch once again. Finally, delete both Linode instances, so you’re not charged when instances are idle and are not in use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sjO7_uhM7hkPpJfb77Z2jQ.png" /><figcaption>Delete the Linode isntance</figcaption></figure><h3>Wrap up</h3><p>At this point you’re done with the configuration of your new shiny instance. It’s secure enough and has reasonable configuration. You can be proud of yourself making all this progress. Congratulations!</p><p>Now, whenever you need to spin up a new instance for some experiments or whatever purposes, just restore it from the existing image, and don’t spend time setting it up everything scratch any more.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c9c78342ad24" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Manage Git work tree and index using “git restore” command]]></title>
            <link>https://medium.com/@alexeysamoshkin/manage-git-work-tree-and-index-using-git-restore-command-d90d90f04c4f?source=rss-c22efb938ed8------2</link>
            <guid isPermaLink="false">https://medium.com/p/d90d90f04c4f</guid>
            <category><![CDATA[tools]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[information-technology]]></category>
            <category><![CDATA[git]]></category>
            <dc:creator><![CDATA[Alexey Samoshkin]]></dc:creator>
            <pubDate>Mon, 19 Jul 2021 08:46:40 GMT</pubDate>
            <atom:updated>2021-07-19T08:46:40.246Z</atom:updated>
            <content:encoded><![CDATA[<p>If you’re working with Git from a command line, as I used to in my everyday work, most likely you find it tough to remember all those commands, that are related to managing working tree and index.</p><h3>Old school way</h3><p>Let’s take a look at each typical scenario, along with a command that handles it.</p><p>I have a bunch of new files/changes in a working tree. I want to split them in several commits. So I need to stage several files first, commit, then stage the next couple of files, commit, and so on. This is obvious and straightforward. Just use “git add” command. Nothing new here.</p><pre>$ git add &lt;file1&gt; &lt;file2&gt;</pre><p>I have accidentally staged a wrong file, which I’d like to postpone for the next commit, and now I want to un-stage it.</p><pre>$ git reset —- &lt;path&gt;</pre><p>Note, that “git reset” command comes with two completely different responsibilities:<br>1. Un-stage files from index. It takes file from HEAD and put them to the index, effectively un-staging them. You still have the same file in working tree. You’re working on a file-level, and you’re not allowed to specify commit (it always uses HEAD). It does not move branch pointer to another commit — your HEAD is left intact.</p><p>2. Move a branch pointer to another commit and overwrite the working tree and/or index to completely match the given commit. You a working at a commit-level, and cannot specify a single file. All files are affected.</p><p>The latter case comes in following forms:</p><pre>$ git reset —-soft &lt;commit&gt;<br>$ git reset —-hard &lt;commit&gt;<br>$ git reset —-mixed &lt;commit&gt; # — mixed is the default</pre><p>Next scenario. I’ve changed several files in my working tree, but then I realised it’s all crap, and I want to revert one or more files.<br>This command takes file from Git index and puts it to the working tree.</p><pre>$ git checkout -— &lt;path&gt;</pre><p>Alternatively, you might want to revert file to one of your previous commit. Note, that this command would also update file in both index and working tree to match the given commit.</p><pre>$ git checkout &lt;commit&gt; -- &lt;path&gt;</pre><p>Note, that same to git reset, git checkout command is overloaded with two responsibilities: switching branches and restoring files.</p><p>To sum up all these commands’ zoo, I’ve created this neat diagram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8cVGKYskMsYu-1eLy2SUIQ.png" /><figcaption>Manage work tree and index in an old school way</figcaption></figure><h4>Index stages and merge conflicts</h4><p>During a merge, in case of merge conflict, Git uses index to keep the different revisions of conflicting files from a three-way merge. To be more correct, it has a concept of different index stages. Think of them like the copies of regular Git index, with each copy given a dedicated number.</p><p>#1. Keeps the files from common merge-base commit in a three-way merge<br>#2. Ours. Files from our current branch (e.g master)<br>#3. Theirs. Files from the branch we’re going to merge (e.g feature)</p><p>There’re several commands, that let you interact with index stages.</p><p>Suppose, when a merge conflict happens, I don’t want to resolve the merge, but just take “ours” version of the given file (e.g from a master) and completely discard theirs one (from a feature branch).</p><pre>$ git checkout --theirs -- &lt;path&gt;</pre><p>Or vice versa, take “theirs” version of a file and completely discard “ours” one.</p><pre>$ git checkout --ours -- &lt;path&gt;</pre><p>Or, another use case. What if you messed up while trying to resolve the conflict within a single file. Now, you’re completely lost and start to panic. Don’t run “git merge— abort” just to repeat the whole merge process rerunning “git merge” again. Just restore merge conflict markers using following command:</p><pre>$ git checkout --merge -- &lt;path&gt;</pre><p>There’s no built-in command to peek into the file version from a common merge-base, but you can do it with some low-level Git plumbing:</p><pre>$ git cat-file -p :1:[file] &gt; [file]</pre><h3>New shiny way with “git restore”</h3><p>Notice, how we used several commands with subtle differences in arguments passed to manage our working tree and index. That’s a cumbersome experience. I constantly forget those commands/arguments variations. To tackle the issue, I’ve added a bunch of shell aliases, but besides usability benefits, they just hide the root problem — inconsistency and complexity of the basic Git commands.</p><pre>alias giA=&#39;git add -A&#39;<br>alias gia=&quot;git add&quot;<br>alias gir=&quot;git reset --&quot;<br>alias gwu=&quot;git checkout --&quot;<br># etc</pre><p>Let’s look at how “git restore” command brings more consistency handling those use cases.</p><p>I have accidentally staged a wrong file, which I’d like to postpone for the next commit, and now I want to un-stage it.</p><pre>$ git restore --staged &lt;path&gt;</pre><p>I’ve changed several files in my working tree, but then I realised it’s all crap, and I want to revert one or more files. This command takes file from Git index and puts it to the working tree.</p><pre>$ git restore &lt;path&gt;</pre><p>You want to revert file to the version from HEAD. Note, that this command updates file in both index and working tree to match the given commit.</p><pre>git restore --staged --worktree &lt;path&gt;</pre><p>If you want to revert a file to the version from one of your previous commit:</p><pre>git restore --staged --worktree -s &lt;commit&gt; &lt;path&gt;</pre><p>Moreover, you can even take file from the given commit, but update working tree only, without touching index. It’s something, which was not possible in an “old school” way.</p><pre>git restore --worktree -s &lt;commit&gt; &lt;path&gt;</pre><p>To work with index stages during merge conflict, use the same “git restore” command.</p><pre>git restore --theirs &lt;path&gt;<br>git restore --ours &lt;path&gt;<br>git restore --merge &lt;path&gt;</pre><p>To sum up, let’s look at the diagram again, now updated with a “git restore” usage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zxbGCDOn2TKzB17rGkR5hQ.png" /><figcaption>Manage work tree and Git index with “git restore” command</figcaption></figure><h3>Conclusion</h3><p>Notice, how “git restore” command gives you the unified solution to manage your working tree and index.</p><p>It makes your life a bit easier, since you don’t need to remember all those git reset, git checkout commands, don’t need to recall to delimit command from “pathspecs” using -—, don’t keep in mind the difference between git reset &lt;path&gt; and git reset --hard|soft|mixed &lt;commit&gt; command flavours.</p><p>That’s great. Hope you enjoy my new post.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d90d90f04c4f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to setup OBS on MacOS for screen recording]]></title>
            <link>https://medium.com/@alexeysamoshkin/how-to-setup-obs-on-macos-for-screen-recording-a064c4f90fc3?source=rss-c22efb938ed8------2</link>
            <guid isPermaLink="false">https://medium.com/p/a064c4f90fc3</guid>
            <category><![CDATA[installation]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[guides-and-tutorials]]></category>
            <category><![CDATA[information-technology]]></category>
            <category><![CDATA[tools]]></category>
            <dc:creator><![CDATA[Alexey Samoshkin]]></dc:creator>
            <pubDate>Sat, 17 Jul 2021 18:02:15 GMT</pubDate>
            <atom:updated>2021-07-17T18:02:15.016Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://obsproject.com/">OBS (Open Broadcaster Software)</a> is a free and open source software for video recording and live streaming. This post is a step-by-step installation guide for MacOS.</p><p>Use it when you need to:</p><ul><li>record a video of a meeting or a call with your customers or colleagues</li><li>record a video tutorial on any “how to” question (e.g how to run guest MacOS 10.14 in a VirtualBox running on host MacOS 10.15)</li><li>stream your screen and/or camera in a real-time (e.g might be used to share your game play in a real time)</li></ul><p>Limitations:</p><ul><li>OBS is not a video editor. It cannot cut/splice/edit recording in anyway, or remove noise from the audio. Use another tools. For example, use <a href="https://shotcut.org/">ShotCut</a> for video editing, and/or <a href="http://Audacity ® | Free, open source, cross-platform audio software for multi-track recording and editing. https://www.audacityteam.org/">Audacity</a> for working with audio channels.</li></ul><h3>Step 1. Download and install</h3><p>Navigate to <a href="https://obsproject.com/">https://obsproject.com/</a> and download “DMG” file for MacOS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8aPeQKN_0_CIG4oPUaAlJg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5I9ldudQUBHUlh89Bp4I5A.png" /></figure><p>Next, open “Applications” directory and find “OBS.app”. Click it to open.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GA-fPsIAZyqNTTzaAVwk4g.png" /></figure><p>If you prefer using Terminal rather than the graphical user interface, you can install OBS with the following command (given you have <a href="https://brew.sh/">Brew</a> already installed):</p><pre># check if you have brew installed<br>$ brew --version</pre><pre># Install OBS<br>$ brew install --cask obs</pre><pre># Launch OBS<br>$ open /Applications/OBS.app/</pre><p>Once OBS is launched, here is how the initial screen looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c2-qjWegcUHb_riQik9REA.png" /></figure><h3>Step 2. Install virtual audio capture device</h3><p>On MacOS, by default, OBS is not able to capture native audio signal from your machine. In the out-of-box installation of OBS, if you record your screen while watching YouTube video, OBS will capture just your screen, but not the audio signal, which you would otherwise hear through the headphones or internal speakers.</p><p>To gap a bridge between default MacOS audio output and OBS, we’ll need to install virtual audio device driver — <strong>iShowU Audio Capture</strong>.</p><p><a href="https://downloads.shinywhitebox.com/driver/latest/">Download the installer</a>. It supports all recent MacOS versions, except for the BigSur (v11.x). Installation process depends on your MacOS version, so choose and follow the right guide:</p><ul><li>Sierra (v10.12) and High Sierra (v10.13), <a href="https://support.shinywhitebox.com/hc/en-us/articles/204161459-Installing-iShowU-Audio-Capture-Mojave-and-earlier-">step-by-step installation guide</a></li><li>Mojave (v10.14) and Catalina (v10.15), <a href="https://support.shinywhitebox.com/hc/en-us/articles/360030800592-Install-iShowU-Audio-Capture-Mojave-Catalina">step-by-step installation guide</a></li></ul><p>It’s best to restart your system upon successful installation.</p><p>To verify you’re all set, navigate to “System Preferences → Sound”, and check the “Output” tab. It should list the new virtual audio output device — “iShowU Audio Capture”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HiwwBbekgJ3LQYjOkHBTnQ.png" /></figure><h3>Step 3. Create multi-output audio device</h3><p>“IShowU Audio Capture” device we’ve just created will be used by OBS to capture the audio signal. The last thing to do is to pipe the audio signal from our Internal speakers or headphones (what we hear as humans) to “IShowU Audio Capture” virtual device (to let OBS hear that as well).</p><p>Navigate to “Applications → Utilities → Audio MIDI setup” and open it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CwNSVdSWOeKAj9kYddWhWw.png" /></figure><p>Here you will see all audio input and outputs, registered in the system (incl. iShowU audio device we’ve just installed).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r1DQtlmeeYhDjgkfJvq7gw.png" /></figure><p>Click the “+” icon in the left-bottom corner and select “Create Multi-Output Device”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pAoyKMZM7Hf7YO7WSzZQTg.png" /></figure><p>Now configure the multi-output device, as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QlC4e9-0LngyI3QjxpJu8Q.png" /></figure><p>Finally, go back to “System Preferences → Sound”, and choose the “multi-output device” as the primary output device, and whatever microphone you’re going to use as an input device.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BLMtOuk3HhkcfdrGeayiXw.png" /></figure><h3>Step 4. Configure audio settings in OBS</h3><p>Go back to OBS application. Open “Settings”, pick “Audio” menu item. Choose your desired microphone in the “Mix/Auxiliary Audio” input. Leave all other inputs “Disabled” by default.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iGZvzm99BxRFwSfl5Ax7NQ.png" /></figure><p>Apply changes and close “Settings” popup. In the main screen, in the “Sources” widget click “+” icon and add new “Audio Output Capture” source.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*swLL1WfMvzbWJaESOaP8Fg.png" /></figure><p>In a popup in a “Device” dropdown, select “iShowU Audio capture” device. Click OK.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_SQ-z9RiNdnIDgxC8O3lSw.png" /></figure><p>To verify you’re all set, launch some video or music on your local machine (e.g watch a video on YouTube). Talk to your microphone. Make sure that OBS detects your audio input/output signals, by looking at the green bars in the “Audio Mixer” section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l68ZC_Xcp3ivEWiHLNTYxQ.png" /></figure><h3>Step 5. Configure video settings in OBS</h3><p>Go back to Settings in OBS, and choose “Video” menu item. Configure output video resolution (HD or FullHD), and frame per seconds (30 fps, 60fps) settings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wW0r8bt7CRCH1ie0Oq2VPQ.png" /></figure><p>In the same “Settings” popup choose “Output” menu item. Configure recording quality and the directory to store your video files. Make sure to choose “MKV” video format, in order to be able to resume recordings where you left off in case of disruption or power outage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KT6PEujlxVVT3CDiZ1yG_A.png" /></figure><h3>Step 7. Capture your display</h3><p>Go back to the main screen. In the “Sources” widget click “+” icon and select “Display Capture” source. If you have multiple displays, choose one you’d like to record.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LfpzqRgW4VkuB8RCwiNpPg.png" /></figure><p>Once you see your screen instead of the black area, right click on it to bring the context menu, and choose “Transform → Fit to screen”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H52pSoq8YFIKq8zhPigfvg.png" /></figure><p>Finally, you should see the following screen. What’s happening is OBS app records my primary screen with OBS app shown inside, which is recording itself, and so on and so on. The recursion is infinite and looks like this. That’s OK.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xIzIopEaFip5BcKheba1sg.png" /></figure><p>Now you’re all set up with OBS and ready to make your first recording. Hide the OBS app for now to avoid distraction and overloading your CPU.</p><h3>Step 8. Making your first recording</h3><p>First, make sure your “System preferences → Sound” settings are all correct. Occasionally, MacOS might change them whenever you plug or unplug headphones or your AirPods. Make sure to choose “Multi-Output device” in the “Output” tab, and your mic (I prefer Internal Microphone for better quality than the AirPods mic) in the “Input” tab.</p><p>Now prepare you desktop for the recording. Launch the app you want to record (e.g browser tab with a Google meet). Optionally, make it fullscreen. Don’t switch back to the OBS app. Just hover on the top of the MacOS menu bar, and choose “Start recording”. In the same way, choose “Stop recording” once you’re done.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_dxvBP-T9ru-DxDzOsPu2A.png" /></figure><p>After you stopped the recording, open the directory where you choose to store video files, and ensure your new recording is there. Open in it your favourite video player to test.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rJMeZXyTKtNulNli_SG0FA.png" /></figure><h3>Step 9. Convert from MKV to MP4 video format</h3><p>Open back OBS application, and choose “File → Remux Recordings” menu. Select just recorded MKV file as a source, and click “Remux”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zX1Oid6bQAjxb_icddEZ6w.png" /></figure><p>Now you’ll get your recoding in an MP4 format in the same directory. Open it and play.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qGAntK8fe4jnJ8jUfAeKiQ.png" /></figure><p>Additionally, you can explore video/audio codecs, resolution and other media info via mediainfo command in a terminal (install it via Brew first).</p><pre>$ brew install media-info<br>$ mediainfo &lt;path_to_video_or_audio_file&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gkx0aNQ5eiQvOxO4rfxpkQ.png" /></figure><h3>CONGRATULATIONS!!</h3><p>That’s it. You’re done with your first recording. Share the video with a friends, or upload it on YouTube. Then take a break and brew a cup of coffee. That’s was pretty tough to configure, wasn’t it?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a064c4f90fc3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why you should be using fzf, the command line fuzzy finder]]></title>
            <link>https://medium.com/free-code-camp/fzf-a-command-line-fuzzy-finder-missing-demo-a7de312403ff?source=rss-c22efb938ed8------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7de312403ff</guid>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Alexey Samoshkin]]></dc:creator>
            <pubDate>Fri, 19 Apr 2019 12:25:50 GMT</pubDate>
            <atom:updated>2019-05-01T16:38:27.755Z</atom:updated>
            <content:encoded><![CDATA[<h4>Missing demo found</h4><p>In this tutorial, I’ll help you take your command line habits to a next level with fzf . Start searching files like a pro. Learn less known features, like changing directory, searching through a command history, looking up the host name to SSH into, killing a process, having instant file’s preview with syntax highlighting, and more…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*LTR424sh7y8E8rUzsUnFsQ.gif" /><figcaption>quick demo</figcaption></figure><p>Today I’m going to tell you about a tool that can change your tech life into “before” and “after” parts. Trust me — it’s not a hyperbole. Meet <a href="https://github.com/junegunn/fzf">fzf</a>, which stands for fuzzy finder. As the definition claims, <strong>it’s a general purpose command-line fuzzy finder</strong>. That doesn’t sound very descriptive and attractive for those who are hearing about it for the first time. But it’s a very popular project ranked at around 21,000 stars <a href="https://github.com/junegunn/fzf">on the Github</a>. So it’s time to clear the fog and get a deeper insight.</p><p>This post accompanies <a href="https://www.youtube.com/watch?v=qgG5Jhi_Els">my recent screencast</a> on the topic. So, if you’re a person who learns by watching, check it out. Otherwise, check it out too 😃, because tools like fzf are best introduced with a live demo rather than tons of text.</p><p>As the fzf project page doesn’t have any demo video so far, I’ve called this post “missing demo found”. But now, this video has already been pulled into the fzf repo and has become a part of the <a href="https://github.com/junegunn/fzf#demo">project’s readme</a>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FqgG5Jhi_Els%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DqgG5Jhi_Els&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FqgG5Jhi_Els%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5e3dd9f5c420a0cb198a5f74fdd884e1/href">https://medium.com/media/5e3dd9f5c420a0cb198a5f74fdd884e1/href</a></iframe><h3>Searching for files</h3><p>People who are used to a command line environment are likely familiar with the concept of Unix filters. This is when several independent utilities are composed together into a pipeline to produce the desired output step by step. For instance, this pipeline produces a list of strings:</p><pre>$ yes | head -10 | awk &#39;{ print NR, NR % 2 == 0 ? &quot;even&quot; : &quot;odd&quot; }&#39;</pre><pre>1 odd<br>2 even<br>3 odd<br>4 even<br>5 odd<br>6 even<br>7 odd<br>8 even<br>9 odd<br>10 even</pre><p>Each program acts as a filter. In simple terms, fzf is just another Unix filter. It reads lines from stdin, launches an interactive finder dialogue, and finally writes selected items to stdout. The key point and difference from tools like GNU find, is its interactive finder dialogue that filters items instantly as you type.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*LTR424sh7y8E8rUzsUnFsQ.gif" /><figcaption>“fzf” as a interactive unix filter</figcaption></figure><p>It might not sound too practical so far, but the primary use case for fzf is to search for files on the command line. With fuzzy matching and instant feedback, you’re only few characters away from finding the right file no matter how deeply it’s lost in the directory hierarchy. No need to jump back to your file manager, traverse the directory hierarchy, copy a file’s path and paste it back to the shell. Compare “file manager” vs “fzf” workflows below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*Yp4AN6dCJeGhXbXHOXc4yA.gif" /><figcaption>fzf vs file manager comparison</figcaption></figure><p>fzf supports fuzzy matching so you can just type several characters in a row and it will match lines with those characters scattered across the string. Alternatively, prefix a search term with a single quote, like &#39;string, to opt for exact matches only, or run as fzf --exact .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Li_Dq0rvvAcG441bzFSanw.png" /><figcaption>fuzzy vs exact match</figcaption></figure><p>It does not support regular expressions or glob patterns, so *.sh pattern would not work. But keep it simple — productivity and speed is your goal today. You don’t have spare time to compose and type correct regular expressions. Instead just type several words, or even parts of the words, delimited by a space, and that would cover &gt;90% of the use cases. For the remaining 10%, use ^ and $ to match the start and end of the string respectively, and use ! to negate matching.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*z789AD9GLIsmy1ncME3Dqw.gif" /><figcaption>fzf matching syntax</figcaption></figure><p>Printing selected files to a command line is not super useful, so usually search is combined with some further action. For instance, you might open it with Vim, or pipe selected items to the next program.</p><pre># Open file in a Vim<br>vim -o `fzf`</pre><pre># Print info for each selected file<br>fzf | xargs ls -l</pre><h3>Fuzzy completion for bash and zsh</h3><p>To make it more convenient, fuzzy completion can be triggered if the word before the cursor ends with the trigger sequence which is by default **. For example, type vim ~/path/** and press TAB . Voilà, fzf steps in!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*L53twoIb0JoOsdq1lpeIqg.gif" /><figcaption>Double star fuzzy completion</figcaption></figure><p>** sequence triggers fzf finder and it resembles * which is for native shell expansion. At some point, you might even forget about fzf’s presence and get the impression it’s a native shell feature.</p><p><a href="https://github.com/junegunn/fzf#fuzzy-completion-for-bash-and-zsh">Format is as follows</a>, where FUZZY_PATTERN is optional.</p><pre>COMMAND [DIRECTORY/][FUZZY_PATTERN]**&lt;TAB&gt;</pre><p>If you don’t like the ** sequence, you can use keyboard shortcuts. CTRL+T triggers fzf and pastes the selected file onto the command line, whereas ALT+C changes into the selected directory.</p><h3>Changing directory</h3><p>Ok, enough about searching files. Let’s talk about other useful applications. Changing your working directory is such a common operation. But nevertheless I’m always stuck a bit trying to recall and type the right directory’s path, making several mistakes along the way. It slows down my tempo. Even zsh completions do not compensate for it. But with fzf, changing directories is a breeze, no matter how deep and far it is. Just type cd ** and you’re almost there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*TmD0RtJb59-oJqv8Q18p-g.gif" /><figcaption>changing directory is like a breeze</figcaption></figure><p>Notice that while fzf is indexing your directory tree, you can start searching right away. Changing directories is my favourite use case, and fuzzy matching really excels here. It requires from you the same effort to change a directory whether it’s one level deep or ten levels deep.</p><p>ALT+C shortcut is an alternate way to trigger fzf in a change directory mode.</p><h3>Command history</h3><p>You might be using Ctrl+R keyboard shortcut to search through your command history. Great, but how about supercharging it with fuzzy finder? Look and compare.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*toZ2nnqRBovAatH-3ksYNA.gif" /><figcaption>Searching through command history</figcaption></figure><p>It pastes the selected item on the command line, so you can tweak it further.</p><h3>Search the host name to SSH into</h3><p>If you’re a backend developer and working with a number of remote servers, you might appreciate ssh+fzf combo. Use the same double star trigger sequence and type ssh ** . It pulls recently used IP addresses and host names from your ~/.ssh/config and brings up an interactive finder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*3D89-7AhPcHSOQMkwbSC_g.gif" /><figcaption>“fzf + ssh” combo</figcaption></figure><h3>Send a signal to a process</h3><p>Sometimes we want to send a signal to a process, but first we need to obtain its PID by name. Usually one uses pgrep &lt;process_name&gt; to resolve PID followed by a kill &lt;process_pid&gt; referring that PID. With fzf you can combine both steps. Type kill &lt;TAB&gt; and fzf steps in listing all your processes. No need to switch to a dedicated process monitor any more, like “Activity Monitor” on Mac.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*tWvjkw_HIU1AWvd8M8yHTA.gif" /><figcaption>Find a process and send a signal</figcaption></figure><h3>Preview files as you search</h3><p>Suppose you’re searching for files, but sometimes the file name itself does not tell you enough. So you might want to peek into a file’s content for a moment to make a decision. fzf covers you here as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*dpDNa5iXqh4C5cE_GmCwdA.gif" /><figcaption>Preview window</figcaption></figure><p>By default I have preview window off, but I toggle it on when I want to peek into the files. Plus, I’ve enhanced it with a coloured output and syntax highlighting using <a href="https://github.com/sharkdp/bat">bat</a> as a preview command.</p><h3>Customization</h3><p>There are two main ways to customize the stock version of fzf:</p><ul><li>Change the behaviour of the finder dialogue (preview window, keyboard shortcuts, dimensions, custom actions, etc). See FZF_DEFAULT_OPTS environment variable.</li><li>Change the underlying find backend. By default it uses GNU find utility, but you can switch to the more advanced tools like <a href="https://github.com/sharkdp/fd">fd</a> or <a href="https://github.com/BurntSushi/ripgrep">ripgrep</a>. First, it’s faster than the stock find utility. Second, those tools respect .gitignore rules, so you don’t get node_modules or .git files in your search results. You <a href="https://github.com/junegunn/fzf#git-ls-tree-for-fast-traversal">can also use</a> git ls-tree to list files when you’re in a Git repo. See FZF_DEFAULT_COMMAND environment variable.</li></ul><p>Configuration is done through the environment variables. Here is the snippet from my setup. Not sure if it would work as is when copy-n-pasted, most likely there’re some other missing dependencies. But this is just to give you an idea of how the configuration looks.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7a5d275501fb0faeb87d0c3f13b69c32/href">https://medium.com/media/7a5d275501fb0faeb87d0c3f13b69c32/href</a></iframe><h3>Fzf and Vim</h3><p>So far we’ve seen command-line usage only. But fzf is also shipped as a <a href="https://github.com/junegunn/fzf.vim">Vim plugin</a>. And trust me, if you are a Vimmer, its worth adding to your vimrc. Vim usage is out of the scope of this post, otherwise no one would read it 😃. But I talk about it thoroughly in the second part of the <a href="https://www.youtube.com/watch?v=qgG5Jhi_Els">my video</a>, also linked at the beginning of the post.</p><p>If you’re impatient, here is a small excerpt to draw your attention. Use :grep command for project-wide text search, open matches in fzf window in a full screen mode, further filter matches within fzf, and jump to the selected match. And don’t forget about instant file preview at the precise line position. That’s awesome, isn’t it?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*bPgXLwaHNwKevvoKmja9Mw.gif" /><figcaption>fzf ❤️ vim</figcaption></figure><h3>Conclusion</h3><p>I hope you’re impressed of how super awesome <a href="https://github.com/junegunn/fzf">fzf</a> is. Chances are you already willing to install and give it a try.</p><p>Keep in mind that fzf is not only for file searching, although it’s the primary feature of it. As you can see, “<strong>search + action</strong>” is a general principle, but it’s abstract and powerful enough to work with any kind of list: files, directories, processes, host names, etc. Remember at the beginning of this post — <strong>fzf is just a Unix filter</strong>. Unlock your imagination, feed it any list, and tailor it for your own needs. You might take some inspiration from a <a href="https://github.com/junegunn/fzf/wiki/examples">wealth of examples</a>.</p><p>BTW, appreciate how superior Unix philosophy is. fzf is a great example of a program that adheres to these principles.</p><ul><li>let the program to do one thing and do it well (single responsibility principle)</li><li>make it abstract enough to be agnostic of irrelevant details or data types</li><li>compose separate individual programs using standard well-defined interfaces.</li></ul><p>Stick with those principles in your software development career. Knowing fundamental principles that stand behind the rapid birth and death of multiple tools, languages, and frameworks out there is what differentiates professional developers from amateurs.</p><h3>Resources</h3><p>junegunn/fzf: A command-line fuzzy finder — <a href="https://github.com/junegunn/fzf">https://github.com/junegunn/fzf</a></p><p>Vim universe. fzf — command line fuzzy finder — YouTube — <a href="https://www.youtube.com/watch?v=qgG5Jhi_Els">https://www.youtube.com/watch?v=qgG5Jhi_Els</a></p><p>My Youtube channel. There’re only a few videos, as I’m taking my first steps into making screencasts. Be patient, I promise to make more videos. Alexey Samoshkin — YouTube — <a href="https://www.youtube.com/channel/UCfju8u-YOpNMO4CbyzIsc9Q">https://www.youtube.com/channel/UCfju8u-YOpNMO4CbyzIsc9Q</a>.</p><p>sharkdp/fd: A simple, fast and user-friendly alternative to ‘find’ — <a href="https://github.com/sharkdp/fd">https://github.com/sharkdp/fd</a></p><p>BurntSushi/ripgrep: ripgrep recursively searches directories for a regex pattern — <a href="https://github.com/BurntSushi/ripgrep">https://github.com/BurntSushi/ripgrep</a></p><p>junegunn/fzf.vim: fzf vim — <a href="https://github.com/junegunn/fzf.vim">https://github.com/junegunn/fzf.vim</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7de312403ff" width="1" height="1" alt=""><hr><p><a href="https://medium.com/free-code-camp/fzf-a-command-line-fuzzy-finder-missing-demo-a7de312403ff">Why you should be using fzf, the command line fuzzy finder</a> was originally published in <a href="https://medium.com/free-code-camp">We’ve moved to freeCodeCamp.org/news</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[JavaScript type coercion explained]]></title>
            <link>https://medium.com/free-code-camp/js-type-coercion-explained-27ba3d9a2839?source=rss-c22efb938ed8------2</link>
            <guid isPermaLink="false">https://medium.com/p/27ba3d9a2839</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Alexey Samoshkin]]></dc:creator>
            <pubDate>Wed, 17 Jan 2018 19:29:59 GMT</pubDate>
            <atom:updated>2018-02-04T22:35:41.728Z</atom:updated>
            <content:encoded><![CDATA[<h4>Know your engines</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7awmfn1lq2McPz8ggapndw.png" /><figcaption>Weird things can happen in JavaScript</figcaption></figure><p><strong>[Edit 2/5/2018]</strong>: This post is now <a href="https://medium.com/@sergeybulavyk/%D0%BF%D1%80%D0%B5%D0%BE%D0%B1%D1%80%D0%B0%D0%B7%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5-%D1%82%D0%B8%D0%BF%D0%BE%D0%B2-%D0%B2-javascript-35a15ddfc333">available in Russian</a>. Claps to <a href="https://medium.com/u/8ce091dcbd92">Serj Bulavyk</a> for his efforts.</p><p><strong>Type coercion</strong> is the process of converting value from one type to another (such as string to number, object to boolean, and so on). Any type, be it primitive or an object, is a valid subject for type coercion. To recall, primitives are: number, string, boolean, null, undefined + Symbol (added in ES6).</p><p>As an example of type coercion in practice, look at the <a href="https://dorey.github.io/JavaScript-Equality-Table/">JavaScript Comparison Table</a>, which shows how the loose equality == operator behaves for different a and b types. This matrix looks scary due to implicit type coercion that == operator does, and it’s hardly possible to remember all those combinations. And you don’t have to do that — just learn the underlying type coercion principles.</p><p>This article goes in-depth on how type coercion works in JavaScript, and will arm you with the essential knowledge, so you can feel confident explaining what following expressions calculate to. By the end of the article I’ll show answers and explain them.</p><pre>true + false<br>12 / &quot;6&quot;<br>&quot;number&quot; + 15 + 3<br>15 + 3 + &quot;number&quot;<br>[1] &gt; null<br>&quot;foo&quot; + + &quot;bar&quot;<br>&#39;true&#39; == true<br>false == &#39;false&#39;<br>null == &#39;&#39;<br>!!&quot;false&quot; == !!&quot;true&quot;<br>[‘x’] == ‘x’<br>[] + null + 1<br>[1,2,3] == [1,2,3]<br>{}+[]+{}+[1]<br>!+[]+[]+![]<br>new Date(0) - 0<br>new Date(0) + 0</pre><p>Yes, this list is full of pretty silly things you can do as a developer. In 90% of use cases it’s better to avoid implicit type coercion. Consider this list as a learning exercise to test your knowledge on how type coercion works. If you’re bored, you can find more examples on <a href="https://wtfjs.com/">wtfjs.com</a>.</p><p>By the way, sometimes you might face such questions on the interview for a JavaScript developer position. So, keep reading 😄</p><h3>Implicit vs. explicit coercion</h3><p>Type coercion can be explicit and implicit.</p><p>When a developer expresses the intention to convert between types by writing the appropriate code, like Number(value), it’s called <strong>explicit type coercion</strong> (or type casting).</p><p>Since JavaScript is a weakly-typed language, values can also be converted between different types automatically, and it is called <strong>implicit type coercion</strong>. It usually happens when you apply operators to values of different types, like<br>1 == null, 2/’5&#39;, null + new Date(), or it can be triggered by the surrounding context, like with if (value) {…}, where value is coerced to boolean.</p><p>One operator that does not trigger implicit type coercion is ===, which is called the strict equality operator. The loose equality operator == on the other hand does both comparison and type coercion if needed.</p><p>Implicit type coercion is a double edge sword: it’s a great source of frustration and defects, but also a useful mechanism that allows us to write less code without losing the readability.</p><h3>Three types of conversion</h3><p>The first rule to know is there are only three types of conversion in JavaScript:</p><ul><li>to string</li><li>to boolean</li><li>to number</li></ul><p>Secondly, conversion logic for primitives and objects works differently, but both primitives and objects can only be converted in those three ways.</p><p>Let’s start with primitives first.</p><h3>String conversion</h3><p>To explicitly convert values to a string apply the String() function. Implicit coercion is triggered by the binary + operator, when any operand is a string:</p><pre>String(123) // explicit<br>123 + &#39;&#39;    // implicit</pre><p>All primitive values are converted to strings naturally as you might expect:</p><pre>String(123)                   // &#39;123&#39;<br>String(-12.3)                 // &#39;-12.3&#39;<br>String(null)                  // &#39;null&#39;<br>String(undefined)             // &#39;undefined&#39;<br>String(true)                  // &#39;true&#39;<br>String(false)                 // &#39;false&#39;</pre><p>Symbol conversion is a bit tricky, because it can only be converted explicitly, but not implicitly. <a href="https://leanpub.com/understandinges6/read/#leanpub-auto-symbol-coercion">Read more</a> on Symbol coercion rules.</p><pre>String(Symbol(&#39;my symbol&#39;))   // &#39;Symbol(my symbol)&#39;<br>&#39;&#39; + Symbol(&#39;my symbol&#39;)      // TypeError is thrown</pre><h3>Boolean conversion</h3><p>To explicitly convert a value to a boolean apply the Boolean() function.<br>Implicit conversion happens in logical context, or is triggered by logical operators ( || &amp;&amp; !) .</p><pre>Boolean(2)          // explicit<br>if (2) { ... }      // implicit due to logical context<br>!!2                 // implicit due to logical operator<br>2 || &#39;hello&#39;        // implicit due to logical operator</pre><p><strong>Note</strong>: Logical operators such as || and &amp;&amp; do boolean conversions internally, but <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Expressions_and_Operators#Logical_operators">actually return the value of original operands</a>, even if they are not boolean.</p><pre>// returns number 123, instead of returning true<br>// &#39;hello&#39; and 123 are still coerced to boolean internally to calculate the expression<br>let x = &#39;hello&#39; &amp;&amp; 123;   // x === 123</pre><p>As soon as there are only 2 possible results of boolean conversion: true or false, it’s just easier to remember the list of falsy values.</p><pre>Boolean(&#39;&#39;)           // false<br>Boolean(0)            // false     <br>Boolean(-0)           // false<br>Boolean(NaN)          // false<br>Boolean(null)         // false<br>Boolean(undefined)    // false<br>Boolean(false)        // false</pre><p>Any value that is not in the list is converted to true, including object, function, Array, Date, user-defined type, and so on. Symbols are truthy values. Empty object and arrays are truthy values as well:</p><pre>Boolean({})             // true<br>Boolean([])             // true<br>Boolean(Symbol())       // true<br>!!Symbol()              // true<br>Boolean(function() {})  // true</pre><h3>Numeric conversion</h3><p>For an explicit conversion just apply the Number() function, same as you did with Boolean() and String() .</p><p>Implicit conversion is tricky, because it’s triggered in more cases:</p><ul><li>comparison operators (&gt;, &lt;, &lt;=,&gt;=)</li><li>bitwise operators ( | &amp; ^ ~)</li><li>arithmetic operators (- + * / % ). Note, that binary+ does not trigger numeric conversion, when any operand is a string.</li><li>unary + operator</li><li>loose equality operator == (incl. !=). <br>Note that == does not trigger numeric conversion when both operands are strings.</li></ul><pre>Number(&#39;123&#39;)   // explicit<br>+&#39;123&#39;          // implicit<br>123 != &#39;456&#39;    // implicit<br>4 &gt; &#39;5&#39;         // implicit<br>5/null          // implicit<br>true | 0        // implicit</pre><p>Here is how primitive values are converted to numbers:</p><pre>Number(null)                   // 0<br>Number(undefined)              // NaN<br>Number(true)                   // 1<br>Number(false)                  // 0<br>Number(&quot; 12 &quot;)                 // 12<br>Number(&quot;-12.34&quot;)               // -12.34<br>Number(&quot;\n&quot;)                   // 0<br>Number(&quot; 12s &quot;)                // NaN<br>Number(123)                    // 123</pre><p>When converting a string to a number, the engine first trims leading and trailing whitespace, \n, \t characters, returning NaN if the trimmed string does not represent a valid number. If string is empty, it returns 0.</p><p>null and undefined are handled differently: null becomes 0, whereas undefined becomes NaN.</p><p>Symbols cannot be converted to a number neither explicitly nor implicitly. Moreover, TypeError is thrown, instead of silently converting to NaN, like it happens for undefined. See more on Symbol conversion rules on <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Symbol#Symbol_type_conversions">MDN</a>.</p><pre>Number(Symbol(&#39;my symbol&#39;))    // TypeError is thrown<br>+Symbol(&#39;123&#39;)                 // TypeError is thrown</pre><p>There are two <strong>special rules</strong> to remember:</p><ol><li>When applying == to null or undefined, numeric conversion does not happen. null equals only to null or undefined, and does not equal to anything else.</li></ol><pre>null == 0               // false, null is not converted to 0<br>null == null            // true<br>undefined == undefined  // true<br>null == undefined       // true</pre><p>2. NaN does not equal to anything even itself:</p><pre>if (value !== value) { console.log(&quot;we&#39;re dealing with NaN here&quot;) }</pre><h3>Type coercion for objects</h3><p>So far, we’ve looked at type coercion for primitive values. That’s not very exciting.</p><p>When it comes to objects and engine encounters expression like [1] + [2,3], first it needs to convert an object to a primitive value, which is then converted to the final type. And still there are only three types of conversion: numeric, string and boolean.</p><p>The simplest case is boolean conversion: any non-primitive value is always <br>coerced to true, no matter if an object or an array is empty or not.</p><p>Objects are converted to primitives via the internal [[ToPrimitive]] method, which is responsible for both numeric and string conversion.</p><p>Here is a pseudo implementation of [[ToPrimitive]] method:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e0737349ce9276bbea1b0c437f3fbc0/href">https://medium.com/media/6e0737349ce9276bbea1b0c437f3fbc0/href</a></iframe><p>[[ToPrimitive]] is passed with an input value and preferred type of conversion: Number or String. preferredType is optional.</p><p>Both numeric and string conversion make use of two methods of the input object: valueOf and toString . Both methods are declared on Object.prototype and thus available for any derived types, such as Date, Array, etc.</p><p>In general the algorithm is as follows:</p><ol><li>If input is already a primitive, do nothing and return it.</li></ol><p>2. Call input.toString(), if the result is primitive, return it.</p><p>3. Call input.valueOf(), if the result is primitive, return it.</p><p>4. If neither input.toString() nor input.valueOf() yields primitive, throw TypeError.</p><p>Numeric conversion first calls valueOf (3) with a fallback to toString (2). String conversion does the opposite: toString (2) followed by valueOf (3).</p><p>Most built-in types do not have valueOf, or have valueOf returning this object itself, so it’s ignored because it’s not a primitive. That’s why numeric and string conversion might work the same — both end up calling toString().</p><p>Different operators can trigger either numeric or string conversion with a help of preferredType parameter. But there are two exceptions: loose equality == and binary + operators trigger default conversion modes (preferredType is not specified, or equals to default). In this case, most built-in types assume numeric conversion as a default, except Date that does string conversion.</p><p>Here is an example of Date conversion behavior:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3428dfe2d8b702efd7d007a9265aa2fe/href">https://medium.com/media/3428dfe2d8b702efd7d007a9265aa2fe/href</a></iframe><p>You can override the default toString() and valueOf() methods to hook into object-to-primitive conversion logic.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b9f550241447c9ab1262051e38f734df/href">https://medium.com/media/b9f550241447c9ab1262051e38f734df/href</a></iframe><p>Notice how obj + ‘’ returns ‘101’ as a string. + operator triggers a default conversion mode, and as said before Object assumes numeric conversion as a default, thus using the valueOf() method first instead of toString().</p><h3>ES6 Symbol.toPrimitive method</h3><p>In ES5 you can hook into object-to-primitive conversion logic by overriding toString and valueOf methods.</p><p>In ES6 you can go farther and completely replace internal[[ToPrimitive]] routine by implementing the[Symbol.toPrimtive] method on an object.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/69a4867c26c793ed6a49e2aa589410ff/href">https://medium.com/media/69a4867c26c793ed6a49e2aa589410ff/href</a></iframe><h3>Examples</h3><p>Armed with the theory, now let’s get back to our examples:</p><pre>true + false             // 1<br>12 / &quot;6&quot;                 // 2<br>&quot;number&quot; + 15 + 3        // &#39;number153&#39;<br>15 + 3 + &quot;number&quot;        // &#39;18number&#39;<br>[1] &gt; null               // true<br>&quot;foo&quot; + + &quot;bar&quot;          // &#39;fooNaN&#39;<br>&#39;true&#39; == true           // false<br>false == &#39;false&#39;         // false<br>null == &#39;&#39;               // false<br>!!&quot;false&quot; == !!&quot;true&quot;    // true<br>[&#39;x&#39;] == &#39;x&#39;             // true <br>[] + null + 1            // &#39;null1&#39;<br>[1,2,3] == [1,2,3]       // false<br>{}+[]+{}+[1]             // &#39;0[object Object]1&#39;<br>!+[]+[]+![]              // &#39;truefalse&#39;<br>new Date(0) - 0          // 0<br>new Date(0) + 0          // &#39;Thu Jan 01 1970 02:00:00(EET)0&#39;</pre><p>Below you can find explanation for each the expression.</p><p>Binary + operator triggers numeric conversion for true and false</p><pre>true + false<br>==&gt; 1 + 0<br>==&gt; 1</pre><p>Arithmetic division operator / triggers numeric conversion for string &#39;6&#39; :</p><pre>12 / &#39;6&#39;<br>==&gt; 12 / 6<br>==&gt;&gt; 2</pre><p>Operator + has left-to-right associativity, so expression &quot;number&quot; + 15 runs first. Since one operand is a string, + operator triggers string conversion for the number 15. On the second step expression &quot;number15&quot; + 3 is evaluated similarly.</p><pre>“number” + 15 + 3 <br>==&gt; &quot;number15&quot; + 3 <br>==&gt; &quot;number153&quot;</pre><p>Expression 15 + 3 is evaluated first. No need for coercion at all, since both operands are numbers. On the second step, expression 18 + &#39;number&#39; is evaluated, and since one operand is a string, it triggers a string conversion.</p><pre>15 + 3 + &quot;number&quot; <br>==&gt; 18 + &quot;number&quot; <br>==&gt; &quot;18number&quot;</pre><p>Comparison operator &gt; triggers numeric conversion for [1] and null .</p><pre>[1] &gt; null<br>==&gt; &#39;1&#39; &gt; 0<br>==&gt; 1 &gt; 0<br>==&gt; true</pre><p>Unary + operator has higher precedence over binary + operator. So +&#39;bar&#39; expression evaluates first. Unary plus triggers numeric conversion for string &#39;bar&#39;. Since the string does not represent a valid number, the result is NaN. On the second step, expression &#39;foo&#39; + NaN is evaluated.</p><pre>&quot;foo&quot; + + &quot;bar&quot; <br>==&gt; &quot;foo&quot; + (+&quot;bar&quot;) <br>==&gt; &quot;foo&quot; + NaN <br>==&gt; &quot;fooNaN&quot;</pre><p>== operator triggers numeric conversion, string &#39;true&#39; is converted to NaN, boolean true is converted to 1.</p><pre>&#39;true&#39; == true<br>==&gt; NaN == 1<br>==&gt; false</pre><pre>false == &#39;false&#39;   <br>==&gt; 0 == NaN<br>==&gt; false</pre><p>== usually triggers numeric conversion, but it’s not the case with null . null equals to null or undefined only, and does not equal to anything else.</p><pre>null == &#39;&#39;<br>==&gt; false</pre><p>!! operator converts both &#39;true&#39; and &#39;false&#39; strings to boolean true, since they are non-empty strings. Then, == just checks equality of two boolean true&#39;s without any coercion.</p><pre>!!&quot;false&quot; == !!&quot;true&quot;  <br>==&gt; true == true<br>==&gt; true</pre><p>== operator triggers a numeric conversion for an array. Array’s valueOf() method returns the array itself, and is ignored because it’s not a primitive. Array’s toString() converts [&#39;x&#39;] to just &#39;x&#39; string.</p><pre>[&#39;x&#39;] == &#39;x&#39;  <br>==&gt; &#39;x&#39; == &#39;x&#39;<br>==&gt;  true</pre><p>+ operator triggers numeric conversion for []. Array’s valueOf() method is ignored, because it returns array itself, which is non-primitive. Array’s toString returns an empty string.</p><p>On the the second step expression &#39;&#39; + null + 1 is evaluated.</p><pre>[] + null + 1  <br>==&gt;  &#39;&#39; + null + 1  <br>==&gt;  &#39;null&#39; + 1  <br>==&gt; &#39;null1&#39;</pre><p>Logical || and &amp;&amp; operators coerce operands to boolean, but return original operands (not booleans). 0 is falsy, whereas &#39;0&#39; is truthy, because it’s a non-empty string. {} empty object is truthy as well.</p><pre>0 || &quot;0&quot; &amp;&amp; {}  <br>==&gt;  (0 || &quot;0&quot;) &amp;&amp; {}<br>==&gt; (false || true) &amp;&amp; true  // internally<br>==&gt; &quot;0&quot; &amp;&amp; {}<br>==&gt; true &amp;&amp; true             // internally<br>==&gt; {}</pre><p>No coercion is needed because both operands have same type. Since == checks for object identity (and not for object equality) and the two arrays are two different instances, the result is false.</p><pre>[1,2,3] == [1,2,3]<br>==&gt;  false</pre><p>All operands are non-primitive values, so + starts with the leftmost triggering numeric conversion. Both Object’s and Array’s valueOf method returns the object itself, so it’s ignored. toString() is used as a fallback. The trick here is that first {} is not considered as an object literal, but rather as a block declaration statement, so it’s ignored. Evaluation starts with next +[] expression, which is converted to an empty string via toString() method and then to 0 .</p><pre>{}+[]+{}+[1]<br>==&gt; +[]+{}+[1]<br>==&gt; 0 + {} + [1]<br>==&gt; 0 + &#39;[object Object]&#39; + [1]<br>==&gt; &#39;0[object Object]&#39; + [1]<br>==&gt; &#39;0[object Object]&#39; + &#39;1&#39;<br>==&gt; &#39;0[object Object]1&#39;</pre><p>This one is better explained step by step according to operator precedence.</p><pre>!+[]+[]+![]  <br>==&gt; (!+[]) + [] + (![])<br>==&gt; !0 + [] + false<br>==&gt; true + [] + false<br>==&gt; true + &#39;&#39; + false<br>==&gt; &#39;truefalse&#39;</pre><p>- operator triggers numeric conversion for Date. Date.valueOf() returns number of milliseconds since Unix epoch.</p><pre>new Date(0) - 0<br>==&gt; 0 - 0<br>==&gt; 0</pre><p>+ operator triggers default conversion. Date assumes string conversion as a default one, so toString() method is used, rather than valueOf().</p><pre>new Date(0) + 0<br>==&gt; &#39;Thu Jan 01 1970 02:00:00 GMT+0200 (EET)&#39; + 0<br>==&gt; &#39;Thu Jan 01 1970 02:00:00 GMT+0200 (EET)0&#39;</pre><h3>Resources</h3><p>I really want to recommend the excellent book “<a href="https://leanpub.com/understandinges6">Understanding ES6</a>” written by <a href="https://medium.com/u/77c787f473e5">Nicholas C. Zakas</a>. It’s a great ES6 learning resource, not too high-level, and does not dig into internals too much.</p><p>And here is a good book on ES5 only - <a href="http://speakingjs.com/">SpeakingJS</a> written by <a href="https://medium.com/u/7fab51e62203">Axel Rauschmayer</a>.</p><p>(<strong>Russian</strong>) Современный учебник Javascript — <a href="https://learn.javascript.ru/">https://learn.javascript.ru/</a>. Especially <a href="https://learn.javascript.ru/object-conversion">these</a> <a href="https://learn.javascript.ru/types-conversion">two</a> pages on type coercion.</p><p>JavaScript Comparison Table — <a href="https://dorey.github.io/JavaScript-Equality-Table/">https://dorey.github.io/JavaScript-Equality-Table/</a></p><p>wtfjs — a little code blog about that language we love despite giving us so much to hate — <a href="https://wtfjs.com/">https://wtfjs.com/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=27ba3d9a2839" width="1" height="1" alt=""><hr><p><a href="https://medium.com/free-code-camp/js-type-coercion-explained-27ba3d9a2839">JavaScript type coercion explained</a> was originally published in <a href="https://medium.com/free-code-camp">We’ve moved to freeCodeCamp.org/news</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>