<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Marek Piechut on Medium]]></title>
        <description><![CDATA[Stories by Marek Piechut on Medium]]></description>
        <link>https://medium.com/@marekpiechut?source=rss-8e0e09168c02------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*-OBENjzVnjmeoaoyvmgJtQ.jpeg</url>
            <title>Stories by Marek Piechut on Medium</title>
            <link>https://medium.com/@marekpiechut?source=rss-8e0e09168c02------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 02:48:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@marekpiechut/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Postgres as document store, or “you don’t need ORM in Node.js”]]></title>
            <link>https://medium.com/@marekpiechut/postgres-as-document-store-or-you-dont-need-orm-in-node-js-177714e6567a?source=rss-8e0e09168c02------2</link>
            <guid isPermaLink="false">https://medium.com/p/177714e6567a</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[mongo]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Marek Piechut]]></dc:creator>
            <pubDate>Fri, 12 Jul 2024 10:25:50 GMT</pubDate>
            <atom:updated>2024-07-12T10:25:50.053Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XeZgEVWWtBmAIXJZg8VmAQ.jpeg" /></figure><p>For quite few years now I’ve been betting on Postgres. After seeing what it’s capable with PostGIS, FTS and other non-standard SQL workloads, I also started using it as my NoSQL database. In this post I want to share with you how I’m using Postgres as my document store for OLTP workloads.</p><p>I started using it that way few years ago, when building <a href="https://crosskeeper.app/">CrossKeeper</a> (now decommissioned), after getting a “query rewrite fatigue” due to constantly changing requirements. This was causing many changes to data models and table schema. We already had a lot of data in Postgres and didn’t want to migrate away from it. We also planned to move back parts of the data to relational model once requirements stabilize a bit. In the end it’s more performant and helps with data consistency.</p><p>We ended up staying with document-like store till the very end of the product. It worked very well for us, even when we stopped moving things around so much. It also worked well for me through the years.</p><h3>Database Schema</h3><p>We wanted things to be simple, so after some time most of our postgres tables looked like this:</p><pre> Column  |            Type             | Nullable | Default <br>---------+-----------------------------+----------+---------<br> id      | uuid                        | not null | <br> created | timestamp without time zone | not null | now()<br> updated | timestamp without time zone |          | <br> data    | jsonb                       | not null |</pre><p>You could event limit that to only id and data columns (heck even only data if you&#39;re brave enough), but storing <em>created</em> and <em>updated</em> stamps outside the main data allows us to do some fancy things like optimistic locking and makes manual querying for suspicious data easier. Just remember that, <strong>application can only mess with </strong><strong>data column</strong> rest is for infrastructure.</p><p>Whenever you need a new <em>aggregate</em>, you create a new table in database with structure like this. The only thing that changes in schema later on are the <strong>indexes</strong>. Of course you need to manage them to match your querying patterns if you start doing something more than CRUD by id.</p><h3>Repositories — you don’t need ORM</h3><p>ORMs are very useful tools, especially outside of JavaScript, where JSON is a native data structure. Manually mapping all fields from classes/objects into JSON is tedious and boring job. Nobody wants to do that. But ORMs have a tendency to leak into your domain and application logic. After some time you start seeing queries inside a controller. You start to notice persistence/ORM related annotations in your entities and aggregates. And after a while there’s no way you’re going to migrate to some other storage or change database layout in any meaningful way. What I’m proposing here, is to follow Repository pattern from DDD and keep all your database related logic contained in one place for each aggregate. Even if you don&#39;t strictly follow DDD.</p><p>This way you’ll be able not only to easily localize all queries and serialization logic, but also swap database layout without changing any code outside of single file.</p><p>There’s also one more benefit of designing your application layer like this — most of repository code can be reused. And thanks to dynamic structure of JavaScript and JSON being it’s native data type, you can just pass objects as is to and from database. No need for complex field mapping.</p><h3>Base Repository</h3><p>Let’s start with base repository. This one does not handle any JSON mapping yet. It’s just a base logic to handle things like transaction and pooling of DB connections:</p><pre>export abstract class Repository {<br>  protected pgPool: pg.Pool<br>  protected tx?: pg.ClientBase<br><br>  constructor(pgPool: pg.Pool) {<br>    this.pgPool = pgPool<br>  }<br><br>  //Subclasses need to override this<br>  protected abstract clone(): this<br><br>  //Returns new instance bound to transaction<br>  public withTx(tx: pg.ClientBase): this {<br>    const repo = this.clone()<br>    repo.tx = tx<br>    return repo as this<br>  }<br><br>  //Method overloads left out for brevity<br>  protected async execute&lt;T&gt;(<br>    fnOrQuery: ((client: pg.ClientBase) =&gt; Promise&lt;T&gt;) | string | pg.QueryConfig,<br>    ...args: unknown[]<br>  ): Promise&lt;T&gt; {<br>    const releaseClient = !this.tx<br>    const client = this.tx || (await this.pgPool.connect())<br>    try {<br>      if (typeof fnOrQuery === &#39;string&#39;) {<br>        return (await client.query(fnOrQuery, args)) as T<br>      } else if (typeof fnOrQuery === &#39;object&#39;) {<br>        return (await client.query(fnOrQuery)) as T<br>      } else {<br>        return await fnOrQuery(client)<br>      }<br>    } finally {<br>      if (releaseClient) {<br>        await (client as pg.PoolClient).release()<br>      }<br>    }<br>  }<br>}</pre><h3>JSON Repository</h3><p>Now let’s get to the meat. We’ll create a base class for all the JSON repositories in system. All aggregate repositories will extend it, adding just domain specific queries.</p><p>First we need few basic types to make our repository type-safe:</p><pre>export type Entity = {<br>  id: Id<br>}<br>export type SavedEntity&lt;T extends Entity&gt; = T &amp; {<br>  created: Date<br>  updated: Date<br>  //This hidden field we&#39;ll use to decide if we should insert or update row in DB<br>  //There are many ways you can solve it, but this one is simple and effective<br>  __managed: boolean<br>}<br>export type JsonData&lt;R extends Entity&gt; = Exclude&lt;R, Entity&gt;<br>export type JsonRow&lt;D&gt; = {<br>  id: Id<br>  created: Date<br>  updated: Date<br>  data: D<br>}</pre><p>And a helper function to generate base queries for given table, so we don’t need to create them each time we query something:</p><pre>const buildQueries = (table: string) =&gt;<br>({<br>  __table: table,<br>  fetchById: `SELECT * FROM ${table} WHERE id=$1`,<br>  deleteById: `DELETE FROM ${table} WHERE id=$1`,<br>  insert: `INSERT INTO ${table} (id, tenant_id, data) VALUES ($1, $2, $3) RETURNING *`,<br>  update: `UPDATE ${table} SET updated=NOW(), data=$2 WHERE id=$1 RETURNING *`,<br>  _upsert: `INSERT INTO ${table}<br>  (id, created, updated, tenant_id, data) VALUES ($1, $2, $3, $4, $5)<br>  ON CONFLICT (uuid) DO UPDATE SET updated=$3, data=$4<br>  RETURNING *<br>`,<br>}) as const<br>type Queries = ReturnType&lt;typeof buildQueries&gt;</pre><p>And here’s base for your repository:</p><pre>export abstract class JsonRepository&lt;<br>  T extends Entity,<br>  R = JsonData&lt;T&gt;,<br>&gt; extends Repository {<br>  private query: Queries<br><br>  constructor(tableNameOrQueries: string | Queries, pool: pg.Pool) {<br>    super(pool)<br>    if (typeof tableNameOrQueries === &#39;string&#39;) {<br>      this.query = buildQueries(tableNameOrQueries)<br>    } else {<br>      this.query = tableNameOrQueries<br>    }<br>  }<br><br>  protected deserialize(row: JsonRow&lt;R&gt;): SavedEntity&lt;T&gt; {<br>    return {<br>      ...row.data,<br>      __managed: true,<br>      id: row.id,<br>      created: row.created,<br>      updated: row.updated,<br>    } as unknown as SavedEntity&lt;T&gt;<br>  }<br><br>  protected serialize(data: T): R {<br>    return omit(data, &#39;__managed&#39;) as unknown as R<br>  }<br><br>  async fetch(id: Id): Promise&lt;T | undefined&gt; {<br>    const { rows } = await this.execute(client =&gt;<br>      client.query(this.query.fetchById, [id])<br>    )<br>    const entity = rows.map(this.deserialize)[0]<br>    if(entity) entity.__managed = true<br>    return entity<br>  }<br><br>  async save(entity: T | SavedEntity&lt;T&gt;): Promise&lt;SavedEntity&lt;T&gt;&gt; {<br>    if ((entity as SavedEntity&lt;T&gt;).__managed) {<br>      return this.update(entity as SavedEntity&lt;T&gt;)<br>    } else {<br>      return this.insert(entity)<br>    }<br>  }<br><br>  async insert(entity: T): Promise&lt;SavedEntity&lt;T&gt;&gt; {<br>    const serialized = this.serialize(entity)<br>    const res = await this.execute(client =&gt;<br>      client.query(this.query.insert, [entity.id, this.tenantId, serialized])<br>    )<br>    if (res.rowCount !== 1) {<br>      throw new Error(&#39;Insert failed&#39;)<br>    }<br>    return this.deserialize(res.rows[0])<br>  }<br><br>  async update(entity: SavedEntity&lt;T&gt;): Promise&lt;SavedEntity&lt;T&gt;&gt; {<br>    const serialized = this.serialize(entity)<br>    const res = await this.execute(client =&gt;<br>      client.query(this.query.update, [entity.id, serialized])<br>    )<br>    if (res.rowCount !== 1) {<br>      throw new Error(&#39;Insert failed&#39;)<br>    }<br>    return this.deserialize(res.rows[0])<br>  }<br><br>  async delete(entity: SavedEntity&lt;T&gt;): Promise&lt;T&gt;<br>  async delete(id: Id): Promise&lt;number&gt;<br>  async delete(idOrEntity: Id | SavedEntity&lt;T&gt;): Promise&lt;number | T&gt; {<br>    if (typeof idOrEntity === &#39;string&#39;) {<br>      return this.deleteById(idOrEntity)<br>    } else {<br>      const res = await this.execute(client =&gt;<br>        client.query(this.query.deleteById, [idOrEntity.id])<br>      )<br>      if ((res.rowCount = 0)) {<br>        throw new Error(<br>          `Entity with id ${idOrEntity.id} not found in ${this.query.__table}`<br>        )<br>      } else if (res.rowCount &gt; 1) {<br>        throw new Error(<br>          `More than one entity with id ${idOrEntity.id} found in ${this.query.__table}`<br>        )<br>      } else {<br>        const copy = { ...idOrEntity }<br>        copy.__managed = false<br>        return copy<br>      }<br>    }<br>  }<br>}</pre><p>…Phew 😮‍💨 that was a lot of code. But it will handle all CRUD operations on all aggregates in your system. There’s only one more thing to add and we can start using it. We need a way to create new transactions and bind our repositories to it. I know it somewhat diverges from what DDD preaches. You should make your aggregate a transaction boundary, but that makes a lot of things complicated. Using cross-repository transactions is much easier. And doesn’t create so much headaches in small-ish systems. So here it goes:</p><pre>export const withTx = async &lt;T&gt;(<br>  client: pg.Pool | pg.ClientBase,<br>  fn: (client: pg.ClientBase) =&gt; Promise&lt;T&gt;<br>): Promise&lt;T&gt; =&gt; {<br>  let release = false<br>  if (client instanceof pg.Pool) {<br>    client = await client.connect()<br>    release = true<br>  }<br><br>  await client.query(&#39;BEGIN&#39;)<br>  try {<br>    const result = await fn(client)<br>    await client.query(&#39;COMMIT&#39;)<br>    return result<br>  } catch (e) {<br>    await client.query(&#39;ROLLBACK&#39;)<br>    throw e<br>  } finally {<br>    if (release) {<br>      ;(client as pg.PoolClient).release()<br>    }<br>  }<br>}</pre><h3>Aggregate repositories</h3><p>So how do you use something like this? Here’s a simple repository:</p><pre>export class CarRepository extends JsonRepository&lt;Car&gt; {<br>  constructor(pool: pg.Pool) {<br>    super(&#39;cars&#39;, pool)<br>  }<br><br>  protected clone(): this {<br>    return new CarRepository(this.pgPool) as this<br>  }<br><br>  public async fetchBeforeYear(year: number): Promise&lt;Car[]&gt; {<br>    const { rows } = await this.execute(client =&gt;<br>      client.query(`SELECT * FROM cars WHERE (data-&gt;&#39;year&#39;)::int &lt; $1`, [year])<br>    )<br>    return rows.map(this.deserialize)<br>  }<br>}</pre><p>And here’s how you can use these repositories in your use cases. With shared transaction:</p><pre>const changeOwnerUseCase =<br>  (pool: pg.Pool, cars: CarRepository, users: UserRepository) =&gt;<br>  async (regNo: CarRegNo, user: Email): Promise&lt;Car&gt; =&gt; {<br>    return withTx(pool, async tx =&gt; {<br>      const owner = await users.withTx(tx).fetchByEmail(user)<br>      const car = await cars.withTx(tx).fetchByRegNo(regNo)<br>      const updatedCar = await updateOwnerInDomainLayer(car, owner)<br>      return cars.withTx(tx).save(updatedCar)<br>    })<br>  }</pre><p>You can also use them without transaction by simply calling users.fetchByEmail(user). This will lease connection from pg.Pool, execute query and return connection to pool. It makes your code super concise if you don&#39;t care about transactions.</p><h3>Final Notes</h3><p>I find this way of organizing my database access code really flexible. I can easily add more queries and more aggregates. Whenever I need to customize serialization or deserialization logic, I simply overwrite serialize and deserialize methods. If I need to do it only for specific query, I simply use different serialization function in that particular function. This covers 99% of my needs for ORM in Node.js while giving me a nice way to not use classes (which I&#39;m not a big fan of) in my model.</p><h3>So why are classes here</h3><p>I usually try to build my data model around simple data structures, multiple types protecting invariants and stateless functions that operate on these structures. But here, for repositories I have used classes. The reason for that is simply to improve performance when binding them to transactions while keeping nice ergonomics of just doing withTx(tx).query(...). Without classes or prototypes I would have to copy all repository functions whenever I do that, and as I have tested in &quot;<a href="http://markdown-to-medium.surge.sh/%7B%7B%3C%20ref%20%22/post/2017-08-22_will-chrome-optimize-your-object-factories%22%20%3E%7D%7D">Will Chrome optimize your object factories</a>&quot; all this copying doesn&#39;t seem to be optimized away by the V8 VM.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=177714e6567a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[(Not so) simple Postgres queue system skeleton]]></title>
            <link>https://medium.com/@marekpiechut/not-so-simple-postgres-queue-system-skeleton-ddfb8d11ad51?source=rss-8e0e09168c02------2</link>
            <guid isPermaLink="false">https://medium.com/p/ddfb8d11ad51</guid>
            <category><![CDATA[queue-management-system]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[postgresql]]></category>
            <dc:creator><![CDATA[Marek Piechut]]></dc:creator>
            <pubDate>Thu, 11 Jul 2024 16:57:34 GMT</pubDate>
            <atom:updated>2024-07-11T16:57:34.894Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5GUckkxMsDAVikUrVOI_Iw.jpeg" /></figure><p>Postgres is a very versatile database system. It can be used for so much more than just storing data. In this post I’ll show you how you can create a fully functional job queue using Postgres.</p><p><strong>Note:</strong> This post is loosely based on my <a href="https://github.com/marekpiechut/pgqueue">pgqueue</a> library and code samples might not work out of the bat, as they were adopted to fit the post. When something does not work check library code, or play with it for 5 minutes to find a fix. Also <strong>it’s not a full solution</strong>. I’ve left all the “uninteresting” parts to the reader 🤭.</p><h3>Why not just FOR UPDATE SKIP LOCKED</h3><p>Most of articles out there propose to just use a single table and FOR UPDATE SKIP LOCKED. While it&#39;s valid to build your queue around that, it has few issues I&#39;m trying to solve with my a bit more involved design. Especially:</p><ul><li>Workers should be able to <strong>arbitrarily set job timeouts</strong> (every job could have different timeout)</li><li>Workers should be able to <strong>extend job timeout</strong> in the middle of work</li><li>Workers should be able to <strong>steal timeouted jobs</strong></li><li>Failed worker transaction should <strong>not undo changes on work items</strong></li><li>It should handle <strong>very long</strong> running jobs — longer than tx timeout</li><li>Long running jobs should <strong>survive database restarts</strong></li><li><strong>Job scheduling should be a bit smarter</strong>, than just pick an item from the top</li></ul><p>We’ll actually use the famous FOR UPDATE SKIP LOCKED statement, but only in scheduling part, not in the actual work queue.</p><h3>Database structure</h3><p>There are few tables we need to create to get us started. I would propose you create them in their own schema, separated from your regular application data. Of course you can add more data to these tables if you need, or remove scheduling or retry stuff if you don’t need it. In <a href="https://github.com/marekpiechut/pgqueue">pgqueue</a> there’s a lot more stuff related to scheduling and worker types. But we don’t need that here.</p><pre>CREATE TABLE QUEUES.QUEUE (<br>  id UUID NOT NULL,<br>  created TIMESTAMP NOT NULL DEFAULT now(),<br>  version INTEGER NOT NULL DEFAULT 0,<br>  queue VARCHAR(255) NOT NULL,<br>  state VARCHAR(40) NOT NULL,<br>  tries INTEGER NOT NULL DEFAULT 0,<br>  delay BIGINT,<br>  run_after TIMESTAMP,<br>  payload BYTEA,<br>  payload_type VARCHAR(255),<br>  PRIMARY KEY(id)<br>);<br>CREATE INDEX IX_QUEUE_STATE_RUN_AFTER_CREATED ON QUEUES.QUEUE (state, run_after, created);<br><br>CREATE TABLE QUEUES.WORK_QUEUE (<br>  id UUID NOT NULL,<br>  version INTEGER NOT NULL DEFAULT 0,<br>  batch_order INTEGER NOT NULL DEFAULT 0,<br>  started TIMESTAMP,<br>  lock_key VARCHAR(255),<br>  lock_timeout TIMESTAMP,<br>  PRIMARY KEY(id),<br>  FOREIGN KEY(id) REFERENCES QUEUES.QUEUE(id) ON DELETE CASCADE<br>);<br>CREATE INDEX IX_WORK_QUEUE_CREATED_ORDER ON QUEUES.WORK_QUEUE (created, batch_order)</pre><ul><li><strong>QUEUE</strong> — here we’ll be pushing our work items from application.</li><li><strong>WORK_QUEUE</strong> — here our scheduler will push items that our workers should take care of.</li><li><strong>INDEXES</strong> — we create some basic indexes that will speed up processing of queue. You should probably add more, if you plan on showing queue items in UI, sort them and filter, etc.</li></ul><h3>Pushing to queue</h3><p>Pushing is pretty straightforward. You simply insert a new item into queue table and leave rest of the work to schedulers and workers. A simple INSERT INTO QUEUES.QUEUE should be enough.</p><h3>Scheduling work</h3><p>Scheduler process is periodically polling the queue and scheduling work in work_queue. We need it, so we can add some more complex rules on how we want to prioritize work, ex:</p><ul><li>Queue fairness (so high volume queues won’t starve low volume ones)</li><li>Complex runtime priorities that could not be decided when item was pushed to queue</li><li>Throttling rules based on amount of free workers</li></ul><p>and all other weird ideas you might have, that would be hard to implement in simple SELECT * FROM QUEUE ORDER BY CREATED LIMIT 1 FOR UPDATE SKIP LOCKED query.</p><p>So what will our scheduler do? It will run query like this every second:</p><pre>SELECT *<br>  FROM QUEUES.QUEUE<br>  WHERE (state=&#39;PENDING&#39; OR state=&#39;RETRY&#39;) AND (run_after IS NULL OR run_after &lt;= now())<br>  ORDER BY created, id ASC<br>  LIMIT 100 FOR UPDATE SKIP LOCKED;</pre><p>Read the rows from this query, play with them, shuffle, sort by whatever you need and then push to queue in single batch with batch_order field providing ordering for workers:</p><pre>INSERT INTO QUEUES.WORK_QUEUE (id, batch_order)<br>SELECT * FROM UNNEST(<br>  { &#39;uuid1&#39;, &#39;uuid2&#39;, &#39;uuid3&#39; }::uuid[],<br>  { 1, 2, 3 }::integer[]<br>) ON CONFLICT DO NOTHING;</pre><p>On conflict we skip, in case by some random circumstance some other scheduling process has already pushed one of the items we have.</p><p>Usually scheduler shouldn’t have a lot to do, so it should be enough to have only one instance. You could also limit amount of pushed items if you see that work_queue is getting too large.</p><h3>Why polling ❓</h3><p>There are ways to build scheduling and worker logic without polling (ex: by using LISTEN/NOTIFY and triggers), but in a busy system there&#39;s plenty of work. So scheduler and workers won&#39;t probably have a lot of wait time. If query has returned full batch (100 items), we re-run poll immediately. This way, if there&#39;s plenty of work, we don&#39;t wait to process it. If there&#39;s not enough work, we just wait 1 second. Usually not a big deal.</p><h3>Processing work</h3><p>Your worker processes can now pick items from the top, ordering by created and batch_order. You can pull top items while also locking them in single query like this:</p><pre>WITH next AS (<br>    SELECT *<br>    FROM QUEUES.WORK_QUEUE<br>    WHERE lock_key IS NULL OR lock_timeout &lt; now()<br>    ORDER BY created, batch_order ASC<br>    LIMIT 10 FOR UPDATE SKIP LOCKED<br>)<br>UPDATE {{schema}}.WORK_QUEUE as updated<br>SET lock_key = ${nodeId},<br>    started = now(),<br>    version = updated.version + 1,<br>    lock_timeout = (now() + (${timeoutSeconds} || &#39; seconds&#39;)::INTERVAL)<br>FROM next<br>WHERE updated.id = next.id<br>RETURNING updated.*;</pre><p>As you can see, I’m pulling items in batches by 10. You can switch to single item, but it should be a bit easier on the database, to not query it so often. What’s also interesting is that I’m using a node-unique lock_key column value. So I always know which node has locked the item and is currently processing it. Using this lock key you can also restart all items for given node in case it crashes:</p><pre>UPDATE QUEUES.WORK_QUEUE<br>SET lock_key = NULL,<br>    lock_timeout = NULL,<br>    started = NULL<br>WHERE lock_key = ${nodeId}</pre><p>In your processing code you then just:</p><ul><li>Pick item</li><li>Start a new transaction</li><li>Process item</li><li>Rollback or commit transaction</li><li>Delete item from work_queue</li><li>Update or delete item from queue</li></ul><h3>Final notes</h3><p>From here you should be able to extend your queue with things like retries, delays, priorities… You can add queue history to inspect old jobs, etc. Or any other idea you might have.</p><p>Just remember to always use sql parameters to secure yourself from sql injection attacks. Be safe.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ddfb8d11ad51" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[66% Ortholinear Programmable Keyboard — Building first prototype]]></title>
            <link>https://medium.com/real-life-programming/66-ortholinear-programmable-keyboard-building-first-prototype-c722c2bc121e?source=rss-8e0e09168c02------2</link>
            <guid isPermaLink="false">https://medium.com/p/c722c2bc121e</guid>
            <category><![CDATA[electronics]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[mechanical-keyboards]]></category>
            <category><![CDATA[keyboard]]></category>
            <category><![CDATA[embedded-systems]]></category>
            <dc:creator><![CDATA[Marek Piechut]]></dc:creator>
            <pubDate>Fri, 13 Aug 2021 07:04:55 GMT</pubDate>
            <atom:updated>2021-08-13T07:04:55.831Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2WrmVFVH_5cnVP8TQRNjeg.png" /></figure><h3>D1M1x Keyboard — Building first prototype</h3><p>I’ve just finished build of a first prototype of an ortho-linear mechanical keyboard — <a href="https://d1keyboards.com/">D1M1x</a>. It was not an easy route and I’ve found a few issues in the design in the meantime. In this post I’ll be sharing how I was building it, what went totally south and how it finally ended.</p><h3>Just get it done by the PCB factory</h3><p>My original idea was to get all assembly done by the PCB factory. I was ordering them from JLCPCB. I’ve heard a lot good about them, their prices are very good and ordering process is simply great — you upload all the files and can see what the final result will be, how much will it cost and if all parts are in correct spots. It really boosts up confidence that what you have in CAD is what you’ll get from the factory. Unfortunately, it wasn’t possible.</p><p>First issue was that they didn’t have per-key LEDs that I’ve originally designed — SK6812 MINI-e. They are not even listed in their catalog. Fortunately, after 15 minutes of searching I’ve found a good candidate for replacement — Tonyu DY-S1515065. To be honest, they even looked like a better idea — smaller, lower energy consumption and the same communication protocol (it took me quite some time browsing through Chinese datasheet to confirm all that though 🤪). The only problem was that their footprint (PCB connector layout) was totally different. I had to create a new footprint in KiCad, replace it in the PCB design and fix the routing. We have 67 LEDs, so it was not that fast.</p><p>After the change I have prepared all files for assembly and was ready to order. Unfortunately I was also not smart enough to carefully read the capabilities section of the docs. Changing the LEDs made our board 2 side assembly, and it was not supported by JLCPCB (they plan to add it some time this year, but its not there yet). So, I could get 1 side assembly with LEDs they don’t have, or 2 side assembly that they don’t do, but with LEDs they have in stock. Either way, I’m screwed.</p><p>I’ve decided to keep the new LEDs and just solder them on my own. It would be enough to just use a lower temperature soldering paste and a home made reflow owen. JLCPCB would give me fully working prototype without LEDs that I can test and just add them on one or two boards. I’ve modified the assembly files (removing assembly for all the LEDs) and was ready to go. And then…</p><h3>Chip shortage hit №1</h3><p>During 2 days when I was changing the design and learning how to prepare everything for the automated assembly, the already short stock of ATMega32u4-MU chip on JLCPCB has depleted (they are back after more than a month… with 2x the price and still with a very low stock). This was the first time that chip shortage has hit me. I’ve had to rethink my approach. If they would not solder the chip and LEDs it made no sense to pay for them to just do few resistors and an USB socket. MCU with VQFN-44 package was the hardest part to solder anyway. Rest could be done by just adding in a soldering iron and some back pain into the mix. Screw it, I’m ordering just PCBs and will do everything on my own.</p><h3>Chip shortage hit №2</h3><p>Getting all the other parts was quite easy. I’ve reverted LEDs routing, so all the soldering was on just on one side. Changed all resistors and capacitors to 0603 package, so it would be a bit easier to solder. And ordered parts from large electronics supplier. Everything except of the main chip. It simply was nowhere to be found. The only place I could find it was Ali Express. And even there prices went up. So it was the worst of all — expensive parts and a long wait for the package. But there was no other option, so I’ve placed an order for 5 chips and waited.</p><p>After 2 or 3 days I’ve received a message from seller — they don’t have the chip in stock and asked me if I could cancel my order. Damn, I had to find another seller. The only one I’ve found had it for almost 2x the price. It was $6 per piece. For a part that was only $2 just few months ago. You could get a fully assembled ProMicro controller with exactly the same chip on it cheaper than that. Nevertheless I’ve placed an order and waited. (What’s interesting is that the original seller has finally posted the package in few more days. The same time asking me again to cancel the order. I’m still waiting for it. I wonder what will be inside the box…).</p><h3>First assembly</h3><p>PCBs and all other parts have arrived in 2 weeks. I’ve already either had or purchased all the tools. It took more than a month for chips to arrive. Few seconds later I’ve unpacked them, and… it’s not the right model! What I got was an ATMega32u4RC-MU not ATMega32u4-MU. I’ve consulted the datasheet. What’s the difference between the two? It appears, that the RC version has a calibrated, built-in oscillator, so you could use it without external one. Except from that it seemed to be the same chip 😮‍💨. The next day I was ready to build the keyboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G8DUyxCbxYsjbnGb0Ai-fw.jpeg" /><figcaption>Board ready for assembly</figcaption></figure><p>Assembly was not that bad. It was not very easy either, but in the end my hot air station was enough to solder the MCU and all the parts in place. When it was ready, I’ve found some USB-C cable, connected it to my laptop, and…</p><h3>It was a total brick</h3><p>The MU series chips come without bootloader baked in, so you cannot program them via USB. This was something I was aware of, so I’ve designed a programmer pins on the board and built a simple programmer from old Arduino UNO lying around. Also hacked a simple no solder connector I could use without soldering anything on the PCB. Just a 6 pin pogo connector and few cables. My own cheap version of Tag-Connect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2guhR2ZOFWKk3O54aKWH3w.jpeg" /><figcaption>Cheap in system programmer connector</figcaption></figure><p>Flashed UNO with a Arduino ISP firmware, connected the cables and tried to flash one of available bootloaders using <strong>avrdude</strong>. Nothing. Detected chip ID was 0x0000000. It looked like there was nothing connected to the programer. Maybe something was wrong with my UNO? I’ve also had someProMicros for prototyping lying around. Tested with one of them. Nothing. Tried to flash bootloader on one of ProMicros — worked fine. So something has to be wrong with my board.</p><p>I’ve got back to the workbench and started to check everything using a microscope. Soldering looked good. Checked with multimeter if all programmer connections were correct — yup they were right. Right pin was connected to right MCU pin. Connected the board to USB and tested if voltage is on all required pins. Everything seemed just fine.</p><p>I’ve spent 2 days trying to find the cause. No luck, everything seemed fine, yet the board was dead. It had power, but chip was not working. It had to be one of three - my soldering skills are shit and I should’t be even trying, MCU from Ali Express is junk or oscillator is dead. And I had no equipment to test any of these. Ok, I’ll sacrifice the ProMicro and swap chips. We’ll see what’s broken.</p><p>After 15 minutes I’ve had my keyboard with MCU from the working ProMicro. Quick test, and I can flash bootloader. 15 more minutes — ProMicro with chip from Ali Express. Dead. No sign of life. I was able to swap chips more than 4 times and one from ProMicro always worked and one from Ali was always dead. So maybe my soldering is not all that bad and its not the problem. It seemed it should be good enough. Also oscillator has to be good if the other chip worked.</p><p>✅ First takeaway: <strong>Source your components from well known suppliers. Wasting 2 days searching for source of problems will be more expensive than paying a bit more for a chip. (Of course if you can get it from any of well known suppliers).</strong></p><p>So now I had a working bootloader, but…</p><h3><strong>USB connection was not working</strong></h3><p>Without USB there was not easy way to flash firmware or use the actual keyboard. I knew that MCU was fine and checked that current from USB was flowing. I had to somehow check if routing for USB was correct or was it problem with socket itself. USB-C is a bit more complicated than good old USB 2.0. It can not only work as master and slave on both sides, has much faster transfer rates, but also be a 2 way charger with configurable voltage up to 20V and the socket is reversible. For keyboards you’re using USB 2.0 compatibility mode, so most of this stuff is not used, but you need to configure the port anyway.</p><p>I wanted to keep all this out of the equation, so I’ve just baked in a pair of cables for D+ and D- into the circuit board. Right on ESD protection diode.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7RAYXr0wBEAhu3NXa4cMvA.jpeg" /></figure><p>After destroying an old USB cable and connecting straight into the board it worked! I could flash QMK firmware and actually use the keyboard. It took me 2 more tries to actually get the socket right.</p><p>✅ Second takeaway: <strong>USB type C sockets are actually bigger pain to solder correctly than VQFN-44 chip package.</strong></p><h3>Design was more or less fine</h3><p>This was my first keyboard design. To be honest this was my first real electronic device except of some Raspberry PI tinkering. In the end, keyboard is working fine. There are some minor issues with routing and rotary encoder, but you can type and program it just fine. I will be ordering fixed PCBs soon. This time I have managed to get chips from established local supplier.</p><p>So here it is. A 66% Ortho-linear keyboard prototype. Not the most common type of a beast.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fHzzKt8i0OISMAEY-brpJA.jpeg" /><figcaption>Next to Keychron K3, a <a href="https://d1keyboards.com/">D1M1x</a> with high profile switches in a minimalistic 3d printed case.</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c722c2bc121e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/real-life-programming/66-ortholinear-programmable-keyboard-building-first-prototype-c722c2bc121e">66% Ortholinear Programmable Keyboard — Building first prototype</a> was originally published in <a href="https://medium.com/real-life-programming">Real Life Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Selecting Your First Microcontroller — From Software To Hardware]]></title>
            <link>https://medium.com/real-life-programming/selecting-your-first-microcontroller-from-software-to-hardware-4f7be5faa7e6?source=rss-8e0e09168c02------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f7be5faa7e6</guid>
            <category><![CDATA[electronics]]></category>
            <category><![CDATA[hardware]]></category>
            <category><![CDATA[mechanical-keyboards]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Marek Piechut]]></dc:creator>
            <pubDate>Thu, 05 Aug 2021 13:33:02 GMT</pubDate>
            <atom:updated>2021-08-05T13:33:02.973Z</atom:updated>
            <content:encoded><![CDATA[<h3>Selecting Your First Microcontroller</h3><h4>From Software to Hardware</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sbfYegXLGd-fGACJ" /><figcaption>Photo by <a href="https://unsplash.com/@viazavier?utm_source=medium&amp;utm_medium=referral">Laura Ockel</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Passive components are boring</h4><p>After few days of messing with passive components (resistors, capacitors, etc.) and Arduinos or other Raspberry PIs, you get bored. At least I was. I could build some simple stuff with motors and LEDs. Even connect a OLED display or a temperature sensor. But I was always curious what’s inside these micro-controller boards. Can I make my own Arduino? Probably. So I started to look around how one might actually do it, and I found that there are <a href="https://www.arduino.cc/en/uploads/Main/arduino-uno-schematic.pdf">schematics available</a> and you can actually check how it’s all built. And the most interesting part in all this is the MCU or a microcontroller. That’s where all the magic happens.</p><h4>MCUs vs CPUs</h4><p>I’ve got my share of formal CS education, so I already knew how CPUs work. It’s part of your university classes, so you should at least know what it’s capable of, how registers work, how instructions are parsed and executed. But CPUs are raw power horses. They can compute, and they can do it very fast. But that’s not enough to build a embedded device. You would need to build a lot of infrastructure around them to make it work.</p><p>That’s why we need motherboards with a lot of other chips and microcontrollers on it, RAM boards, GPUs, USB controllers, etc. They have almost no output or input built in. They take data and instructions from external memory, calculate stuff and put results back into memory. No persistent storage, no analog interfaces, very limited digital interface. It would require a lot of knowledge and design work to build a working device around just a CPU.</p><p>That’s where <a href="https://en.wikipedia.org/wiki/Microcontroller">MCUs</a> come in. They are a lot less powerful, but consume a lot less power and provide all features one may need when building a stand alone device — processing unit, RAM, persistent flash, IO ports, digital/analog converters, serial interfaces, built in clock source, etc. All built into the chip. It’s like you only need to build a power supply section, add few capacitors and you’re good to go. Flash firmware and they will boot and do work for you.</p><p>So that’s what I started to look into after getting bored with passive components and pre-built microcontroller boards.</p><h4>How to select a microcontroller ?</h4><p>At first it was a bit overwhelming. I didn’t know where even to look for a microcontroller. Then I have found so many to choose from, that it was almost impossible to pick anything. So what I did is I just started checking out development boards I had. What do they use? Here’s what I had in my closet:</p><p><strong>Raspberry Pi 3B:<br></strong>- Broadcom BCM2837B0</p><p>Looks like it’s a pretty fast chip, but it has no built-in RAM or storage. Definitely not something I have skill to build my device around. It’s also not something one can buy off the shelf. There’s no availability on <a href="https://www.mouser.com/">Mouser</a>, <a href="https://www.mouser.com/">DigiKey</a> or my local electronics provider <a href="https://www.tme.eu/en/">TME</a>. It’s even hard to get info about what’s inside the chip.</p><p><strong>BeagleBone Black:<br></strong>- Texas Instruments AM3358</p><p>This also is a rather fast chip using quite a lot of power (nothing like what your PC CPU uses, but still quite a lot in embedded scale). You would not power it from a CR3032 battery. For this one it’s easy to get some documentation: <a href="https://www.ti.com/product/AM3358">https://www.ti.com/product/AM3358</a> and you can actually buy single chip at popular electronics stores. It also seems to have a built in ROM and 64KB of RAM, a lot of interfaces, etc. But it also has a built in memory controller for DDR and it looks like it’s how it’s meant to be used. It’s also very hard to find any easy to follow tutorials and guides. Not something you would start your learning from.</p><p><strong>Arduino Uno:<br></strong>- Microchip ATmega328P</p><p>This is a tiny chip. Low power, low memory low storage, low price 🤪. It has plenty of tutorials to learn from, schematics with examples and guides how to unbrick it when you mess something up. Even if you totally fry it, it’s cheap and available. Has built in analog and digital interfaces and is easy to program with Arduino IDE. Great chip to start with. The only downside is that it has no USB controller built in. So you would need to include another chip to make a device that can be connected to your PC.</p><p><strong>Arduino Leonardo/Pro Micro</strong></p><p>I’ve had a ProMicro clone lying around. It had an ATMega32u4, which is almost the same as ATmega328P, but has a built in USB support. So it’s probably an even better option for a novice to start with.</p><h4>There goes ATMega32u4</h4><p>In the meantime I’ve also got into building custom mechanical keyboards. That’s where I did get my ProMicro from. It’s a very open community with a lot of open source hardware. In many keyboard designs there’s ATMega32u4 used one way or another. Some of them are using just plain chip and some are just wiring around a ProMicro board. I thought that why not use that? ATMega32u4 has plenty of documentation and examples. Most of them open source. Built in USB support and could be soldered by hand. So there it was. I decided that why not build my own programmable keyboard with ATMega32u4 chip on it. It would be a great place to start tinkering with microcontrollers.</p><h4>But there are many other options</h4><p>Microchip MCUs are crazy popular among tinkerers. It makes sense, they are used in Arduino boards — the most popular development boards in the world. There are so many resources on the internet about how to build hardware around them and they are powered with 5v, the same as what USB port provides. What a coincidence that they are so crazy popular 😉. But after some time of messing with hardware I started noticing, that they might not be the best choice. To be honest, they are crazy expensive when compared to competition (expensive in embedded chip scale — they cost around 4$). Especially when you start comparing features. They are pretty slow and have much less memory and interfaces for the price. That’s when I started noticing other options:</p><p><strong>STM chips:<br></strong>STM has a crazy large selection of MCUs. You can probably find any combination of RAM, Flash and interfaces you can think of somewhere in their catalog. For around 4$ you will get 10 times more memory, 32bit CPU and a lot more processing power. All this probably consuming less power and providing twice the GPIO pins for your project. They also provide great prototyping boards (<a href="https://www.st.com/en/evaluation-tools/stm32-nucleo-boards.html">https://www.st.com/en/evaluation-tools/stm32-nucleo-boards.htm)</a> and easy to start with CubeMX IDE based on Eclipse. The only downside (upside?) is that you need to code in C or C++. No Arduino programming language. There are more and more tutorials and guides available. They also have nice Bluetooth enabled chips with examples and videos showing how to use them. All this makes them really nice for a novice like me to learn while using them. Been playing with STM32WB55 and tooling and hardware is really nice.</p><p><strong>Nordic chips:<br></strong>Nordic has many nice chips when you need to work with Bluetooth chips. You can get a cheap nRF52840 boards and pre-assembled modules with built in antenna. Plenty of GPIOs and quite a lot of computing power for just few bucks. Here you also need to get your hands dirty with C, and tooling is not that easy to use as CubeMX from STM. But their price for Bluetooth is really hard to beat. There is also one more caveat. They usually are not in packages you can solder at home. So no way you’re using these if don’t have a hot air soldering station.</p><p>These 3 were the brands I’ve been tinkering with so far, and they are all really nice to prototype with. Its a bit more work to find sample schematics for them then with ATMega and they are powered with 3.3V power, so you need to add voltage regulator to the board. But they also provide a lot more power and are easier to buy in these chip shortage times.</p><p>There are also many more options. Quick search shows that there are many more brands to pick from: TI, NXP, Maxim, Infineon…</p><h4>So which one is the best to use?</h4><p>To be honest… I have no idea. After designing my first keyboard with ATMega32u4 on it and playing with STM and Nordic chips for my second design (a wireless one) I think, that Microchip MCUs are really great to start with. There’s so much easily available examples and schematics to learn from. You can quickly start building. But they are probably not the fastest ones nor have the most features.</p><p>After some time it appeared to me, that all these chips are more or less the same stuff. The only thing that matters is if they have features you need, can be bought somewhere (I’m writing this in the middle of chip shortage crisis), can be soldered with tools you have and have support in software you plan on using. For mechanical keyboard that would be:</p><ul><li>Enough GPIO pins to connect all rows and columns</li><li>USB support so you don’t need to add USB chip</li><li>I2c and Serial support for OLED displays and split boards</li><li>Have support in QMK, ZMK etc…</li></ul><p>Then you look at what packages are available:</p><ul><li>DIP and SOP — easy to solder by hand.</li><li>QFP — can be done by hand with just a pen soldering iron, but requires a bit more skill.</li><li>QFN — only if you have a hot air station or a reflow oven.</li><li>BGA — if you have a full featured PCB re-work workshop or an assembly line somewhere in your home. Just don’t even try.</li></ul><p>You don’t really need to care about memory size or processing power. It’s just a keyboard, any chip is faster than you typing 😀.</p><p>For your project it might be Bluetooth, DAC converter or AI engine. Just pick a chip that you want and has a development board available. Start building your next project and learn while you build. It’s really fascinating what you can do with these tiny, cheap devices.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f7be5faa7e6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/real-life-programming/selecting-your-first-microcontroller-from-software-to-hardware-4f7be5faa7e6">Selecting Your First Microcontroller — From Software To Hardware</a> was originally published in <a href="https://medium.com/real-life-programming">Real Life Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Software to Hardware — Part 1]]></title>
            <link>https://medium.com/real-life-programming/from-software-to-hardware-part-1-e5796bd1c413?source=rss-8e0e09168c02------2</link>
            <guid isPermaLink="false">https://medium.com/p/e5796bd1c413</guid>
            <category><![CDATA[development]]></category>
            <category><![CDATA[hardware]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[electronics]]></category>
            <category><![CDATA[mechanical-keyboards]]></category>
            <dc:creator><![CDATA[Marek Piechut]]></dc:creator>
            <pubDate>Mon, 19 Jul 2021 12:35:59 GMT</pubDate>
            <atom:updated>2021-07-19T12:35:59.627Z</atom:updated>
            <content:encoded><![CDATA[<h3>From Software to Hardware — Part 1 (how it all started)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2TgNF3WQTNXuYMRc" /><figcaption>Photo by <a href="https://unsplash.com/@vishnumaiea?utm_source=medium&amp;utm_medium=referral">Vishnu Mohanan</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>How it all started</h4><p>I’ve always been interested in electronics and computer hardware. I’ve built almost all of my own PCs, done some soldering, replaced broken capacitors, etc. I’ve also tinkered a bit with Arduino and Raspberry Pi, but never really got anything more than recreating some basic examples from the web. Until now.</p><p>Few months ago, probably out of boredom or something, I’ve started to teach myself to touch-type. I never really had issues with typing speed. Software developers don’t need to type that fast. Most of the time you spend on staring at the screen and thinking about “What has just happened here, why this stuff doesn’t work”. But outside of that it started to really bother me, that I was making a lot of mistakes when I was typing fast. So I have started to go through online services teaching touch-typing. It was pretty bad at the beginning, but after few weeks it started to pay off. I’ve got back to my regular typing speed, but now without ever looking at my keyboard and using all my fingers.</p><p>Then I started to feel, that regular keyboard is not really made for real people. I started to feel discomfort in my left hand after longer typing session. It was probably due to fact, that my left hand fingers are not leaning to the left and my elbow is naturally not somewhere near my bellybutton. But that’s what is expected from you, when you touch type using regular — staggered keyboard. Just try to put your fingers on the keyboard according to any hand placement tutorial and press keys with proper fingers. Your right hand fingers are expected to lean towards center of your body, while left hand is leaning to the left side. I know that people are not really symmetrical, but not like that.</p><p>So, when you have learned a bit of touch-typing, and have experienced a bit of typing fatigue, what do you do? Now you have an excuse to buy yourself a…</p><h4>Mechanical Keyboard</h4><p>So I’ve started to look around for a mechanical keyboard. Unfortunately most of them were just regular, staggered keyboards made for gamers. It was not something I’ve even considered. There must be something better, I thought. This way I got into ortho-linear keyboards. My first shot was to get a <a href="https://olkb.com/collections/planck">Planck</a> keyboard kit. I’ve had the idea, that you want not only to have keys in a plain grid, to reduce finger stress, but also that you want to have less keys, so that you have less finger movement while typing.</p><p>But during the search through the web I’ve also found a cheaper clone of Planck — a BM40 keyboard. It’s more or less the same design. The only difference is that you cannot have 2 separate keys in place of space bar, no buzzer for sound and added RGB lighting. All stuff that I don’t care about.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AeMHApADF1-Ox8bp6KjnAQ.jpeg" /></figure><p>While waiting for shipping I’ve also got the idea to try out something more extreme — a minimalistic split keyboard. Something considered an ergonomic holy grail. So I’ve also ordered a kit to build a <a href="https://github.com/foostan/crkbd">Corne</a> keyboard. This was a bit more involved. I’ve received a pack with a lot of tiny pieces and had to solder it all by myself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2KvL-I-AioGwy4_rGXIKgg.jpeg" /></figure><p>It took me few hours, 2 painful de-solderings of 12 pin microcontroller boards (and it’s really a pain to do it with a regular soldering iron), but it was a full success. Now I’ve had a geekiest keyboard of all people I know.</p><h4>It can’t be that hard</h4><p>All this tinkering with electronics, firmware flashing and microcontrollers, got me back into my old hardware interests. I’ve started to get back into Arduino and basics of voltage, current and power. You know… connect a LED to power source, voltage dividers, control a small motor with a microcontroller, etc. But this time I wanted to get a bit more, not just tinker with it. I wanted to understand how all this stuff really works. Didn’t want to just read a tutorial and build something small. I wanted to design it myself.</p><p>So that I’ve started to read about <a href="https://en.wikipedia.org/wiki/Ohm&#39;s_law">Ohm’s</a> and <a href="https://en.wikipedia.org/wiki/Kirchhoff&#39;s_circuit_laws">Kirchhoff’s</a> laws, AC, DC and power. About passive and active electronic components and which where used when and how they could be connected. Went through <a href="http://simonmonk.org/electronics-cookbook/">Electronics Cookbook</a> by Simon Monk and analyzed few simple circuits filling in the gaps in knowledge whenever something was not clear.</p><p>After some time it all started to click and I could finally understand why some stuff did work and why sometimes it didn’t. In the meantime I’ve switched all my typing to new keyboards and started to see some issues within their design. Don’t get me wrong, they are really great for typing. I really like to type using Corne and my 40% ortho. I think they are ideal for text input. But not all programming tools are made for typists. You tend to rely a lot on keyboard shortcuts in tools like IntelliJ Idea or Sublime Text. Most of the people don’t use combination like Alt+Shift+F6 or Ctrl+Cmd+↑. But that’s common for programmers to use stuff like that. And with just 47 keys it’s getting painful. You need to add even more keys to these 3–4 key combos to change layers and access F-keys or numbers.</p><p>And so, I was back on a lookout for a new keyboard. I knew already that I didn’t want a staggered keyboard. It was so much easier on my fingers to type on a regular grid or on a grid with vertical stagger. I was never going back, but needed more keys and better layout.</p><p>After some searching I’ve finally have connected the dots. I was already few months into the electronics. Already had some idea about how all this stuff works, and wanted something different from my keyboard. So, how hard can it be? I thought, and set myself a goal to build my own keyboard. But not just to reproduce what others have made. I wanted to understand it all and build it from ground up. And it appears that keyboard community has a lot to offer to people willing to learn: open source schematics, open source firmware, a lot of documentation and examples.</p><p>And that’s how I got myself, after 13 years of software development, into microcontrollers, PCB design, ESD issues, schematics, datasheets, prototypes and others. Always wanted to do it all — hardware and software. To design it from ground-up. Build a prototype, create firmware and drivers and release it into production. I know that keyboards are simple products, even programmable ones. But now, while typing this, I’m holding a PCB of my first prototype and quoting aluminium machining companies for a case. It was a lot of learning and doing and it feels really good, to build a physical product from nothing. Not only software for it. Let’s see where it ends.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xG39N5cd1YyaB5ywL8ljZg.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5796bd1c413" width="1" height="1" alt=""><hr><p><a href="https://medium.com/real-life-programming/from-software-to-hardware-part-1-e5796bd1c413">From Software to Hardware — Part 1</a> was originally published in <a href="https://medium.com/real-life-programming">Real Life Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Investigate poor performance React components with “React Profiler”]]></title>
            <link>https://medium.com/real-life-programming/investigate-poor-performance-react-components-with-react-profiler-86be7d7364b3?source=rss-8e0e09168c02------2</link>
            <guid isPermaLink="false">https://medium.com/p/86be7d7364b3</guid>
            <category><![CDATA[react]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[reactjs]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[performance]]></category>
            <dc:creator><![CDATA[Marek Piechut]]></dc:creator>
            <pubDate>Wed, 30 Jun 2021 06:32:56 GMT</pubDate>
            <atom:updated>2021-07-09T13:54:04.935Z</atom:updated>
            <content:encoded><![CDATA[<h3>Optimize slow React components with “React Profiler”</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gGmvkmFP39WJj_DjGRjb5w.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@a2eorigins?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Alex Perez</a> on <a href="https://unsplash.com/s/photos/magnify?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p><em>This post is a work in progress chapter from “React Performance” book i’ve started working on and haven’t touched for few months. Looks like it will never be released, so I’ll just post parts that I think might be useful as blog posts.</em></p><p>You’re probably here, because you already know that something is wrong with a part of your application. Maybe after some change, maybe when hit with a big dataset, your component/screen started to get slow. You probably can see it when you invoke some action, that UI is no longer responsive and you’re pretty sure it’s React rendering that takes too long.</p><p>You’re already thinking about adding some useCallback, useMemo, and React.memo calls, but where? Sometimes it&#39;s obvious. Component structure is simple and you have the gut feeling what might be causing the problem. But sometimes you just stare at the code and have no idea where, in this huge component tree, we&#39;re dropping these frames. Which component to wrap with memo or which callback is responsible for component re-rendering regardless of nothing changing in it&#39;s data.</p><h3>Quick Win</h3><p>In this section we’ll try to answer exactly that — <strong>how to find that one component (or group of components) that is causing the slowdown</strong>.</p><p>Let’s start with quickest way we can get to the answer. React already has some nice tools that will help us out. We’ll dive into React Profiler and without much introduction just try to hunt the performance hog.</p><h3>Install React Profiler</h3><p>—</p><p>🤔 Feel free to skip this section if you already have it installed.</p><p>—</p><p>React profiler is a part of totally awesome React Developer Tools package available for Chrome/Chromium and Firefox. You can download it here:</p><ul><li><strong>Chrome: </strong><a href="https://chrome.google.com/webstore/detail/react-developer-tools/fmkadmapgofadopljbjfkapdkoienihi"><strong>https://chrome.google.com/webstore/detail/react-developer-tools/fmkadmapgofadopljbjfkapdkoienihi</strong></a></li><li><strong>Firefox: </strong><a href="https://addons.mozilla.org/en-US/firefox/addon/react-devtools/"><strong>https://addons.mozilla.org/en-US/firefox/addon/react-devtools/</strong></a></li></ul><p>Both are being developed by core React team and built from the same codebase, so they should provide the same functionality. So it shouldn’t matter which one you use. Just select browser you’re most comfortable to work with. Unfortunately there’s no support for other tools as of now. You’ll have to use standalone edition if you need to debug/profile these (we’ll dive into that later on).</p><p>After installation you should be able to access React profiler inside your browser developer tools (we’re looking for the Components and Profiler tabs):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tRUjNDZyP4F0SK9ROrSJtA.png" /></figure><p>When you have it all and running open your application in the browser, navigate to Profiler tab and start profiling (Press blue circle button in the top left corner). Click around your app for few seconds and stop it by pressing red circle button . If you didn&#39;t get any errors we should be all set and ready to dive into it.</p><p>—</p><p>⚠️ <em>I’m getting </em><strong><em>“Profiling not supported.”</em></strong><em> message, what next?</em> — make sure you’re running your React app in development mode. Profiling is disabled by default on production builds. Don’t worry for now. Will handle it later. Just switch to development mode. Most of the performance issues will be visible in both — production and development builds anyways.</p><p>—</p><h3>Investigate slow page</h3><p>Now that we have our profiler running we can go on and try to investigate some slowly loading page in your app. We’ll be investigating a lazy loaded list of old notifications. We expect it to be doing more work than required when each load is finished, but we encourage you to open up application you’re currently developing and check a page that you think might need some performance improvements and it’s not related to network time.</p><p>So let’s dit into it. As always we start with a clean state to make sure we’re touching only code we think we do and that nothing is going to skew our measurements, so:</p><p>✔️ Checkout to a commit you want to profile or stage your current changes</p><p>✔️ Make sure theres not much going on on your machine. CPU or memory hungry applications might skew our results.</p><p>And here’s the screen we’ll be working on:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kSBalgfoidYBv2EDNTI_sA.png" /></figure><p>It’s currently showing the last 20 notifications, but we have many more on the backend and whenever you’re near the bottom we’ll be fetching new ones and appending to the list. Just classic lazy loading infinite (Of course it’s not infinite, we’ll run out of notifications in few screens, but it has enough to make it interesting) scroll.</p><p>What we need to do now, after initial rendering of the page is done, is to gather a profiler baseline, so we know if we’re fixing performance or making it worse. Let’s open React Profiler, start recording and scroll down until we hit the lazy load boundary few times, at least twice. Then immediately stop the profiling session. Output should be similar to what you see on the screenshot below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*85njJ73iHio9nwR_EJvH4w.png" /></figure><p>That’s a lot of output, but we’re not really interested in all of that. We start by investigating the commits chart at the top:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/1*IA1aS4eXD4lAjdPF6HlHSQ.png" /></figure><p>What we can see here, is that we had 20 <em>“commits”</em> — 20 times React has applied changes to the browser DOM. That’s not bad actually when we think about all the loading state changes and that we’ve executed lazy loads few times. What is more interesting though, is that this chart is constantly moving up, so it looks like we’re doing more and more work on each commit. When you hover over the bars, you’ll also see, that we get from around 90ms to almost 200ms time for single render. This is probably now what we were expecting. We are fetching 20 items in each batch and appending them to the list. Of course we’re doing some more work when going through ever-growing list of items, but we should only append new 20 nodes to the DOM tree. It looks suspicious, let’s dig into it a little bit more.</p><p>To do that, we start at the last commit and check what was rendered there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ON1cAc5cV8dud23R6e0j3w.png" /></figure><p>We can ignore all the grey bars — these components did not change during this render. First interesting one is the Notifications component. It did render, and that&#39;s expected. We have changed its state, so this looks fine. We also see, that commit of Notifications component itself did not take a lot of time. It&#39;s only 3.5ms. But take a look on all these small components at the bottom. All of them are colored, so they all have changed and their DOM changes were applied. Compare them to the previous screen. We see that in each commit there&#39;s more and more of these, and all of them are always rendered. If you hover over them, you&#39;ll see, that they are the individual notifications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*q7ZK5JN05zDn4jOIwPEIcA.png" /></figure><p>Also, if you did not refresh the page or change page structure, React Profiler will highlight actual nodes that were rendered during the commit. Scrolling to the top of the page and hovering over one of the first nodes in the flame chart will reveal the truth. We’re re-rendering every notification on each update.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kQoKAaKhq-Er9tSk8TXOZQ.png" /></figure><p>This makes no sense. These components will never change. We only append new notifications at the end. There’s no way data can change and they should not trigger any DOM updates. It looks like we’ve found good place to try out React.memo. Let&#39;s wrap our NotificationItem into it and see if it helps.</p><h3>Memoize Component</h3><p>Digging into the source code it looks like we’re using just a regular functional component for a single notification item. Usually that’s totally fine and in most cases should not matter. But here we’re rendering quite a lot of them and it makes sense to give it few minutes of thought and try to memoize rendered component. Here’s how it look like right now. Actual body of the component does not matter, it’s only important that it is a functional component and that it’s not memoized yet.</p><pre>const NotificationItem = ({ notification, onMarkAsRead, session }: Props) =&gt; &lt;div&gt;...&lt;/div&gt;</pre><p>Ok, let’s wrap it up with React.memo and see if it helps:</p><pre>const NotificationItem = React.memo&lt;Props&gt;(<br>  ({ notification, onMmarkAsRead, session }) =&gt; &lt;div&gt;...&lt;/div&gt;<br>)</pre><p>Now rebuild project, and start a new profiling session. Make sure that you have similar environment (nothing heavy is running on your computer and you gave few seconds for VM to warm up). And check if our fix has helped to mitigate the problem:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OHxDnyKaabqA3Cd4h80jMw.png" /></figure><p>Yup, looks like the problem is fixed. In the commits bar we can see a <em>“rake”</em> pattern. Lower bar being a render of loading indicator, when no notification change, and higher bar being actual rendering of newly fetched data. Also each render takes similar time now — around 60–70ms and looking at the flame chart at the bottom we see, that previously fetched notifications are not re-rendered (they are greyed out).</p><h3>Wrap Up</h3><p>Looks like we have fixed our problem and here we have our quick win. It was enough to simply wrap a component with React.memo. But what&#39;s more important here is that we&#39;ve made an informed change. There was no guesswork about where the problem is. We had hard evidence, that there was unnecessary work being done and then we have fixed it.</p><p>—</p><p>⚠️ There’s some more consideration needed when using React.memo. Sometimes adding it adds more overhead instead of fixing the problem. It might be the case, that it won’t help in your project.</p><p>—</p><h3>Deep dive</h3><p>React Profiler is a nice tool to quickly investigate possible performance issues in rendering process. In this chapter we’ll be doing a deep dive into how it works and what actually it measures. We’ll also take a look at React APIs that are used internally by the profiler and investigate how we might use it ourselves to get more answers than React Profiler provides.</p><h3>Profiler UI</h3><p>Let’s take a detailed look into information available in React Profiler and how it’s presented. To have a better image of what is going on here, we need to remember about 2 stages of React rendering process:</p><ul><li>First stage called <strong>Render</strong> or <strong>Reconciliation</strong> is one that calls all render functions on components, generates a Virtual DOM and compares it to the previous vDOM. This way it knows which nodes have changed and need to be applied to the browser DOM.</li><li>Second stage — <strong>Commit</strong> — responsible for doing actual changes on browser DOM. This has to be done in single call to the browser APIs and will be done in <em>“stop the world”</em> manner. If we would allow it to be done in stages or in background, user would see intermediate stages while we are updating the UI.</li></ul><p>With that in mind we can now take a look at the commits view in the profiler. It’s this small bar chart at the top:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/1*IA1aS4eXD4lAjdPF6HlHSQ.png" /></figure><p>What we see here is a list of all commits that React has flushed to DOM during our profiling session. Each bar is showing a separate commit in sequence of execution. Although it shows commits, it does count the render phase. You can test it out using this simple code:</p><pre>import { useState } from &#39;react&#39;</pre><pre>const SlowComponent = ({ noSlowdown }) =&gt; {<br>  const arr = []<br>  if (!noSlowdown) {<br>    for (var i = 1000000 - 1; i &gt;= 0; i--) {<br>      arr.push(i)<br>    }<br>  }</pre><pre>return &lt;div&gt;I&#39;m slooooooow&lt;/div&gt;<br>}</pre><pre>const FastComponent = () =&gt; {<br>  return &lt;div&gt;I&#39;m fassssst....&lt;/div&gt;<br>}</pre><pre>function App() {<br>  const [dummy, setDummy] = useState()<br>  return (<br>    &lt;div className=&quot;App&quot;&gt;<br>      &lt;SlowComponent noSlowdown={dummy % 2} /&gt;<br>      &lt;FastComponent /&gt;<br>      &lt;button onClick={() =&gt; setDummy(Date.now())}&gt;Render!&lt;/button&gt;<br>    &lt;/div&gt;<br>  )<br>}</pre><pre>export default App</pre><p>Loading it and pressing Render! button few times shows that slow component is impacting profiling results regardless of it returning the same trivial single div. Commit cost of both components is exactly the same, yet profiling results are completely different: fast component renders nearly instantly while slow component takes 32ms out of 33ms total rendering time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b3kjRM_RTm-UsI5jLeHWqg.png" /></figure><p>Traveling back and fourth through the commits you can find some interesting patterns in render times that you didn’t expect. Selecting one of them will show you exact render duration and few more details about it.</p><p>As you have probably already noticed React Profiler marks commits and components with different colors and bar length. Generally the longer the bar and warmer the color (more yellow), the more time it took to render compared to other commits and components. Grey lines did not render at all during given run.</p><p>—</p><p>🤔 <strong>Note:</strong></p><ul><li>Line length shows how long did it take for component to render with all it’s children</li><li>Line color shows how long did component itself render compared to other components (how slow/fast it is)</li></ul><p>—</p><p>There are two more interesting views here: <strong>Flamegraph</strong> and <strong>Ranked</strong> tab. First shows all components in your view in a tree according to your components structure. Second orders components by their rendering time ( <strong>self render time</strong> — how long it took to render component excluding its children time). It’s very useful to catch slowest components — a good place to start searching for problems. Hovering on the bars will highlight it on the page and selecting one of them will show you history of commits of given components and <strong>reason why render was triggered</strong> (ex: state change, parent render, props change, etc.). This information is sometimes enough to get you on track with figuring out what’s wrong. Very often some props change, that we didn’t expect to be any different.</p><p>—</p><p>⚠️ It’s usually good idea to have <em>“Record why each component rendered while profiling.”</em> option enabled. It might add some overhead to profiling process, but it’s usually not a problem if your app is not super-huge. Just press the options cog ⚙️ and enable it now.</p><p>—</p><p>There’s also an option to hide commits that took below given time. It’s not super useful, as we usually are looking for anomalies, and want to see whole picture. It’s better idea to narrow use case to short problematic interaction, but you might also start with longer recording sessions and try to find issues using this filter.</p><h3>React Profiler API</h3><p>DevTools profiler allows to gain a lot of knowledge about performance issues in your app. But if what you need is to profile only selected component and do it in production, there’s also a React Profiler API (<a href="https://reactjs.org/docs/profiler.html">https://reactjs.org/docs/profiler.html</a>). It’s a lightweight component to gather data about it render times, thus giving an insight of performance of its children.</p><p>It will not allow us to get a deep tree of render stats like we have in Profiler view in DevTools, but allows to programmatically get timings for selected components in the app. Usage is simple, just wrap your component and provide onRender callback:</p><pre>&lt;Profiler<br> id=&quot;user-profile&quot;<br> onRender={(<br>  id,<br>  phase,<br>  actualTime,<br>  baseTime,<br>  startTime,<br>  commitTime,<br>  interactions<br> ) =&gt; aggregatePerformance(id, phase, actualDuration, interactions)}<br>&gt;<br> &lt;UserProfile user={user} /&gt;<br>&lt;/Profiler&gt;</pre><p>We’ve got quite a few parameters to the callback, but except of that it looks pretty straightforward. Let’s go through all the data we have in the callback:</p><ul><li><strong>id</strong> — this is exactly the same as what we have passed into the Profiler tag. React does nothing interesting with this parameter. It&#39;s not grouping or overriding profiler tags with same Id. You can have as many as you want with exactly same value. Its role is purely informational, so you can have single callback, reuse it between profilers and still know which measures you have received.</li><li><strong>phase</strong> — component lifecycle phase. This can only be: mount or update and means exactly what you expect. Can be used find out if problem is only appearing during mount - maybe some slow useEffect that is firing only on mount.</li><li><strong>actualTime</strong> — measured duration of rendering. This is a real time of render during given commit. If you’re using memoization this time should go down significantly. If it does not, then you might found the problem.</li><li><strong>baseTime</strong> — estimated time of full render of component subtree. This should roughly be the same time as the <em>actualTime</em> when component did mount if it did not do any heavy initialization. If you’re using memoization correctly, this time should be usually larger.</li><li><strong>startTime</strong> — simply a timestamp when rendering started</li><li><strong>commitTime</strong> — timestamp when rendering was finished. This time is shared between all profilers that took part in the commit. Thanks to that you can group data from separate sub-trees that were rendered due to same change in state.</li><li><strong>interactions</strong> — list of all interactions associated with this render.</li></ul><p>Profiler API is very simple and allows to track performance of our components even on production. Just keep in mind, that it has a cost associated. There’s a small performance penalty for each Profiler instance in your components tree. Also make sure you&#39;re not doing a lot of processing in your callback. It&#39;s best to do any calculations asynchronously, or just dump your data to server and do all heavy-lifting there.</p><h3>Profiling in production builds</h3><p>Due to small performance impact profiler API is disabled in production builds of React (as of May 2021). While it’s usually not a problem, as most of real performance issues will be visible in both production and development builds. Sometimes you want to dig into production bundle and gain some insights on rendering time.</p><p>The way it’s currently being disabled is through build time configuration. So if you have built your app with regular ReactDOM, you’ll have to re-build it. Here’s what to put in Webpack config to switch to profiling versions of libraries:</p><pre>module.exports = {<br>  //...<br>  resolve: {<br>    alias: {<br>      &#39;react-dom$&#39;: &#39;react-dom/profiling&#39;,<br>      &#39;scheduler/tracing&#39;: &#39;scheduler/tracing-profiling&#39;,<br>    }<br>  }<br>};</pre><p>In case you’re using create-react-app script, just add --profile flag during build process:</p><pre># Using Yarn<br>yarn build --profile<br># Using NPM<br>npm run build -- --profile</pre><p>With all this in place profiling should work just fine in production build. Regardless of that we’re not really advising to add it to your build process. It should be enough to run this build locally and connecting it to production backend. It should be trivial if you’re already serving your app from CDN or static folder using your Web server.</p><h3>Performance impact</h3><p>As we have already noticed, profiler builds tend to run slower. Even if you’re not actively profiling. We’ve done a quick benchmark to actually measure it:</p><pre>const ROUNDS = 100_000<br>const Benchmark = () =&gt; {<br>  const [took, setTook] = useState()<br>  const [round, setRound] = useState(1)<br>  const start = useRef(performance.now())</pre><pre>useEffect(() =&gt; {<br>    if (round &lt; ROUNDS) {<br>      setRound(round + 1)<br>    } else {<br>      setTook(performance.now() - start.current)<br>    }<br>  }, [round])</pre><pre>return (<br>    &lt;div className=&quot;App&quot;&gt;<br>      &lt;div&gt;{round}&lt;/div&gt;<br>      &lt;div&gt;Took: {took}ms&lt;/div&gt;<br>    &lt;/div&gt;<br>  )<br>}</pre><p>And results are:</p><p>Development: <strong>13575ms</strong><br>Production: <strong>1478ms</strong><br>Production Profiling: <strong>2179ms</strong></p><p>So it looks like performance hit is not that bad, but it’s still there. We wouldn’t recommend to just push profiling bundles to production. Just build one when needed and profile on your development machine or build it each time and deploy alongside your production build on separate url. This way you can always switch to profiling build and do your analysis while not sacrificing your users performance.</p><h3>Interactions</h3><p>—</p><p>🧨 Watch out, this is an experimental API and might change in future versions of React</p><p>—</p><p>Along with the profiler API React team has introduced an interaction tracking that can be attached to profiling data. Sometimes it’s just hard to find out which exact user action or backend call ended up with very slow render commit. What this API allows, is to attach a bit of semantic context to recorded performance traces.</p><p>Sometimes we can track the component that is slow, and tack slow sessions, but it’s not always easy to find out which exact user interaction is causing this. Especially when we are trying to gather performance data remotely, from user machine, as it’s not showing up on our environment. Thanks to the interactions API we’re able to see exactly which user actions are slow and which are fast, even if they are triggering the same component or we don’t exactly know where to look for the problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BbWu_rO2dXBvCrM59Ld6gg.png" /></figure><p>What we can see on the screen above, is a trace of user interactions on the same component. Based on grid color we can see, that <em>“Enter user name”</em> action is pretty fast. What we should be looking at is the <em>“Randomize button”</em> action. Its marker color is yellow and render duration is much bigger. Also when looking from the commit perspective it can be found, that this action is problematic:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*51LFSP_ZF5Q4L3wqeQOCpg.png" /></figure><p>And here’s how we track interaction in this example:</p><pre>import {<br>  unstable_trace as trace,<br>} from &#39;scheduler/tracing&#39;</pre><pre>&lt;input<br>  type=&quot;text&quot;<br>  value={name}<br>  onChange={e =&gt; {<br>    trace(&#39;Enter user name&#39;, performance.now(), () =&gt;<br>       onChange(e.target.value)<br>    )<br>  }}<br>/&gt;</pre><p>What we do here, is on each character entered in the input we call trace first, passing in high resolution timestamp and a callback to handle actual logic after interaction is recorded. Don&#39;t worry about the performance object. It&#39;s a global available in every major browser now. We use it here, because Date.now() is not good enough to measure code performance. You can read more about it in your browser documentation, ex: <a href="https://developer.mozilla.org/en-US/docs/Web/API/Performance_API">performance-mdn</a>.</p><p>There’s one more thing we’re missing here. It’s how to add interactions to <strong>asynchronous code</strong>. A lot of performance issues are related to data we’re fetching from backend. If we use this simple trace API, our async code calls will not be counted as part of the interaction. That&#39;s simply because of the way async is handled in JS. What we need to do then is to wrap callbacks of async code with container that will bind it to current interaction. Thankfully React already has tools to support it:</p><pre>import {<br>  unstable_trace as trace,<br>  unstable_wrap as wrap,<br>} from &#39;scheduler/tracing&#39;</pre><pre>&lt;button<br>  onClick={() =&gt;<br>    trace(&#39;Fetch data&#39;, performance.now(), () =&gt; {<br>      setLoading(true)<br>      fetchData()<br>        .then(<br>          wrap(data =&gt; {<br>            setData(data)<br>          })<br>        )<br>        .finally(wrap(() =&gt; setLoading(false)))<br>    })<br>  }<br>&gt;<br>  Fetch data<br>&lt;/button&gt;</pre><p>What’s most important here, is the wrap call that is surrounding then and finally handlers in our async call. After that we should see, that all renders related to this call are assigned to the same interaction:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zAmXqDLCJNmDhmHlFL_SKg.png" /></figure><p>We can see here, that <em>“Fetch data”</em> has exactly 3 renders associated. This totally makes sense. First one is sue to loading being set to true, second when we finish fetching the data and third when loading is set back to false. Of course all this code does not need to be inlined in the component. Interaction tracing API can as well be used inside custom hooks:</p><pre>import {<br>  unstable_trace as trace,<br>  unstable_wrap as wrap,<br>} from &#39;scheduler/tracing&#39;</pre><pre>const useData = () =&gt; {<br>  const [loading, setLoading] = useState(false)<br>  const [data, setData] = useState(null)</pre><pre>const fetch = () =&gt; {<br>    setLoading(true)<br>    fetchData()<br>      .then(<br>        wrap(data =&gt; {<br>          setData(data)<br>        })<br>      )<br>      .finally(wrap(() =&gt; setLoading(false)))<br>  }<br>  return { fetch, loading, data }<br>}</pre><pre>const Component = () =&gt; {<br>const { fetch, loading, data } = useData()</pre><pre>return &lt;button<br>  onClick={() =&gt; trace(&#39;Fetch data&#39;, performance.now(), fetch)}<br>&gt;<br>  Fetch data<br>&lt;/button&gt;<br>}</pre><h3>Details and Techniques</h3><p>In this chapter we take a look at other cases (except our <em>“Quick Win”</em>) that make React Profiler worth while. We also talk about cases where it’s not very useful and what to do if we cannot get any meaningful input from it.</p><h3>Using profiler in other browsers</h3><p>While Chrome is the leading browser (at least as of 2021) we sometimes also need to support clients using other agents — noticeable ones being Safari and mobile safari. For this we’ll be using standalone version of React Dev Tools and connect it remotely:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4iwUKG9V8SP0bsKrdLEPfQ.png" /></figure><p>To get standalone tools fetch it with yarn or npm and simply start:</p><pre>npm install -g react-devtools<br>react-devtool</pre><p>What you should get is a welcome screen with information about how to connect. For browsers it should be as easy as adding a script to your page &lt;head&gt; section:</p><pre>&lt;script src=&quot;<a href="http://localhost:8097/">http://localhost:8097</a>&quot;&gt;&lt;/script&gt;</pre><p>and reloading the page. This will open a web-socket connection from your browser to the app used for all communication between debugger/profiler and dev-tools.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JCcc7zm7betW9JTfnWm20Q.png" /></figure><p>Unfortunately it has to run before <em>ReactDOM</em> is loaded, so it cannot be added in dev-tools when page is already shown in your browser. Just make sure you remove it from the code before committing or maybe you already have a separate html template for development and production. Then just put it only in dev one.</p><p>After a few seconds your app should connect and you should be able to use the same tools as in Chrome extension, but within unsupported browsers.</p><p>This solution will also work just fine for iOS Simulator. For Android emulator/device you’ll need another step — connect phone using cable and proxy all requests to localhost from your device to host:</p><pre>adb reverse tcp:8097 tcp:8097</pre><p>Unfortunately there’s no easy way to use it with real iOS device. Due to security considerations React Dev Tools listen only on localhost and there’s no equivalent for adb reverse for iOS.</p><h3>Unexpected renders of memoized components</h3><p>React.memo is usually treated as a holy grail and universal tool to fix all your performance problems. But sometimes it simply doesn&#39;t seem to work. Usually it&#39;s due to misunderstanding of how it works or what is really passed into component props. Let&#39;s try to investigate such case using React Profiler and what to look for when we suspect memoization is not working. Here&#39;s our suspected component:</p><pre>const SlowComponent = React.memo(({ text, onClick }) =&gt; {<br>  const arr = []<br>  for (var i = 1000000 - 1; i &gt;= 0; i--) {<br>    arr.push(i)<br>  }<br>  return (<br>    &lt;div&gt;<br>      &lt;div&gt;I&#39;m slow {text}&lt;/div&gt;<br>      &lt;button onClick={onClick}/&gt;<br>    &lt;/div&gt;<br>  )<br>})</pre><p>It looks like it’s being correctly optimized. But let’s use it in a way that simply passes an inline function as a onChange parameter. This will basically make out React.memo useless, but what we want to do is to check how will this error manifest itself in React Profiler.</p><pre>const App = () =&gt; {<br>  const loadData = id =&gt; ({ id, value: &#39;some&#39; })<br>  const [_, setDummy] = useState()</pre><pre>return (<br>    &lt;div className=&quot;App&quot;&gt;<br>      &lt;button onClick={() =&gt; setDummy(Date.now())}&gt;Render&lt;/button&gt;<br>      &lt;SlowComponent onClick={() =&gt; loadData(&#39;dummy&#39;)} /&gt;<br>    &lt;/div&gt;<br>  )<br>}</pre><p>To investigate problem like this, we need to have at least 2 renders of parent component without any changes in data passed to SlowComponent. In this example it&#39;s actually easy. We&#39;ll just press a button few times and it should re-render the App component. The only prop we pass in is this problematic inline function. Thanks to React.memo it should never re-render. The only moment this component should show up colored in the profiler is its initial mount. Let&#39;s see how it actually behaves in the profiler:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Ezv_GCOaoRFvDaWPgVsww.png" /></figure><p>We can see here, that it has been rendered on each button press, and it has taken quite a lot of time. Thanks to the popup you also see, that it’s due to onClick property changing. This is not what we wanted here, let&#39;s fix this inline function and check how will profiler output look like afterwards. Wrapping it in a simple useCallback should be enough here:</p><pre>const App = () =&gt; {<br>  const loadData = id =&gt; ({ id, value: &#39;some&#39; })<br>  const [_, setDummy] = useState()</pre><pre>return (<br>    &lt;div className=&quot;App&quot;&gt;<br>      &lt;button onClick={() =&gt; setDummy(Date.now())} /&gt;<br>      &lt;SlowComponent<br>        onClick={React.useCallback(() =&gt; loadData(&#39;dummy&#39;), [])}<br>      /&gt;<br>    &lt;/div&gt;<br>  )<br>}</pre><p>And here’s how profiling results look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GR8HS5nMNjkZpLc49dUcSw.png" /></figure><p>SlowComponent did not render anywhere except of initial mount. That&#39;s the result we have expected and it is clearly shown in the profiler data. This example might be trivial, but when investigating performance of theoretically optimized components you&#39;ll be able to easily see them re-rendering in the profiler. Hovering on the component will also show why it has rendered. With this information it should be fairly easy to fix.</p><h4>Using update highlights to find unneeded renders</h4><p>There’s one more quick way to pinpoint components like that. It’s not really a profiler feature, but React Developer Tools have also an option to mark all rendered components with color borders. It’s especially useful to track forms re-renders. Often we’ll see some undesired renders when we were hoping that only single input was updated. Let’s enable the option now and check how it looks like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*Xp4E2ZJv3FOyXK4zfPcO_w.png" /></figure><p>In the screen below we can see a 3 input form. We’re editing first text input with “Highlight updates when components render” option enabled:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pgu5wLKCNx_11WgQnNdkTg.png" /></figure><p>As you can see, on each character we put into the “What’s your name?” box, we have a render on all inputs and parent component. Parent re-rendering is probably fine, as it’s the place where we keep the state for the form. But all other inputs are not really related to the change, so we can optimize it.</p><p>In this particular case it might be an overkill to do it just for 2 inputs. But, especially when working with redux you might see some updates that are totally unexpected. Enabling “Highlight updates when components render” can be the only thing that you need to find performance issues then. It might be worth to enable it from time to time and play around with your app. You might be surprised how many unneeded renders are there.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=86be7d7364b3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/real-life-programming/investigate-poor-performance-react-components-with-react-profiler-86be7d7364b3">Investigate poor performance React components with “React Profiler”</a> was originally published in <a href="https://medium.com/real-life-programming">Real Life Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Corne MX 3.0 Keyboard build guide]]></title>
            <link>https://medium.com/real-life-programming/corne-mx-3-0-keyboard-build-guide-9b5c7eff4178?source=rss-8e0e09168c02------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b5c7eff4178</guid>
            <category><![CDATA[guides-and-tutorials]]></category>
            <category><![CDATA[electronics]]></category>
            <category><![CDATA[mechanical-keyboards]]></category>
            <category><![CDATA[crkbd]]></category>
            <category><![CDATA[qmk]]></category>
            <dc:creator><![CDATA[Marek Piechut]]></dc:creator>
            <pubDate>Wed, 23 Jun 2021 10:18:38 GMT</pubDate>
            <atom:updated>2021-06-23T10:18:38.801Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*htyWEU_h0BkhV6sqAC9gHA.jpeg" /></figure><p><strong>⌨️ Typed with: Corne MX with Gateron Brown &amp; BM40 with ultra-light Gateron Clear</strong></p><p>This is a build guide for <a href="https://github.com/foostan/crkbd">Corne Keyboard (Crkbd)</a> Cherry MX keyboard version 3.0.1 by <a href="https://github.com/foostan">foostan</a>. Original Japanese guide is available <a href="https://github.com/foostan/crkbd/blob/master/corne-cherry/doc/v3/buildguide_jp.md">here</a>. Unfortunately there’s no official guide in English for this particular version, but I’ve been recently building one of them, so here it is. Please keep in mind that this version is not a translation and I don’t have even slightest Japanese knowledge. So all what’s written here is based on my own experience and might differ from the official guide.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DRfcj54VjefTRcr1e1kPHg.jpeg" /><figcaption>Keyboard done</figcaption></figure><h3>Before you start</h3><p>Building a keyboard is not very difficult if you have previous experience with soldering or at least some tinkering skills and messed a bit with electronics. But if you have never soldered or played with electronics it might be best to pick up a YouTube tutorial on soldering first and play with some throwaway electronics you have. It’s possible in the beginning to overheat components and even destroy PCBs.</p><p>Before we start there’s one remark on my side. Please note that <strong>you’re the only responsible person for all damage you do while following this guide. It’s not my fault if you break anything or burn yourself.</strong> If you are not sure you can handle stuff described here on your own, don’t do it. Just order an assembled keyboard or find some qualified electronic engineer to do it for you.</p><h3>Required equipment</h3><ul><li><strong>Soldering iron</strong></li></ul><p>This has to be a pencil type soldering iron for small electronics with a precision tip. Don’t even try with a soldering gun. If you don’t have one I’d suggest that you get a basic soldering station with temperature setting. It will be much easier to work on LEDs if you can lower the temperature. It should not be much more expensive. If you’re going to get one, buy one with more power. It will heat up faster and thanks to temperature setting it will not fry things up.</p><ul><li><strong>Solder wire</strong></li></ul><p>To make things easy get a thin one (0.5mm is best). Don’t be very cheap on it. Poor quality solder wire will make you swear a lot, while working with high quality one is a pleasure.</p><ul><li><strong>Soldering flux</strong></li></ul><p>Just get a “<em>no clean SMD soldering flux</em>” liquid. The smallest amount available. It can be a pen or a bottle with tiny brush.</p><ul><li><strong>Desoldering pump</strong></li></ul><p>Desoldering pump It’s not really required if you don’t do any mistakes. But if you do… you know… get it or you’ll have hard times.</p><ul><li><strong>Precision tweezers — reverse</strong></li></ul><p>You can also try using straight ones if you already have them. You don’t need anything fancy. We won’t be doing any surgery here. Don’t start without them, using only your fingers with SMD components is a bit too much.</p><ul><li><strong>Black insulation tape</strong></li></ul><p>You know… To insulate things.</p><ul><li><strong>Isopropyl alcohol (optional)</strong></li></ul><p>To cleanup the board of the excessive flux. Not really needed if you have used a no clean one. It will leave small stains, but nothing major. If you’re using a case it won’t really be visible.</p><ul><li><strong>Black marker (optional)</strong></li></ul><p>Just a waterproof black marker (or rather of a color matching your PCB)</p><ul><li><strong>Thick sand paper (optional)</strong></li></ul><h3>Required parts</h3><p>There are two ways to get required parts — get a full kit or assemble all parts yourself. It might be easier to just order a kit and have everything done for you. Getting everything on your own is not that difficult, but can take a bit more time. You can grab <em>gerber</em> files from <a href="https://github.com/foostan/crkbd/releases/tag/corne-cherry-v3.0.1">here</a> and order PCBs at one of Chinese suppliers (ex: <a href="https://jlcpcb.com/">https://jlcpcb.com</a>). Rest of the parts should be available on Aliexpress or your local electronics shop. This is mostly common stuff. Maybe Pro Micros or OLEDs won’t be there. Here’s the stuff you need:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8f4297d49f905d66139fc73d2f385cac/href">https://medium.com/media/8f4297d49f905d66139fc73d2f385cac/href</a></iframe><h3>Switches &amp; Keycaps</h3><p>You need to pick correct type of switches for your build. They need to be MX compatible (so low profile ones won’t fit). If you are using a case with top cover you can use 3 or 5 pin switches. If you’re going to go without a case please select 5 pin ones. They will fit firmly into the board while 3 pin ones might disconnect and wobble around. There are also some switches that have a hole for LEDs. These will work best if you plan on using per-key lights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CqmjRQvRfBhPmYvUAmMERA.jpeg" /></figure><p>You will be ok with anything that’s compatible: Gateron, Cherry, Kailh, Blue, Brown, Clear, Red, whatever. Just pick what you like, or get a switch tester and try them out before purchase.</p><p>For the keycaps you should be fine with anything that fits your taste and is <strong>MX compatible</strong>. Just pick something that looks cool.</p><h3>Final notes</h3><p>Take a deep breath, do it well rested (I know, I know, you’ll be doing it through the night regardless of what I say) and let’s get started. Just please think twice and solder once. Some mistakes are easy to fix and some will take time. I’ll be adding remarks on how to fix mistakes and how hard it is to do. But it’s always faster and easier to not do them the first time. It’s also a good idea to read whole guide at least once before soldering anything, and then start following it step by step. This way you’ll know what should be the expected outcome before doing actual work.</p><h3>Prepare PCB</h3><p>Unpack your PCBs and check for defects. It’s pretty rare that they’ll be not ok, but inspect them to find some obvious stuff (you know, like broken in half or something). You will have 2 different pieces, one for left and one for right hand part. You can start with any. Just remember that <strong>side with white markings is the bottom</strong>.</p><p>If you want to have nice, black edges on your PCB, use sand paper and remove leftovers of PCB breakaway connectors (small bumps on the edge), clean it up with alcohol and paint the edge with permanent marker. This way it will look better, especially in dark case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uqToJDqUkax5N0FbftMaWA.jpeg" /></figure><p>Now cover your desk to not burn any holes in it and let’s get to the real work.</p><h3>Soldering diodes</h3><p><strong>Temperature: 300–350℃<br>Difficulty: medium<br>How hard to fix: medium</strong></p><p>Setup your soldering iron to 300 ℃, wait for it to warm up and test on the soldering wire if it melts. Give it few more minutes if it doesn’t. Low power soldering irons need at least 5 minutes to heat up. If it’s still not melting increase temperature a bit and try again. Maybe you got a poor quality soldering wire, or temperature setting is not calibrated. As a rule, try to use lower temperatures whenever possible. This gives you more time before you overheat SMD components.</p><p>Diodes are these small black thingies that allow current flow in only one direction. There’s only one important thing here: ⚠️ <strong>diodes are polarized</strong>. You <strong>must</strong> solder them that line on diode is on the side with line on the PCB. Arrow points into direction of the white line on the diode. Please check twice before soldering the first one and re-check from time to time. Re-soldering one is not a problem, but redoing all 42 will not be fun.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/1*_Pi1eZYmYnmFiuS5FHOOkw.jpeg" /></figure><p>— <br>🤔 <strong>Advice:</strong> The best way to solder these for me is to first apply flux on contacts on the board. Then cover one contact for given diode with just a bit of solder. After that I would heat up the solder on the PCB until it melts and slide diode into it using tweezers.<br>—</p><p>It’s faster to do it in batches. So, after getting comfortable with it, apply flux on 6 contacts, cover 6 with solder and solder 6 diodes on one side. Rotate the board and solder second contact in all 6. Just repeat until you’re done.</p><p>— <br>💣 <strong>How hard to fix:</strong> diodes are not hard to fix or replace. They are also very cheap. If you get one wrong it should be 30 seconds to get it right. Just don’t get them all wrong.<br> —</p><h3>Soldering per-key LEDs</h3><p><strong>Temperature: 250–280 ℃<br>Difficulty: medium<br>How hard to fix: medium</strong></p><p>Per key LEDs are easy to solder. They have a nice hole on the PCB, so they are easy to position and you don’t need to hold them in place while soldering. The only real issue is that they are <strong>heat sensitive</strong>. Just make sure you’re soldering with lowest possible temperature and don’t heat them longer than for 5 seconds. It should be more than enough. If you need to give it more time due to a mistake or something, just make a 30 seconds break and let it cool down. Also try to solder them near the leg end, not near the led itself. It should be enough to touch it for a second with a bit of solder on tip of your soldering pen. Just use flux and it should work.</p><p>They are also <strong>positional</strong> components, so make sure you get them right. There’s one shorter leg that is connected to spot with a small cutout. <strong>Shorter leg goes to the corner marked on the board</strong> and emitting part (transparent) goes into the hole — pointing towards the top of the PCB. They should fit and not move around when in place.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*7zt-BHkkXQrd0bTqdLZyZw.jpeg" /></figure><p>— <br>🤔 <strong>Advice:</strong> My advice here is to just drop few LEDs in correct spots with tweezers, cover connectors with a bit of flux, press it gently with tip of tweezers and solder one side (2 legs) one by one. It should be a quick action, get some solder on tip of your soldering pen and touch end of the leg. Don’t worry, if you won’t get it right the first time, you’ll see it after connecting the keyboard and it will be easy to fix. Finish whole board on one side, rotate PCB and do the same on the other pair of legs.<br> —</p><p>Again, it’s faster to do it in batches. Drop all LEDs into one side, cover 2 rows with flux and solder. Cover next 2 rows, solder…</p><p>Just as a reminder, these components are <strong>heat sensitive</strong>. Use high quality soldering wire and low temperature. Don’t heat it up for long, take a break and let it cool down.</p><p>— <br>💣 <strong>How hard to fix:</strong> LEDs are a bit harder to fix. You’ll need to use sucking pump, heat up solder and remove it from pins before pulling it off.<br> —</p><h3>Soldering underglow LEDs</h3><p><strong>Temperature: 250–280 ℃<br>Difficulty: medium+<br>How hard to fix: medium</strong></p><p>The only reason this step is marked harder is that these LEDs don’t have long legs. This makes them a bit harder to solder correctly and easier to overheat. I hope you’ve got few spares, as there’s a chance you’ll break one or two. Except of that it’s similar to the per-key LEDs.</p><p>These are also <strong>positional</strong> components and are also <strong>heat sensitive</strong>. On one corner of the LED there’s a marker or a cut corner. This spot has to go into the white corner on the board.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*dyQd48yS4ZIOwrYtzsgOBQ.jpeg" /></figure><p>— <br>🤔 <strong>Advice:</strong> Cover connectors with flux first, then position the LED. Press it firmly with tip of the tweezers so it doesn’t move. Then solder only one leg and move to next LED so it has time to cool down.<br> —</p><p>These I would do one by one, each pin at a time. So solder the first leg, move to next LED, solder the first leg, etc. Finish whole board with only one leg in each then move to second leg. This way you give them some time to cool down. Also please make sure you’re not touching them with your soldering iron for <strong>more than 5 seconds</strong>. Just take a short break if you need to retry. It will cool down in 30 seconds and you can try again.</p><p>— <br>💣 <strong>How hard to fix:</strong> For me these are a bit easier to replace than per-key ones. Just heat up each leg and suck solder with pump. It should be easier to have a firm catch with tweezers on these.<br> —</p><h3>Soldering hotswap sockets</h3><p><strong>Temperature: 300–350 ℃<br>Difficulty: easy<br>How hard to fix: medium</strong></p><p>After all the LEDs and diodes, with their overheating problems, low temperature and tiny size, this will be an easy task. Sockets are big, hard to overheat and easy to position. The only thing to watch out for is to have them in place while soldering. Just press them firmly into the board and keep it pressed while soldering. You can even do it with your finger. This way you also make sure you won’t overheat them. If your finger burns, it means you’re too slow 🤪.</p><p>— <br>🤔 <strong>Advice:</strong> Press sockets with your finger or tweezers when soldering, to make sure they are in place.<br> —</p><p>Start by dropping all sockets in place for whole board. Apply some flux on few of them and put your soldering iron tip with some solder straight into the hole over the PCB connector. Theres a small cut in the metal part of the socket, so solder will get through it and join it to the board. If you can’t get enough solder this way, just add it while soldering. Theres a lot of space now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7ZDdCug9dsAuBtClduDr0A.jpeg" /></figure><p>Finish the whole board with one side of the socket, then rotate the PCB and repeat on the other side. You should be done in no time.</p><p>— <br>💣 <strong>How hard to fix:</strong> Not very hard, just use solder pump and pull it off while heating. But there should be no need to do it anyway. They’re hard to break.<br> —</p><h3>Soldering Pro Micro pins</h3><p><strong>Temperature: 300–350 ℃<br>Difficulty: easy<br>How hard to fix: VERY HARD!</strong></p><p>This part is a bit tricky. It’s pretty easy to solder in the pins. They tend to sit firmly in the board and are impossible to overheat. But please make sure you solder it <strong>on the correct side of the board</strong>. Pins are cheap, but if you don’t have a hot air soldering station, they are <strong>real pain to remove</strong>. So make sure you have flipped the PCB to the other side. Pro Micro goes to the top of the board. All other parts we have been soldering till now were on the bottom. This is the side you want to be looking at:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fYEMK8hldBrVOaozEA3aog.jpeg" /></figure><p>Now put the pins in and flip the board. It might need a bit of force to get them in place, as we are inserting square pins into round holes 🤪.</p><p>We’ll be soldering on the bottom, but controller itself goes to the top. Please take a 5 minute break and <strong>double check</strong> that you have it right. This is really important.</p><p>.</p><p>.</p><p>.</p><p>⏱ After 5 minute break…</p><p>Now check again if you have everything right, apply some flux on both pin lanes, grab your soldering iron in one hand and soldering wire in second and solder each pin individually.</p><p>— <br>🤔 <strong>Advice:</strong> It’s easiest to heat up the pin and apply wire so that it touches both: pin and iron tip. This way it will melt quickly and go to right spot. This should be like a walk in the park after all the stuff we’ve already done.<br> —</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JZsdxu1OR-ywQ-yEGik_ww.jpeg" /></figure><p>This is how it should look like from the bottom after soldering (sorry I don’t have a photo of the pins before soldering controller). You might also like to socket the Pro Micros instead of soldering them in. The Micro-USB connector on them is really fragile. Its soldered only on the surface and easy to break. I wanted mine to be as low as possible and couldn’t get sockets that were low enough. Also I am using a magnetic connector anyway, but maybe socketing is an option for you. <a href="https://docs.splitkb.com/hc/en-us/articles/360011263059-How-do-I-socket-a-microcontroller-">Here</a> is a guide how you might want to do it.</p><p>— <br>💣 <strong>How hard to fix:</strong> ⚠️ Watch out here. It’s really hard to desolder these if you don’t have a hot air station. You will spend quite some time fighting it, and then will have problems to fit new socket into the board, as it’s hard to clean the holes from all the solder. So <strong>double check</strong> and then check it once more. To make sure you have everything placed correctly before soldering.<br> —</p><h3>Soldering TRRS socket and reset switch</h3><p><strong>Temperature: 300–350 ℃<br>Difficulty: easy<br>How hard to fix: medium</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bjpGcspdEy2wsnUPzup66w.jpeg" /></figure><p>You can start with one you like. They are both quite easy. The only thing that can go wrong is that you solder them <strong>on the wrong side of PCB</strong>. They both go on the top side — the same side as the microcontroller sockets. You also can’t solder them the wrong way. Tactile switch is not positional and TRRS connector simply won’t fit in any incorrect way. Just put them into the holes on the board. You can stick them in place with some tape, but I like to just keep my finger on it while soldering on the other side. Flip the board and apply some solder. You can use some flux to make it easier.</p><p>— <br>💣 <strong>How hard to fix:</strong> Not very hard, just use solder pump on all the pins and pull it off. TRRS is a bit more work as it has more pins.<br> —</p><h3>Soldering OLED pins</h3><p><strong>Temperature: 300–350 ℃<br>Difficulty: easy<br>How hard to fix: medium</strong></p><p>It’s basically the same as with the Pro Micro. Just make sure you’re doing it on the correct side of the PCB. Longer part of the ping should be located on the top side of the board. If you have the cover, it might be worth to just put the pins into the holes, screw spacers in and check if it fits. In my build spacers were 0.5mm short and I had to lower the pins a bit. You can just move the plastic part on the pins up and down by applying some force with small pliers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0vy6P5Cs7tC-zoohFpIxGQ.jpeg" /></figure><h3>Programming Pro Micros (optional)</h3><p><strong>Temperature: None<br>Difficulty: easy<br>How hard to fix: easy</strong></p><p>Now this step is a bit tricky, as we’re going to be shortening pins on the board manually. That’s why I marked it optional. You can do all this using reset button we have soldered into the main board, but it’s good to test our microcontrollers before soldering them in. Desoldering is a big pain here, we want to make sure they are working correctly. We’ll start with default firmware, as it has all the features enabled and we’ll be able to test our keyboard. You can flash anything you wan later on, but <strong>don’t use the ‘<em>common</em>’ firmware from QMK configurator for now</strong>. It has OLED support disabled and you might spend quite some time (as I did) trying to figure out why they don’t work.</p><p>But first a <strong>BIG WARNING</strong>:</p><p>— <br>⚠️ Pro Micros are <strong>sensitive to static current</strong>. Its this small shock you sometimes feel when touching things out of metal. It can destroy your microcontroller. So before taking it from the antistatic bag and each time before you touch it, <strong>touch something that’s out of metal and is grounded.</strong> Your room radiator will be best. Sometimes you’ll feel the shock. That’s the one you saved your controller from.<br> —</p><p>Now grab the QMK toolbox here:</p><p><a href="https://github.com/qmk/qmk_toolbox/releases">Download QMK toolbox</a></p><p>We will be using it to flash our controller. And base firmware here:</p><p><a href="https://github.com/foostan/qmk_firmware-hex/releases/download/release-20201208/crkbd_rev1_common_via.hex">Download CRKBD firmware</a></p><p>(If link doesn’t work, just search for <em>‘crkbd_rev1_common_via.hex’</em> on <a href="https://github.com/foostan/crkbd/blob/master/doc/firmware_jp.md">this</a> site)</p><p>Connect your Pro Micro using Micro USB cable (be very gentle with it, this socket is very fragile). Start QMK toolbox app and:</p><ul><li>Click Open button and select downloaded <em>‘.hex’</em> file.</li><li>Make sure you have <em>‘atmega32u4’</em> controller selected</li><li>Use jumper wire or tweezers to short RST and GND pins on Pro Micro for a second. Make sure you don’t touch any other pieces of the board with your wire.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/1*cTCJO6ka0IB1MmELgP6zZw.jpeg" /></figure><p>If you do it right, QMK toolbox should show a yellow message about that it detected your controller. It should look similar to what you see below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xrjo5mFf-TB4oI4VCYaqLQ.png" /></figure><p>Device name might be different (depending on which Pro Micro clone you have), but there should be “<em>something connected</em>” 🤪.</p><p>Now press the <strong>Flash</strong> button and it should start baking firmware into your controller. The board will be in the programmable state only for few seconds, so if you get a “Device disconnected” warning, just short the RST and GND again. After few seconds, if the flashing was successful, you should see something like on the screen below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j8QtH6ESchKGD5X4d_ZmRg.png" /></figure><p>Now your Pro Micro should show up as a regular keyboard (HID device).</p><p>— <br>🤔 <strong>Advice:</strong> This part might be a bit tricky. You really have to watch out not to touch any other part of the board with your wire or tweezers while shortening the pins. If you don’t feel like you want to do this, just connect the Pro Micros and check if LEDs on them lightens up. Probably they are all fine and you will be able to flash them when soldered in.<br> —</p><h3>Soldering Pro Micro controllers</h3><p><strong>Temperature: 300–350 ℃<br>Difficulty: easy<br>How hard to fix: VERY HARD</strong></p><p>Now insert your controllers onto the pins on the board facing down. The socket and all chips on the controller board should face the main PCB. You should see the empty bottom of the Pro Micro when it’s on the board. Looking from the side, a Micro USB socket should almost touch the main PCB:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jJTxRkGQrc_9Bt0Y-0uFLA.jpeg" /></figure><p>Make sure it’s in place and <strong>double check if it’s positioned correctly</strong>, desoldering it will be very painful. Apply some flux on both lanes of pins and solder each one of them, just like with the pins themselves.</p><p>— <br>💣 <strong>How hard to fix:</strong> ⚠️ Watch out here. It’s really hard to desolder it if you don’t have a hot air station. You will spend quite some time fighting it, and then will have problems to fit new socket into the controller, as it’s hard to clean the holes from all the solder. So <strong>double check</strong> and then check it once more. To make sure you have everything placed correctly before soldering.<br> —</p><p>As you can see in the photo above. I have longer pins on one side. That’s because I’ve got it wrong and had to re-solder them. Didn’t have any more short pins and had to use a longer set. It was really pain in the a** to desolder it. Hope you’ll have it right the first time.</p><h3>Soldering OLEDs</h3><p><strong>Temperature: 300–350 ℃<br>Difficulty: easy<br>How hard to fix: hard</strong></p><p>After you have your controllers in place it’s time to solder in the OLEDs. First <strong>place some insulating tape on your Pro Micro</strong>. It shouldn’t be needed, because theres nothing really conductive there, but better safe than sorry. Then put your OLED in place and check if everything is where it should be. And yes, you want display of the module facing up 🤪.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dWgLWPQNS9KqEJpd39wgkg.jpeg" /></figure><p>— <br>🤔 <strong>Advice:</strong> If you have a OLED cover it might be a good idea to check now if it will fit. It will be easier to replace or shorten the pins now.<br> —</p><p>If everything is in place press gently in the middle of the display with your finger and make sure it’s in place and positioned horizontally to the Pro Micro. It might be a bit hard to level it as it has some outstanding components at the bottom. Just do one pin first and check it it’s correct. Then apply some flux and solder rest of the pins.</p><h3>Test OLEDs and LEDs</h3><p><strong>Temperature: None<br>Difficulty: easy</strong></p><p>We’re almost done, but now is a good moment to test what we already have. Theres a chance that some LEDs won’t be lightening up, and now it’s a bit easier to fix them than when we have everything in place.</p><p>If you have not flashed your controllers, now is the time to do this. Just follow the “<strong><em>Programming Pro Micros (optional)”</em></strong> section and instead of shortening pins on the board press the reset button on the PCB. Remember, that you have to flash <strong>both halves</strong> of the keyboard.</p><p>After flashing disconnect USB, connect both sides of the keyboard with TRRS cable (always do it <strong>before</strong> you connect your keyboard via USB, this cable has no protection and can short circuit the controller while being inserted or removed) and connect the USB cable. Your keyboard should lighten up. You should see a layer indicator and a keylogger on left half and a Corne logo on the right one. Also all the LEDs should be glowing in red.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E0n_vE81SG48qSAzdMkdfg.jpeg" /></figure><p>As you can see. I have messed up some LEDs on the left half. Don’t worry, it’s easy to fix. If you also got something messed up, check the “<strong><em>Troubleshooting”</em></strong> section.</p><p>— <br>🤔 <strong>Advice:</strong> Use a magnetic connector on the Pro Micro USB. Its socket is really fragile. It usually breaks, does it pretty soon and desoldering a Pro Micro is a pain in the a** as you already know.<br> —</p><h3>Case and switches</h3><p><strong>Temperature: None<br>Difficulty: easy<br>How hard to fix: easy</strong></p><p>If you have a case, now is the moment to assemble it. If your case is similar to mine, then first assembly the top part and connect all the switches. Test if they all work. If not — check the “<strong><em>Troubleshooting”</em></strong> section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KLWwP3pqA10aEaypJ3eHNg.jpeg" /></figure><p>— <br>🤔 <strong>Advice:</strong> There are two ways to insert the switches. First is to insert 4 switches into corners of the case, put it into the board, screw the case and then insert others one by one. It usually works great, but with Corne it worked better for me to just insert all switches into the case and connect them all at once. YMMW, just try it out.<br> —</p><p>When you have all your switches in and your case assembled, just insert your keycaps. You’re done.</p><h3>The end</h3><p>Now you should have a fully functional keyboard. If something doesn’t work check the “<strong><em>Troubleshooting”</em></strong> section at the end. I hope it was a fun journey. You can check out <a href="https://qmk.fm/">QMK</a> firmware page to start customizing your layout or firmware.</p><h3>Troubleshooting</h3><h4>Some LEDs are not glowing or are not red</h4><p>If one or more LEDs is totally dark, then probably you have an unsoldered leg on the first one that is not glowing or on the previous one. LEDs are connected in series — previous one is powering next. So there are only 2 to check. The same is for LEDs that have different color than the rest. You only need to check faulty one or the previous. Unfortunately numbers on the PCB are not the same as the connection sequence. Check one of the pictures below from the Foostan guide to see the connection ordering:</p><p><a href="https://user-images.githubusercontent.com/736191/54487452-7b955c80-48d9-11e9-98f7-87490a584274.png">Per-key left part</a></p><p><a href="https://user-images.githubusercontent.com/736191/54487455-7b955c80-48d9-11e9-9498-c841747c5dbc.png">Per-key right part</a></p><p>For underglow you just need to guess or consult the schematics. But there’s only 6 of those, so it won’t be hard. If no under-key LED is working, check either the first one or last underglow LED.</p><p>First check for soldering issues. Maybe one of connections looks bad. If you think that soldering is fine replace the LED. It might be overheated or faulty.</p><h4>No LED is glowing</h4><p>This can be one of 4 problems. In order of probability:</p><ul><li>You have disabled LEDs by using a keyboard shortcut. Check keymap for combination to turn them on again.</li><li>You have flashed firmware without LED support. Check that you have RGBLIGHT_ENABLE = yes in rules.mk file for your keymap and check for define DISABLE_RGB_MATRIX_ALPHAS_MODS lines in config.h. Re-flash and check again.</li><li>LED no. 1 is not soldered correctly or broken. Check soldering on LED number 1. If it’s all good, replace the LED.</li><li>You have an unsoldered pin in your Pro Micro. Check all connections of Pro Micro. Maybe theres a pin that looks bad.</li></ul><h4>One of the buttons is not working</h4><p>It can be one of following, check in following order:</p><ul><li>It’s mapped to some modifier key in firmware. Check keymap.</li><li>Switch is not fully in the socket. Check if it’s touching the PCB.</li><li>One of the switch legs is bent. Remove the switch and straighten the leg.</li><li>Diode on the switch is soldered incorrectly. Make sure it’s in the same direction as rest of the diodes. If not desolder the diode and solder it back correctly.</li><li>Hot-swap socket is soldered badly. Re-solder.</li><li>Switch is faulty, try another switch.</li><li>Hotswap socket is either faulty or you have overheated it. Desolder and replace.</li><li>Diode is faulty, desolder and replace.</li></ul><h4>Whole row or column of the keys is not working</h4><p>You have a bad connection on the microcontroller. Check soldering on Pro Micro. Or you have messed up all the diodes on the row. Second one should be easy to spot.</p><h4>Keyboard is not recognized in the OS</h4><p>You have messed up firmware flashing or have a faulty Pro Micro. Start by reprogramming your controllers.</p><h4>My keyboard halves are mixed</h4><p>If you’re pressing a button that should be on your left hand, but it is on the right and vice-versa, it means that you have connected wrong half to the USB. Unplug cable from the side it is in and plug it into the second half. You can also change firmware settings and re-flash if you want to keep cable in the current half.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b5c7eff4178" width="1" height="1" alt=""><hr><p><a href="https://medium.com/real-life-programming/corne-mx-3-0-keyboard-build-guide-9b5c7eff4178">Corne MX 3.0 Keyboard build guide</a> was originally published in <a href="https://medium.com/real-life-programming">Real Life Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Get lazy loading cheap with IntersectionObserver in React]]></title>
            <link>https://medium.com/real-life-programming/get-lazy-loading-cheap-with-intersectionobserver-in-react-8ca50fe1730d?source=rss-8e0e09168c02------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ca50fe1730d</guid>
            <category><![CDATA[react]]></category>
            <category><![CDATA[reactjs]]></category>
            <category><![CDATA[front-end-development]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Marek Piechut]]></dc:creator>
            <pubDate>Tue, 07 Jul 2020 08:01:56 GMT</pubDate>
            <atom:updated>2020-07-07T08:01:56.746Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OpMxzQS7iJQRwfDOqjokKQ.jpeg" /><figcaption>Photo by Sébastien LAVALAYE on Unsplash</figcaption></figure><h3>Get lazy loading scrolling cheap with IntersectionObserver in React</h3><p>Getting lazy loaded web page elements based on scroll position has been a performance or maintenance nightmare. You either loosen components encapsulation and build a complex mechanism with single scroll handler notifying elements when needed, or end up with huge amount of onScroll handlers that will kill page performance. Thanks to <a href="https://www.w3.org/TR/performance-timeline-2/#the-performanceobserver-interface">W3C working draft from 2019</a> being already implemented in all major browsers we can get rid of all this stuff. Just use IntersectionObserver and get fast and elegant lazy loads when needed.</p><p>So if you no longer care about IE compatibility (and please stop if you do, we need to kill that monster), then stay with me and check out how to cheaply get onScroll lazy loads in React. We’ll base our example on exercise library we’ve added to <a href="http://crosskeeper.app/">CrossKeeper</a> - product we are working on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T9SivkV8XSrDJHOL5ZIfHg.png" /></figure><h3>The Problem</h3><p>Problem with this UI is that it can get slow pretty fast if we simply loaded all exercises with images and dumped them into the page at once. It’s also worth noting, that in this example we don’t exactly know which thumbnails to show based on just exercises. We have to query media source (which can be our custom media server or YouTube API) to fetch it. The only way to do it is one by one, no batch loading. So you might already guessed, firing few hundred HTTP calls each time user opens it up might feel slow. Caching helps a bit, but first load will be slow anyway and keeping all this stuff around if someone never scrolls to the bottom is also not most elegant.</p><p>On the image below you can see how it looks like performance wise when we go with render-all-at-once solution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5a6b40Gro4n90DIUY47RQA.png" /></figure><p>We have more than 7 seconds of network calls to fetch exercise media and thumbnails. Also there’s only so many connections you can initiate to each server. Depleting connection pool makes jumping to some other section of the app feel slow. We’re not allowed to fetch it’s data while waiting for media sources.</p><h3>Using IntersectionObserver</h3><p>Let’s try to cheaply fix it using IntersectionObserver so we only fetch metadata and images for stuff we really need.</p><p>First we need some UI we’ll be rendering and store URL somewhere and make sure it’s rendered when ready. Div with background image and useState hook should fit in nicely:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/78a8dccca17be9ee0dcd11895d1f3894/href">https://medium.com/media/78a8dccca17be9ee0dcd11895d1f3894/href</a></iframe><p>We’ll keep ExerciseItem self contained and make itself load required data from media sources, so we add effect to fetch data and do it only on mount and when media identifier might change:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e379ee403eb95257349cd66ae680495c/href">https://medium.com/media/e379ee403eb95257349cd66ae680495c/href</a></iframe><p>What we have now is fully functional UI component that will give us exactly performance we see in our problem description. All exercises will start fetching media metadata and images as soon as they are added to DOM. This is not what we’re expecting.</p><p>Let’s fix it by adding IntersectionObserver and connecting it to DOM element via ref:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e4478323e226e0de2912c3c0da845c53/href">https://medium.com/media/e4478323e226e0de2912c3c0da845c53/href</a></iframe><p>This should work pretty nicely. The only thing that’s awkward is that images start to load when user does already see them, giving him impression that page does not keep up with his scrolling speed.</p><p>Fortunately there’s something baked into IntersectionObserver that can help us out. If you check API for it, you’ll notice we can pass some options to the constructor (<a href="https://developer.mozilla.org/en-US/docs/Web/API/Intersection_Observer_API#Intersection_observer_options">Intersection observer options</a>) and rootMargin is what we need.</p><p>Let’s make sure that we start loading stuff 200px ahead of user scroll (format is like in CSS margins value). This should cover it up nicely:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1a93bf47480964d076f4adc30e86f44e/href">https://medium.com/media/1a93bf47480964d076f4adc30e86f44e/href</a></iframe><p>For final touches we might add graceful fallback to previous solution on old browsers and here’s what we have:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a51835ade3b4ef36729bb9563e92447f/href">https://medium.com/media/a51835ade3b4ef36729bb9563e92447f/href</a></iframe><h3>Cleanup</h3><p>It doesn’t look that bad, but let’s give it a bit of refactoring, so we’re not overwhelmed with details each time we open up this component.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b2fcafef0750d06b64a3bd6127939bd0/href">https://medium.com/media/b2fcafef0750d06b64a3bd6127939bd0/href</a></iframe><h3>Result</h3><p>Well, there you have it. Fully functional and reusable hook to handle lazy loading using IntersectionObserver. Of course you can easily add loading and error indicator or generalize it even further to make it reusable (ex: passing root margin and loader function from the outside). But even like that it&#39;s a huge improvement over original.</p><p>Below you can see how it looks like to open exercise library now. We’re able to show fully functional UI with all images nearby in 1 second — same machine, same server and same network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ppN1SALQD9mZ1saV6vZMpQ.png" /></figure><p>All this with just few lines of code and modern browser. Great stuff going on in the web technologies and let’s hope that, thanks to browsers auto-update, we’ll be able to use more and more of these in our day to day work.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ca50fe1730d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/real-life-programming/get-lazy-loading-cheap-with-intersectionobserver-in-react-8ca50fe1730d">Get lazy loading cheap with IntersectionObserver in React</a> was originally published in <a href="https://medium.com/real-life-programming">Real Life Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Authentication and Authorization in NodeJS GraphQL API]]></title>
            <link>https://medium.com/real-life-programming/authentication-and-authorization-in-nodejs-graphql-api-58528f6fce5f?source=rss-8e0e09168c02------2</link>
            <guid isPermaLink="false">https://medium.com/p/58528f6fce5f</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[graphql]]></category>
            <category><![CDATA[authentication]]></category>
            <dc:creator><![CDATA[Marek Piechut]]></dc:creator>
            <pubDate>Mon, 21 Jan 2019 13:29:06 GMT</pubDate>
            <atom:updated>2019-01-21T13:29:06.122Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AasVs894VEy5lON0kgvfMA.jpeg" /></figure><p>Most of GraphQL APIs that are developed are probably not meant for public access without any authorization. Sooner or later you’ll need to somehow limit access to only authenticated users or limit resources so that only allowed users are able to see them.</p><p>In this post, we’ll take a look at how you could implement GraphQL security in applications using NodeJS, Passport and Apollo Server.</p><h3>Ways to implement authentication</h3><p>There are a few ways you could add access rights to your GraphQL APIs:</p><ul><li>If your requirements are simple, you can just allow all access to logged in users and decline it to the general public. This could be easily done in a context factory (we’ll go through that later on).</li><li>You could check access rights in your resolvers and fine grain it based on requested data, current session and returned objects.</li><li>Move security code into your domain logic and fail there if access should not be granted.</li><li>Add directives to your schema and make security declarative.</li><li>Leave all security to other services if your GraphQL is only a facade to other systems.</li></ul><h3>Where to keep security code</h3><p>I think, that security is rarely something domain specific but rather part of system infrastructure and does not belong to domain code. Moving it to domain layer would also mean you make it dependent on session data and web application concepts. That’s not what we usually want.</p><p>Keeping it in directives may look very elegant at first, but it makes things really complex if you need something more than simple role-based authorization. You might end up with a brand new <a href="https://en.wikipedia.org/wiki/Domain-specific_language">DSL</a> on top of GraphQL schema language to get something more from it.</p><p>What we have left is to keep it in GraphQL resolvers and that’s what we’ll focus on in this article. Keeping authorization code there is very elastic and does not pollute your domain logic.</p><h3>Starting Point</h3><p>Technologies we’ll be using:</p><ul><li><a href="http://nodejs.org/">NodeJS</a></li><li><a href="http://expressjs.com/">ExpressJS</a></li><li><a href="http://www.passportjs.org/">Passport</a></li><li><a href="https://www.apollographql.com/docs/apollo-server/">Apollo Server 2.0</a></li><li><a href="https://www.apollographql.com/docs/apollo-server/features/graphql-playground.html">GraphQL Playground</a></li></ul><p>To make things simpler I assume you already have Passport configured and some basic way to authenticate in your system. If you need help with that, check out <a href="https://reallifeprogramming.com/node-authentication-with-passport-postgres-ef93e2d520e7">this</a> article on <a href="https://reallifeprogramming.com/node-authentication-with-passport-postgres-ef93e2d520e7">RealLifeProgramming</a> to get you up and running.</p><p>I also assume you have some knowledge about GraphQL and some GraphQL API that we’re going to secure. Make also sure you have <a href="https://www.apollographql.com/docs/apollo-server/features/graphql-playground.html">GraphQL Playground</a> or some other way to test things out.</p><h3>Know your user</h3><p>To do any access right checks we first need to know who is trying to access the resources. When using Passport we already have everything in place and can retrieve the user from the session (or get nothing if the user is not authenticated).</p><p>To retrieve Passport session data we need to first make sure to setup Passport before registering Apollo middleware, so when we hit any resolver we’re already authenticated. We just need to make sure our code is in right order:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8e004177974c1c05932f463a83f7d91f/href">https://medium.com/media/8e004177974c1c05932f463a83f7d91f/href</a></iframe><p>With this code in place, our GraphQL API is still publicly available. What we can do now, is pass logged in user (if available) down to all resolvers using Apollo context.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e89eef5180a014eb926b1b5379c83cc8/href">https://medium.com/media/e89eef5180a014eb926b1b5379c83cc8/href</a></iframe><p>If all or nothing authorization scheme is what you need, then you can also authorize all requests here. What you can do, is simply throw AuthenticationError from apollo-server-express package whenever there&#39;s no user, or your user is not allowed to access graph:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7e45d7cfa2473316f5b9bdbbe9a83c68/href">https://medium.com/media/7e45d7cfa2473316f5b9bdbbe9a83c68/href</a></iframe><p>Although this might solve your current problems I don’t really recommend it. It’s not very elastic and is hard to extend later on. It also forbids any public APIs you might want to add in the future, so it might be better to move to the second step and add some security logic directly to graph resolvers.</p><h3>Testing things out</h3><p>But first, let’s check if everything works as expected. We can use GraphQL Playground to test things out. You should see something similar when trying to query without login:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*etzoq5m6QPxkX68C0FWLcg.png" /><figcaption>GraphQL Playground — Auth Error</figcaption></figure><p>Just one small note before we continue: before you try to access secured resource make sure you have request.credentials option set to at least same-origin in playground options. Without it, there will be no authentication data sent to your backend when you&#39;ll be executing your queries.</p><h3>Securing resolvers</h3><p>Let’s start with moving our all or nothing code into resolvers, so we can choose on per resolver basis if we allow access or not. What we need to check is a user property in context object (3rd argument to resolver):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/02fe8c550d4de8e128b2fea165edbada/href">https://medium.com/media/02fe8c550d4de8e128b2fea165edbada/href</a></iframe><p>Ok, this solves our problem, but copying all this stuff around would be very cumbersome. Why not take advantage of that Javascript is a functional language and we can pass functions around. Let’s refactor this code to a higher order function that we can later apply on any resolver we like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/46d07e67a8b265712bec6f110da8e700/href">https://medium.com/media/46d07e67a8b265712bec6f110da8e700/href</a></iframe><p>Now we only need to wrap our resolver with requiresLogin call:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/99a77f3d9d8cbfbd255828a4105ac2cf/href">https://medium.com/media/99a77f3d9d8cbfbd255828a4105ac2cf/href</a></iframe><h3>Role-based authentication</h3><p>By changing only a few lines here, we could also get a very nice role-based authentication using this pattern:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d8dc078e95cc12e177fd6a8956859ac5/href">https://medium.com/media/d8dc078e95cc12e177fd6a8956859ac5/href</a></iframe><p>And use it in similar manner:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/14c7af51f4747b22758e8670def4e8ab/href">https://medium.com/media/14c7af51f4747b22758e8670def4e8ab/href</a></iframe><h3>Modularized resolvers</h3><p>All this can get you far, but will probably also get cumbersome after some time. When you start to divide your resolvers, you’ll end up wanting to setup access rights for whole groups in one place instead of wrapping functions one by one. Again we would end up with a lot of copied code.</p><p>Why not simply make our authorization function support whole objects and wrap all resolvers with auth code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/981d5f41bd7f854c5c51690c7733f4c6/href">https://medium.com/media/981d5f41bd7f854c5c51690c7733f4c6/href</a></iframe><p>Now we can wrap our resolvers with requiresRole on any level and it will recursively secure everything according to the declaration:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4519d664309c1652b4b0529448f91a8c/href">https://medium.com/media/4519d664309c1652b4b0529448f91a8c/href</a></iframe><p>In this example, both: templates and tasks queries will require the current user to be a &#39;<em>member</em>&#39; before he can access them.</p><p>P.S. If you don’t modularize your resolvers yet, check out <a href="https://blog.apollographql.com/modularizing-your-graphql-schema-code-d7f71d5ed5f2">Modularizing your GraphQL schema code</a> by @dpandya.</p><h3>What next</h3><p>Keeping authorization logic in resolvers is very elastic and we’ve only touched surface here. Thanks to that resolvers are asynchronous in nature and can return <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise">Promises</a> we can implement pretty complex authorization schemes with pre and post validation and even custom per-entity ACLs. But we’ll cover these in another article.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58528f6fce5f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/real-life-programming/authentication-and-authorization-in-nodejs-graphql-api-58528f6fce5f">Authentication and Authorization in NodeJS GraphQL API</a> was originally published in <a href="https://medium.com/real-life-programming">Real Life Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Get rid of some fat from your React production build]]></title>
            <link>https://medium.com/real-life-programming/get-rid-of-some-fat-from-your-react-production-build-3a5bb4970305?source=rss-8e0e09168c02------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a5bb4970305</guid>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[reactjs]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[react]]></category>
            <dc:creator><![CDATA[Marek Piechut]]></dc:creator>
            <pubDate>Thu, 02 Nov 2017 10:40:08 GMT</pubDate>
            <atom:updated>2017-11-02T10:45:59.932Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MgThmCPzRGXeGmFl3WVTnQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/UzbSX5-PeTE?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Fabian Blank</a> on <a href="https://unsplash.com/?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Get rid of some fat from your React production build with Webpack</h3><p>Who wouldn’t want to get something for free? If you’re using <a href="http://webpack.github.io/">Webpack</a> with <a href="https://facebook.github.io/react/">React</a> you can get size and load time optimizations with just a few small configuration changes. Almost for free, without messing with your code.</p><p>Here I’m showing few of these. They are really easy to do and can give you nice results regarding application size and startup time.</p><h3>Switch Node to production</h3><p>When switched to production, Webpack will add optimizations not executed otherwise. What’s even more important — <a href="https://github.com/mishoo/UglifyJS">UglifyJS</a> will remove all React debugging and introspection code. As it becomes effectively a <a href="https://en.wikipedia.org/wiki/Dead_code_elimination">dead code</a>. It was helping you to find errors in development, but only causes slowdown and bundle bloat in production. Stuff like invariants checking, props validation, etc. There’s a lot of code like this in React:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/31648f2c7559372e9b599dc04f7fbe73/href">https://medium.com/media/31648f2c7559372e9b599dc04f7fbe73/href</a></iframe><p>and it will be completely removed in production mode. Thanks to our little environment flag.</p><p>Add this code to production plugins in webpack.conf.js to enable it:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b14d59db87e81add7030f9b8bdc90a5a/href">https://medium.com/media/b14d59db87e81add7030f9b8bdc90a5a/href</a></iframe><p>and remember to launch webpack with -p and --env.production options:</p><pre>webpack --env.production -p</pre><h3>Strip all unneeded stuff with UglifyJS</h3><p>Your production build probably includes a lot of stuff that’s not really required during execution. With some more aggressive UglifyJS configuration you can also get rid of these:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8c1c4d114807d310c8b68c05d75828fb/href">https://medium.com/media/8c1c4d114807d310c8b68c05d75828fb/href</a></iframe><p>It will wipe a lot of stuff you don’t really need. Some of it might be useful during development but is only a waste when you release. I personally like screw_ie8 the most. It’s just something we all should have done quite some time ago. This option should be enabled by default.</p><h3>Load only required Lo-Dash modules</h3><p>If you’re building a larger application, chances are, you’re already using <a href="https://lodash.com/">Lo-Dash</a>. You can get some bundle size reduction by using <a href="https://github.com/lodash/babel-plugin-lodash">babel-plugin-lodash</a> to enable <a href="https://webpack.js.org/guides/tree-shaking/">tree shaking</a> for Lo-Dash imports. It analyzes your code, to check which functions are imported and which can be skipped. What it means, is that you’ll only include stuff you are really using (and it’s dependencies), not everything that’s in the package. And it doesn’t matter if you’re using import map from &#39;lodash/map or import {map} from &#39;lodash&#39; notation. It’s smart enough to figure it out.</p><h3>Get rid of unused moment locales</h3><p><a href="https://momentjs.com/">Moment</a> is a great library, that has a huge set of locale-specific settings, that you might actually never use.</p><p>It might be useful to get rid of ones you don’t use. If you have support for only subset of languages or are building an internal app and you need no internalization at all, there’s no point in keeping them around.</p><p>Here’s the difference it can make (file sizes before minification, so final results might vary):</p><ul><li>With <strong>all</strong> locales: moment: 455.55 KB (35.9%)</li><li>With only <strong>en</strong> locale: moment: 125.92 KB (13.4%)</li><li>With <strong>en</strong> and <strong>pl</strong> locales: moment: 129.59 KB (13.8%)</li></ul><p>As you can see, leaving only a handful of these can save a bit of space. Here’s how you set it up:</p><ul><li>First, filter out all locales using webpack-ignore-plugin (it’s built-in, so nothing to install):</li></ul><pre>new webpack.IgnorePlugin(/^\.\/locale$/, /moment$/),</pre><ul><li>Then, in case you need something more than plain en, require it explicitly. You can do it by adding something like this somewhere in your code (probably near app initialization, where you setup moment stuff):</li></ul><pre>import &#39;moment/locale/pl&#39;</pre><p>Webpack will make sure it’s included in the output, even if it was ignored before.</p><h3>Split bundle to vendor and app code</h3><p>If you have more than one entry-point and there are parts of it that some users won’t ever access (like admin panel). Or you have more than one app in a single codebase (like variants of the app for different departments). Then it might be useful to split distribution package to separate bundles and keep external dependencies apart. This way users that don’t need admin module won’t download and parse its code. And your dependencies will be in a separate file, that browser can cache and reuse when switching between apps.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b263d5020d41a432b30806dcfb176224/href">https://medium.com/media/b263d5020d41a432b30806dcfb176224/href</a></iframe><p>You should probably also have your entry-points declared somewhere in configuration object:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/655b292e112a07f3a159afcd3a6b43ee/href">https://medium.com/media/655b292e112a07f3a159afcd3a6b43ee/href</a></iframe><p>Just make sure you include vendor script before your application in entrypoint html file.</p><p>There’s also a chance, that after some time, your dependencies stop changing so often and you might release updates without even changing vendor bundle. This way clients won’t need to fetch it again.</p><p>Of course you can experiment with CommonsChunkPlugin to get even better results. Maybe you’ll want to keep separate bundle with dependencies used in each variant of the app or extract only stuff that’s really shared between entry-points. Check out docs here: <a href="https://webpack.js.org/plugins/commons-chunk-plugin/">commons-chunk-plugin</a> to see what else you can get from it.</p><h3>Bonus — how to analyze bundle size</h3><p>So now you have set up some optimizations that should lower your bundle size and decrease startup time. But what more can you do to make it smaller and faster? Get rid of stuff that’s taking a lot of space and doesn’t give much value. Sometimes you include library and use only a tiny fraction of it just to find out that it’s a half of your application size.</p><p>To know which libraries consume most space, we need to look into Webpack bundle and analyze what’s there. There are two nice tools to get some insight:</p><ul><li><a href="https://www.npmjs.com/package/webpack-bundle-analyzer">Webpack bundle analyzer</a></li></ul><p>and</p><ul><li><a href="https://github.com/robertknight/webpack-bundle-size-analyzer">Webpack bundle size analyzer</a></li></ul><p>First of them gives you nice visual map to click through and check what takes up most of the space and what it depends on.</p><figure><img alt="Bundle size browser" src="https://cdn-images-1.medium.com/proxy/1*RNpMU6T0e6NAw04CttNW3w.png" /></figure><p>It’s very good if what you’re doing is a hunt for space hogs. I highly recommend you run it at least once on your project. You might be surprised. It’s not always obvious what is the biggest part of your app.</p><p>On the other hand, if you only want to check what’s the impact of stuff you just added or quickly get some insight <a href="https://github.com/robertknight/webpack-bundle-size-analyzer">Webpack bundle size analyzer</a> might be the way to go. You just pipe Webpack JSON output to it and you’re done:</p><pre>webpack -p --json | webpack-bundle-size-analyzer | less</pre><p>and here’s the output:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/53c1aefc03dca43e767ca45185a2dbbb/href">https://medium.com/media/53c1aefc03dca43e767ca45185a2dbbb/href</a></iframe><h3>Set the options, but what are the results?</h3><p>Just to sum things up. Here are bundle sizes for <a href="http://github.com/dayonepl/dos-react">DOS</a> React starter kit with:</p><pre>Version: webpack 3.6.0<br>Time: 13194ms<br>     Asset       Size  Chunks                    Chunk Names<br>   main.js    7.29 kB       0  [emitted]         main<br> vendor.js     390 kB       1  [emitted]  [big]  vendor<br>styles.css    2.25 kB    0, 1  [emitted]         main, vendor<br>index.html  463 bytes          [emitted]</pre><p>and without configuration changes applied:</p><pre>Version: webpack 3.6.0<br>Time: 10899ms<br>     Asset       Size  Chunks                    Chunk Names<br>   main.js     571 kB       0  [emitted]  [big]  main<br>styles.css    5.57 kB       0  [emitted]         main<br>index.html  463 bytes          [emitted]</pre><p>As you can see, we have stripped almost 200kB from final size without any changes in source code or removing any libraries. It was probably worth these few minutes to put it in the config files.</p><p>By the way, if you want a React boilerplate to speed up your next project ramp-up, give <a href="http://github.com/dayonepl/dos-react">DOS</a> a try. It has all these optimizations already configured and ready to be used.</p><p><strong>And here is a full webpack config file for reference: </strong><a href="https://github.com/DayOnePl/dos-react/blob/master/webpack.config.js"><strong>webpack.config.js</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a5bb4970305" width="1" height="1" alt=""><hr><p><a href="https://medium.com/real-life-programming/get-rid-of-some-fat-from-your-react-production-build-3a5bb4970305">Get rid of some fat from your React production build</a> was originally published in <a href="https://medium.com/real-life-programming">Real Life Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>