<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[JobTeaser Engineering - Medium]]></title>
        <description><![CDATA[JobTeaser&#39;s Engineering Team sharing tips and thoughts about programming - Medium]]></description>
        <link>https://medium.com/jobteaser-dev-team?source=rss----bd77d16a0035---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>JobTeaser Engineering - Medium</title>
            <link>https://medium.com/jobteaser-dev-team?source=rss----bd77d16a0035---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 23:22:58 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/jobteaser-dev-team" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Stop Defaulting to "use client": A Practical Mental Model for React Server Component Architecture]]></title>
            <link>https://medium.com/jobteaser-dev-team/stop-defaulting-to-use-client-a-practical-mental-model-for-react-server-component-architecture-d94dd0a73364?source=rss----bd77d16a0035---4</link>
            <guid isPermaLink="false">https://medium.com/p/d94dd0a73364</guid>
            <category><![CDATA[react]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[front-end-development]]></category>
            <category><![CDATA[react-server-component]]></category>
            <category><![CDATA[nextjs]]></category>
            <dc:creator><![CDATA[Clarisse Leduc]]></dc:creator>
            <pubDate>Tue, 21 Apr 2026 11:14:11 GMT</pubDate>
            <atom:updated>2026-04-21T11:14:10.539Z</atom:updated>
            <content:encoded><![CDATA[<p>How thinking in server-first component boundaries can simplify data flow and reduce client JavaScript in Next.js applications.</p><p>When I first started working with the Next.js App Router, I kept running into the same situation.</p><p>Components would suddenly break — usually after adding a hook or a click handler. The fix felt obvious: add &quot;use client&quot;.</p><p>But after doing this a few times, I realised something: I was slowly turning my application back into a <strong>traditional client-side React app</strong>.</p><p>Which defeats the whole point of <strong>React Server Components</strong>.</p><p>The real challenge isn’t learning how to use Client and Server Components.<br>It’s <strong>learning how to decide where code should run</strong>.</p><p>Once that mental model clicked, structuring Next.js applications became much clearer — and often more performant.</p><h3>The Mental Model</h3><p>React Server Components introduce a simple but powerful separation of responsibilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dJG6AIIkuhLvF1E7eONBwQ.png" /></figure><p>A helpful rule of thumb:</p><blockquote><strong><em>Default to Server Components.<br>Use Client Components only when interactivity is required.</em></strong></blockquote><p>Think of it like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_qn9KUGVwYa2cYr4etlnBA.png" /></figure><p>This small shift in thinking has a big impact on how you design component trees.</p><h3>A Simple Example</h3><p>Consider a typical product page.</p><pre>export default async function ProductPage({ params }) {<br>  const product = await getProduct(params.id, { cache: &quot;no-store&quot; })<br>  <br>  if (!product) {<br>    notFound() // Next.js helper for 404 pages<br>  }<br><br>return (<br>    &lt;&gt;<br>      &lt;ProductDetails product={product} /&gt;<br>      &lt;AddToCartButton productId={product.id} /&gt;<br>    &lt;/&gt;<br>  )<br>}</pre><p>Here:</p><ul><li>ProductPage → Server Component</li><li>ProductDetails → Server Component</li><li>AddToCartButton → Client Component</li></ul><p>Only the <strong>interactive part</strong> runs in the browser. Everything else stays on the server, meaning <strong>less JavaScript shipped to the user</strong>.</p><h3>The &quot;use client&quot; Boundary</h3><p>One subtle detail that often surprises developers is that &quot;use client&quot; creates a <strong>boundary in the component tree</strong>.</p><p>If a component is marked as a Client Component, <strong>all of its children become Client Components too</strong>.</p><p>Example:</p><pre>&quot;use client&quot;<br><br>export default function Page() {<br>  return (<br>    &lt;Layout&gt;<br>      &lt;ProductInfo /&gt;<br>      &lt;AddToCartButton /&gt;<br>    &lt;/Layout&gt;<br>  )<br>}</pre><p>This architecture effectively becomes:</p><pre>Page (Client)<br> ├── Layout (Client)<br> ├── ProductInfo (Client)<br> └── AddToCartButton (Client)</pre><p>Even if Layout and ProductInfo don’t need interactivity, they are now part of the <strong>client bundle</strong>.</p><p>A better approach is to move the client boundary closer to the interactive component:</p><pre>export default function Page() {<br>  return (<br>    &lt;Layout&gt;<br>      &lt;ProductInfo /&gt;<br>      &lt;AddToCartButton /&gt;<br>    &lt;/Layout&gt;<br>  )<br>}</pre><pre>&quot;use client&quot;<br><br>export function AddToCartButton() {<br>  // interactive logic<br>}</pre><p>Now the architecture becomes:</p><pre>Page (Server)<br> ├── Layout (Server)<br> ├── ProductInfo (Server)<br> └── AddToCartButton (Client)</pre><p>Only the interactive surface is in the browser bundle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pY2yM28tmCOMh5U5r00Xhg.png" /></figure><h3>A Useful Pattern: Server Shell + Client Islands</h3><p>One way to structure applications with React Server Components is to think in terms of a <strong>server-rendered shell with small interactive islands</strong>.</p><p>This idea is close to the “islands architecture” popularised by frameworks like Astro, but applied at the component level in React.</p><p>In practice, it looks like this:</p><ul><li>The <strong>server</strong> fetches data and renders most of the UI</li><li>The <strong>browser</strong> only hydrates the parts that actually need interactivity</li></ul><pre>Server<br> ├── fetch data<br> ├── render UI<br> └── send HTML</pre><pre>Browser<br> └── hydrate interactive components</pre><p>The key is that interactivity is not spread across the tree — it’s <strong>isolated into small, explicit Client Components</strong>.</p><p>For example:</p><pre>export default function ProductPage({ product }) {<br>  return (<br>    &lt;ProductLayout<br>      actions={&lt;AddToCartButton productId={product.id} /&gt;}<br>    &gt;<br>      &lt;ProductDetails product={product} /&gt;<br>    &lt;/ProductLayout&gt;<br>  )<br>}</pre><p>Here, the page and layout stay on the server, while the interactive part (AddToCartButton) is injected as a <strong>client island</strong>.</p><p>This keeps the client bundle small and makes the boundary between server and client explicit.</p><p>It’s less about following a strict pattern and more about adopting a habit: keep the server in charge by default, and carve out small client islands only where interaction is required.</p><h3>When Server Components Are the Wrong Choice</h3><p>Server Components are powerful, but not always the right tool.</p><p><strong>Highly interactive UI:</strong> drag-and-drop, rich editors, complex forms, dashboards.<br><strong>Real-time or local state:</strong> live search, optimistic UI, WebSocket-driven features.<br><strong>Browser APIs:</strong> localStorage, window/document, geolocation, canvas.<br><strong>Third-party libraries:</strong> charts, animations, UI frameworks.</p><p><strong>Practical rule:</strong></p><pre>Server → data and structure<br>Client → interaction</pre><p>Server Components handle <strong>what the page displays</strong>, Client Components handle <strong>how the user interacts</strong>.<br>In practice, trying to eliminate Client Components completely is a mistake — you end up fighting the framework. What actually works is keeping them small and intentional.</p><h3>Production Lessons Learned</h3><h3>1. Defaulting to Server Components simplifies data flow</h3><p>Fetching data server-side avoids client request waterfalls and reduces loading states.</p><h3>2. Small Client Components scale better</h3><p>Isolating interactivity keeps bundles smaller and components easier to maintain.</p><h3>3. Component boundaries matter</h3><p>Moving &quot;use client&quot; higher can unintentionally convert large parts of the UI into Client Components. Be deliberate about boundaries.</p><h3>4. Think in rendering responsibilities</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Y2CPDFlOq9Zvha2ybg3Zw.png" /></figure><p>This mindset keeps architecture simple and predictable.</p><h3>Why This Matters for Performance</h3><p><strong>Smaller bundles:</strong> only interactive components ship to the browser.<br><strong>Faster page loads:</strong> browser receives rendered HTML immediately.<br><strong>Less hydration work:</strong> fewer components to hydrate.<br><strong>Better SEO:</strong> content rendered on the server.</p><h3>Final Thoughts</h3><p>React Server Components encourage a <strong>server-first architecture mindset</strong>.<br>At our team, adopting a <strong>server-first mindset</strong> when building with Next.js has helped us think more carefully about where logic should live and how much JavaScript we ship to users.</p><p>When reviewing pull requests in Next.js projects, one thing I often look for is whether &quot;use client&quot; could be pushed deeper into the component tree.</p><p>The question isn’t <em>“server or client?”</em> — that’s already the wrong framing.</p><p>A better question is:</p><blockquote><strong><em>Does this UI need to run in the browser at all?</em></strong></blockquote><p>If the answer is no, it belongs on the server.<br>In practice, the apps that scale well are the ones where Client Components are small, deliberate, and isolated.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d94dd0a73364" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jobteaser-dev-team/stop-defaulting-to-use-client-a-practical-mental-model-for-react-server-component-architecture-d94dd0a73364">Stop Defaulting to &quot;use client&quot;: A Practical Mental Model for React Server Component Architecture</a> was originally published in <a href="https://medium.com/jobteaser-dev-team">JobTeaser Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Conduct a technical migration
within less than 6 months — Part 3 (final)]]></title>
            <link>https://medium.com/jobteaser-dev-team/conduct-a-technical-migration-within-less-than-6-months-part-3-final-7968c007f770?source=rss----bd77d16a0035---4</link>
            <guid isPermaLink="false">https://medium.com/p/7968c007f770</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[nextjs]]></category>
            <category><![CDATA[developer-experience]]></category>
            <category><![CDATA[project-management]]></category>
            <dc:creator><![CDATA[Fabien RAYNAUD]]></dc:creator>
            <pubDate>Mon, 02 Dec 2024 09:47:16 GMT</pubDate>
            <atom:updated>2024-12-02T10:16:45.057Z</atom:updated>
            <content:encoded><![CDATA[<h3>Conduct a technical migration<br>within less than 6 months — Part 3 (final)</h3><h4>based on Next.js migration, performed at JobTeaser</h4><p>What is this 3-parts article about:</p><ul><li><strong>sharing my own experience</strong> with handling a technical migration</li><li>📖 some <strong>additional elements</strong> that I find relevant and could help (identified with 📖 emoji) but not personally tested</li></ul><p>What this presentation is NOT about:</p><ul><li><strong>technical step-by-step</strong> Next.js migration</li><li><strong>the perfect and only way</strong> to do a migration</li></ul><p>I tried to be as exhaustive as possible, but I didn’t even complete all the points personally. So the <strong>presentation is a mix of my personal experience and all the things that could help to do it “qualitatively”. </strong>Pick what is necessary for you and your ecosystem.</p><p>3 main parts:</p><ol><li><a href="https://medium.com/jobteaser-dev-team/conduct-a-technical-migration-within-less-than-6-months-part-1-c1215057a515"><strong>Buying time and sponsorship</strong></a> to be able to perform such a migration = creating a good environment to navigate easily to success</li><li><a href="https://medium.com/jobteaser-dev-team/conduct-a-technical-migration-within-less-than-6-months-part-2-c4f2cd1feea4"><strong>Organize your project</strong></a> = keep an eye on and communicate about what was done and what needs to be done</li><li><strong>▶️ Preserve the developer experience</strong> = ensure use, sustainability, and future maintenance of the migration</li></ol><p><em>You are now ready to focus on the implementation but do not hesitate to make a step aside and check if the learning curve is not too big for newcomers on the migrated part.</em></p><h3>Preserve the Developer’s Experience</h3><p>Finally, it is important to <strong>preserve the Developer’s Experience</strong> throughout the migration process. This means <strong>minimizing disruption</strong> to the development environment and <strong>ensuring that developers can continue to work efficiently</strong>. It also means <strong>preparing the next step</strong> of the project (outside migration) and making it efficient as soon as possible.</p><p><strong>Analyze and preserve the current</strong> development environment of the team</p><ul><li><strong>Check the used tools</strong> and the <strong>compatibility after the migration</strong>, and communicate if you need to remove one</li><li><strong>Keep automation as much as possible</strong>, nobody wants to get back to manual stuff after a migration</li><li>Take time to <strong>document </strong>(update) what you are keeping, newcomers would thank you for that</li></ul><p><strong>Train the team</strong> to be able <strong>to be efficient quickly</strong> with new technologies or tools.</p><ul><li>📖 <strong>Develop training materials and resources</strong> to help developers learn how to use new tools or technologies. Sometimes official documentation is not enough.</li><li>Provide <strong>opportunities for developers to ask questions and receive support</strong> as they learn. You can open a support Slack channel.</li></ul><p><strong>Prepare “after the migration” step</strong>: it’s not an ON/OFF period, it’s a <strong>transitional period</strong> for everyone</p><ul><li><strong>Communicate</strong> and share about the experience (internally and externally)</li><li>Find a way to <strong>get out of the project</strong> (avoid being a SPOC — <em>Single Point of Contact</em>)<br>- if you organize properly and <strong>imply other teams early</strong>, should be naturally OK<br>- <strong>share all your knowledge</strong> with other team leads<br>- <strong>document</strong> what is still in your brain</li><li><strong>Monitor your migration</strong> to validate the success and/or analyze the mistakes, so you still learn<br>- Do not hesitate to <strong>wait for the first “real” use case</strong> of the migration, to validate and celebrate properly</li><li>Define <strong>rules to move debt</strong> alongside the migrated part properly:<br>- <strong>What to do with “old stuff”?</strong> Migrate, kill, keep as it… and for how long?<br>- Who is the <strong>owner of the remaining tasks</strong>?</li></ul><p><em>🔍 Concrete “real life” example:</em></p><ul><li>CI/CD offers the same experience as before (i18n, feature env, linting, and testing …)</li><li>Recurring weekly point to discuss what was new on the stack (in Chapter Life — 5/10 min)</li><li>Hard for teams to learn Next.js and TypeScript at the same time</li><li>Currently defining a plan to migrate old CSR into Next.js ecosystem without redoing them from scratch</li><li>Do not forget to communicate the end of your part of the migration and celebrate!</li></ul><p><em>Even if this part is the last one, it’s probably the most stringent part of your migration project. If you keep the teams with you, it’s already a success.</em></p><p><em>🔍 Concrete “real life” outcomes:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9Y2bR7LkQv1tax3H" /><figcaption><em>Before/After results from </em><a href="https://developer.chrome.com/docs/lighthouse"><em>Lighthouse</em></a><em> and </em><a href="https://www.websitecarbon.com/"><em>Website Carbon Calculator</em></a></figcaption></figure><ul><li>All our squads are now using the new technical stack with better productivity</li><li>Pages are easier to enhance regarding performance because we also manage the server part</li></ul><p>A<strong> successful technical migration requires careful planning, organization, and communication.</strong> By following these steps, we can <strong>ensure that the migration process runs smoothly</strong> and <strong>the project is completed on time and within budget</strong>. <strong>Do not forget to celebrate</strong> at the end!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/0*m_Yl7BggHoUW-iM2" /></figure><p>First part is about <a href="https://medium.com/jobteaser-dev-team/conduct-a-technical-migration-within-less-than-6-months-part-1-c1215057a515"><strong>“Buying time and sponsorship”</strong></a><br>Previous part is about <a href="https://medium.com/jobteaser-dev-team/conduct-a-technical-migration-within-less-than-6-months-part-2-c4f2cd1feea4"><strong>“Organize your project”</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7968c007f770" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jobteaser-dev-team/conduct-a-technical-migration-within-less-than-6-months-part-3-final-7968c007f770">Conduct a technical migration
within less than 6 months — Part 3 (final)</a> was originally published in <a href="https://medium.com/jobteaser-dev-team">JobTeaser Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Conduct a technical migration
within less than 6 months — Part 2]]></title>
            <link>https://medium.com/jobteaser-dev-team/conduct-a-technical-migration-within-less-than-6-months-part-2-c4f2cd1feea4?source=rss----bd77d16a0035---4</link>
            <guid isPermaLink="false">https://medium.com/p/c4f2cd1feea4</guid>
            <category><![CDATA[developer-experience]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[nextjs]]></category>
            <category><![CDATA[project-management]]></category>
            <dc:creator><![CDATA[Fabien RAYNAUD]]></dc:creator>
            <pubDate>Mon, 02 Dec 2024 09:46:56 GMT</pubDate>
            <atom:updated>2024-12-02T10:16:21.195Z</atom:updated>
            <content:encoded><![CDATA[<h3>Conduct a technical migration<br>within less than 6 months — Part 2</h3><h4>based on Next.js migration, performed at JobTeaser</h4><p>What is this 3-parts article about:</p><ul><li><strong>sharing my own experience</strong> with handling a technical migration</li><li>📖 some <strong>additional elements</strong> that I find relevant and could help (identified with 📖 emoji) but not personally tested</li></ul><p>What this presentation is NOT about:</p><ul><li><strong>technical step-by-step</strong> Next.js migration</li><li><strong>the perfect and only way</strong> to do a migration</li></ul><p>I tried to be as exhaustive as possible, but I didn’t even complete all the points personally. So the <strong>presentation is a mix of my personal experience and all the things that could help to do it “qualitatively”. </strong>Pick what is necessary for you and your ecosystem.</p><p>3 main parts:</p><ol><li><a href="https://medium.com/jobteaser-dev-team/conduct-a-technical-migration-within-less-than-6-months-part-1-c1215057a515"><strong>Buying time and sponsorship</strong></a> to be able to perform such a migration = creating a good environment to navigate easily to success</li><li><strong>️️▶️ Organize your project</strong> = keep an eye on and communicate about what was done and what needs to be done</li><li><a href="https://medium.com/jobteaser-dev-team/conduct-a-technical-migration-within-less-than-6-months-part-3-final-7968c007f770"><strong>Preserve the developer experience</strong></a> = ensure use, sustainability, and future maintenance of the migration</li></ol><p><em>Now that you have convinced the right people and made a big-picture plan, time to jump into the project, whether you are part of the implementation team or not.</em></p><h3>Organize your project</h3><p>The second step is to <strong>organize</strong> the project. This includes <strong>cleaning the scope</strong> of the migration regarding <strong>the most important tasks and secondary ones</strong> that could be done later. It’s also about having a clean place to <strong>organize, prioritize, and communicate</strong> the project execution.</p><p>Define the <strong>“real” scope of the migration = MVP</strong> with only the mandatory tasks: Be prepared for the next steps.</p><ul><li><strong>You won’t be able to perform all the tasks</strong> from the initial plan by yourself or with the initial team, only the most important ones</li><li><strong>Listing and specifying secondary tasks</strong> is important for the next steps of the project (after the migration)</li><li><strong>Continue your technical watch</strong> on the subject and <strong>adjust the migration</strong> if necessary, nobody wants an already deprecated solution.</li><li>Assign specific tasks to team members based on their skills and expertise, and <strong>do not hesitate to raise some hands for (external) help</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b_piLXpQciv1mLxbFq5i4w.png" /><figcaption><em>Tasks list from JIRA project</em></figcaption></figure><p>Break down the project into <strong>manageable tasks</strong> and set <strong>realistic deadlines</strong> for each stage.</p><ul><li>Develop a <strong>detailed project plan</strong> that outlines the steps involved in the migration process and the timeline for each stage. Could be done on a ticketing tool.</li><li>📖 Set <strong>realistic deadlines for each task</strong> and ensure that team members have adequate time to complete their work</li><li>Start as soon as possible to <strong>engage other teams to perform secondary tasks</strong> or participate in primary task discussion</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aon8UYTtCWqMIVRH" /><figcaption><em>Weekly communication on Slack</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*_MRJtr_TNW5glq9b_AqaOg.png" /><figcaption><em>None technical and popularized talks about the migration, done during department meetings</em></figcaption></figure><p><strong>Establish clear communication</strong> channels and regularly check in with the team to ensure that the project is on track.</p><ul><li><strong>Communicate quickly</strong>, and <strong>avoid funnel symptoms</strong> (even if it’s real): you need to find <strong>intermediary steps</strong> that could be quickly <strong>valuable</strong> and <strong>create reinsurance</strong></li><li>Hold <strong>regular status</strong> meetings to discuss progress and any issues that may arise. Could be ad hoc ones if async communication is good.</li><li>Encourage <strong>open communication between team members</strong> and provide opportunities for feedback and suggestions.</li><li><strong>Communicate</strong> not only at the engineering level but <strong>also at the business/product level</strong>, so they could anticipate the new possibilities offered by the migration.</li><li><strong>Popularize</strong> what you are currently doing and the expected outcomes</li></ul><p><em>🔍 Concrete “real life” example:</em></p><ul><li>We met external team leads to discuss this kind of migration and hired a freelancer</li><li>JIRA board to follow main tasks and save secondary tasks for later</li><li>Next 13 was released during the migration with its app router</li><li>Communication example with weekly reporting</li><li>Bad communication was done about how teams could help during this step</li></ul><p>Previous part is about <a href="https://medium.com/jobteaser-dev-team/conduct-a-technical-migration-within-less-than-6-months-part-1-c1215057a515"><strong>“Buying time and sponsorship”</strong></a><br>Next and last part is about <a href="https://medium.com/jobteaser-dev-team/conduct-a-technical-migration-within-less-than-6-months-part-3-final-7968c007f770"><strong>“Preserve the Developer’s Experience”</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4f2cd1feea4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jobteaser-dev-team/conduct-a-technical-migration-within-less-than-6-months-part-2-c4f2cd1feea4">Conduct a technical migration
within less than 6 months — Part 2</a> was originally published in <a href="https://medium.com/jobteaser-dev-team">JobTeaser Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Conduct a technical migration
within less than 6 months — Part 1]]></title>
            <link>https://medium.com/jobteaser-dev-team/conduct-a-technical-migration-within-less-than-6-months-part-1-c1215057a515?source=rss----bd77d16a0035---4</link>
            <guid isPermaLink="false">https://medium.com/p/c1215057a515</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[nextjs]]></category>
            <category><![CDATA[developer-experience]]></category>
            <dc:creator><![CDATA[Fabien RAYNAUD]]></dc:creator>
            <pubDate>Mon, 02 Dec 2024 09:46:26 GMT</pubDate>
            <atom:updated>2024-12-02T10:15:59.350Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Conduct a technical migration<br>within less than 6 months — Part 1</strong></h3><h4>based on Next.js migration, performed at JobTeaser</h4><p>What is this 3-parts article about:</p><ul><li><strong>sharing my own experience</strong> with handling a technical migration</li><li>📖 some <strong>additional elements</strong> that I find relevant and could help (identified with 📖 emoji) but not personally tested</li></ul><p>What this presentation is NOT about:</p><ul><li><strong>technical step-by-step</strong> Next.js migration</li><li><strong>the perfect and only way</strong> to do a migration</li></ul><p>I tried to be as exhaustive as possible, but I didn’t even complete all the points personally. So the <strong>presentation is a mix of my personal experience and all the things that could help to do it “qualitatively”. </strong>Pick what is necessary for you and your ecosystem.</p><p>3 main parts:</p><ol><li><strong>▶️ Buying time and sponsorship</strong> to be able to perform such a migration = creating a good environment to navigate easily to success</li><li><a href="https://medium.com/jobteaser-dev-team/conduct-a-technical-migration-within-less-than-6-months-part-2-c4f2cd1feea4"><strong>Organize your project</strong></a> = keep an eye on and communicate about what was done and what needs to be done</li><li><a href="https://medium.com/jobteaser-dev-team/conduct-a-technical-migration-within-less-than-6-months-part-3-final-7968c007f770"><strong>Preserve the developer experience</strong></a> = ensure use, sustainability, and future maintenance of the migration</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nFPqhW7DkxDgSUPR" /><figcaption><em>Timeline of the migration</em></figcaption></figure><p><em>🔍 Concrete “real life” timing</em></p><ul><li>2 weeks to build an audit in August, done in autonomy and presented to frontend chapter, management, and tech leadership</li><li>3 months to put a minimalist ecosystem into production, from November to January, done mainly in autonomy with the help of the Devex and Infra team, and also from hyped frontend engineers</li><li>2 months for the release of the first “real life” use case, from February to March, by a squad of 3 engineers + 2 others squads starting their own features on the new stack</li></ul><h3>Buying time and sponsorship to do the migration</h3><p>The first step is buying <strong>time</strong> and <strong>sponsorship</strong> for the migration. This is essential to ensure that <strong>the necessary resources and funding will be available</strong> for the initiative, aka <strong>the right environment</strong>, to perform the migration properly. This part is more about <strong>digging into the pros and cons</strong> of this migration and <strong>being able to explain it</strong> so people will <strong>acknowledge your idea</strong>.</p><p><strong>Communicate the benefits of the migration</strong>, clearly, to stakeholders and decision-makers, such as improved performance, scalability, or security.</p><ul><li>Provide <strong>concrete examples</strong> (best from competitors) of <strong>how the migration will improve</strong> the project, such as reduced downtime, increased speed, reduced TTM (Time To Market), or improved user experience.</li><li>Find <strong>someone who will sponsor you</strong> in high-level meetings in the company (CTO position is the best for that but non-tech people could a be great asset too). Convince (harass) people until you find somebody! It will be important when you will face moments of doubt (and it will happen).</li></ul><p>For that, you need <strong>a solid plan</strong> to be able to argue and then build your plan.</p><ul><li>Could be an audit, a Design Document, a Request For Change, an Architecture Design Record, or a mix</li><li>For me, my base documentation was a complete study about “Frontend <strong>state of art</strong>” and “Frontend <strong>technical debt</strong> at JobTeaser”. I performed it in August in 2 weeks.</li><li>It allows me to <strong>study competitor</strong>s, analyze our technical stack versus the state of the art, and create <strong>some data to validate the vision</strong>.</li></ul><p>Available in the document:</p><ul><li>Results and analysis from an internal survey about pain points regarding frontend stuff (whole department = not only engineers)</li><li>Cleaning concepts about our current <strong>technical debt</strong>, what is considered as “legacy stuff”, like Rails pages and the recent features, build with React</li><li>Study different module audiences and number of users, to be able to prioritize incoming changes</li><li>Details about our last initiatives regarding frontend autonomy (= being able to build AND run React features outside our Rails monolith)</li><li><strong>Comparison</strong> between our current technical stack VS state of the art VS our competitors</li><li>Performances comparison with our competitors (Core Web Vitals dashboard + Lighthouse report) on main pages</li><li>Specific part about improving our internal Design System/UI kit library (Figma/React sync, Storybook documentation, Visual Test regression, Accessibility…)</li><li><strong>Proposal plan with several initiatives</strong> with priority, sizing, objective(s), why, main tasks, and success metrics for each</li></ul><p>Create <strong>a migration plan</strong> that outlines the <strong>timeline</strong> (even macro), <strong>budget</strong>, and expected <strong>outcomes</strong> (again) of the project.</p><ul><li>Include <strong>a breakdown of the tasks involved in the migration</strong> and the resources required to complete them.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*elLCvZZv_wxmTYKFtSA5UA.png" /><figcaption><em>One initiative from the audit</em></figcaption></figure><ul><li>Provide <strong>regular updates on the progress</strong> of the migration and <strong>any changes to the plan</strong>.</li><li>📖 <strong>Identify</strong> and address any potential roadblocks or <strong>risks</strong> that may arise during the migration process.</li><li>Conduct a thorough risk assessment to <strong>identify potential issues</strong>, such as compatibility problems, data loss, or security vulnerabilities.</li><li><strong>Develop a contingency plan</strong> to address any potential issues that may arise.</li></ul><p><em>🔍 Concrete “real life” example:</em></p><ul><li>CTO was already convinced because did this kind of migration in his previous company</li><li>Output initiatives were:<br>- more quality/compliance monitoring on frontend side<br>- UI-kit improvements<br>- local env improvements<br>- Next.js and TypeScript setup</li><li>Data were not so fresh when the migration starts</li><li>Some actions were outdated (sometimes because already done!)</li></ul><p>Next part is about <a href="https://medium.com/jobteaser-dev-team/conduct-a-technical-migration-within-less-than-6-months-part-2-c4f2cd1feea4"><strong>“Organize your project”</strong></a><br>Last part is about <a href="https://medium.com/jobteaser-dev-team/conduct-a-technical-migration-within-less-than-6-months-part-3-final-7968c007f770"><strong>“Preserve the Developer’s Experience”</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1215057a515" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jobteaser-dev-team/conduct-a-technical-migration-within-less-than-6-months-part-1-c1215057a515">Conduct a technical migration
within less than 6 months — Part 1</a> was originally published in <a href="https://medium.com/jobteaser-dev-team">JobTeaser Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spring Cleaning for z-indexes]]></title>
            <link>https://medium.com/jobteaser-dev-team/spring-cleaning-for-z-indexes-42beba8e526a?source=rss----bd77d16a0035---4</link>
            <guid isPermaLink="false">https://medium.com/p/42beba8e526a</guid>
            <category><![CDATA[css]]></category>
            <category><![CDATA[z-index]]></category>
            <category><![CDATA[stacking-context]]></category>
            <dc:creator><![CDATA[Jean-Jacques Royneau]]></dc:creator>
            <pubDate>Tue, 05 Nov 2024 10:06:48 GMT</pubDate>
            <atom:updated>2024-11-05T10:06:48.197Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*CxTRU6ST8uiyT51lDFAMfQ.png" /></figure><p>Managing z-index is a classic challenge for front-end developers. As a project grows, display conflicts and bugs related to z-index can quickly turn into serious headaches.</p><p>I recently took some time to clean up the z-indexes in one of our projects.<br>In this article, I’ll share the approach I used for this cleanup (while also laying the foundation for keeping things manageable in the long run).</p><h3><strong>1. Analyzing the existing z-indexes</strong></h3><p>The first step was to perform a thorough audit of all the z-index declarations in the project. This gave me an overview of how many z-indexes we had and what values were being used.</p><p>Here’s what I found:<br>- August 2023: 57 declarations with 8 distinct values.<br>- February 2024: 87 declarations with 19 distinct values.</p><p>This evolution showed me that the use of z-indexes had increased significantly. And there was no reason to believe it would stop.</p><p>With that in mind, my goal became to centralize and standardize these values.</p><h3><strong>2. Centralizing z-indexes with CSS Custom Properties</strong></h3><p>I centralized all the z-index values in a single file using CSS custom properties.</p><p>This allowed me to have an overview of the different layers of elements on the page, helping me get a (slightly) clearer picture of the task ahead.</p><p>If you’d like to get an idea (or maybe scare yourself a bit), here’s what that first file looked like:</p><pre>:root {<br>  - z-index-second-basement: -2;<br>  - z-index-basement: -1;<br><br>  - z-index-ground: 0;<br><br>  - z-index-floor: 1;<br>  - z-index-second-floor: 2;<br>  - z-index-third-floor: 3;<br><br>  - z-index-job-ads-secondary-filters--selects: 10;<br>  - z-index-job-ads-results-sort: 10;<br>  - z-index-job-ads-primary-filters: 20;<br><br>  - z-index-fo-header: 30;<br>  - z-index-above-fo-header: 31;<br>  - z-index-fo-header--dropdowns: 100;<br><br>  - z-index-career-center--login-modal--autocomplete-list: 99;<br><br>  - z-index-bo-drawer-backdrop: 999;<br>  - z-index-bo-drawer: 1000;<br><br>  - z-index-feature-env-switcher: 9999;<br><br>  - z-index-notifications-panel: 10000;<br><br>  - z-index-msw-tools-panel--open-button: 99999;<br>  - z-index-msw-tools-panel: 999999;<br>}</pre><h3><strong>3. Understanding Stacking Contexts</strong></h3><p>While trying to make sense of each of these z-index values, I had to revisit an important CSS concept: stacking contexts.</p><p>Z-indexes are just a tool tied to this concept.</p><p>In my experience, z-indexes can be tricky to master because they interact with stacking contexts in TWO ways<br>- a z-index can <strong>modify</strong> the stacking order (which is fine, that’s its purpose)<br>- a z-index can <strong>create</strong> a new stacking context (and this is where things can go wrong)</p><p>The real challenge isn’t just about the z-index values themselves, but also the <strong>unintentional</strong> creation of stacking contexts. Many z-indexes end up creating stacking contexts that weren’t needed, which complicates the layering of elements.</p><p>For the next phase of my work, I paid special attention to these unintentional stacking context creations.</p><h3><strong>4. Removing, Reducing, and Grouping</strong></h3><p>With a better understanding of stacking contexts, I was able to identify unnecessary or redundant z-index values.</p><p>Quite often, the natural order of elements in the DOM is enough to ensure correct layering without needing a z-index.</p><p>So, I <strong>removed</strong> z-indexes where elements were already stacking correctly based on their HTML order (and in some cases, I even reordered the HTML itself to avoid needing a z-index 😉).</p><p>I also <strong>reduced</strong> the value of some z-indexes when they were higher than necessary.</p><p>This step is crucial because I believe that the lower and more limited z-index values are, the more we can control them over time.</p><p>This reduction also helped me achieve one of my goals: ensuring all z-index values were increments of 1 (I mean: -1, 0, 1, 2, 3…).</p><p><strong>Why increments of 1? Because you need to control your z-indexes.</strong></p><p>No more randomly picking a value (“There was space between 20 and 30, so I went with 27!”).</p><p>Finally, I <strong>grouped</strong> CSS custom properties that had identical values, which allowed me to identify some useful abstractions.</p><h3><strong>5. Defining a Visual Priority Order for Recurring Elements</strong></h3><p>We defined a visual hierarchy for elements that frequently appear on a page: menus, modals, dev tools, and notifications.</p><p>The order we chose is as follows:<br>page content &lt; website menus &lt; modals &lt; dev tools &lt; notifications.</p><p>From the previous work, the highest z-index value that belonged to page content (i.e. not including elements that appear on top) was 3.</p><p>This allowed me to make the priority logic dynamic:</p><pre>:root {<br>  …<br>  --z-index-website-menu: 4;<br><br>  --z-index-modal: calc(var(--z-index-website-menu) + 1);<br><br>  --z-index-dev-tools: calc(var(--z-index-modal) + 1);<br><br>  --z-index-notifications-panel: calc(var(--z-index-dev-tools) + 1);<br>}</pre><p>Now, if the value of --z-index-website-menu ever needs to change (to decrease, of course!), all the following values will update automatically, without developers having to think about it.</p><h3><strong>6. Defining Utility Values</strong></h3><p>At the end of step 4, I had grouped together the CSS custom properties that had identical values. Some of these values turned out to be the recurring elements we discussed earlier.</p><p>Others didn’t correspond to a specific type of element but were often used because they’re <strong>useful</strong> for fine-tuning the stacking order in specific contexts.</p><p>I took these values to define utility custom properties:</p><pre>:root {<br>  --z-down-in-the-current-stacking-context: -1;<br>  <br>  --z-reset-in-the-current-stacking-context: 0;<br><br>  --z-up-in-the-current-stacking-context: 1;<br>  …<br>}</pre><p>These values are used to manage finer adjustments in any current stacking context.</p><h3><strong>7. Discouraging Hardcoded Values</strong></h3><p>To ensure this approach remains sustainable, I added a rule to our CSS linter, Stylelint, which prevents the use of hardcoded z-index values. This rule encourages developers to either use the centralized variables or define new ones when needed.</p><pre>// .stylelintrc<br>&quot;rules&quot;: {<br>  // ...<br>  &quot;scale-unlimited/declaration-strict-value&quot;: &quot;z-index&quot;<br>  // ...<br>}</pre><p>However, be careful: it should be understood as a <strong>guideline</strong> and not a rule. It’s fine to silence the linter and use a hardcoded value. You just have to do it consciously.</p><h3><strong>Final thoughts</strong></h3><p>While all these steps seem to flow nicely in theory, that’s not always the case in real life.<br>In practice, I went back and forth between the different steps multiple times before reaching a satisfactory result. 😉</p><p>By centralizing values, eliminating unnecessary stacking contexts, and reducing the number of different z-index values, I was able to simplify and make z-index management more maintainable. This process lightened the codebase and made the handling of element layering more predictable.</p><p>If this approach seems helpful to you, I encourage you to take inspiration from it and apply it to your own projects. A well-structured z-index management strategy can make a real difference, making maintenance easier and avoiding subtle bugs related to element stacking.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42beba8e526a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jobteaser-dev-team/spring-cleaning-for-z-indexes-42beba8e526a">Spring Cleaning for z-indexes</a> was originally published in <a href="https://medium.com/jobteaser-dev-team">JobTeaser Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making sense of TypeScript generics]]></title>
            <link>https://medium.com/jobteaser-dev-team/making-sense-of-typescript-generics-6b830e66eeff?source=rss----bd77d16a0035---4</link>
            <guid isPermaLink="false">https://medium.com/p/6b830e66eeff</guid>
            <category><![CDATA[typescript]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[generics-in-typescript]]></category>
            <dc:creator><![CDATA[Clément Gateaud]]></dc:creator>
            <pubDate>Thu, 04 Jul 2024 16:12:05 GMT</pubDate>
            <atom:updated>2024-07-05T09:09:20.142Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CtIyJH2hOCDuLgjAG2j3JA.png" /></figure><p>TypeScript significantly improved JavaScript development with enhanced code reliability and maintainability, ensuring fewer runtime errors and greater productivity for developers.</p><p>But when you start working with it, the syntax can sometimes feel overwhelming.</p><p>One of the most intimidating features in TypeScript is “generics”. Look at the example below 🤯 aren’t you afraid ?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CvAXU2ZEmaU0rTS6m4xMkQ.png" /><figcaption>Types declarations from @tanstack/react-query (library) that can feel overwhelming</figcaption></figure><p>But don’t worry, we will go step by step and once you understand generics, you’ll see how powerful they are.</p><h3>The best any alternative</h3><p>Imagine this very simple JavaScript function that returns the first element of an array. (I agree, this function is not very useful, but it’s for the sake of explanation).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g-ZDjlxsD-5zWUbHI7PVqg.png" /></figure><p>How could we type it? 🤔 What would be the type of the array? Well, in this function it could be an array of anything (a string, a number, an object). So it could be tempting to type it like this, using the type whose name must not be spoken: any 😱</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3aU_hkvBonNqUhC-TJ6G-w.png" /></figure><p>But this is very bad typing. Why? Because TypeScript won’t be able to properly infer the return type of this function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vmC4plRaQLweS9BLrIaj9A.png" /></figure><p>So maybe we could specify all the different types it could be?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7nsZJGZFHuOAqO3zAVjzrA.png" /></figure><p>Not a good idea. First, because it would force you to maintain a huge list of all the possible types for this function. But more importantly, because it doesn’t solve our inference issue. TypeScript is not smart enough to guess the type when calling the function.</p><p>So maybe we could use casting with as to help TypeScript?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ndnqqR6BXYdltLMulQg9YA.png" /></figure><p>But that’s cheating! With this type assertion, you are telling TypeScript: “Stop type checking and trust me, I know what I’m doing”. This weakens type safety which is the core purpose of using TypeScript.</p><p>Don’t do that! There’s a way safer and cleaner way to type this. <br>Guess how it’s named? ✨ Generics✨</p><h3>Introducing generics</h3><p>Let’s keep our previous example.</p><p>What we want to tell our function is: <em>“You will take as a parameter an array of SOMETHING, and you will return a SOMETHING element”</em>.</p><p>That’s exactly what generics are all about.</p><p>It will look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H4LUXoQl765v-cSl_UJBwA.png" /></figure><p>Let’s zoom into parts of this code:</p><p><strong>🟣</strong> Instead of typing our parameter as any[] like before, we define it using a generic: we will name it SomeType. <strong>A generic is not a type itself but a type parameter, a placeholder for a type that will be specified when the function is called</strong>. You can name it whatever you want (as long as it&#39;s not a reserved keyword or an already imported type name). You will see that we often use a single letter for generics, like T. We are telling Typescript “the function will take as parameter an array of some type”.</p><p>🟢 This specifies the return type of the function. We are telling Typescript “the function will return an element of the same type as the elements in the parameter’s array”.</p><p><strong>🟡</strong> This part just before the function parentheses is listing the generics we are about to use for this function, inside angle brackets. Here we will only be using one generic (SomeType), but I’ll show you later that we can have multiple.</p><p>How we typed the function using a generic just means: <em>“The parameter is an array of some type, and the function will return something of this same type”</em>.</p><p>A more common way of writing this function would be with T (so that the syntax looks less cumbersome).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w7ozrMnGNrmhGpa7bReelg.png" /></figure><p>I hope this introduction could help you grasp the essence of generics.</p><h3>Generics are everywhere</h3><p>If you’ve already worked with Typescript, you’ve probably used generics without even knowing it. For example, consider the querySelector function in the DOM API:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wZXRhhc5LFTklnNkyTjCeQ.png" /></figure><p>In this example, querySelector uses generics to ensure that the returned element is of type HTMLElement. This type information allows TypeScript to provide accurate autocompletion and type checking for the returned element. In TypeScript, native types like HTMLElement are defined through type declarations in the standard library, specifically in lib.dom.d.ts.</p><p>Here’s what using the generics allows you to do with the element:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x1gqeyA5ezcuqkdRHK6qxA.png" /></figure><p>Here’s the difference without using generics:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hdVu2UqUQkj-GppaWqBvyg.png" /></figure><p>Without the generic type, TypeScript would not know the specific type of element, you would not get helpful autocompletion and you would get an error (fixable with casting, but still not a good idea).</p><p>Using generics allows TypeScript to understand exactly what kind of element you’re working with, providing better type safety and developer experience. So it’s no surprise generics are widely used in native APIs and libraries.</p><h3>Going deeper with generics</h3><p>Generics are not limited to simple functions. They can be used in various ways to create more flexible and reusable code.</p><p>Let’s take a new example that is closer to a real-world issue than just getting the first element of an array. We’ll work with API responses.</p><p>Let’s take this code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gBd1mWWX2jHmza5FC1b6OA.png" /></figure><p>It defines an ApiResponse type that will always return a number status, a string message, and some data that can be anything.</p><p>But as we saw in the first part of this article, using any is not a good idea as it prevents TypeScript from inferring our types properly.</p><p>Let’s use generics to improve that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e1I3Al63_LKpngnNsJdkZg.png" /></figure><p>Let’s zoom a bit on this code.</p><p>🟡 First, you notice a new syntax here. It’s a generic type. We are defining a type that is using a generic. It works kind of the same as the function example we saw before:</p><ul><li>We are listing the generics used, inside angle brackets</li><li>We are using this generic inside the type definition</li></ul><p>🟣 So now, every time we will use this type, we will have to precise in angle brackets what is the type of T. We are specifying that the data of our API response for this fetch will be of type { name: string; age: number }. This is a type argument.</p><p>And now, as our fetchData function is well typed, TypeScript will be able to properly infer it’s return type.</p><h3>Constraining generics</h3><p>Sometimes, you want your generic to not accept any type, but to have more constraints.</p><p>Keeping our example of API response, we might want our data to always be an object and nothing else.</p><p>This is possible with the extends keyword.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WJ0MWOpi5L4tdnswBIykCA.png" /></figure><p>If we try to call the function with something else than an object as a type argument, TypeScript will throw an error.</p><p>We can even go further by constraining an object with specific properties. Let’s say we want our API data to be an object that always contains an id:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9i8sUayW7noWq1eWqrIDmg.png" /></figure><h3>Default types for generics</h3><p>Another cool feature of generics is the ability to give them a default type.</p><p>In all the examples we saw so far, we had to precise the type argument when using the generic type, like: ApiResponse&lt;{ name: string; age: number }&gt; .</p><p>But if you often use the same type for your generic, you can specify a default value in the type definition, with =.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UVkadPA17OM5td0C-ZfYeg.png" /></figure><p>You can even combine it with a constraint:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q7ZD1S25FoFKyk0xL9v81g.png" /></figure><h3>Multiple generics</h3><p>Sometimes, one generic is not enough, and you need 2 different ones (or more).</p><p>Inside the angle brackets, you can just list all the generics you will use, separated with commas. And when you call a function, do the same for your types arguments.</p><p>Let’s see an example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YVmHbHsNMQaoVHo1QMFgfA.png" /></figure><p>Generics can be intimidating at first, but once you grasp how they work, you quickly realize their power and understand why they are at the cornerstone of the TypeScript ecosystem.</p><p>By allowing functions and components to operate with a variety of types while maintaining strong type safety, TypeScript generics significantly enhance our ability to write robust and maintainable code, while making the developer experience fantastic.</p><p>Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6b830e66eeff" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jobteaser-dev-team/making-sense-of-typescript-generics-6b830e66eeff">Making sense of TypeScript generics</a> was originally published in <a href="https://medium.com/jobteaser-dev-team">JobTeaser Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[7 tips to survive as a junior developer]]></title>
            <link>https://medium.com/jobteaser-dev-team/7-tips-to-survive-as-a-junior-developer-b15b47586fb4?source=rss----bd77d16a0035---4</link>
            <guid isPermaLink="false">https://medium.com/p/b15b47586fb4</guid>
            <category><![CDATA[junior-developer]]></category>
            <category><![CDATA[careers]]></category>
            <category><![CDATA[junior]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[developer]]></category>
            <dc:creator><![CDATA[Clément Gateaud]]></dc:creator>
            <pubDate>Tue, 23 May 2023 14:06:01 GMT</pubDate>
            <atom:updated>2023-05-23T14:06:01.705Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Illustration of a boy entering a maze" src="https://cdn-images-1.medium.com/max/1024/1*tHgmdkN3adeL_rv3MaDAGg.png" /><figcaption>©<em> Midjourney</em></figcaption></figure><p>Whether you’re fresh out of school or transitioning from another career, landing a job as a junior developer is a great accomplishment. But it’s only the first step on your path to a successful career.</p><p>Finding your marks in this new position is the next big challenge. In this article I’ll share with you 7 tips I would have loved to receive when I started as a junior developer, one year ago.</p><h3><strong>1. Learn to walk before you run</strong></h3><p>During your first days as a “professional” developer, you will most likely feel overwhelmed by the size and complexity of your company’s codebase. “What are all those files? Do we really need millions of lines of code to run our application?”. It will for sure look very different from the small side projects you’ve worked on so far.</p><p>Don’t bother trying to understand the whole codebase and how everything is structured. Just focus on what you need for your current task. If your first task involves a minor fix to a specific behavior, focus on the file that’s responsible for that behavior and forget about the rest. It’s already a significant challenge to understand the necessary code for your initial task, especially since you’ll also be learning about the company’s processes, people, and products.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FwTCmLy60bq3dWGoHPF%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FwTCmLy60bq3dWGoHPF%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FwTCmLy60bq3dWGoHPF%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/12a11f60dbd4c74090fdadd0e0f0fded/href">https://medium.com/media/12a11f60dbd4c74090fdadd0e0f0fded/href</a></iframe><p>Don’t get me wrong, getting a bigger picture is important, but take it one step at a time. You will get more comfortable with the rest of the codebase over time. If you try to assimilate everything, you will assimilate nothing. Choose your fights. Time will do the rest.</p><h3><strong>2. Getting stuck is great (but not for too long)</strong></h3><p>You’ll get stuck on some issues when coding (just like every developer in the history of mankind). It may sound counterintuitive but getting stuck is a chance. Why? Because it is in those situations that you’ll learn the most.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2Fe7C2wQpE2UwvqZCdzh%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Fe7C2wQpE2UwvqZCdzh%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2Fe7C2wQpE2UwvqZCdzh%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/2776dac640fcf556b334de07293fd1e4/href">https://medium.com/media/2776dac640fcf556b334de07293fd1e4/href</a></iframe><p>Do not ask for help right away. Do your best to understand how things work and try different solutions. But if you see that you’ve been going in circles for 30 minutes and you don’t see what next you could try, it’s time to ask for help.</p><p>Other developers from your team will probably be happy to help and they might unblock the situation in a snap! The solution they will help you find is important, but the way they will find this solution will be even more valuable for you. Don’t hesitate to ask them to explain their reasoning.</p><h3><strong>3. Master the art of searching</strong></h3><p>The web is full of resources to help you solve your issues, it would be silly not to take advantage of it. After all, what is a developer if not an expert in googling issues and copy-pasting code from Stack Overflow? This is for the joke but honestly, searching online is an essential skill. Nothing to be ashamed of.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FNcx0rtwxScNt9lJztD%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FNcx0rtwxScNt9lJztD%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FNcx0rtwxScNt9lJztD%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="297" frameborder="0" scrolling="no"><a href="https://medium.com/media/d2f77f2c74f5cbc1383578a72f66cc85/href">https://medium.com/media/d2f77f2c74f5cbc1383578a72f66cc85/href</a></iframe><p>As I write this article, there is currently quite a hype around the new AI tools, especially ChatGPT. And I think it can be a game-changer for junior developers. Ask it to explain some code, write functions or even draft some unit tests for you, the outcomes can be stunning! Be as precise as possible to get the best answers and use its conversational capabilities to ask for corrections or precisions.</p><p>But generative AI tools must be used wisely. Most importantly, make sure you don’t share sensitive data in your prompts. Use them sparingly to avoid regressing intellectually, and always keep a critical eye on their answers. These tools should not make you forget the value of asking other developers. Your colleagues will often be the best solution as they know about your specific codebase.</p><h3><strong>4. You’re not just here to code</strong></h3><p>Being a developer is not just about writing code. It is a common misconception that a developer’s only responsibility is churning out lines of code, but in reality, there is much more to the job.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FaNqEFrYVnsS52%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FaNqEFrYVnsS52%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FaNqEFrYVnsS52%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/a6ddde4268cec451227a99fb1ae37bc1/href">https://medium.com/media/a6ddde4268cec451227a99fb1ae37bc1/href</a></iframe><p>You have the opportunity to contribute to the product discussions and bring a valuable point of view. Don’t hesitate to share your ideas and suggestions, even if you are junior and new to the company. Your fresh perspective and unique insights may provide clever ideas or identify potential opportunities for improvement.</p><p>In addition to that, you need to take ownership of your code and subjects. Make sure to communicate regularly your progress and any potential issues to your team and stakeholders. I’ll also recommend you, if you have the opportunity, to present and demonstrate to your stakeholders the features you’ve been working on. It is an excellent opportunity to showcase your work, receive valuable feedback, and build trust with your colleagues.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FRa6oLi1NYiCGjMXyQy%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FRa6oLi1NYiCGjMXyQy%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FRa6oLi1NYiCGjMXyQy%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/a8807abe4755623ac66dc99c33b3b82d/href">https://medium.com/media/a8807abe4755623ac66dc99c33b3b82d/href</a></iframe><h3><strong>5. Learn to grow, and share to learn</strong></h3><p>As a developer, keeping your skills up to date is crucial as the tech industry is constantly evolving. As important as it is, it’s also essential to approach it strategically. During your first months, I advise you to focus on mastering the company’s stack and familiarizing yourself with the codebase.</p><p>Once you’ve established a solid foundation, you can then expand your knowledge and skills to new areas (learning a new language, discovering a library, a new tool, etc.). In some companies like JobTeaser, you’ll be lucky enough to have specific time slots dedicated to learning and sharing.</p><p>If you have the opportunity, I highly advise you to share your learnings, even if you are a junior. You can even learn something just for the occasion of this sharing. You can make a presentation about a specific subject you’ve learned, or even write an article on your company’s blog 😉. Sharing is a great way to consolidate your knowledge and gain credibility. Even the most senior developers can get to learn something from you.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F3orieVM9saOSlxu2cg%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2F3orieVM9saOSlxu2cg%2Fgiphy.mp4&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2F3orieVM9saOSlxu2cg%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/66df4e1c27c08e4b4ee1c9bbe13ef43f/href">https://medium.com/media/66df4e1c27c08e4b4ee1c9bbe13ef43f/href</a></iframe><h3><strong>6. Code can go beyond the office</strong></h3><p>As with the learning part, I think it’s important to keep having side projects, but only when the first wave of panic is over and you feel at ease with your job’s stack and codebase.</p><p>Coding side projects outside of work will probably differ from the code you are doing at work. Starting a project from scratch is a different feeling than optimizing a big codebase. Keeping these two different facets of coding is to me the best way to keep your passion for coding alive. On top of that, it is a great way to try new technologies and keep learning to become an even greater developer.</p><p>I’m not telling you that you must code every night after work or every week. The best rhythm I found for me is to do a small side-project every 3–6 months, but it depends on you. Find the right balance so that you don’t get bored with what you do at work and you don’t do an overdose of code. Coding must remain a pleasure, and if you don’t feel at all like coding outside the office, there is no shame about that, it’s totally fine.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F3EiNL9hmebH6GgNy75%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2F3EiNL9hmebH6GgNy75%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2F3EiNL9hmebH6GgNy75%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/58caa628cb3ed782f3511b034ebe159f/href">https://medium.com/media/58caa628cb3ed782f3511b034ebe159f/href</a></iframe><h3><strong>7. Don’t hold yourself back</strong></h3><p>Being a junior developer is not always a long quiet river. You will sometimes feel light-years away from reaching the level of senior developers. This is totally normal. Remember that nobody expects you to be an expert.</p><p>It often happens that junior developers get intimidated by years of experience and wonder how they could contribute to a team where all members are way more senior. But good ideas can come from anyone. And so do bad ideas. Junior engineers can propose ingenious solutions that everyone missed. And more experienced engineers sometimes suggest insane ideas that are way too complex and over-engineered.</p><p>So don’t be afraid to speak up, share your ideas, ask questions and challenge the status quo. Your fresh perspective and willingness to learn can be valuable assets to the team. And remember, it’s not about being the best developer in the room, it’s about collaborating with others to create great software.</p><p>Be proud of what you do. Celebrate your accomplishments. Communicate your frustrations. Keep pushing. And everything is gonna be fine 💪</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F12XDYvMJNcmLgQ%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2F12XDYvMJNcmLgQ%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2F12XDYvMJNcmLgQ%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="325" frameborder="0" scrolling="no"><a href="https://medium.com/media/3036ebf1372b812578b622f81b6c5de0/href">https://medium.com/media/3036ebf1372b812578b622f81b6c5de0/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b15b47586fb4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jobteaser-dev-team/7-tips-to-survive-as-a-junior-developer-b15b47586fb4">7 tips to survive as a junior developer</a> was originally published in <a href="https://medium.com/jobteaser-dev-team">JobTeaser Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data replication across backend services with Kafka and Protobuf]]></title>
            <link>https://medium.com/jobteaser-dev-team/data-replication-across-backend-services-with-kafka-and-protobuf-cad1d9fd9f90?source=rss----bd77d16a0035---4</link>
            <guid isPermaLink="false">https://medium.com/p/cad1d9fd9f90</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Emmanuel Joubaud]]></dc:creator>
            <pubDate>Fri, 14 Apr 2023 10:01:57 GMT</pubDate>
            <atom:updated>2023-04-14T10:01:56.929Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mlj2AU5OfqdmoOS3hpgqJA.png" /></figure><p>The Jobteaser application contains a lot of different relatively independent modules to help universities provide career guidance to students: a job board, a career event management system, a career advice appointment management system…</p><p>When we decided to migrate our application’s backend from a monolith to a service-oriented architecture, we strived to keep each module as isolated as possible from the others in the event of an incident. If the career appointment system was down, students should still be able to browse and apply to job ads.</p><p>That isolation is achieved through what we’ve called our <a href="https://medium.com/jobteaser-dev-team/layers-vs-silos-a-tale-of-2-microservice-architectures-50c21016e9b1">Silos architecture</a>. The gist is we avoid synchronous API calls between backend services, and prefer asynchronous data integration between services.</p><p>There are a lot of different ways to implement asynchronous communication between services, and few companies share the details of theirs so we had to figure out a lot of stuff on our own. Now that we’ve refined our system, we thought we’d share the details of our approach, based on <strong>simple data replication using Kafka and Protobuf.</strong></p><h3>How it works</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mlj2AU5OfqdmoOS3hpgqJA.png" /><figcaption>When a write operation happens, data is changed first on the Source of Truth service, who then publishes it to a Kafka topic that consumers can subscribe to, in order to replicate their own local copy in their database</figcaption></figure><h4>Source of Truth and consumers</h4><p>Every table in our data model has an owner service, also called the Source of Truth (SoT) for that data, or the producer. For instance, the job board service may be the SoT for the job ad data.</p><p>The owner service has 3 main responsibilities:</p><ul><li>it receives and validates all write requests for the data it owns (Create, Update, Delete)</li><li>it stores the data to its own database, i.e. the Source of Truth, the authoritative state of the data</li><li>if it’s data that other services might want to access, <strong>it publishes its latest state as a message into a Kafka topic</strong></li></ul><p>Consumer services that are interested in accessing a given table can then subscribe to the Kafka topic for that table, and receive a message for each state update, so they can keep a local copy up-to-date in their own database by replaying the changes.</p><h4>Protobuf messages</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9moPzjrru7LecTBsCFiyCQ.png" /><figcaption>The Protobuf serialization format is statically typed. The producer and the consumer both need access to the schema to respectively serialize and deserialize the data.</figcaption></figure><p>Each message contains the whole new state of the object, serialized using the Google Protobuf serialization format.</p><p>Protobuf is a statically typed format. The producer and the consumer both need access to a shared schema definition that tells them how to serialize and deserialize the data.</p><p>In a given Kafka topic, every message is always serialized using the same Protobuf message type, so that all the objects have a predictable format.</p><p>That creates an API contract, effectively turning our Kafka topics into asynchronous APIs. We version those APIs by adding a version into each data replication topic and each protobuf message name. For instance the jobad.v1.main topic only contains messages of Protobuf type jobads.v1.JobAd . Protobuf graciously handles non-breaking changes like adding new fields, but if we need to perform breaking changes on our API format, we’ll have to create a new topic and a new message.</p><h4>Double-write and transactions</h4><p>So whenever we make a database change to a replicated table, we want to make sure we also push a message to Kafka.</p><p>But we can’t push them both atomically. No transaction can span both the database and Kafka.</p><p>Instead, we need to pick our poison between two types of possible discrepancies, depending on whether we first commit the change to the database or to Kafka:</p><pre>// option 1: push to Kafka within the transaction<br>database.begin_transaction()<br>database.save(record)<br>kafka_topic.produce(record.to_proto()) // Failure here would rollback transaction<br>// Failure here would cause the change to be pushed to Kafka but not the DB<br>database.commit_transaction()<br></pre><pre>// option 2: push to Kafka after the DB transaction<br>database.begin_transaction()<br>database.save(record)<br>database.commit_transaction()<br>// Failure here would cause the change to be pushed to the DB but not Kafka<br>kafka_topic.produce(record.to_proto())</pre><p>As you can see, we’re facing a conundrum:</p><ul><li>With option 1, failures will be rare but more consequential when they happen: It takes the DB connection to drop precisely between the push to Kafka and the transaction commit to have a problem, but then you basically end up pushing false information into Kafka, data that doesn’t reflect the Source of Truth, and you have no easy way to recall it.</li><li>With option 2, failures are much more likely. All it takes is for Kafka to be down and your change will be stored in the DB but won’t make it to Kafka. But this is also easy to fix: you can just reproduce the missing message at a later time. You haven’t pushed any wrong information, just delayed its transmission.</li></ul><p>We opted for option 2, because we care more about the accuracy of information than its timing.</p><p>Note that there are other options available, like the <a href="https://microservices.io/patterns/data/transactional-outbox.html">outbox pattern</a>, <a href="https://martinfowler.com/articles/patterns-of-distributed-systems/two-phase-commit.html">2-phase commits</a> or <a href="https://microservices.io/patterns/data/event-sourcing.html">event sourcing</a> (discussed in more details below), but they introduce more complexity. So far this compromise in simplicity hasn’t proven a major source of trouble for us.</p><h4>Ordering and idempotency</h4><p>On the consumer side, we basically have a loop that listens to all new messages for a given topic, deserializes the Protobuf message and upserts the resulting object into its own local database.</p><p>But one complication is that the order of messages is not guaranteed. You can get some weak ordering guarantees in Kafka, but it’s always possible to reconsume old messages.</p><p>When that happens, if you just upsert any incoming messages to the DB, you risk overwriting a more up-to-date version of your record with an older message.</p><p>In order to fix that issue, we add an integer field called seq on each table that we want to replicate. That number is incremented before each update of the record, and included in the message pushed to Kafka.</p><p>That way, when the consumer processes an incoming message, if it already holds a copy of the same object (identified by its id) that has a higher seq than the incoming records, it knows that it already holds a more recent version of the object, and can just skip the message.</p><p>That, combined with the fact that we always produce the full state of the record, gives us a very important property called idempotency: no matter in what order the update messages arrive for a given record, the record will always end up in the same state in the consumer database.</p><h4>Reproducing historic data</h4><p>This system works fine for propagating changes in real-time as they happen, but we haven’t touched the problem of propagating historic data.</p><ul><li>What happens when you create a new service that needs its local copy of the job ad objects. If you just plug it to the system, it will only get new updates about records, not the old existing ones.</li><li>What if you add a new field to the Source of Truth table, or start producing a new field that had values in the source database but wasn’t deemed useful to expose to the rest of the system before?</li><li>What if there was a bug in your producer that led you to produce faulty data, how do you fix that?</li><li>What if some changes failed to be propagated due to a Kafka outage?</li></ul><p>We address all of the above problems with a common tool, called a <strong>reproduce task</strong>: a script that iterates on a Source of Truth table, and reproduces each record into its corresponding Kafka topic.</p><p>It’s a crude but simple tool that lets us bootstrap new services, fix discrepancies and add new fields to our replication mechanism.</p><p>Note that republishing a big table can put a non-trivial amount of strain on the sytem. It can imply pushing millions of records that will then have to be processed by each consumer of the topic (possibly tens of services), eating up a lot of computing resources and likely introducing queue build-ups and delays into the replication process.</p><p>Thankfully, we rarely need a republish to impact all the consumers. When we’re bootstrapping a new service, or an existing service needs to consume a new field, we can usually get away by reproducing the data into a dedicated topic that only one single consumer service will listen to.</p><h4>Soft-Deletion</h4><p>In this replication-oriented system we only ever push one type of payload: the full object’s state. There’s no metadata, no event type. It’s always treated by consumers in the same way: upsert (with seq check).</p><p>That means we have to pass information about the deletion as object state. We do so by adding a soft-delete flag, typically called delete_time to each produced object.</p><p>Our design even forces us to implement soft-delete logic in both the consumer and producer services. If we hard-delete records in the producer, we’d lose the information that the record ever existed and we couldn’t reproduce a deletion event that failed to propagate. If we hard-delete records in the consumer, we lose their seq and we may find ourselves reconsuming an old message, effectively recreating a record that should remain deleted.</p><p>So we never actually delete replicated records, but instead store a deletion flag, and code the application to treat records with that flag as deleted, e.g. to ignore records with the soft-delete flag in DB queries. When we need to anonymise data for GDPR compliance, we do so by overwriting any sensitive fields in the data rather than by deleting the whole records.</p><p>Of course other strategies would be possible, like using the outbox pattern or dedicated deletion events, but each would introduce its own set of complexities into the replication mechanism. Our default approach may also cause data storage issues, for instance in cases where we’d need to handle huge volumes of short-lived records. But we can always resort to ad-hoc solutions in these special cases, like hard-deleting cold records after a while.</p><h4>Keep it super simple</h4><p>So our main use-case with asynchronous communication is replicating raw data.</p><p>Speed is a desirable attribute of this system. While it is built with eventual consistency in mind and can tolerate some delays, the longer the delays, the more the impact risks being felt by end users.</p><p>Even more important is robustness. In the event of an error, the consumer will keep retrying the messages until they succeed.</p><p>That’s why we strive to keep our consumers dead simple. They’re just dumb pipes focused on doing one thing fast and reliably: data replication.</p><p>They won’t perform any kind of validation. Validation should be performed by the Source of Truth before committing the change. Once the change is made to the Source of Truth and pushed to Kafka, a consumer attempting to reject it would just be committing a denial of reality.</p><p>For this reason, we also avoid database constraints as much as possible in consuming tables, besides basic type casting. No NOT NULL, no FOREIGN KEY (those don’t tend to play well with eventual consistency), ideally no size limits.</p><p>Any logic included in the consumer is a possible source of bugs, failures or delays and is actively avoided.</p><h3>Alternatives</h3><p>This is just one way to handle async communications in a distributed system. Here are some of the alternatives we’ve considered and why we preferred that system.</p><h4>Why not a CDC/Debezium?</h4><p>There are generic tools, called Change Data Capture, that can listen to database changes and replicate them to other databases. Using them would save us from having to implement the producers and consumers, and the mapping to Protobuf, in each service’s application code.</p><p>Our main issue with those tools is that they always introduce tight coupling to the source database schema. If you change the source database schema, say remove a column, the tool will either break, or propagate the schema changes to the target databases, hence breaking their own internal logic that depends on that schema.</p><p>If your data team has ever used such tools to populate their data warehouse (as is fairly common), then you’re probably familiar with that kind of incident where a backend developer changing his application’s schema breaks your data team’s BI reports. The postmortems tend to lead to gatekeeping processes, cross-team coordination efforts breaking the independence of your teams or even schema freezes that can hinder your ability to change your application.</p><p>This kind of problem is the very reason why the industry has adopted the concept of versioned API contracts.</p><p>When several applications depend on the same database, then the database schema becomes a de facto API contract.</p><p>Here, the conversion to Protobuf provides us with a layer of indirection between the API schema and the API contract used for replication. It also helps us enforce API versioning, which gives us a way forward to change our model by creating a new API version, with a new Protobuf version and a new topic, and deprecating the old version.</p><p>Sadly we haven’t found any off-the-shelf tool that can plug into the database and react to low-level database changes, then map them to a high-level versioned “business” intermediary representation then use it to replicate that versioned flow into the consumer databases. But we’ve built some internal libraries to help automate the consuming and the reproducing of objects in our services.</p><h4>Why not Event-Sourcing?</h4><p>Our replication-based system has some superficial resemblance with the <a href="https://microservices.io/patterns/data/event-sourcing.html">Event Sourcing</a> pattern. But there are also crucial differences:</p><ul><li>In Event Sourcing, the event stream is the Source of Truth, rather than the database of the source service. You need to persist the whole chain of events to reconstruct your state. By contrast, we use Kafka as a temporary communication rather than a persistent storage mechanism.</li><li>The consumers need to understand the semantics of each type of event they’re interested in, in order to be able to reconstruct state from the events. Each event type reflects specific actions that happened to your system, like ArchivedJobAds or DeletedUser, and the consumers require specific logic to derive the state of an object by applying different types of events differently. This creates coupling between producer and conumers: If you introduce a new ArchivedJobAds event, you first need to update all the consumers to undertand and process that new event, or they won’t be able to reconstruct an accurate representation of the JobAd object.</li><li>Order and exhaustiveness matter with Event Sourcing. Your system will end up in a different state if you apply a TimeFive and a PlusThree event in different orders, or skip one. Event Sourcing events also don’t typically contain the full state of the object they contain, but only the relevant data for the event’s application, kind of like a diff, so if you miss one the changes it was carrying might be lost. By contrast, we produce the full state of the object in each of our change events, so it doesn’t matter if you miss an intermediary event or if you consume them out of order, you can still easily reconstruct the final state of the record.</li></ul><p>For all these reasons, we feel that a lower-level replication mechanism gives us a much simpler and less coupled system that fits well the majority of use-cases.</p><p>There are however cases when we do miss the semantics provided by event types. That’s typically when we want to trigger additional treatments, besides replication, in reaction to a specific event like a status change. For instance a service may react to a job ad becoming “published” by pushing it to outside sources.</p><p>We can address those needs by 2 different mechanisms:</p><ul><li>Inferring the event from the state change, by comparing the new consumed state with the old existing state stored in the consumer’s data. For instance if we consume a job ad in the “published” state and see that we were already holding a “draft” version of that same job ad, we can treat that as a publication event. That adds unwelcome complexity and error risk to the consumers, so we tend to offload these “reactive” treatments to background jobs triggered by the consumers, so that they won’t cause errors or slow down the more critical replication mission of the consumer.</li><li>Ad-hoc semantic Domain Events, distinct from the replication messages, propagated via dedicated Kafka topics or by some other means like synchronous API calls. These are very similar to Event Sourcing events, except they’re not the Source of Truth and can be reserved to specific use-cases where they add value. Our use-cases for them include analytics tracking events, anonymization, CRM integration…</li></ul><h3>Summary</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XhVMiGsLwGcHdbfw" /></figure><p>In summary, when we need to expose data across services, the Source of Truth service will produce a versioned Protobuf message containing the full new state of the object into a Kafka topic for each change of the object.</p><p>The message will include the following fields:</p><ul><li>id to identify records within a stream, and find out which replication events apply to which object</li><li>seq, incremented at each update on a given object, to store the version of the object and ensure we don’t overwrite newer versions with older versions</li><li>delete_time to convey that an object has been deleted</li><li>all the fields of the object that we may want to expose, i.e. the latest full object state</li></ul><p>Consumer services can subscribe to the topic to upsert the objects in their own local database, maintaining their own local copy of the data.</p><p>The consumer code is kept fast, simple and focused on replication, and any follow-up reactive processing is offloaded to background jobs or cron tasks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cad1d9fd9f90" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jobteaser-dev-team/data-replication-across-backend-services-with-kafka-and-protobuf-cad1d9fd9f90">Data replication across backend services with Kafka and Protobuf</a> was originally published in <a href="https://medium.com/jobteaser-dev-team">JobTeaser Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Layers vs Silos, a tale of 2 microservice architectures]]></title>
            <link>https://medium.com/jobteaser-dev-team/layers-vs-silos-a-tale-of-2-microservice-architectures-50c21016e9b1?source=rss----bd77d16a0035---4</link>
            <guid isPermaLink="false">https://medium.com/p/50c21016e9b1</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[ruby]]></category>
            <dc:creator><![CDATA[Emmanuel Joubaud]]></dc:creator>
            <pubDate>Fri, 07 Apr 2023 10:01:02 GMT</pubDate>
            <atom:updated>2023-04-07T10:01:02.289Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pK1PrZOPe-79wTZQFrZSyA.png" /></figure><p>When it comes to the communication between microservices, there are 2 possible extremes:</p><ul><li><strong>All-sync:</strong> whenever a service needs data from another service, it fetches it via a synchronous API call (REST, gRPC, GraphQL). Service calls service calls service… which tends to evolve into <strong>layers</strong> of APIs, where each layer has dependencies on the next.</li><li><strong>All-async:</strong> no sync calls between services, all communication is event-driven or in the form of data replication. This tends to form <strong>silos:</strong> independent services that are fed all the data they need by an async mechanism and can function independently of any other service’s availability.</li></ul><p>Of course real-world systems rarely fit either of these 2 extremes perfectly. But whether you plan for it or not, you’ll probably end up with a dominant mode or, worse, a random mix.</p><p>Imagining a spectrum between these 2 extremes can offer a helpful mental model to design a more deliberate architecture, where you control the set of circumstances under which you tend towards one or the other, and the associated trade-offs.</p><h3>Layers</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vsHastgbOW5oAl-FescQCA.png" /><figcaption>An exemple of layers architecture, for a Career Service Management System</figcaption></figure><h4>Business layer</h4><p>So you’ve started building a bunch of business services for your microservice application. They implement business logic and store data. For isolation, you chose to go <a href="https://microservices.io/patterns/data/database-per-service.html">database-per-service</a>: each service stores a bunch of business objects for which it is the Source of Truth.</p><p>If you opt for sync-only communications between your services, that means any time one service needs to read another’s data, it has to make a synchronous API request to the Source of Truth service, introducing a dependency between them.</p><p>And if you map those dependencies on a graph, you’ll likely notice that <strong>software systems tend to contain a few business objects that are depended on way more than others.</strong> They’re the core of the dependency graph. Often they’re your users, your referential data (categories), the items of your e-commerce site, etc.</p><p>So even within your “business layer”, your API should naturally tend to structure itself into sub-layers, with for instance some “core dependency” services being depended on by many of the other services.</p><h4>Composition layer</h4><p>To display data to your users, the frontend will need to make queries to some of your backend services. And it will often need to join the data of multiple services. Doing so in the client would incur a performance penalty: you’d need to make a bunch of client-server road-trips to each service…</p><p>So the natural evolution from there is to introduce one or more <a href="https://microservices.io/patterns/data/api-composition.html">Composition services</a> in front of your system to merge the data from several backend services into a single payload for your frontend.</p><p>Maybe you’re using a few <a href="https://www.codemag.com/Article/2203081/A-Deep-Dive-into-the-Back-End-for-Front-End-Pattern">Backend-for-Frontend</a> services, maybe it’s a single <a href="https://www.apollographql.com/docs/federation/">GraphQL federation gateway</a>, maybe you just call these services “frontend services”. In any case, you end up with what we will call here a “Composition layer”, where data integration between your services happens upstream of the end user’s API request, before it hits your business services.</p><h3>Pitfalls of the Layers architecture</h3><p>The Layers architecture is simple, it resembles a lot how we structure regular software in a monolithic application, with API calls instead of function calls. But let’s take a step back and consider some of its weaknesses.</p><h4>Resilience</h4><p>A major selling point of the microservice architecture is its resilience benefits: if I have a job board module and a career appointment module, and they live in the same monolithic application, an outage of your monolith means that both modules are down. If they’re independent backend services, the users of your job board module could still access it even if your career appointment booking system is down.</p><p>But if both your services actually depend on a third “core dependency” service, for instance the user service, then when that service is down, both services stop working.</p><p>The “core dependency” services tend to become Single Points of Failure, just like your old monolith was.</p><h4>Fat composition layer, thin DB wrappers</h4><p>Now that you have a composition layer, exposing a new endpoint might actually force you to implement 2 endpoints: one in the business service that owns the data, and one in the composition layer.</p><p>A typical approach to mitigate the problem is then to make your business service APIs as generic as possible, so they can easily accommodate a wide range of use-cases, and you only need to implement new specific use-cases in endpoints of the composition layer.</p><p>Designing generic APIs is hard. You need to accommodate not only the use-cases you have at hand, but all the possible use-cases the future might hold in store for you.</p><p>A common trick is to provide a set of very simple primitives, typically basic <a href="https://en.wikipedia.org/wiki/Create,_read,_update_and_delete">CRUD operations</a> for data. Then you can combine those simple building blocks together in a higher-level layer to assemble more complex and powerful features: access control, sending emails, orchestration, sagas…</p><p>For instance, depending on whether it’s initiated by an admin or a basic user, an object creation may involve different ACL checks, input validations or different follow-up actions like sending emails to different people. A simple fix is to only handle the basic insertion of the object in the domain service, with perhaps a few invariant validations, and let the composition layer handle the use-case-specific ACL checks, validations and email-sending.</p><p>You can see how that creates a gravitational pull towards managing more business logic inside the composition layer, and turning business services into <a href="https://martinfowler.com/bliki/AnemicDomainModel.html">“anemic”</a> database wrappers providing simple CRUD operations.</p><h4>Joins</h4><p>Performing and optimizing joins over API requests is typically much harder than with a SQL query within a single database. What if you need to join and paginate the results from 2 different sources while filtering on attributes of both? Or more complex queries involving 5 or 6 data sources? Those queries can be really hard to write and harder yet to optimize and cache correctly.</p><h4>The distributed monolith</h4><p>In the end that fat composition layer fetching data from various services via API calls ends up looking a lot like a good old monolith fetching data from its database through ORM calls.</p><p>Your business services may be simple and have well-defined APIs, but most of the actual business logic is stitched together in the fat composition layer and you don’t get much benefits in terms of resilience.</p><p>But you also bought all the complexities of operating a distributed system: the increased latency and network unreliability, the complex API joins, distributed transactions… Observability is complicated by cascading errors, you need tracing to find the source of incidents. You can’t really run your stack on your local environment anymore without running all the services or resorting to complex mocking…</p><p>Then is that complexity really avoidable? How can you truly decouple services? It can feel pretty hopeless. That’s where the “Silos” architecture comes in.</p><h3>Silos</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ivHxMmgRGMaLL9VMe49wAQ.png" /><figcaption>An exemple of layers architecture, for a similar Career Service Management System</figcaption></figure><h4>Data replication</h4><p>Now let’s try to imagine what the opposite of the “Layers” architecture would look like. Say we don’t allow any sync calls between services. Each service then needs to contain all the data it needs to perform its function without depending on any other service being available. <strong>I call such a service a “Silo”</strong>.</p><p>But without sync calls, how can your service access another’s data? One way or another you’ll need to dispatch the data across the services using some form of <strong>asynchronous data replication.</strong></p><p>A typical implementation will use some sort of <a href="https://www.confluent.io/learn/publish-subscribe/">pub/sub or message queue</a>, for instance Kafka, where all your data, either raw or in the guise of events, can be pushed and dispatched to “consumer services” interested in using it, so they can keep their own readonly copy of the data up-to-date in their local database. In this model, write operations typically go through a request to a “Source of Truth” service that handles the relevant validations and produces the data into the message queue. For instance, when a new user is created or updated, the user service will push a message to the “user” topic in Kafka, containing all the attributes of the new user. Then if another service needs read access the user data, it can subscribe to the “user” topic and replicate a readonly copy of the users in its own database, so that it can read and join the user data via regular SQL queries.</p><p>The main perk of this approach is that even if the “Source of Truth” service is down — or even the message queue — the “consumer” services can continue to work. Their data might be slightly outdated but in most cases it’s better than an outage.</p><p>Having their own copy of the data also gives your consumers a lot of flexibility in how they can store and query that data. Performing joins and complex queries is then as easy as if it was your service’s own data.</p><h4>Inner-layers</h4><p>As you may have noticed from the schema, our silos have layers too, but they’re inner-layers, inside each service. They‘re reminiscent of the “layers” architecture:</p><ul><li>the replicated data, consumed via the pub/sub and readonly, is akin to the “core dependencies” layer</li><li>the service’s own data (the data for which it is the Source of Truth and receives write operations) can depend on the replicated data</li><li>the service’s business logic depends on both replicated and own data</li><li>some endpoints may expose or write only the service’s own data, executing the service’s business logic</li><li>finally some composition endpoints might expose a join of replicated and own data, possibly by calling the service’s own endpoints as a composition service would</li></ul><h3>Pitfalls of the Silos architecture</h3><h4>Replication costs</h4><p>The biggest drawback of an architecture based on data replication is that implementing a replication mechanism is costlier and more complex than just querying an endpoint.</p><p>To consume a new table, you need to update the database schema of the consumer service, implement the replication logic, backfill the historic data and ensure each change will be propagated.</p><p>You need to minimize, detect and fix inconsistencies, handle deletion events, idempotence and resilience to ordering issues. And think carefully about the design, scope and size of your replication channels or topics: what data should go into each stream (a problematic that’s analog to API design and <a href="https://www.dddcommunity.org/library/vernon_2011/">aggregate design in Domain Driven Design</a>).</p><p>Tooling and automation can help but, depending on the level of flexibility and robustness you’re after, you might have to implement your own. Built-in database replication mechanisms like <a href="https://www.postgresql.org/docs/current/logical-replication.html">Postgres logical replication</a> only work if all your services use the same DB engine. And off-the-shelf tools like the <a href="https://debezium.io/">Debeizum CDC</a> tend to work at the database schema level rather than through a versioned API, so they can’t help you without introducing strong coupling between the DB schemas of the source service and the consumers.</p><p>We will dive into ways of handling those challenges in a <a href="https://medium.com/jobteaser-dev-team/data-replication-across-backend-services-with-kafka-and-protobuf-78ff91d027ec">subsequent article</a>.</p><h4>Resource usage</h4><p>Replicating data across services can take up a non-trivial amount of DB storage space, network bandwidth and computing power.</p><p>Unless you have a massive scale, your pub/sub is unlikely to be the bottleneck here: tools like Kafka can scale to deliver enormous amonts of data to a lot of consumers very fast. But the regular flow of upserts can take a toll on the database of smaller consumer services.</p><h4>Limitations to service granularity</h4><p>Because of replication costs and resource usage, this approach alone doesn’t lend itself well to splitting your system into a large number of very small microservices. You would end up replicating a lot of data in each service — and often the same tables in services within related domains.</p><p>One solution is to accept coarse-grained modular services, where each physical service contains several submodules that depend on the same parts of the data model. This is more akin to a Service-Oriented Architecture than true micro services, but that’s fine. SOA may actually be the right granularity for many companies that have yet to reach the bigger scales where real microservices really shine.</p><p>You can even let data <a href="https://en.wikipedia.org/wiki/Cohesion_(computer_science)">cohesion</a> guide you to find the right seems to break down your system into services: if 2 modules depend on a lot of the same data, they might be better off as two submodules of the same service, that replicates the data they both need into its local database. If they have very different data requirements, it’s a good sign they actually tackle different subdomains and belong to different services.</p><p>Or another approach might be to mix the two: define a bunch of domains, each with a core service that contains the most central entities and replicates data asynchronously, surrounded by a few satellite services that contain the logic of satellite submodules, integrating with the core through synchronous APIs.</p><h4>Eventual consistency</h4><p>Replicating data means you’re entering the wonderful world of distributed systems, and the <a href="https://www.educative.io/blog/what-is-cap-theorem">CAP theorem</a> tells you that, if you want to contain failures to single subsystems, your data cannot be always up-to-date across all your services. <strong>Your system needs to become tolerant to replication lag</strong>, and be designed accordingly.</p><p>If specific use-cases where strong consistency is required, you might have to fall back to synchronous API calls.</p><p>Of course choosing to go microservice with different databases means that you’re working with distributed systems anyway and need to give up on strong transactional consistency across your whole data model, even in a “layers” architecture.</p><h3>Combining layers and silos</h3><p>As you may have picked up by now, at Jobteaser we’ve chosen the <strong>“Silos”</strong> architecture as our default stance for business services. We mostly avoid synchronous calls between backend services and rely on <a href="https://medium.com/jobteaser-dev-team/data-replication-across-backend-services-with-kafka-and-protobuf-78ff91d027ec">data replication</a> as much as possible instead.</p><p>But layers and synchronous calls are so important to software architecture that it’s not desirable to avoid them altogether. That’s why we also sprinkle bits of “Layers” where appropriate:</p><ul><li><strong>Avoid business logic duplication:</strong> One downside of replicating raw data is that it doesn’t transmit the associated business logic to manipulate that data. In a lot of cases that’s not a big deal, because we only need simple display or querying logic that actually belongs in the consumer service. Other times we can side-step the issue by producing pre-calculated values into the Kafka topic (typically calculations that only involve the state of the business object, with no other input parameters). Other times yet, when we need to reuse some presentation logic across different parts of the app, we can just expose the raw data in the client-facing endpoints and leave its manipulation to the frontend’s monorepo (e.g. i18n). And sometimes the logic is so trivial and non-vital that it’s not a big deal to duplicate it in a couple of places. Yet, when we need to <a href="https://en.wikipedia.org/wiki/Don%27t_repeat_yourself">DRY</a>-up business logic and none of those workarounds make sense, we do resort to synchronous API calls between business services.</li><li><strong>Strong consistency requirements:</strong> For some use-cases, dealing with data that’s not quite up-to-date is not an option. In those cases, we sometimes fallback to synchronous communications and have to accept outages when the upstream dependency is down.</li><li><strong>Tech bricks:</strong> Besides our business services, that hold business logic and data, we also have a bunch of technical services that hold little to no business logic and serve as sorts of libraries or interfaces between our infrastructure and clients or external services. We call those “tech bricks”. We mostly interact with tech bricks through sync API calls, and they tend to cluster into layers upstream and downstream of our business services. For instance, our apigw and session management service live in an “access layer” that depends on our business services, and our email service — used to send transactional emails — lives in an outer layer that our business services depend on. Our “Silos” architecture has layers too, just not layers of business services depending on each other.</li><li><strong>Mobile BFF:</strong> Our mobile apps require especially stable APIs because, unlike browser SPAs, we can’t roll out upgrades instantly to all clients. We need stable APIs to support older versions of the mobile apps. That’s why our mobile team chose to create a <a href="https://www.codemag.com/Article/2203081/A-Deep-Dive-into-the-Back-End-for-Front-End-Pattern">Backend For Frontend</a> service that maintains a specific API for the iOS and Android apps. It doesn’t have its own database and performs sync API calls on many backend services, in a typical “composition layer” fashion.</li></ul><h3>Summary</h3><p>The “layers” architecture, based on synchronous calls between services, is relatively simpler and cheaper to implement, but it tends to evolve into a distributed monolith and won’t unlock all the resilience benefits that a service-oriented architecture can provide.</p><p>The “silos” architecture, based on asynchronous integration, tends to be costlier and more complex to build, and to nudge you towards bigger services, but it gives you more resilience and isolation between services.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=50c21016e9b1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jobteaser-dev-team/layers-vs-silos-a-tale-of-2-microservice-architectures-50c21016e9b1">Layers vs Silos, a tale of 2 microservice architectures</a> was originally published in <a href="https://medium.com/jobteaser-dev-team">JobTeaser Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Our microservice stack]]></title>
            <link>https://medium.com/jobteaser-dev-team/our-microservice-stack-db50aa2b17d6?source=rss----bd77d16a0035---4</link>
            <guid isPermaLink="false">https://medium.com/p/db50aa2b17d6</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[ruby]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Emmanuel Joubaud]]></dc:creator>
            <pubDate>Fri, 31 Mar 2023 10:34:09 GMT</pubDate>
            <atom:updated>2023-03-31T10:34:08.910Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CXavpk2DGQUjEEE08Hf-0g.png" /></figure><p>This is an introduction to how we’ve implemented microservices at a mid-size scale-up called <a href="https://www.jobteaser.com/">Jobteaser</a>, with a mix of Go and Ruby service chassis, gRPC APIs and data replication via Kafka.</p><h3>Foundation: The service chassis</h3><p>Back in early 2019, when Jobteaser decided to get serious about breaking up its decade-old Rails monolith into microservices, we assembled a Foundation team that started working on an in-house <a href="https://microservices.io/patterns/microservice-chassis.html">service chassis</a>.</p><p>It was soberly coined service and came in two flavours: the rb-service framework in <a href="https://www.ruby-lang.org/en/">Ruby</a> and the go-service framework in <a href="https://go.dev/">Go</a>. Four years later, they still form the foundation of our fleet of about 20 services.</p><p>They provide a lean set of consistent features across the Ruby and Go flavours:</p><ul><li>a <a href="https://grpc.io/">gRPC</a> server component to power our gRPC APIs (more below)</li><li>a <a href="https://kafka.apache.org/">Kafka</a> consumer, to consume messages from a Kafka message queue</li><li>a <a href="https://prometheus.io/">Prometheus</a> exporter, to expose monitoring metrics</li><li>a low-level lib (<a href="https://martinfowler.com/bliki/OrmHate.html">no ORM</a>) to interact with Postgres, that would be the default database of each service, and optionally Redis for services who require one</li><li>consistent logging, metrics and error reporting for those components</li><li>a bunch of CircleCI, Dockerfiles and Kubernetes (k8s) config files to enable automatic deployment in staging and production</li><li>an executable to start the service</li><li>a generator script, to scaffold a new service’s <a href="https://wiki.c2.com/?WalkingSkeleton">Walking Skeleton</a> with all of the above and get it deployed to prod in less than a day</li></ul><p>For a while, the various workloads would all run as threads inside a single process: each instance of a service would run a thread pool for its gRPC server, one for its Kafka consumers, one for its Prometheus exporter, one for its background jobs, etc.</p><p>That was great for the local dev environment because you didn’t need to run a different Docker image for each workload, but we quickly came to split the types of workloads into different pods in production, to avoid issues where a buggy background job could take down the API.</p><h3>Cloud-native Infrastructure</h3><p>All our services are containerized and run on two <a href="https://kubernetes.io/">Kubernetes</a> clusters, one for production and one for staging, setup and operated with <a href="https://github.com/kubernetes/kops">kops</a> on our AWS accounts. Stateful servers like our services’ Postgres databases and Redis are not managed via Kubernetes but provisioned straight in <a href="https://aws.amazon.com/rds/">Amazon RDS</a> or <a href="https://aws.amazon.com/elasticache/">Elasticache</a>.</p><p>The staging cluster is identical to the prod one, except it runs fewer replicas and slightly less powerful versions of the service pods.</p><p>We do Continuous Deployment: whenever we merge a feature branch to master, our CI builds a new Docker image and deploys it to staging then production with the help of <a href="https://helm.sh/">helm</a>.</p><p>We also have a script that lets us deploy a service’s Git branch to staging to test changes ahead of merging them. But the next merge on master will trigger a deploy that will override it, so prod and staging always end up running the same code again.</p><p>Our secrets are stored in <a href="https://www.vaultproject.io/">Vault</a>. Our monitoring and alerting handled by <a href="https://prometheus.io/">Prometheus</a>, <a href="https://grafana.com/">Grafana</a> and <a href="https://github.com/grafana/loki">Loki</a>. Gitops-deployed <a href="https://www.terraform.io/">Terraform</a> templates describe our cloud resource provisioning (RDS DBs, S3 buckets) but also our k8s and Github teams and permissions.</p><h3>APIs: gRPC and Protobuf</h3><p>You may have noticed one thing missing in my list of features for our service chassis: an http server.</p><p>By default, all our APIs are served over Google’s gRPC and Protobuf, rather than the more common http/json.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*i8AyNbbYVxXB2ioyPMQcMA.png" /><figcaption>gRPC is a Google framework that lets a client service call a function in a remote server service. It opens persistent http2 connections to offer high performance by skipping the TCP handshake on each request</figcaption></figure><p>gRPC serializes the API request and response payloads using a binary format called Protobuf, that forces us to define our API’s request and response formats in .proto files using Protobuf’s Interface Description Language.</p><p>So when we want to create a new endpoint, the first step is always to make a Pull Request to our proto Github repository, defining a new rpc line for the new endpoint, along with the types of its request and response formats.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C6STHZbpucyLRkIFgX6u6Q.png" /><figcaption>An example of .proto definition. rpc = endpoint, message = type.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zWHzFWsLl0YsxsClJC6USw.png" /><figcaption>A ruby and a Go library are generated from the .proto files. They contain the stubs, i.e. generated Go and Ruby classes that let us call or implement the endpoints and enforce the API’s type-safety. The libraries are then included in each service using the language’s dependency management (Rubygems or Go packages)</figcaption></figure><h4>Benefits</h4><p>The main perks of this setup come from the centralized proto repo with all the API definitions:</p><ul><li>it gives us a centralized catalog documenting all our API endpoints, necessarily up-to-date</li><li>every API change starts with a discussion of the interface in a dedicated Pull Request that both front and backend devs can review</li><li>linters on the Proto repo catch any breaking change, forcing us into proper API versioning practices and enforcing naming conventions</li></ul><p>We also get performance benefits from gRPC’s persistent http2 connections and Protobuf’s small binary payloads.</p><h4>Costs</h4><p>The Protobuf step adds a bit of work to all API changes: you need to make a PR to the proto repo, wait for the CI to generate the proto-ruby and proto-go library releases then upgrade the libs in the client and server services.</p><p>You also pay a small exoticism tax: gRPC is not as widespread as REST or GraphQL, which means you don’t get quite as much tooling and integrations. There’s a bit of an onboarding curve for newcomers. And off-the-shelf components do not always offer a gRPC interface so you can’t always preserve its performance benefits across your whole infrastructure.</p><p>Finally, gRPC <a href="https://www.useanvil.com/blog/engineering/load-balancing-grpc-in-kubernetes-with-istio/">complicates load-balancing</a>. Persistent http2 connections are tricky to load-balance and support in infrastructure tools like Kubernetes services has been slow to catch-up. So far we’ve resorted to DNS-based client-side load-balancing via <a href="https://kubernetes.io/docs/concepts/services-networking/service/#headless-services">Kubernetes headless services</a>, which doesn’t share the load optimally and requires retries to handle failovers. We’re considering adopting a service mesh to support more advanced gRPC load-balancing, among other features, but it comes at the cost of more complexity.</p><h3>The API Gateway</h3><p>gRPC works great for communication between persistent services, but it wasn’t designed to work in the browser. We still want our frontend <a href="https://reactjs.org/docs/glossary.html#single-page-application">React Single-Page Applications</a> (SPA) to query the gRPC API endpoints of our backend services using good old http/json requests.</p><p>We also need to think about access control. Some API endpoints should only be called inside our infrastructure and not be exposed to the world. Others must verify that the request comes from an authorized user.</p><p>So our Foundation team implemented an API Gateway in Go, soberly named apigw , built on top of the go-service chassis, and performing the following duties:</p><ul><li>Conversion of the incoming http/json requests of the frontend into a gRPC/Protobuf request that can be dispatched to the gRPC endpoint of a target service (and conversion of the resulting response from proto to json)</li><li>Identification, by checking the incoming request’s session cookie against our authentication service, and getting the corresponding user_id</li><li>Routing to the right service’s endpoint, by checking the request’s path against a whitelist of endpoints in config files, and performing a gRPC/proto request decorated with the user_id in the request’s metadata, so the target service can identify the user making the request.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XhIVK5WBwD3wRtrCRLHXlA.png" /><figcaption>Typical flow of a request to a backend service coming from the browser through our apigw: Conversion, Identification and Routing</figcaption></figure><p>Like all our other services, the apigw needs the latest version of our Proto library to send gRPC requests to the new endpoints we add. That part was automated so that whenever we merge a new update of our proto repo, our CI triggers an upgrade of the apigw with the newly generated release of theproto-go library and deploys it to production.</p><h3>Product services and tech bricks</h3><p>We tend to classify our services in two broad categories:</p><ul><li><strong>Business services</strong>: contain business logic and data, roughly map to modules or domains of our product, built around one main business entity: one service for our job board, one for our career appointment booking module, one for our career event management platform…</li><li><strong>Tech bricks: </strong>utility services that hold little to no business logic, a bit like libraries that could almost be open-sourced, and typically serve as interfaces between our infrastructure and clients or external services.</li></ul><p>Examples of tech bricks include:</p><ul><li>our API Gateway that handles incoming API requests from our clients</li><li>our transactional email service that sends emails via an external service (<a href="https://sendgrid.com/">Sendgrid</a>) and handles templating, i18n, scheduling and retries, so we don’t have to integrate libs for those concerns into each service. Business services can just make gRPC calls to that service using a generic API</li><li>our CMS service interfaces with a headless Content Management System (<a href="https://kontent.ai/">Kontent.ai</a>) to expose articles produced by our internal teams of guidance experts into the Jobteaser product using gRPC APIs</li></ul><h3>Granularity</h3><p>Finding the right seems to break down a system is one of the biggest challenges with a service-oriented architecture.</p><p>We’re not (yet) a tech giant, and despite all the tooling we deployed over the years, creating and maintaining a new service still involves a non-trivial amount of overhead for us, so we had to take a pragmatic approach to keep efficiently delivering value to our users.</p><p>That’s why we haven’t gone very micro with our services. A more accurate way to describe our architecture would be “Service-Oriented”, though “microservice” has come to convey the general idea more broadly. A lot of our services are actually small modular monoliths that include several submodules, and we’re intentionally staying very far from the oft-touted ideal of “services you can rewrite in 2 weeks”.</p><p>The biggest drawback is that we sometimes end up with several teams working at the same time on a given service, raising similar issues to those you may encounter on a monolith, though at a lesser scale and frequency. This seems like an acceptable trade-off at our current scale, though it may evolve in the future.</p><h3>Integration: sync and async</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CXavpk2DGQUjEEE08Hf-0g.png" /><figcaption>Integration between business services is mostly accomplished by ways of asynchronous messaging using Kafka and Protobuf rather than synchronous gRPC dependencies</figcaption></figure><p>We use the <a href="https://microservices.io/patterns/data/database-per-service.html">database-per-service pattern</a> for isolation so each business service has its own database. That way, if one service is down, it shouldn’t keep the other modules from functioning. And each service can change its internal database schema without impacting the others, as long as we avoid breaking changes in the service’s APIs.</p><p>Then of course we often need to integrate the data from several services to produce useful features and that constitutes one of our main challenges.</p><p>Communication between services is achieved by a mix of synchronous gRPC APIs, and asynchronous communication via Kafka.</p><p>The messages we push into Kafka to propagate changes are also encoded using Protobuf, which provides an API contract for those async messages.</p><p>So you can think of gRPC as the service’s synchronous API, and Kafka as its asynchronous API, and the two combined form each service’s public API.</p><p>But integration is such a vast topic that we’ll dig into it in a subsequent post.</p><p>Stay tuned!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db50aa2b17d6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jobteaser-dev-team/our-microservice-stack-db50aa2b17d6">Our microservice stack</a> was originally published in <a href="https://medium.com/jobteaser-dev-team">JobTeaser Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>