<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Shippeo Tech Blog - Medium]]></title>
        <description><![CDATA[Shippeo is a global leader in real-time shipment visibility, across all transport modes, helping shippers and 3/4PLs to run more collaborative, automated, sustainable, profitable and customer-centric supply chains with highly accurate, real-time data and advanced AI/ML engines. - Medium]]></description>
        <link>https://medium.com/shippeo-tech-blog?source=rss----25b860162bea---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Shippeo Tech Blog - Medium</title>
            <link>https://medium.com/shippeo-tech-blog?source=rss----25b860162bea---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 22:49:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/shippeo-tech-blog" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Angular Testing Mastery: The Best Practices to Skyrocket Your Skills! — Examples]]></title>
            <link>https://medium.com/shippeo-tech-blog/angular-testing-mastery-the-best-practices-to-skyrocket-your-skills-examples-0a43220b949d?source=rss----25b860162bea---4</link>
            <guid isPermaLink="false">https://medium.com/p/0a43220b949d</guid>
            <category><![CDATA[angular]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[front-end-development]]></category>
            <dc:creator><![CDATA[Antoine Audusseau]]></dc:creator>
            <pubDate>Tue, 27 Feb 2024 13:34:07 GMT</pubDate>
            <atom:updated>2024-02-27T13:44:02.803Z</atom:updated>
            <content:encoded><![CDATA[<h3>Angular Testing Mastery: The Best Practices to Skyrocket Your Skills! — Examples</h3><p>This article is here to provide you with some tips for testing Angular components, services, or other classes. It follows a previous article where you can find tips and best practices: <a href="https://medium.com/shippeo-tech-blog/angular-testing-mastery-the-best-practices-to-skyrocket-your-skills-369ef71a4456">https://medium.com/shippeo-tech-blog/angular-testing-mastery-the-best-practices-to-skyrocket-your-skills-369ef71a4456</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xEaPH3wXFa7ysBMKlZEQ-A.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/fr/@kellysikkema">Kelly Sikkema</a> from <a href="https://unsplash.com/fr/photos/papier-dimprimante-blanc-4JxV3Gs42Ks?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p>Before beginning, I want to clarify that these examples are purely subjective; they are based only on my personal experience as an Angular frontend developer and on what we learn at Shippeo.</p><p>These tests are made with the <a href="https://angular.io/">Angular</a> framework, combined with the <a href="https://jestjs.io/">Jest testing framework</a>. Jest is not the default testing framework for Angular; it is a good choice as a substitute for Karma, which will be removed in future Angular versions. In addition, I use the <a href="https://github.com/ngneat/spectator">Spectator</a> and <a href="https://github.com/help-me-mom/ng-mocks">ng-mocks</a> libraries, which simplify the Angular tests a lot by providing very useful tools. I highly recommend these libraries; Angular tests are quite verbose right now, and I hope they will evolve in the future.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kFsjDk2rlkPqmCNuFJbRiA.png" /><figcaption>Angular, Jest and Spectator logos</figcaption></figure><p><em>All the code you will see is available in this repository: </em><a href="https://github.com/a-audusseau/angular-tests-examples"><em>https://github.com/a-audusseau/angular-tests-examples</em></a></p><h3>Testing components</h3><p>First of all, remember that the purpose of a component is (in most cases) to deliver HTML to the user. So you need to test the template and interact with it. Do not skip it by directly testing component methods or properties. All public methods and properties are supposed to be used in the template; if they don’t, they should be private and not available directly for the tests.</p><p>Ok, now let’s see our component 👀</p><pre>@Compontyent({<br>  selector: &#39;app-pizza&#39;,<br>  standalone: true,<br>  imports: [AsyncPipe, PizzaIngredientListComponent],<br>  styles: `<br>    .red { color: red; }<br>  `,<br>  template: `<br>    &lt;h2 class=&quot;title&quot; [class.red]=&quot;!pizza.isAvailable&quot;&gt;{{ pizza.name }}&lt;/h2&gt;<br><br>    &lt;app-pizza-ingredient-list [pizza]=&quot;pizza&quot; /&gt;<br><br>    @if (isInCart$ | async) {<br>    &lt;p&gt;In the cart&lt;/p&gt;<br>    } @else { @if (pizza.isAvailable) {<br>    &lt;button (click)=&quot;addToCart()&quot; data-test=&quot;add-to-card&quot;&gt;Add to card&lt;/button&gt;<br>    } @else {<br>    &lt;p&gt;Currently not available&lt;/p&gt;<br>    } }<br>  `,<br>})<br>export class PizzaComponent {<br>  @Input({ required: true }) pizza!: Pizza;<br><br>  public readonly isInCart$: Observable&lt;boolean&gt; = this.pizzaService.cart$.pipe(<br>    map((cart) =&gt; !!cart.find(({ id }) =&gt; id === this.pizza.id)),<br>    distinctUntilChanged()<br>  );<br><br>  constructor(private readonly pizzaService: PizzaService) {}<br><br>  public addToCart(): void {<br>    if (this.pizza.cost &gt; 12) {<br>      console.log(&#39;wow this is expensive&#39;);<br>    }<br><br>    this.pizzaService.addToCard(this.pizza);<br>  }<br>}</pre><p>This component takes a pizza as input, displays some information depending on the state, and can render a button to add the pizza to the cart.</p><p>Let&#39;s start by configuring our tests for this component. The first step is to mock all dependencies by checking the constructor and the template. It is not always required, but this way we avoid the complexity of our dependencies, and it can make the tests faster.</p><p>So for this component, we have a dependency on the PizzaService and on the PizzaIngredientsListComponent component. Here are the results with comments:</p><pre>describe(&#39;PizzaComponent&#39;, () =&gt; {<br>  let spectator: Spectator&lt;PizzaComponent&gt;;<br><br>  const cart$ = new BehaviorSubject&lt;Pizza[]&gt;([]);<br>  // createSpyObject let us create a PizzaService instance with all methods replaced by Jest spy. We can also overridres properties, so now we have the control of the cart$ Observable.<br>  const pizzaService = createSpyObject(PizzaService, { cart$ });<br>  const pizza: Pizza = {<br>    id: 2,<br>    name: &#39;Pepperoni Paradise&#39;,<br>    cost: 12,<br>    ingredients: [&#39;Tomato&#39;, &#39;Mozzarella&#39;, &#39;Pepperoni&#39;],<br>    isAvailable: true,<br>  };<br><br>  const createComponent = createComponentFactory({<br>    component: PizzaComponent,<br>    // Mock services<br>    providers: [{ provide: PizzaService, useValue: pizzaService }],<br>    // Mock components<br>    overrideComponents: [<br>      [<br>        PizzaComponent,<br>        {<br>          add: { imports: [MockComponent(PizzaIngredientListComponent)] },<br>          remove: { imports: [PizzaIngredientListComponent] },<br>        },<br>      ],<br>    ],<br>  });<br><br>  // the pizza Input is required, so we need to fill it when creating the component.<br>  beforeEach(() =&gt; (spectator = createComponent({ props: { pizza } })));<br><br>  it(&#39;should create the component&#39;, () =&gt; {<br>    expect(spectator.component).toBeTruthy();<br>  });<br>});</pre><p>Okay, now let’s just look at the template and create tests for everything we see. The most important things are the dynamic ones, like if/else or for statements. For instance, it can be:</p><pre>it(&#39;should display the pizza name&#39;, () =&gt; {...});<br>describe(&#39;when the pizza is not available&#39;, () =&gt; {<br>  it(&#39;should add the class red to the title &#39;, () =&gt; {...})<br>});<br>it(&#39;should display the pizza ingredient list&#39;, () =&gt; {...});<br>describe(&#39;when the pizza is in the cart&#39;, () =&gt; {<br>  it(&#39;should display a message to indicate it&#39;, () =&gt; {...})<br>});<br>describe(&#39;when the pizza is not in the cart&#39;, () =&gt; {<br>  describe(&#39;and it is available&#39;, () =&gt; {<br>    it(&#39;should display the add to card button&#39;, () =&gt; {...})<br>    describe(&#39;and the add to card button is clicked&#39;, () =&gt; {<br>      it(&#39;should call the PizzaService addToCard method&#39;, () =&gt; {...})<br>    });<br>  });<br>  describe(&#39;and it is unavailable&#39;, () =&gt; {<br>    it(&#39;should display the unavailable message&#39;, () =&gt; {...});<br>  });<br>});</pre><p>Just by looking at the template, we have a nice list of tests to do 😍, so let’s write them now:</p><pre>// You can add specific attributes to target the HTML elements, like the data-test attribute. It can help for your Angular tests and also for E2E tests if you have some.<br>  const addToCardSelector = &#39;[data-test=&quot;add-to-card&quot;]&#39;;<br>  const titleSelector = &#39;.title&#39;;<br><br>  // the pizza Input is required, so we need to fill it when creating the component.<br>  beforeEach(() =&gt; (spectator = createComponent({ props: { pizza } })));<br><br>  it(&#39;should display the pizza name&#39;, () =&gt; {<br>    expect(spectator.query(titleSelector)).toBeTruthy();<br>    expect(spectator.query(titleSelector)).toHaveExactText(pizza.name);<br>    expect(spectator.query(titleSelector)).not.toHaveClass(&#39;red&#39;);<br>  });<br><br>  describe(&#39;when the pizza is not available&#39;, () =&gt; {<br>    beforeEach(() =&gt;<br>      spectator.setInput({ pizza: { ...pizza, isAvailable: false } })<br>    );<br><br>    it(&#39;should add the class red to the title &#39;, () =&gt; {<br>      expect(spectator.query(titleSelector)).toHaveClass(&#39;red&#39;);<br>    });<br>  });<br><br>  it(&#39;should display the pizza ingredient list&#39;, () =&gt; {<br>    expect(spectator.query(PizzaIngredientListComponent)).toBeTruthy();<br>    expect(spectator.query(PizzaIngredientListComponent)?.pizza).toEqual(pizza);<br>  });<br><br>  describe(&#39;when the pizza is in the cart&#39;, () =&gt; {<br>    beforeEach(() =&gt; {<br>      cart$.next([pizza]);<br>      spectator.detectChanges();<br>    });<br><br>    it(&#39;should display a message to indicate it&#39;, () =&gt; {<br>      expect(spectator.query(byText(&#39;In the cart&#39;))).toBeTruthy();<br>      expect(spectator.query(byText(&#39;Currently not available&#39;))).toBeFalsy();<br>      expect(spectator.query(addToCardSelector)).toBeFalsy();<br>    });<br>  });<br><br>  describe(&#39;when the pizza is not in the cart&#39;, () =&gt; {<br>    beforeEach(() =&gt; {<br>      cart$.next([]);<br>      spectator.detectChanges();<br>    });<br><br>    describe(&#39;and it is available&#39;, () =&gt; {<br>      beforeEach(() =&gt;<br>        spectator.setInput({ pizza: { ...pizza, isAvailable: true } })<br>      );<br><br>      it(&#39;should display the add to card button&#39;, () =&gt; {<br>        expect(spectator.query(addToCardSelector)).toBeTruthy();<br>        expect(spectator.query(addToCardSelector)).toHaveExactText(<br>          &#39;Add to card&#39;<br>        );<br>        expect(spectator.query(byText(&#39;In the cart&#39;))).toBeFalsy();<br>        expect(spectator.query(byText(&#39;Currently not available&#39;))).toBeFalsy();<br>      });<br><br>      describe(&#39;and the add to card button is clicked&#39;, () =&gt; {<br>        beforeEach(() =&gt; spectator.click(addToCardSelector));<br><br>        it(&#39;should call the PizzaService addToCard method&#39;, () =&gt; {<br>          expect(pizzaService.addToCard).toHaveBeenCalledWith(pizza);<br>        });<br>      });<br>    });<br><br>    describe(&#39;and it is unavailable&#39;, () =&gt; {<br>      beforeEach(() =&gt;<br>        spectator.setInput({ pizza: { ...pizza, isAvailable: false } })<br>      );<br><br>      it(&#39;should display the unavailable message&#39;, () =&gt; {<br>        expect(spectator.query(byText(&#39;Currently not available&#39;))).toBeTruthy();<br>        expect(spectator.query(addToCardSelector)).toBeFalsy();<br>        expect(spectator.query(byText(&#39;In the cart&#39;))).toBeFalsy();<br>      });<br>    });<br>  });</pre><p>Now if we take a look at the coverage, we can see that we already cover almost all the component code. But we also can see that one branch is still not covered; this is when the coverage report is useful.</p><p>So let&#39;s add the test to fix this:</p><pre>describe(&#39;when the pizza is not in the cart&#39;, () =&gt; {<br>    [...]<br>    describe(&#39;and it is available&#39;, () =&gt; {<br>      [...]<br>      describe(&#39;and the add to card button is clicked with a pizza cost over 12&#39;, () =&gt; {<br>        const consoleSpy = jest.spyOn(console, &#39;log&#39;).mockImplementation();<br><br>        beforeEach(() =&gt; {<br>          spectator.setInput({<br>            pizza: { ...pizza, isAvailable: true, cost: 18 },<br>          });<br>          spectator.click(addToCardSelector);<br>        });<br><br>        it(&#39;should call the PizzaService addToCard method and log a message&#39;, () =&gt; {<br>          expect(consoleSpy).toHaveBeenCalledWith(&#39;wow this is expensive&#39;);<br>          expect(pizzaService.addToCard).toHaveBeenCalledWith(pizza);<br>        });<br>      });<br>    });<br>  });</pre><p>Congrats 👏, now we have a pretty well-tested component!</p><h3>Testing Services</h3><p>After the component, let’s see the other main Angular class, services! In our demo app, we have one service used to get the list of pizzas and manage the cart.</p><pre>@Injectable({ providedIn: &#39;root&#39; })<br>export class PizzaService {<br>  private readonly pizzas$$ = new BehaviorSubject&lt;Pizza[]&gt;(PIZZA_LIST);<br>  public readonly pizzas$ = this.pizzas$$.asObservable();<br>  private readonly cart$$ = new BehaviorSubject&lt;Pizza[]&gt;([]);<br>  public readonly cart$ = this.cart$$.asObservable();<br><br>  public addToCard(pizza: Pizza): void {<br>    const index = this.pizzas.findIndex(({ id }) =&gt; id === pizza.id);<br><br>    if (index === -1 || !pizza.isAvailable) {<br>      return;<br>    }<br><br>    this.cart$$.next([...this.cart, pizza]);<br>  }<br><br>  public removeFromCard(pizza: Pizza): void {<br>    this.cart$$.next([...this.cart.filter(({ id }) =&gt; id !== pizza.id)]);<br>  }<br><br>  private get cart(): Pizza[] {<br>    return this.cart$$.getValue();<br>  }<br><br>  private get pizzas(): Pizza[] {<br>    return this.pizzas$$.getValue();<br>  }<br>}</pre><p>At Shippeo, we decided to test services method by method (or properties) to improve readability. Let’s see the result with the PizzaService:</p><pre>describe(&#39;PizzaService&#39;, () =&gt; {<br>  let spectator: SpectatorService&lt;PizzaService&gt;;<br><br>  const pizza: Pizza = {<br>    id: 2,<br>    name: &#39;Pepperoni Paradise&#39;,<br>    cost: 12,<br>    ingredients: [&#39;Tomato&#39;, &#39;Mozzarella&#39;, &#39;Pepperoni&#39;],<br>    isAvailable: true,<br>  };<br><br>  const createService = createServiceFactory(PizzaService);<br><br>  beforeEach(() =&gt; (spectator = createService()));<br><br>  describe(&#39;The addToCard method&#39;, () =&gt; {<br>    describe(&#39;when the pizza is not in the pizza list&#39;, () =&gt; {<br>      it(&#39;should not add it to the cart&#39;, async () =&gt; {<br>        spectator.service.addToCard({ ...pizza, id: -1 });<br>        expect(await firstValueFrom(spectator.service.cart$)).toHaveLength(0);<br>      });<br>    });<br><br>    describe(&#39;when the pizza is not in available&#39;, () =&gt; {<br>      it(&#39;should not add it to the cart&#39;, async () =&gt; {<br>        spectator.service.addToCard({ ...pizza, isAvailable: false });<br>        expect(await firstValueFrom(spectator.service.cart$)).toHaveLength(0);<br>      });<br>    });<br><br>    describe(&#39;otherwise&#39;, () =&gt; {<br>      it(&#39;should add the pizza to the cart&#39;, async () =&gt; {<br>        spectator.service.addToCard(pizza);<br>        expect(await firstValueFrom(spectator.service.cart$)).toContain(pizza);<br>      });<br>    });<br>  });<br><br>  describe(&#39;The removeFromCard method&#39;, () =&gt; {<br>    beforeEach(() =&gt; spectator.service.addToCard(pizza));<br><br>    it(&#39;should remove the pizza from the card&#39;, async () =&gt; {<br>      expect(await firstValueFrom(spectator.service.cart$)).toContain(pizza);<br>      spectator.service.removeFromCard(pizza);<br>      expect(await firstValueFrom(spectator.service.cart$)).not.toContain(<br>        pizza<br>      );<br>    });<br>  });<br>});</pre><p>We can see that we use a description for both the methods addToCart and removeFromCart, and a describe for every if statement possibility. The coverage report is great in services for seeing which branch is not covered, to not forget any case. If you have some private method with a branch uncovered, it can also be done by adding a new describe and targeting the public method that uses the private method.</p><p>We could have added specific tests for the service’s Observables, but they are quite simple and implicitly tested by other tests. We use the await/sync with the firstValueFrom function from RxJS to get the Observable value. It is a good way to test Observables, as we are sure that it emits a value and we follow the <a href="https://github.com/jest-community/eslint-plugin-jest/blob/main/docs/rules/no-done-callback.md">Jest recommendation</a> by avoiding the done callback.</p><p>Our service doesn’t have any dependencies, but like for the component, we could use Spectator to set providers with mocks.</p><h3>Other classes</h3><p>Testing other classic Angular classes should be quite simple. For guards and resolvers, they are just specific services. For pipes and directives, you can take a look at the <a href="https://github.com/ngneat/spectator">Spectator documentation</a> which contains great examples.</p><p>Routed components can be painful, but once again Spectator can help by providing a nice helper to mock the ActivatedRoute object.</p><p>All the examples are available in this repository: <a href="https://github.com/a-audusseau/angular-tests-examples">https://github.com/a-audusseau/angular-tests-examples</a></p><h3>Conclusion</h3><p>In this article, we talked about best practices we try to follow at Shippeo, but we know that these practices can evolve over time, especially with tools that change quickly as Angular and the JS ecosystem do. As an example, in my opinion the Angular testing toolbox has a complex and verbose syntax, and I will not be surprised if it evolves for a more pleasant syntax like we have with Spectator. Moreover, we still use Observables a lot as you can see in the above examples, but since Angular 17, <a href="https://angular.io/guide/signals">Signals</a> have been introduced and it change the way we think reactivity a lot. I’m looking forward to see all these great things coming!</p><p>Thanks for reading! I hope all these tips will make you love testing your code 💜</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0a43220b949d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/shippeo-tech-blog/angular-testing-mastery-the-best-practices-to-skyrocket-your-skills-examples-0a43220b949d">Angular Testing Mastery: The Best Practices to Skyrocket Your Skills! — Examples</a> was originally published in <a href="https://medium.com/shippeo-tech-blog">Shippeo Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Angular Testing Mastery: The Best Practices to Skyrocket Your Skills!]]></title>
            <link>https://medium.com/shippeo-tech-blog/angular-testing-mastery-the-best-practices-to-skyrocket-your-skills-369ef71a4456?source=rss----25b860162bea---4</link>
            <guid isPermaLink="false">https://medium.com/p/369ef71a4456</guid>
            <category><![CDATA[front-end-development]]></category>
            <category><![CDATA[angular]]></category>
            <category><![CDATA[testing]]></category>
            <dc:creator><![CDATA[Antoine Audusseau]]></dc:creator>
            <pubDate>Tue, 27 Feb 2024 13:33:21 GMT</pubDate>
            <atom:updated>2024-02-27T13:43:30.232Z</atom:updated>
            <content:encoded><![CDATA[<p>The goal of this post is to help you comprehend the benefits of testing and make you comfortable with writing or reading tests 👩‍💻👨‍💻. I will share some well-known best practices, tips, and advanced techniques, making it relevant for both beginner and senior developers. It is important to note that the post is focused on frontend development, as I am a dedicated frontend developer.</p><p>Testing code is often not a primary consideration when you begin developing an app or a library. Initially, the focus is on delivering new features to the customer swiftly, fixing bugs promptly, and deploying them in production 🚀. Consequently, your product works, and you may not have invested time in writing tests. However, as the codebase expands, making changes becomes more complex, and confidence diminishes. New developers struggle to comprehend the code 🥵, especially if there are no specifications, leading to a process of reverse engineering.</p><p>Remember: Tests are not just lines of code; they’re the guardians of your developer sanity. Treat them well, and they will make your coding adventures legendary!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lon0momogDXqKg5wZp_cWg.jpeg" /><figcaption>Photo by Possessed Photography from <a href="https://unsplash.com/fr/photos/vue-de-dessous-photographie-du-batiment-de-mur-rideau-M7V9rglHaFE">Unsplash</a></figcaption></figure><h3>Why Testing Is Great</h3><p>The benefits list is very long, but based on my past and current experiences, here are the main advantages I see in having great tests:</p><h4>🐛 Error Detection</h4><p>The primary goal of writing tests is to ensure your code works well. Tests help to identify scenarios with errors that you have not seen during the development. This ensures no regression has been introduced elsewhere in the application or in the code of the feature/fix being developed.</p><h4>🧱 Code Quality and Reliability</h4><p>Testing encourages developers to write modular, well-organized, and maintainable code. It is easier to test small methods than a large one with hundreds of lines and conditional statements. Using a well-tested class makes you feel confident to use it, as tests anticipate various scenarios.</p><h4>🧗 Confidence</h4><p>When delivering well-tested code, you feel a lot more confident, as writing tests forces you to consider numerous possibilities, scenarios, and error handling. It makes you an expert of your changes! Writing tests can also be seen as a form of development, so you gain more experience with your ecosystem, how it works.</p><p>Regarding refactoring, it becomes an easy task because you clear up any doubt when you are sure that your method or component delivers the exact same result as before.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F3og0IQVQ5YwY6Lwk6c%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2F3og0IQVQ5YwY6Lwk6c%2Fgiphy.gif&amp;image=https%3A%2F%2Fmedia1.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExYTNzZmRjdmcxanV2cWY2YnNlZTBsMWNsYmNpZm04aWNydGJkemI2ciZlcD12MV9naWZzX2dpZklkJmN0PWc%2F3og0IQVQ5YwY6Lwk6c%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="322" frameborder="0" scrolling="no"><a href="https://medium.com/media/3a8812b22e8c1521506f7d83405c2a37/href">https://medium.com/media/3a8812b22e8c1521506f7d83405c2a37/href</a></iframe><h4>📖 Code Sharing and Readability</h4><p>This is, in my opinion, an underestimated advantage of having tests. Because tests contain human-readable phrases, such as “when … it should …”, this helps any developer understand the code’s purpose more quickly. In most cases, the code with good naming and comments should be enough, but sometimes you cannot avoid complexity and tests could clearly act like technical specifications. Test Driven Development (TDD) has pushed this concept further, by writing tests before writing code. If you are interested in this approach, there are a lot of articles about it online.</p><h3>Best Practices</h3><p>Here are some main best practices and principles from my experience as a frontend developer. But it surely can apply to other types of development.</p><h4>🕙 Test Whenever You Can</h4><p>Even if it seems trivial, it is always a good thing to add tests; it will force you to reread your code and avoid mistakes. If your class does practically nothing, the tests should be straightforward. If you think you don’t have time, remember that testing reduces the overall cost of software development, so the time to write tests should be considered when estimating tasks. Consider finding a bug during your development; it will save time for the QA (and/or the customer support team if the bug has shipped into production), and of course also for you, because you do not have to come back to this subject and do a fix.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F5FoxedYWNVmO4%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2F5FoxedYWNVmO4%2Fgiphy.gif&amp;image=https%3A%2F%2Fmedia2.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExc2ZlNm5rNXBpeGliMjRqa2MwcDMzZ3d5NW50bnhpamp4dGF5djBwZiZlcD12MV9naWZzX2dpZklkJmN0PWc%2F5FoxedYWNVmO4%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="218" frameborder="0" scrolling="no"><a href="https://medium.com/media/1dce6832795601e993990825873b5f3a/href">https://medium.com/media/1dce6832795601e993990825873b5f3a/href</a></iframe><h4>☮️ Tests should be easy to write</h4><p>As said before, tests can improve code quality. One great principle I apply to myself is that tests should be (reasonably) easy to write. If it is hard, it probably means that the code can be refactored to be more modular and well-organized. Modern frontend frameworks allow you to split the application into small code chunks; take advantage of this!</p><blockquote><em>The man who moves a mountain begins by carrying away small stones.</em></blockquote><blockquote><em>― Confucius, Confucius: The Analects</em></blockquote><h4>🎨 Mock the dependencies</h4><p>Sometimes, tests can be painful, not because of the code complexity, but because of dependencies. This problem leads us to another good practice: mock the dependencies. This is always the first thing I do when starting to test a class. I check the constructor, then mock everything in it. For components, I also check its template to mock complex components, directives, or pipes that are used. This helps isolate tests correctly and simplifies them. It also prevents errors like a missing provider needed in a dependency. When your mocks are set, it is also easy to define the context of the tests by setting what the dependencies should do.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2Fd89BjPGXijE0imFOxA%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Fd89BjPGXijE0imFOxA%2Fgiphy.gif&amp;image=https%3A%2F%2Fmedia0.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExZG8wcWNsbjR2aDh4NDR5NnFwa3pkaW50bHprenMxdmNwcjkzaTJqOCZlcD12MV9naWZzX2dpZklkJmN0PWc%2Fd89BjPGXijE0imFOxA%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="333" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef93632b00366ffb025268365f8fb5c0/href">https://medium.com/media/ef93632b00366ffb025268365f8fb5c0/href</a></iframe><h4>📈 Do not rely only on coverage</h4><p>Coverage is a great tool to determine if a codebase is mostly tested or not, by helping to not forget to test some cases or parts of the code. Nevertheless, you should not rely only on coverage when writing tests. For instance, you can trigger a method without testing the result; the coverage will be good, but maybe the method result is wrong. So keep in mind that human eyes are still needed to ensure a well-tested codebase.</p><h4>🤝 Don’t hesitate to test other developers’ code</h4><p>When you discover someone else’s code that is not tested, it can be a great opportunity for knowledge transfer. By testing it, you will learn how this part of the codebase works, and it will be easy for you to make changes in the future. It is also an opportunity to confidently refactor old code and possibly find bugs.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F6FymBmqKeBrl6%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2F6FymBmqKeBrl6%2Fgiphy.gif&amp;image=https%3A%2F%2Fmedia1.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExbXNqNHV3MDNvb2tyMXhtOWJmM21mOWo0cGdob2NiYTV5ejNhaDNuZCZlcD12MV9naWZzX2dpZklkJmN0PWc%2F6FymBmqKeBrl6%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="383" frameborder="0" scrolling="no"><a href="https://medium.com/media/611a95cf12fae96416d7d3fb21f414c8/href">https://medium.com/media/611a95cf12fae96416d7d3fb21f414c8/href</a></iframe><h3>To Go Further</h3><p>These are some tips and guidelines I wanted to share. I intentionally omitted some topics that are also very interesting, for instance, how to integrate tests in the continuous integration and deployment process (CI/CD). You can find a lot of great articles about this online; in this article, I only wanted to encourage you to enjoy writing tests!</p><p>At Shippeo, we are trying to improve our tests every day by increasing our coverage and by making tests mandatory in all our pull requests! 💪</p><p>Thanks for reading. Now, if you want to go further and see technical examples, you can go to the second article: <a href="https://medium.com/shippeo-tech-blog/angular-testing-mastery-the-best-practices-to-skyrocket-your-skills-examples-0a43220b949d">https://medium.com/shippeo-tech-blog/angular-testing-mastery-the-best-practices-to-skyrocket-your-skills-examples-0a43220b949d</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=369ef71a4456" width="1" height="1" alt=""><hr><p><a href="https://medium.com/shippeo-tech-blog/angular-testing-mastery-the-best-practices-to-skyrocket-your-skills-369ef71a4456">Angular Testing Mastery: The Best Practices to Skyrocket Your Skills!</a> was originally published in <a href="https://medium.com/shippeo-tech-blog">Shippeo Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Antifragile Supply Chains with Visibility: Why resilience is no longer enough]]></title>
            <link>https://medium.com/shippeo-tech-blog/building-antifragile-supply-chains-with-visibility-why-resilience-is-no-longer-enough-aae130d0e32a?source=rss----25b860162bea---4</link>
            <guid isPermaLink="false">https://medium.com/p/aae130d0e32a</guid>
            <dc:creator><![CDATA[Lucien Besse]]></dc:creator>
            <pubDate>Fri, 24 Nov 2023 17:06:22 GMT</pubDate>
            <atom:updated>2023-11-28T12:40:16.619Z</atom:updated>
            <content:encoded><![CDATA[<p>In the not so distant past, supply chains were experiencing a level of pressure they had never experienced before. In the US, the Federal Reserve Bank of New York’s famous <a href="https://www.newyorkfed.org/research/policy/gscpi#/interactive">Global Supply Chain Pressure Index</a> showed clearly that pressure on supply chains was the highest in 30 years.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_DuO9opA3t2UDXpL" /></figure><p>When we fast forward to today, this pressure has dissipated for a number of reasons. Consumer demand has gone back to pre-Covid levels and supply is no longer a bottleneck. Transportation prices are decreasing as a result, supplier shortages are under better control, and some lead times are compressing. But what has not changed is the extreme volatility in the supply chains in the face of geopolitical tensions and a cost of living crisis. What has to be accepted as a new normal is that supply chains will always be under some kind of pressure.</p><p>Now, whether the pressure is going up or down, it is not so much the pressure itself that is driving the stress in supply chains; all supply chain leaders know they need to be good at dealing with pressure. The real cause of the stress is volatility and with volatility comes uncertainty. It becomes very difficult to know in which direction things are going to evolve tomorrow when you are operating in uncertain times. You never know if you are out of the woods or just in the calm before the storm.</p><p>It is blindingly obvious to say that the best way to deal with uncertainty is to become as certain as you can be. In supply chain, this means being able to understand and manage demand fluctuations, supply disruptions, and market trends on a continual basis. Because the cadence between planning and execution has shrunk so much, having real-time information on the happenings in the supply chain are crucial for a supply chain leader. Managing this is not manually possible and this is why supply chain leaders need to understand the fierce urgency of investing in digitalization and real-time technologies.</p><p>Such an investment to prepare for and weather the storms of tomorrow is not a question of if but rather when. Many leaders wonder when is the right moment to invest, and our thesis is that the best time to build resilience is before the next crisis hits — to use an analogy, you don’t invest in anti hacking technology while you are being hacked — and given supply chain disruptions are unlikely to settle down, it needs to be now. This is the fierce urgency facing supply chain leaders today.</p><h3>But is resilience enough?</h3><p>While supply chain resilience has been a main focus in recent times, especially throughout the pandemic, we believe it does not go far enough. There is a step beyond resilience that organizations should be aspiring to achieve. It goes further than just protecting ourselves from a disruption to get back to the position we started from — the classic bend but not break scenario. It refers to a state where disruptions and uncertainty are embraced, one where supply chains thrive, strengthen, and ultimately land in a better position compared to where they started.</p><p>Such a framework comes from Nassim Nicholas Taleb, who in 2012 published the New York Times bestseller ‘<a href="https://en.wikipedia.org/wiki/Antifragile_(book)">Antifragile: Things that gain from disorder</a>”, and coined the term <em>antifragility,</em> which he defined <em>as a property of systems that can increase their capacity to thrive as a result of shocks, attacks, failures, or any kind of volatility</em>. As the old adage goes, antifragility embodies the spirit of ‘what doesn’t kill you makes you stronger’.</p><p>In the context of supply chains, all organizations fall into one of four potential categories, from fragile through to antifragile.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ASgt3biMYHXEi3rD" /></figure><ul><li><strong>Fragile: </strong>Supply chains in this state lack the systems and processes to resist a disruption. As a result, their supply chains simply break when a disruption hits.</li><li><strong>Robust:</strong> Organizations have implemented some systems and processes to help their supply chain to resist a disruption, but they are unable to or are not flexible enough to adapt to the disruption easily enough. Their survival may purely be down to luck or to manual processes within their organization, and they could end up in a worse position than they were when they started, especially if the manual processes don’t scale.</li><li><strong>Resilient: </strong>Resilient supply chains have the systems, processes and mindset to adapt to a disruption, allowing them to ‘bend but not break’ and return to the pre-disruption state. But they lack the ability to make structural changes in upstream processes to avoid disruptions from repeating.</li><li><strong>Antifragile:</strong> A combination of advanced systems providing visibility into real-time operations at all times, dynamic processes and strategy allows a supply chain to thrive under uncertainty and disruptions, converting challenges into strengths and greater success. Such a supply chain is not only resilient but also learns and adapts upstream processes as well, thereby fixing both the immediate and the structural issues all in one go.</li></ul><p>We firmly believe that this notion of antifragile supply chains will inspire supply chain leaders to think differently about building resilience. At Shippeo, we are building visibility solutions to help our customers foster antifragile and sustainable supply chains. By combining end-to-end high-quality visibility data with advanced systems, services and strategies, we believe supply chain management can become far more adaptive, and supply chains can increase their value as a competitive advantage.</p><h3>What is required to achieve antifragility?</h3><p>We believe that there are two key attributes are needed to enable antifragile supply chains:</p><ol><li><strong>High data quality</strong>: Your internal teams and ecosystem partners need to be able to rely upon your data in order to increase agility, responsiveness, and their knowledge of the impact any number of uncertainties could have. This high quality data is also the foundation that enables trust between an organization and its ecosystem to enable end-to-end decision alignment and process orchestration. As the old adage goes, ‘what you can’t see, you can’t fix,’ and high quality data allows you to see what is going on in the supply chain at all times.</li><li><strong>A true system of engagement</strong>: Relying on a system of information won’t suffice. Just knowing that something happened in real time, while useful, doesn’t help. You have to be able to make a judgment on whether this disruption has a material impact or if it’s merely an outlier: you must be able to convert this visibility data into actionable insights and probability calculations to fix the issue on hand urgently but also constantly evaluate what <em>might</em> happen in the future to better prepare for it.</li></ol><p>In order to help organizations build antifragile strategies, Shippeo has adopted a two-pronged strategy. We believe the combination helps customers get the greatest value from their digital supply chain investments.</p><h3>1. Focus on network depth first, then breadth</h3><p>Should visibility providers focus their investments on building a large visibility network as quickly as possible? Maybe helped by some mergers and acquisitions, to acquire some data sources? Unfortunately this is usually at the expense of data quality.</p><p>Over the past 10 years, we have built one of the largest multimodal visibility networks on Earth, enabling tracking in more than 110 countries around the world, via any mode of transport. However, our difference is that we build deeper integrations so that our network contains richer visibility data. We are willing to take additional time to create a reliable integration with a carrier’s TMS in order to gain additional contextual data around every shipment.</p><p>For customers, this means they get so much more than live shipment locations on a map. The context is provided in the form of order information and status changes as shipments move from A to B. When paired with GPS location data from telematics devices or mobile apps, a more comprehensive and accurate view of movements is attained. For example, when a truck arrives at a delivery site, the telematics system uses GPS to determine that the truck is onsite. However only by accessing the schedule via TMS integration can a visibility platform determine if the truck has actually started unloading. Historical contextual data also feeds our machine learning algorithms, which enables exceptionally high data quality and predictive ETAs essential for antifragile supply chains.</p><h3>2. Build a system of engagement, not just information</h3><p>When it comes to product strategy, should visibility data only remain in a system of information, showing dots on a map, or should this data be transformed into actionable insights which can be shared across your ecosystem, boosting collaboration and streamlining processes?</p><p>We announced the foundation of our product vision earlier this year when we introduced our <a href="https://www.shippeo.com/transportation-process-automation">Transportation Process Automation™</a> (TPA™) innovation. TPA relies on high quality data to automate transportation processes and improve operational efficiency. Underpinned by high-quality visibility data, TPA streamlines end-to-end transportation processes to reduce decision-making latency and improve operational efficiency with AI-enabled recommendations and collaborative workflows. Some customers are already benefiting from their first automated processes, including global automotive OEM Renault Group. This has the potential to bring enormous benefits to organizations and will be a core component to enable antifragile supply chains.</p><h3>Helping companies build antifragile and sustainable supply chains</h3><p>Our mission is to help our customers build their own antifragile supply chains, built solidly on a foundation of market-leading high quality data. By combining our focus on deeper network integrations with a product roadmap towards the most collaborative, engaging and automated platform experience, we believe our approach to visibility will continue to set us apart.</p><p>What makes us so confident? Well, since our company was founded in 2014, we have invested heavily to continually improve our data quality for customers. For the past three years, we are delighted to have consistently scored the highest reviews and recommendation scores in analyst reports and on peer review websites, with customers and analysts alike attesting to this. In fact, we were named a Customers’ Choice in the 2023 Gartner Peer Insights Voice of the Customer Report: Real-Time Transportation Visibility Platforms, with the highest overall recommendation rate in the market, at 96%.</p><p>While Shippeo is doing part of the job, we rely on an ecosystem of partners to accelerate this transformation and unlock additional value from visibility. Whether we’re talking about TMS, supply chain planning solutions, WMS, control towers, digital twins, or others, using visibility data to automate transportation processes can only be achieved in collaboration with the wider ecosystem, and we’re thankful to have the support of <a href="https://www.shippeo.com/en/network/partners">over 1000 integration partners</a> to date, with many more to come.</p><h3>With uncertainty comes opportunity</h3><p>Antifragile supply chains can help organizations make the best of bad situations they find themselves in by embracing uncertainty in a way that unlocks new opportunities. This entails promoting a methodology centered on a new suite of best practices, including things like:</p><ul><li>Frequent experimentation with scenario planning</li><li>Prediction of disruptive events and ability to manage them in real-time</li><li>Learning system to ensure structural upstream processes are adjusted dynamically</li><li>Process automation to enable dynamic decision making</li><li>Stochastic demand and supply planning</li><li>Flexible systems capable of adjusting to uncertainties</li></ul><p>So what are the five steps to build your antifragile supply chain?</p><ol><li><strong>Define</strong> what fragile, robust, resilient and antifragile states look like across your supply chain, so you can assess how well uncertainty is dealt with. This definition is not a ‘one size fits all’ and it needs to be relevant to your supply chain</li><li><strong>Get executive-level support</strong> to champion antifragility as a key pillar of your supply chain strategy, and ensure wider teams understand the concept and benefits</li><li><strong>Work together with tech leaders</strong> to plan how to leverage digital tools to increase supply chain antifragility</li><li><strong>Trust the ecosystem </strong>within your supply chain to help you make dynamic decisions with digital tools</li><li><strong>Start your journey with real-time visibility. </strong>Let the data from a visibility platform inform your end to end processes, take guesswork out to make the uncertain certain, and enable an antifragile supply chain</li></ol><p><strong>To learn more about how your organization can take advantage of end-to-end real-time visibility and Transportation Process Automation™ as part of your antifragile supply chain strategy, </strong><a href="https://www.shippeo.com/en/book-a-demo"><strong>get in touch with one of our experts today</strong></a><strong>.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aae130d0e32a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/shippeo-tech-blog/building-antifragile-supply-chains-with-visibility-why-resilience-is-no-longer-enough-aae130d0e32a">Building Antifragile Supply Chains with Visibility: Why resilience is no longer enough</a> was originally published in <a href="https://medium.com/shippeo-tech-blog">Shippeo Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlocking the power of toggle features: Elevating frontend development with Angular and…]]></title>
            <link>https://medium.com/shippeo-tech-blog/unlocking-the-power-of-toggle-features-elevating-frontend-development-with-angular-and-2a79c37cf494?source=rss----25b860162bea---4</link>
            <guid isPermaLink="false">https://medium.com/p/2a79c37cf494</guid>
            <category><![CDATA[continuous-delivery]]></category>
            <category><![CDATA[real-time-visibility]]></category>
            <category><![CDATA[angular]]></category>
            <category><![CDATA[supply-chain]]></category>
            <category><![CDATA[launchdarkly]]></category>
            <dc:creator><![CDATA[Imane Benmelouk]]></dc:creator>
            <pubDate>Thu, 09 Nov 2023 13:58:48 GMT</pubDate>
            <atom:updated>2023-11-09T13:58:48.365Z</atom:updated>
            <content:encoded><![CDATA[<h3>Unlocking the power of toggle features: Elevating frontend development with Angular and LaunchDarkly</h3><h4>Boost your software development game with Angular’s skilful toggle feature management, driven by the power of LaunchDarkly</h4><h3>Welcome to the realm of toggle features in Angular applications</h3><p>In the world of software development, releasing new features and updates is mandatory for staying ahead of the competition. However, no matter how well your QA processes, there is always an element of risk when rolling out new features into the final environment. This is where <strong>toggle features</strong> come into play.</p><p>Toggle features, also known as<strong> feature flags</strong>, offer a powerful solution to mitigate risk when rolling out new features and manage their activation within applications.</p><p>Whether you are a <strong>developer</strong> seeking ways to improve the release process or a manager looking for effective feature management strategies, this article is for you. We will explore how<strong> LaunchDarkly</strong> empowers software developers to effortlessly implement and manage toggle features, revolutionizing your <strong>development workflow</strong> 🤯.</p><p>By the end of this article, you will gain a better understanding of toggle features and how they can transform your development habit.</p><p><em>So, let’s dive into it</em> 🚀.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7oAy-B6FP_iDCpqV2KV1fA.jpeg" /><figcaption>Source: <a href="https://unsplash.com/fr/photos/IU0K4L7c6do">https://unsplash.com/fr/photos/IU0K4L7c6do</a></figcaption></figure><h3>What are toggle features?</h3><p>Toggle features are a software development technique that allows developers to turn specific features <strong>ON </strong>or <strong>OFF</strong> for specific audiences, based on parameters set. These parameters can include anything from a user’s information, such as email address, to any other kind of data, depending on what you provide your configuration tool with.</p><p>While it is true that toggling the activation and deactivation of features is <strong>codeless</strong>, implementing toggle features does require integrating the provided SDK and adding code to test and manage the flag activation. The inclusion of this additional code requires a bit of work upfront.</p><p>Hence, each use of a LaunchDarkly flag first requires a <strong>small code implementation</strong>, which must then be removed once the feature is fully deployed, tested and approved within the production environment. Adding toggle features unavoidably adds technical debt that must be managed over time.</p><p>However, once the toggle features are set up, it allows for a diverse set of stakeholders, including developers, product managers, and QA teams, to easily control the visibility and functionality of app features. This can be achieved without the need to release a new version of the application, thanks to the user interface provided by the toggle feature tool you use 😎.</p><h3>What are toggle features useful for?</h3><blockquote><strong><em>Controlled Rollouts:</em></strong><em> Developers can use toggle features to introduce new features to a limited group of users, giving them the opportunity to test and evaluate before releasing them to a wider audience. This controlled approach helps </em>minimize potential risks<em> and can prevent issues or bugs from affecting a significant number of users.</em></blockquote><blockquote><strong><em>Continuous Deployment:</em></strong><em> Toggle features support a </em>continuous delivery approach<em> that enables the introduction of updates and new features to users in a smooth and agile manner. This ensures that users always have access to the latest version of the app and can enjoy the best user experience.</em></blockquote><h3>They have their downsides too...</h3><p>Nevertheless, it’s important to note that using too many toggle features for minor updates or small changes can make the code harder to understand in the future. This can lead to longer development cycles and increased costs. For example, it is advisable to prioritize utilizing <strong>toggle features for larger-scale page switches</strong> rather than relying on numerous conditional statements (“if” statements) fragmented throughout components, services, or pipes. This approach ensures that the code remains easy to read and avoids unnecessary complexity.</p><p>If toggle features are not managed properly, they can create significant technical debt, which refers to the extra development work required to fix issues that arise as a result of implementing those <strong>flags</strong> (toggle features).</p><p>In addition, with a dynamic flag implemented, we have to call LaunchDarkly each time we need the current flag value. It can lead to a lot of queries for a value that rarely changes. As a result, using too many toggles can impact performance on production even when no new features is being released for the user.</p><pre>&lt;ng-container *ngIf=&quot;featureFlagActivated&quot;&gt;<br>  &lt;!-- New feature to display if flag is activated --&gt;<br>&lt;/ng-container&gt;</pre><h3>Why did we choose LaunchDarkly ?</h3><p>When it comes to implementing toggle features, <a href="https://launchdarkly.com/">LaunchDarkly</a> is a popular tool that developers use to manage features. LaunchDarkly offers a wide range of possibilities, including the ability to create and manage toggle features, segment users based on specific criteria, and monitor <strong>flag performance</strong> in real time. LaunchDarkly also provide integrations to a variety of other tools and platforms, making it a valuable addition to any app development team.</p><p>That is how all these factors motivated us at <strong>Shippeo</strong> to explore this tool a little deeper. Then, we had to make sure that <strong>all Shippeo tools complied with LaunchDarkly</strong> to ensure its integration made sense.</p><p><em>Now that all the key points have been checked. Let’s talk about LaunchDarkly 🤗.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RlHCbpSlyVe9JO11PbG9Uw.png" /></figure><h3>Single point of failure: a risk to manage</h3><p>When adopting a new deployment process like the integration of LaunchDarkly, it is crucial to address the risk of a single point of failure.</p><blockquote><em>💡</em><strong><em>A single point of failure occurs when a toggle feature controls critical functionality, and its failure can result in the entire application becoming unavailable or experiencing critical errors.</em></strong></blockquote><p>To minimize this risk, it is vital to follow <strong>best practices</strong>, such as implementing a backup plan in case of failure and carefully evaluating which features are essentials for the application’s functionality.</p><p>And that is why we chose to add the use of the LaunchDarkly Relay Proxy.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FPApUm1HPVYlDNLoMmr%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FPApUm1HPVYlDNLoMmr%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FPApUm1HPVYlDNLoMmr%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="301" frameborder="0" scrolling="no"><a href="https://medium.com/media/874b3cb914050977ae717d509b73a6b0/href">https://medium.com/media/874b3cb914050977ae717d509b73a6b0/href</a></iframe><h3>Using a relay proxy with LaunchDarkly at Shippeo?</h3><p>Implementing a <a href="https://docs.launchdarkly.com/home/relay-proxy">relay proxy when using LaunchDarkly</a> can provide a number of benefits, including increased security, improved scalability, and easier maintenance and management.</p><p>If you are using LaunchDarkly in a production environment, or with <strong>sensitive data</strong>, a relay proxy can be a valuable tool for ensuring the reliability of your app while improving your network’s speed. Thereby, it allows us to retrieve LaunchDarkly toggle feature informations much <strong>faster</strong>.</p><blockquote><em>💡 </em><strong><em>A relay proxy is a lightweight server that sits between your app and the LaunchDarkly SDK, intercepting requests and responses in order to forward</em></strong><em> </em><strong><em>them to the appropriate endpoint.</em></strong></blockquote><p>This can be particularly useful for <strong>security</strong> purposes, as it can prevent sensitive data from being sent directly to the LaunchDarkly servers. Instead, the relay proxy handles the data and ensures that it is secure.</p><p>In addition to security benefits, a relay proxy can also <strong>reduce the load</strong> on your app and the LaunchDarkly servers. By handling requests and responses on its own, the relay proxy can help improve overall performance and reduce the risk of service interruptions.</p><h3>The Angular side</h3><p><em>Now, let’s accommodate front-end developers by speaking their language 🤖 !</em></p><p>To start on our <strong>Angular</strong> app and to ensure an easy integration, we encapsulated the LaunchDarkly snippet within an Angular service, providing a simplified and efficient implementation.</p><pre>  const options: LDOptions = {<br>    allAttributesPrivate: true,<br>    baseUrl: &#39;relayproxy-client-url&#39;,<br>    bootstrap: &#39;localStorage&#39;,<br>    eventsUrl: &#39;relayproxy-client-url&#39;,<br>    streamUrl: &#39;relayproxy-client-url&#39;,<br>  };<br><br>  let userContext: LDContext = {<br>    anonymous: true,<br>    kind: &#39;Your organization name&#39;,<br>  };<br><br>  this.client = initialize(launchdarklyClientId, userContext, options);<br>  this.client.on(&#39;ready&#39;, () =&gt; this.isReady$.next(true));<br>  this.client.on(&#39;change&#39;, () =&gt; this.refresh$.next());<br>  this.client.identify(userContext).then();ty</pre><p>Once the SDK is integrated, you can use LaunchDarkly’s toggle feature management tools to control the visibility and functionality of certain features within your app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3T8AAvHIkd_j538Goz0F_g.png" /></figure><p>In addition, LaunchDarkly’s real-time monitoring and analytics tools can help you <strong>track the performance</strong> of your features and quickly identify any issues or bugs that may arise. This can be particularly useful for catching issues early on.</p><h3>What’s next for Shippeo’s deployment process?</h3><p>In conclusion to this article, adopting a new workflow that incorporates toggle features, such as LaunchDarkly, brings <strong>significant advantages to the development cycle</strong>.</p><p><strong>Shippeo </strong>has already implemented LaunchDarkly to continue deploying code without impacting production. This usage primarily focuses on the development of significant new features. The ultimate goal is to completely transform the <strong>production deployment process</strong> by replacing the delivery of large code quantities with the <strong>simple activation</strong> of desired features, with the assurance that they have been throughly tested directly in the production environment through LaunchDarkly.</p><p>In fact, once a feature is successfully deployed and tested in the development environment, we will automate its entire process from deployment to production while making sure the feature is deactivated using Launchdarkly. This way, we can activate it when needed, without waiting for a new code release.</p><p>All of this will be achieved through this new automated process, facilitated by the numerous tools provided by LaunchDarkly, including GitHub, GCP, and Jira.</p><p>Hopefully, with these new automated processes, developers will no longer have “bugs in production” nightmares 😁.</p><p><strong>***</strong></p><p><strong>Thanks for reading! We hope our new approach to feature deployment will inspire your future projects!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a79c37cf494" width="1" height="1" alt=""><hr><p><a href="https://medium.com/shippeo-tech-blog/unlocking-the-power-of-toggle-features-elevating-frontend-development-with-angular-and-2a79c37cf494">Unlocking the power of toggle features: Elevating frontend development with Angular and…</a> was originally published in <a href="https://medium.com/shippeo-tech-blog">Shippeo Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to structure your Angular project with multiple apps within a monorepo?]]></title>
            <link>https://medium.com/shippeo-tech-blog/how-to-structure-your-angular-project-with-multiple-apps-within-a-monorepo-a01c9f409f95?source=rss----25b860162bea---4</link>
            <guid isPermaLink="false">https://medium.com/p/a01c9f409f95</guid>
            <category><![CDATA[angular]]></category>
            <category><![CDATA[nx]]></category>
            <dc:creator><![CDATA[Denis Martinho]]></dc:creator>
            <pubDate>Wed, 18 Jan 2023 12:37:22 GMT</pubDate>
            <atom:updated>2023-01-18T12:37:22.241Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Shippeo’s journey to large Angular architecture</h3><p>When your project becomes bigger and bigger, you will have to reorganize your folders to answer an important question asked by any developers on the project: “Where should I put my code ? 🤔”. Since you want to help them and still have a highly modular application, you should read this article for fresh ideas. I’ll take you through Shippeo’s architecture.</p><blockquote><strong><em>💡 You may wondering what “large” means?<br></em></strong><em>We are around 5k files (Html, ts, js, scss) which represent more than 200k lines of codes. Everything is split into ~110 libraries and 4 apps.</em></blockquote><h3>Context</h3><p>At Shippeo, we had one AngularJS app on one repository and three Angular apps on another repository. We had to switch from one repository to another following where the features may be implemented and sometimes we had to implement it twice. So we decided to move both apps into one repository and we start to migrate the old AngularJS to Angular. This migration will not be elaborated in this article but feel free to comment if you want one.</p><p>As a result, we had multiple modules in every app and we had two types of apps in the same repository. How can we share code among apps without copy-pasting every time?</p><h3>First: assemble the avenger 🦸‍♀️</h3><figure><img alt="A plant in a zoom meeting. The plant is an avenger too" src="https://cdn-images-1.medium.com/max/1024/1*u_HrqfYGCiQR-z02SNH6QQ.jpeg" /><figcaption>The plant is an avenger too</figcaption></figure><p>We started a meeting with only avengers (they are frontend developers) and we started talking. We ended up with this request list:</p><blockquote><em>By looking at the structure you can already tell what the app is all about;<br>Higher modularity;<br>Multiple frameworks compatible;<br>Easier code navigation;<br>Higher level of abstraction;<br>Separates both features and layers;<br></em><strong><em>More readable and maintainable structure;<br></em></strong><em>More cohesion;<br>Much easier to scale;<br>Less chance of accidentally modifying unrelated classes or files;<br>Much easier to add or remove application features;<br>And much more reusable modules.</em></blockquote><p>As you can see, we were asking for more <strong>maintainability</strong>, <strong>consistency</strong>, <strong>modularity, </strong>and <strong>separation of concerns</strong>. Saying things and writing them down was a big step: everyone agreed to change our project and we decided to start looking for some tools.</p><h3>What we have done?</h3><p>TLDR: Look at the schema</p><h4>It’s dangerous to go alone! Take CLIs</h4><figure><img alt="A workbench with tools for woodcrafting. Don’t use this tools on your computer." src="https://cdn-images-1.medium.com/max/1024/1*W9-4-mwdIo1dC6biuFtbOA.jpeg" /><figcaption>Don’t use the tools in this picture they will harm your computer</figcaption></figure><p><strong>Always look at Angular CLI</strong></p><p>The first evident weapon is Angular CLI. With the CLI you can create a library where you can share your code among your apps. The Angular documentation has a great article on <a href="https://angular.io/guide/creating-libraries">how to create a library</a>. We began to divide our apps into large feature libraries, with a core module library. We started to handle <strong>modularity</strong> by having these libraries and <strong>consistency</strong> by using the generator provided by Angular. We were almost pleased until we had circular dependencies and a very long building time. As the saying goes: <em>a bad workman blames his tools</em> and we were bad. 😅</p><p><strong>Step up your game with Nx</strong></p><p>We wanted to manage our code to avoid circular dependencies and speed up our building, testing, and linting so we ended up using <a href="https://nx.dev/">Nx from Nrwl</a>.</p><blockquote><strong><em>❓ What is Nx ?<br></em></strong><em>Nx is an opinionated smart, fast and extensible build system with first class monorepo support and powerful integrations (Angular, React, jest, etc.).</em></blockquote><p>As we began using Nx instead of Angular CLI, we saw how much time that will save us by using cache. Plus, <a href="https://nx.dev/using-nx/updating-nx">Nx helped us to migrate</a> our code to newer versions of Angular. Let’s check: <strong>maintainability</strong>.</p><p>But first, we need to tackle our dependency problem. By using the enforce-module-boundaries we started adding tags to enforce the<strong> separation of concerns </strong>and we ended a lot of undesired circular dependencies. We sliced each of our large libraries into three smaller libraries :</p><figure><img alt="One feature have two angular layers and a no framework layer" src="https://cdn-images-1.medium.com/max/277/1*6iyc7hi0CY2xNsc6k81G-g.png" /><figcaption>One feature have two angular layers and a no framework layer</figcaption></figure><p>As you can see, each layer relies on another one:</p><ul><li><strong>Components library</strong> can import an API library, a model library, or another component library. This library will have components or directives.</li><li>API <strong>library</strong> can import a model library or another API library. Note: we wrote API and not service because it handles API calls and it exposes business logic for other libraries.</li><li><strong>Model library</strong> is not an angular library but a typescript only library. No need to add more weight to classes and interfaces. It can import another model library</li></ul><p>No more unexpected import from a big barrel file.</p><blockquote><em>🕵️‍♂️: </em>Hey guys, why dont you use the <a href="https://nx.dev/structure/library-types">Nx recommandation about library types </a>?</blockquote><p>Good question <em>inspector </em>! Because, we wanted to separate the data access (API for us) from the model. Which will allow us to have multiple frameworks working with the same models. Maybe one day, to have these model libraries auto-generated. Remember: we want <strong>modularity</strong>.</p><p>Furthermore, we have an UI library which some components can be promoted to our design system (which is in another repository).</p><p>We are almost good: we have a solution to <strong>separate the concerns</strong>, to handle full <strong>modularity </strong>and <strong>consistency</strong>. Now, let’s assemble all the avengers with their new weapons for the final battle.</p><h3>A schema is worth a thousand words 🖼</h3><p>We have features in which we have 3 layers. Each layer is a library. Each layer can only import from a layer under him. You will end with this kind of structure :</p><figure><img alt="An Angular app using two features" src="https://cdn-images-1.medium.com/max/426/1*UxS6CcIadcFho1Am3ynqsw.png" /><figcaption>An Angular app using two features</figcaption></figure><p>It’s simple and elegant. Let’s add an app and some features with dependencies between them:</p><figure><img alt="An app using two libraries and another app using 4 libraries" src="https://cdn-images-1.medium.com/max/967/1*sOjdM-X26CG9AtHZMQwv2w.png" /><figcaption>An app using two libraries and another app using 4 libraries</figcaption></figure><h3>Your project is living 🌻</h3><p>A project is a living thing that is composed of humans and code. Since humans change and the code too, we should live together and not fight when it’s possible.</p><p>It’s important that everybody understand and be comfortable with the project architecture: we decided to do an annual review of the project structure. It’s easy, we ask our developers, before the meeting, to think of any problems they got or where they made a workaround. They write somewhere what is it then at the meeting we discuss it and we write what we can do to fix it. If the majority agrees we will add an issue to refactor our code and warn every developer of the decision.</p><p>Of course, we tackle every refactoring issue all along the year, this meeting is a health checkup.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tm057T83GgCnkmKI8Raw4Q.jpeg" /><figcaption>You can wear a lab coat at the meeting, we won’t judge</figcaption></figure><h3>Epilogue: What’s next for us?</h3><p>As you can see, we have a clear structure. But today we don’t use the angular schematics at full power. We want to create our <a href="https://nx.dev/packages/nx-plugin">Nx plugin</a> to generate our files as we want to follow our architecture.</p><p>We also tested to have multiple frameworks in the same repository and app. We had a ReactJs component library in our app and it worked like a charm. Our project’s <strong>modularity </strong>and <strong>maintainability</strong> is ready for the next best js/ts framework. 😎</p><p><strong>***</strong></p><p><strong>Thanks for reading! We hoped our architecture will help you in your projects as it helped us!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a01c9f409f95" width="1" height="1" alt=""><hr><p><a href="https://medium.com/shippeo-tech-blog/how-to-structure-your-angular-project-with-multiple-apps-within-a-monorepo-a01c9f409f95">How to structure your Angular project with multiple apps within a monorepo?</a> was originally published in <a href="https://medium.com/shippeo-tech-blog">Shippeo Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing state management in an Angular complex app]]></title>
            <link>https://medium.com/shippeo-tech-blog/implementing-state-management-in-an-angular-complex-app-a613ebe76c5d?source=rss----25b860162bea---4</link>
            <guid isPermaLink="false">https://medium.com/p/a613ebe76c5d</guid>
            <category><![CDATA[redux]]></category>
            <category><![CDATA[state-management]]></category>
            <category><![CDATA[angular]]></category>
            <category><![CDATA[ngrx]]></category>
            <dc:creator><![CDATA[Iker Bedecarrax]]></dc:creator>
            <pubDate>Mon, 28 Nov 2022 09:49:36 GMT</pubDate>
            <atom:updated>2022-11-28T09:49:35.954Z</atom:updated>
            <content:encoded><![CDATA[<p>This article explains what is a state manager, how does it work and why should complex applications use it. Here we will focus on Angular frontend complex applications as we will give an overview on the most famous state management solutions in Angular.</p><h3>Context</h3><h4>State management ? 🤔</h4><p>Nowadays, modern web applications are single pages composed of dynamic web components. Application components are conditionally displayed in the view regarding user’s permissions and navigation. Moreover they adapt their own rendering related to their specific data. The data linked to a component that defines its rendering is then called a <strong>state</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*FHlaBpm_2ahhyA3B1Wonng.png" /></figure><p>Then the state management is basically the management of all data states within an application.</p><h4>Avoid the mess 🤯🧹</h4><p>State management can quickly become a big issue if developers do not have scalability in mind as the application grows.</p><p>In small applications, component states can be accessed and modified internally. However, the complexity increases with data sharing between components. In large applications, components often manipulate and share the same data sources by reference, and you can easily lose the current data state.</p><p>As the application grows, the number of states grows as well and become more complex as well. They might be accessed from plenty of components or controls among several libraries. At this point, the management of the states becomes a hassle to maintain.</p><h4>A good SM recipe 👨‍🍳</h4><p>A recipe for a good state management should at least gather the three following ingredients:</p><ul><li><strong>Performance</strong>: Actions on state have to be asynchronous as the action queue asked by components does not lead to performance leaks</li><li><strong>Consistency: </strong>is fundamental to make all components share the same data state at every moment</li><li><strong>Visibility</strong>: Each data update has to be noticed to let developers time travel through these changes and debug easily</li></ul><h3>Solutions💡</h3><h4>Patterns</h4><p><strong>Observer design pattern</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*zvi8AKA6AmAlYjD_W_7WRA.png" /></figure><p>With the power of <a href="https://en.wikipedia.org/wiki/Observer_pattern">observer pattern</a>, we can define shared data as observables where a modification made by one component on a specific observable updates all components that observe the succession of events emitted by this observable. Such components are called observers and the succession of events is simply the succession of data states.</p><p>This solution brings asynchronism and guarantees <strong>performances </strong>while scaling.</p><p><strong>Reducers</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xmjsLiRnPD3z92Ki0dOgUQ.png" /></figure><p>Based on <a href="https://redux.js.org/">Redux</a> library for React, a reducer is a state manager that provides a single <strong>Store </strong>to held the global application state as an object tree. Whenever the store is updated, it will update the components subscribed to it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*ofQijbSE0fRehP3BFArCjg.png" /></figure><p>A reducer also let developers define <strong>Actions</strong> that are static information about the event that initiate a state change. Actions are linked to pure reducer functions in order to mutate the state. A reducer function will consider the previous state and action as parameters, and return the application state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*l7BJobZMxM345hYELV_exQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/471/1*U0pDk-kfLmNdQbzkU4jVZA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/1*z6g21NIpGEvENbcMQeE1_w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/993/1*9A7TC42ogDPGfHqw93NqgA.png" /></figure><p><a href="https://www.educative.io/blog/understanding-redux">Learn more about Redux</a></p><h4>Implementations in Angular</h4><p><strong>Rxjs</strong></p><p><a href="https://angular.io/">Angular</a> needs the help of <a href="https://rxjs.dev/"><strong>Rxjs</strong></a><strong> </strong>to implement the <em>Observer pattern</em> and solve multi directional data access and modification issues.</p><p>It is possible to manage multiple states using only Rxjs by reproducing based on a Subject the Actions and the Selectors that a reducer library provides. But this custom way might be messy to manage in large applications due to the lack of visibility and the complexity to debug.</p><p><strong>NgRx / Ngxs / Akita</strong></p><p>In large applications, centralize states into one single application store can be very powerful and enhance scalability. Three main Angular libraries implement reduce state management to achieve this goal :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/1*1rwXLhN9hLp4vXzuDKBZ0Q.png" /></figure><ul><li><a href="https://v8.ngrx.io/docs">NgRx</a>: This is the original state management library for Angular and remains the most exhaustive and customizable solution. Consequently it needs a big boilerplate to use it. As the leading solution, it brings lot of dedicated tools and a great community.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/411/1*OqpHBP_UjI5CvzikoGV0Qg.png" /></figure><ul><li><a href="https://www.ngxs.io/">Ngxs</a>: Ngxs is the second famous solution to manage states in Angular. As it is more recent than NgRx, it is based on modern Typescript features that let reduce boilerplate.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*cmGurT-OyVqKccKeQyYUww.png" /></figure><ul><li><a href="https://opensource.salesforce.com/akita/">Akita</a>: Akita is newer than the other solutions and is more common in smaller applications as it is light and encourages simplicity with small boilerplate.</li></ul><h3>Choose a reducer</h3><h4>Benchmarks of Angular reducers</h4><p>A <a href="https://ordina-jworks.github.io/angular/2018/10/08/angular-state-management-comparison.html">helpful article</a> shows up benchmarks for these solutions that we can analyse thinking about our application :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ngj7zn3yfP01fxUxiM9Nlg.png" /></figure><h4>The choice of Shippeo</h4><ol><li>First, as our application is vast and complex, we need the most customizable solution with great tools and features package, that is why we won’t choose <strong>Akira</strong>.</li><li>Then, <strong>Ngrx</strong> is for now the only solution supported by <a href="https://nx.dev/">Nx</a> that we use as project manager. Using <strong>Ngrx</strong> will let us take advantage of Nx powerful machinery as schematics and upgrade automatization.</li></ol><p>As a result, we decided in consortium to choose <strong>Ngrx</strong> that seems the best to fit our specific needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JUkj73DMqOqXm6Wllv2yAw.png" /></figure><h3>Summary</h3><ul><li>State management becomes necessary as the application grows and increases its complexity. We avoid the mess accessing components data and maintain performance in scalability.</li><li>Angular already uses the capabilities of the observer pattern provided by <strong>Rxjs</strong> library giving us access to Subject and Observable flows</li><li><strong>Ngrx</strong> is an Angular state management solution based on Redux pattern paired with <strong>Rxjs</strong>. We basically chose this solution among the others because it is the most complete and the only one supported by <strong>Nx</strong> until now.</li></ul><h3>Sources</h3><ul><li><a href="https://ordina-jworks.github.io/angular/2018/10/08/angular-state-management-comparison.html">NGRX vs. NGXS vs. Akita vs. RxJS: Fight! - Orjan De Smet</a></li><li><a href="https://www.educative.io/blog/understanding-redux">Understanding Redux: Beginner&#39;s guide to modern state management</a></li><li><a href="https://redux.js.org/tutorials/essentials/part-1-overview-concepts">Redux Essentials, Part 1: Redux Overview and Concepts | Redux</a></li><li><a href="https://dev.to/this-is-angular/choosing-a-state-management-library-for-progressive-reactivity-in-angular-19j8">Choosing a State Management Library for Progressive Reactivity in Angular</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a613ebe76c5d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/shippeo-tech-blog/implementing-state-management-in-an-angular-complex-app-a613ebe76c5d">Implementing state management in an Angular complex app</a> was originally published in <a href="https://medium.com/shippeo-tech-blog">Shippeo Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tableau Relationships vs. Joins (2/2)]]></title>
            <link>https://medium.com/shippeo-tech-blog/tableau-relationships-vs-joins-2-2-cf1d48c8057c?source=rss----25b860162bea---4</link>
            <guid isPermaLink="false">https://medium.com/p/cf1d48c8057c</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[tableau]]></category>
            <category><![CDATA[supply-chain-analytics]]></category>
            <category><![CDATA[real-time-visibility]]></category>
            <dc:creator><![CDATA[Mathieu De Launay]]></dc:creator>
            <pubDate>Thu, 06 Oct 2022 09:43:11 GMT</pubDate>
            <atom:updated>2022-10-06T09:43:11.823Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why the choice is not always obvious</h3><p>In the first part of this story <a href="https://medium.com/shippeo-tech-blog/tableau-relationships-vs-joins-1-2-b78dd9cc373d">here</a>, we’ve outlined what improvements Tableau relationships bring and how convenient they can be compared to the old-fashioned Tableau joins. But, with a bit of experience on the matter, we would like to highlight a few pain points that you may need to be aware of. We won’t cover all of them in this article, but <strong>here are some of the limitations and tips we’ve learned from our own experiences using Tableau solutions.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xkNpKdlrVbVhmG5n.jpg" /><figcaption>Photo by Vladislav Babienko on <a href="https://unsplash.com/photos/KTpSVEcU0XU">Unsplash</a></figcaption></figure><h3>Watch out for performance</h3><p>One of the significant benefits of relationships is that Tableau autonomously generates and optimizes the queries to maintain the right granularity for each of your KPIs. This capacity to remove useless joins in the queries is called <em>join culling</em>. Yet, in some conditions, the query generation can be unexpectedly complex, leading to heavy and slow queries. The more complex the relationships are, the more Tableau will struggle to eliminate the useless joins and filters in the queries.</p><p>One of the needs we have at Shippeo is to display the right data to our customers. Here is an example similar to our use case.</p><p>Let’s say:</p><ul><li>we are a retailer with several channels: shops, corner shops, marketplaces, websites, etc.,</li><li>we want to display to internal users only the data of channels they can see,</li><li>our data model is made of:<br>a <strong><em>sales order</em></strong> table, with the history of the sales orders on the different channels,<br>a <strong><em>basket</em></strong> table, containing the details of the baskets of the sales orders,<br>a <strong><em>product</em></strong> table, containing the product catalog,<br>a <strong><em>user channel</em> </strong>table, containing the list of channels each user can follow and related to the channels in the <em>sales order</em> table,<br>a <strong><em>channel</em></strong> table, containing the list of existing channels.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YKED4lzAmBxF-e-I.png" /><figcaption>ER diagram</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hy7nuLK1ydAmv_aj.png" /><figcaption>Tableau relationships diagram</figcaption></figure><p>Let’s first have a look at the query generated by Tableau if we drag and drop the fields to compute in our visualization:</p><ul><li>the count of baskets per product,</li><li>with canceled orders removed,</li><li>for some specific products.</li></ul><p>It looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3160414fd3b8e3721c7c524931458396/href">https://medium.com/media/3160414fd3b8e3721c7c524931458396/href</a></iframe><p>Now, let us add our user as a filter, to filter the results and count only the baskets for channels the user can follow:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e0431687a95680b564b2e9358b830a16/href">https://medium.com/media/e0431687a95680b564b2e9358b830a16/href</a></iframe><p>As you can see, some joins and filters are replicated into several subqueries, and Tableau struggles to factorize them. Indeed, the query could have been tweaked to look more like this (while keeping the logic of each relationship):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/073659613cec9d91fe4c428a9bd987ad/href">https://medium.com/media/073659613cec9d91fe4c428a9bd987ad/href</a></iframe><p>With multiple queries running at the same time on a dashboard, this behavior may compromise the performance with a longer loading time and the user experience at the same time.</p><p>On the other hand, if you skip the logical layer of the relationships and work directly with the joins, you can use an option, called <em>Assume Referential Integrity,</em> to avoid useless joins in your queries when using fact and dimension tables in your data model. It means that when the granularity of the query results is supposed to be unchanged before and after a join, the join itself is skipped (assuming the keys are consistent).<em> </em>Compared to relationships, here join culling is pretty straightforward.</p><p>Relationships are praised for generating shorter queries than joins because they call on only the tables and fields involved in the visualization. Nevertheless, it can become untrue according to what you need to do.</p><blockquote>Using data source filters [with relationships] will limit Tableau’s ability to do join culling in the data. Join culling is a term for how Tableau simplifies queries by removing unnecessary joins.</blockquote><p>Another example: beware of the types of filters when using relationships. Indeed, the way you define your filters may also have an impact on the number, and the orchestration, of the joins in the query. One issue we experienced is the extra joins applied by Tableau when using filters in data sources directly, resulting in additional or reordered steps, slowing down queries. Tableau also warns about the risk of stacking up different types of filters with the relationships (data source filters, context filters, etc.).</p><h3>Abstraction: a double-edged feature</h3><p>In general, the abstract layer of relationships may prevent them from having the same behavior as the joins for some visualizations and functionalities, leading sometimes to heavy hacks to have the same results. One that proved troublesome for us was the impossibility to use RAWSQL() in published data sources when involving relationships. You can find more about RAWSQL() <a href="https://help.tableau.com/current/pro/desktop/en-us/functions_functions_passthrough.htm">here</a>, but the main purpose is to leverage extra capabilities of your data warehouse, such as some system-defined functions and user-defined functions.</p><p>For instance, Tableau cannot handle timezone conversion on its own, while most database management systems offer this capability. Replicating this kind of function within Tableau calculated fields (for possible several dashboards) can be very cumbersome. In general, it is ideal to have dedicated and reusable functions for all kinds of conversions.</p><h3>Interrelating fact and dimension tables</h3><p>It’s important to note that relationships can only be defined between two tables. This means that it is not possible:</p><ul><li>to use keys from more than two tables in a relationship,</li><li>once you have your main (and first) fact table on your data source, adding relationships can only be done by adding new tables.</li></ul><p>This can be a bit of a nuisance when having to deal with multi-dependencies between at least three entities. In that case, you will need to change and adapt your data model to Tableau relationships. So, if you use the same tables with other tools, relationships will drive your data mart design.</p><h3>A personal summary of the lessons we’ve learned</h3><p>Now that you have read an overview of the various pros and cons related to relationships and joins, you can see that the main takeaway is to be careful with the solution you pick according to your objectives and their complexity. You must find a good trade-off between feasibility, sustainability, and performance of your product. As a reminder, all the tips and conclusions listed here may not be exhaustive and are just a few based on our own experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mkr4JJ5FpnJoYl-i.jpg" /><figcaption>Photo by Susan Q Yin on <a href="https://unsplash.com/@syinq">Unsplash</a></figcaption></figure><p>Relationships present in a common sense simpler usability, and, in most cases, don’t require too much thought about what’s happening under the hood to operate. I would favor them for citizen data analysts or ad-hoc analyses. But watch out, as stated above, there can be some use cases where you still need to take extra care, even more, when you need your analytics to be scalable (with a consistently growing number of concurrent users).</p><p>Even if relationships are generally preferred, do not forget to counterbalance them with joins when dashboard performance is key and when the data stakeholder is an expert on data granularity. You could actually even combine them in the same data source and use one or the other when it looks more relevant to you. Even two years after their release, relationships could still be improved — with a less blurry underlying behavior — and we look forward to seeing their full potential.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf1d48c8057c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/shippeo-tech-blog/tableau-relationships-vs-joins-2-2-cf1d48c8057c">Tableau Relationships vs. Joins (2/2)</a> was originally published in <a href="https://medium.com/shippeo-tech-blog">Shippeo Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tableau Relationships vs. Joins (1/2)]]></title>
            <link>https://medium.com/shippeo-tech-blog/tableau-relationships-vs-joins-1-2-b78dd9cc373d?source=rss----25b860162bea---4</link>
            <guid isPermaLink="false">https://medium.com/p/b78dd9cc373d</guid>
            <category><![CDATA[tableau]]></category>
            <category><![CDATA[embedded-analytics]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[real-time-visibility]]></category>
            <category><![CDATA[dashboard]]></category>
            <dc:creator><![CDATA[Mathieu De Launay]]></dc:creator>
            <pubDate>Fri, 05 Aug 2022 13:27:27 GMT</pubDate>
            <atom:updated>2022-10-06T09:44:49.447Z</atom:updated>
            <content:encoded><![CDATA[<h3>What are the key differences?</h3><p>To offer advanced analytics for real-time transportation visibility on our platform, we rely on <a href="https://www.tableau.com/">Tableau</a>, a world leader in analytics tools. One of the keys to state-of-the-art dashboards lies in the data modeling hiding behind the visualizations. It is a cornerstone for building high-performing and sustainable products and avoiding misleading metrics computation. And to set your data model, Tableau provides two ways to link tables together from your data warehouse: ‘relationships’ and ‘joins’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RuX1UGKSI52MAgCJ.jpg" /><figcaption>Photo by Lukas Blazek on <a href="https://unsplash.com/@goumbik">Unsplash</a></figcaption></figure><p>You may have previously seen articles and posts praising relationships at the expense of joins but, in this two-part article, we would like to highlight why you still need to be careful. <strong>This first part will focus on the presentation of the solutions and their core differences.</strong></p><h3>A bit of background on Tableau</h3><p>Tableau is an analytics software for building advanced visualizations, providing tailor-made tools to explore data and drive business decisions. The first step for building visualizations and dashboards on Tableau is to set up your data source, whether it is a local file, a Google spreadsheet, or a dedicated data warehouse. You can learn more about Tableau connections <a href="https://help.tableau.com/current/pro/desktop/en-us/basicconnectoverview.htm">on their online documentation</a>.</p><p>Within our data warehouse, which relies on the Snowflake data platform, operation and usage data is collected and transformed into data marts, which are in turn made available to analytics tools. You can find out more about how we pull the transactional data into Snowflake <a href="https://medium.com/shippeo-tech-blog/debezium-to-snowflake-lessons-learned-building-data-replication-in-production-a5430a9fe85b">here</a>. Once Tableau is connected to the data warehouse, we first select the tables that we will need to compute the KPIs and link them together to our data model. Lastly, we build the visualizations and Tableau generates tailor-made queries to be run onto Snowflake.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SIiyi9k_pZ27mtFm.png" /><figcaption>Overview of Shippeo’s embedded Analytics Architecture</figcaption></figure><p>Originally, the links between tables on Tableau were defined by joins between physical tables, resulting in a single table, common for all worksheets using the same Tableau data source. Joins are seen as a static combination of data displayed with <a href="https://en.wikipedia.org/wiki/Venn_diagram">Venn diagram</a> icons between tables. To define the joins, you need to provide the fields to be joined between the tables and the types of the joins: left, right, full outer, and inner joins.</p><p>The main drawbacks of joins have been the following:</p><ul><li>there is a unique granularity (and a single table) resulting from the joins,</li><li>a direct consequence is that they result in duplicates of aggregate values, meaning you cannot build KPIs or lists at different levels of detail, without relying on counting distinct values and performing aggregations,</li><li>in a nutshell, the definition of your joins requires extra care to keep the right granularity you will need for your metrics.</li></ul><p>Then, why would you build a dimensional model (also known as star-snowflake schemas) when you could have a flat table with all the data? The simple reason is that it allows you to make your tables reusable. In your data preparation, you may want to build different tables at different levels of detail, and each of them is to be used in different analytics tools. Building dedicated tables for each tool is cumbersome and very costly in terms of processing for your data warehouse.</p><h3>Introduction of ‘relationships’ by Tableau</h3><p>Tableau released ‘relationships’ in 2020 as another way to combine data coming from multiple tables. Instead of having a single table at the end, you define <em>contracts — </em>to use Tableau’s terminology — between two tables based on matching fields and key constraints. The queries run in your data warehouse will depend on:</p><ul><li><strong>the match between the keys from the two tables</strong> — which keys are supposed to match and whether all values should match between tables or not — will guide the kind of join required between the tables to ensure data integrity,</li><li><strong>the cardinality of the relationships</strong> (basically, if it is a one-to-one, a one-to-many, or a many-to-many relationship) will specify how much data needs to be aggregated to keep the right level of details.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Wf2a1H-9mcGw7lhc.png" /><figcaption><a href="https://help.tableau.com/current/pro/desktop/en-us/datasource_datamodel.htm">The difference in Tableau UI between relationships and joins</a></figcaption></figure><p>With relationships, Tableau has introduced a ‘logical layer’ for the data source. You combine data between tables in the logical top-layer using relationships and Tableau handles the conversion to the underlying physical layer. ‘Joins’ remain available on Tableau though. They allow you to avoid resorting to the logical layer and instead allow you to dive directly into the physical one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hgXAR7R2tDOotI2Y.png" /><figcaption><a href="https://help.tableau.com/current/online/en-us/datasource_relationships_learnmorepage.htm">Layers of Tableau data modeling</a></figcaption></figure><blockquote>The contract is a relationship that is theoretical and will be physically applied only if needed.</blockquote><p>Compared to the joins modeling, Tableau will actually select the tables following the fields used in the visualization and generate the query according to the relationships you’ve defined, assuming there are no problems with the data integrity by default. So, if the visualization doesn’t require the processing of a table, this table won’t appear in the eventual query.</p><p>Take, for example, if we’re using a data model with a <em>sales_order</em> table and a <em>delivery</em> table, and we want to count the number of sales orders and sales orders only. The <em>delivery</em> table won’t be involved in the final query.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yDVThlKC_RwKSatANXmcUw.png" /><figcaption>Tableau data source based on relationships</figcaption></figure><p>On top of that, relationships preserve the level of detail within your KPI. Namely, if you define the count of sales orders as a measure on Tableau, even if an order can have 0 to N deliveries, the count of orders will be correct regardless of the number of deliveries.</p><p>In a nutshell, you are not supposed to think of the underlying joins to make sure that your KPI will reflect the records you want to count or sum. You just let Tableau do the switch from the logical to the physical layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*qlCIYih-7mT0EPYR89XnjA.png" /><figcaption>(Count) Measures defined at each level of granularity</figcaption></figure><h3>Why were these relationships released?</h3><p>The main purpose of ‘relationships’ was to make the data source design more user-friendly and more accessible to citizen data analysts. The latter can focus on constraints and cardinality, without thinking of what a join might result in.</p><p>But a core reason behind offering these relationships is to avoid, as much as possible, having to build complex calculations to deal with different granularities or levels of detail for the visualizations.</p><ul><li><strong>Say goodbye to handling duplicates and counting distinct values in case of 1 to N or N to N relationships!</strong> With the joins, counting two levels of details means having to count distinct values at least once. With relationships, you can build two different measures by counting respectively records from different tables, without creating duplicated lines. Maintaining the counts of distinct values is a resource-intensive process that should be limited as much as possible.</li><li>Similarly, some Tableau LOD calculations can be avoided in many cases. You can see more about Tableau LOD <a href="https://help.tableau.com/current/pro/desktop/en-us/calculations_calculatedfields_lod.htm">here</a>. They are counter-intuitive, hard to maintain and keep track of, and require a lot of computation. Yet, LOD calculations can still be helpful in some cases.</li></ul><p>As a corollary, relationships allow a sustainable usage of multi-fact tables in your reporting without having to worry about duplicated data (provided your relationships are correctly set up).</p><h3>Is using relationships really better?</h3><p>From the previous statements, we are easily seduced by the relationships and tempted to give up on the joins forever. As a matter of fact, and in most cases, relationships turn out to be more convenient when building the visualizations. But, their design still shows some limitations that will prevent you from doing everything you would expect with the relationships, and this is what we had to face when improving our analytics.</p><p><a href="https://medium.com/@mathieu.delaunay_83757/cf1d48c8057c">Read the second part of this article here</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b78dd9cc373d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/shippeo-tech-blog/tableau-relationships-vs-joins-1-2-b78dd9cc373d">Tableau Relationships vs. Joins (1/2)</a> was originally published in <a href="https://medium.com/shippeo-tech-blog">Shippeo Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing the performance and the scalability of embedded dashboards]]></title>
            <link>https://medium.com/shippeo-tech-blog/optimizing-the-performance-and-the-scalability-of-embedded-dashboards-5fa865e5ab2a?source=rss----25b860162bea---4</link>
            <guid isPermaLink="false">https://medium.com/p/5fa865e5ab2a</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[dashboard]]></category>
            <category><![CDATA[supply-chain-visibility]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[data-analytics]]></category>
            <dc:creator><![CDATA[Yuting Huang]]></dc:creator>
            <pubDate>Mon, 13 Jun 2022 13:34:46 GMT</pubDate>
            <atom:updated>2022-06-13T13:34:46.024Z</atom:updated>
            <content:encoded><![CDATA[<h4>Migrating our data warehouse to Snowflake</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RTUlFfXwH1Lg06NL" /><figcaption><em>Photo by</em><a href="https://unsplash.com/@isaacmsmith?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText"><em> Isaac Smith</em></a><em> on</em><a href="https://unsplash.com/s/photos/dashboard?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText"><em> Unsplash</em></a></figcaption></figure><p>As the data team from a SaaS company offering advanced analytics on transportation and real-time visibility, we commit to constantly improving the performance of the dashboards embedded on our platform. In this article, we discuss the challenges faced, why we decided to migrate our data warehouse from PostgreSQL to Snowflake, the insightful conclusions we have drawn on data modeling and configuration on Snowflake, and what the final achievements were.</p><h3><strong>1. Our business context and analytics architecture</strong></h3><p>In the Data-as-a-Service team, we build multiple data products, among which, are embedded dashboards for our end users to use. The challenge is to offer various analytics products to monitor their supply chain from numerous perspectives (including operational performance, tracking compliance and data quality, and feature usage and value) with a high data refresh rate relying on a single data source: our data warehouse.</p><p>Here is an overview of our embedded analytics architecture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NA2P2s91f55fOHiQ" /><figcaption><em>Overview of Shippeo’s embedded Analytics Architecture</em></figcaption></figure><p>In a nutshell, we load the data from the operational database to the data warehouse (previously PostgreSQL, now Snowflake) through the Debezium and Kafka connection (you can see more info on the data loading process in <a href="https://medium.com/shippeo-tech-blog/debezium-to-snowflake-lessons-learned-building-data-replication-in-production-a5430a9fe85b">this article</a> from our data engineer). Within the data warehouse, the preparation of data marts is completed with the help of the <a href="https://www.getdbt.com/">dbt</a> (data transformation tool). Then, we connect Tableau to our data warehouse and build all the dashboards on Tableau. Finally, we embed those dashboards on our platform, where customers can access them.</p><h3>2. What are the pain points and where are they coming from?</h3><p>There are three main pain points that we pay the most attention to in order to offer the best data products to our customers.</p><h4>❗️Dashboard performance with higher demand for complex analysis</h4><p>With the continuous deployment of new features and transportation flows on our platform, coupled with the increasing maturity of our customers, the demand for delivering more complex analytics is growing. Inevitably, more complex analyses mean more elaborate modeling of the data warehouse to ensure dashboard responsiveness.</p><h4>❗️Scalability with fast-growing data volume</h4><p>The business expansion keeps bringing more and more customers, so the number of users and the volumes of ingested and generated data are increasing accordingly. We need to make sure that our analytics architecture will be able to scale seamlessly.</p><h4>❗️Data operations optimization</h4><p>With the growing number of external and internal users, data operations have become crucial to avoid the slowness or unavailability of dashboards. Therefore, we need a tool to simplify the data warehouse management process by allocating proper resources for each kind of usage. This ensures the relevant computation capacity enabling multiple simultaneous dashboard accesses and data loading in parallel.</p><h4>❓Where are the pain points coming from?</h4><p>After having performed all the possible optimizations for Tableau dashboards, and for Tableau server settings (to process queries more quickly, and with more in parallel), the improvements were hardly noticeable, confirming the fact that the bottleneck was indeed caused by the performance limitation of PostgreSQL. Of course, increasing PostgreSQL CPU resources indefinitely couldn’t be the long-term solution, and limiting the data horizon could not be considered either.</p><p>PostgreSQL was originally built as an OLTP database, meaning it’s optimized to do operational work, not complex analytics, which is the case for our dashboards. Therefore, migration from an OLTP database to an OLAP database dedicated to analytics queries seemed to be the best solution.</p><h3>3. Migration from PostgreSQL to Snowflake ❄️</h3><p>Snowflake is built as an OLAP database and with a fully managed cloud service, it can scale up resources automatically when needed and comes with a powerful engine to optimize query performance.</p><p>So how did we come up with an optimized data infrastructure on Snowflake?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/327/1*IxVjzLkJex3bDmm9tr-jvg.png" /><figcaption>The art of balance of cost and performance</figcaption></figure><h4>1. Why is denormalizing not always the best solution on Snowflake?</h4><p>Snowflake is an OLAP system and the data is mostly used for analysis on the dashboard in our case, so we assumed that a denormalized model, namely a flat table containing all columns and lines, would be faster and more efficient compared to a dimensional model also known as star-snowflake schemas.</p><p>The dimensional models reduce redundancy and eliminate inconsistent dependency by dividing the data into multiple tables. Conversely, the denormalized models produce redundancy by combining the data into a single table to make data retrieval faster. It’s commonly used in OLAP systems as it performs better by avoiding complex joins.</p><p>The results were as interesting as expected, with <strong>a dashboard loading time around 10% faster for the denormalized model</strong> in Shippeo’s use case. However,<strong> the denormalized model takes almost 1500% more time than the dimensional model when generating or updating the data marts</strong> in Snowflake.</p><p>Considering the high update frequency of our dashboard service, a better balance between performance and cost was the main principle to apply. Actually, joins are not as expensive as we imagined on Snowflake. Instead, table scanning consumes the most resources (when no clustering is used), whilst <strong>clustering smaller fact tables and then joining them to dimension tables is actually less costly than clustering a fully denormalized table</strong>.<strong> </strong>Therefore, we decided, in the end, to go with the dimensional model, which shows a better balance between performance and cost in our case.</p><h4>2. What to cluster and why?</h4><p>To be more specific in the costs generated by the denormalized table, we want to emphasize that it is the clustering that generates the high cost.</p><p>According to the documentation of Snowflake:</p><blockquote><a href="https://docs.snowflake.com/en/user-guide/tables-clustering-keys.html"><strong>A clustering key</strong></a> is a subset of columns in a table (or expressions on a table) that are explicitly designated to co-locate the data in the table in the same <a href="https://docs.snowflake.com/en/user-guide/tables-clustering-micropartitions.html">micro-partitions</a>.</blockquote><blockquote>Source: <a href="https://docs.snowflake.com/en/user-guide/tables-clustering-keys.html">https://docs.snowflake.com/en/user-guide/tables-clustering-keys.html</a></blockquote><p>For example, if we create a cluster on a date-based field, it’s going to be more efficient when running time-series analysis over a specific horizon. However, although clustering can substantially improve the performance and reduce the cost of some queries, <strong>the compute resources used to perform clustering consume credits and it takes more time to generate the table with the clustering.</strong></p><p>Besides, <strong>the micro-partitions have a minimum size that prevents making clusters with lower granularity keys useless</strong>. For instance, when there is already a clustering on one date-based field, if we compute a clustering based on date plus any other string field, the number of micro-partitions may not change, which means adding the clustering on a second field doesn’t really help. Therefore, we need to <strong>limit the number of columns to be clustered and ensure a good query performance at the same time.</strong></p><h4>3. How to optimize warehouse size and warehouse clusters?</h4><blockquote>Snowflake warehouse provides the required resources, such as CPU, memory, and temporary storage, to perform the following operations in a Snowflake session.</blockquote><blockquote>Source: <a href="https://docs.snowflake.com/en/user-guide/warehouses.html">https://docs.snowflake.com/en/user-guide/warehouses.html</a></blockquote><p><a href="https://docs.snowflake.com/en/user-guide/warehouses-overview.html#warehouse-size"><strong>Warehouse size</strong></a> specifies the set of computing resources available in a warehouse. Resizing a warehouse (Scale-up) is to be addressed with bigger machines, but<strong> that is not to say bigger is always better</strong>. After the test with different sizes, we found out that even though it’s faster to run a single query with the bigger warehouse, the total loading time when opening a dashboard is actually longer because the bigger the machines are, the longer the provisioning time is.</p><p><a href="https://docs.snowflake.com/en/user-guide/warehouses-multicluster.html"><strong>Multi-cluster warehouse</strong></a> (Scale-out) is more designed for handling queuing and performance issues related to large numbers of concurrent users and queries and this is definitely the case for embedded analytics made for a large volume of users. Considering this is an option not included in the base contract with Snowflake, we need to keep in mind that there is also an art of balance here because more clusters enable more parallel queries but cost more credits at the same time. Therefore, <strong>the customization of the number of clusters should be always based on the real business need</strong>. In our case, we took the option and the option has turned out useful and cost-effective.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/0*AqfwM6yjegsyNqrW" /><figcaption><em>Snowflake warehouse default configuration interface</em></figcaption></figure><h4>4. Optimize auto suspension: user access habits analysis</h4><p>Snowflake works with a per-use pricing framework and when there is no usage, it offers the possibility to suspend the warehouse if it is inactive for a specific period (1 minute by default). Auto-suspend ensures that you do not leave a warehouse running and consuming credits when there are no incoming queries. On the other side, suspending the warehouse drops the existing cache layers, which make queries faster. So in our use case, having the warehouse active means faster queries, while when there is no prior activity, the first queries are relatively slower.</p><p>Given the Snowflake pricing methodology, one of our recommendations would be to<strong> analyze users’ access habits (access rush hour &amp; access frequency) before setting up the auto-suspension</strong>. This is what we did to better assess when the user experience needs to be kept at a certain level of satisfaction.</p><h3>4. Where do we stand now after the migration?</h3><p>Now if we look back again at our three pain points: <strong>dashboard performance</strong>, <strong>scalability</strong>, and <strong>data operations</strong>, what are the achievements after the migration?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3JiW7GTRTx_GACli" /><figcaption><em>Photo by</em><a href="https://unsplash.com/@krakenimages?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText"><em> krakenimages</em></a><em> on</em><a href="https://unsplash.com/s/photos/achievement?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText"><em> Unsplash</em></a></figcaption></figure><p>✅ One of the most obvious results is that the <strong>average load time when opening a dashboard was cut in half</strong> for all of our customers and even more noticeably for large-scale customers. This indeed contributes a lot to customer satisfaction and enhances user stickiness to our data product.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bv1z1FgVM7phTrwY-57zRQ.png" /><figcaption><em>Dashboard average loading time evolution</em></figcaption></figure><p>✅ The mastery of various properties of the data warehouse and the significant improvement in the performance for large-scale customers are also <strong>strong proof of our scalability with the expansion of the company</strong> with more clients, more data, more use cases, and more complex analytics.</p><p>✅ By establishing dedicated data warehouses with the customized configurations according to different use cases, we won’t be trapped by the disturbance between use cases, because the <strong>data loading processes are all independent</strong>. In other words, we can freely work with production data without compromising any key feature on the platform.</p><p>📝 After a thorough phase of investigation and testing, we have confirmed that <strong>a data warehouse from the new generation with an agile and efficient analytics infrastructure is able to give businesses a competitive advantage and provide customers with fast and easy access to business intelligence, while seamlessly scaling to meet growing demands</strong>. At the same time, we also need to keep in mind that there are some limitations of the current pay-per-use model which require extra care. As of now, we are continuously tracking the performance of the dashboards and the cost spent on Snowflake with our internal monitoring tool to keep delivering data products in an efficient, safe and convenient way.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5fa865e5ab2a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/shippeo-tech-blog/optimizing-the-performance-and-the-scalability-of-embedded-dashboards-5fa865e5ab2a">Optimizing the performance and the scalability of embedded dashboards</a> was originally published in <a href="https://medium.com/shippeo-tech-blog">Shippeo Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Debezium to Snowflake: Lessons learned building data replication in production]]></title>
            <link>https://medium.com/shippeo-tech-blog/debezium-to-snowflake-lessons-learned-building-data-replication-in-production-a5430a9fe85b?source=rss----25b860162bea---4</link>
            <guid isPermaLink="false">https://medium.com/p/a5430a9fe85b</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[debezium]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[change-data-capture]]></category>
            <dc:creator><![CDATA[Omar Ghalawinji]]></dc:creator>
            <pubDate>Wed, 01 Jun 2022 14:51:43 GMT</pubDate>
            <atom:updated>2022-06-07T19:04:01.179Z</atom:updated>
            <content:encoded><![CDATA[<p>With the well-deserved position that Snowflake and Debezium have gained in the modern data stack, it is now fairly easy to find online resources on using these technologies. <strong>In this blog, we take it one step further by reflecting on the lessons we (at Shippeo) have learned using Debezium to replicate data at scale in near real time to Snowflake.</strong></p><p><em>Data is the blood of Shippeo’s business. As a data aggregator in supply chain, fresh and reliable data is crucial to provide accurate ETA predictions and performance insights.</em></p><p><strong>This is not a tutorial.</strong> We are not going to explain the different steps to implement data replication to Snowflake using Debezium and Kafka. The available resources online as well as the official documentation are largely sufficient. What we want to share with you in this article is the <strong>cherry on the cake</strong>; what to do and what not to do if you want to go beyond a simple PoC of these technologies to the actual implementation of a reliable system in production.</p><h3>Overall Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*DylOUtQn__8bj6sfUubQcw.png" /><figcaption>Overview of the KafkaConnect-based (Debezium) data replication pipe</figcaption></figure><p>We use <a href="https://debezium.io/">Debezium</a>, an excellent open-source tool to perform log-based Change Data Capture; which is the process of using the logs of an OLTP database to track and stream all the changes in a database, as events. These events are sent in near real time to Kafka to be consumed by several Kafka Sink Connectors and pushed to OLAP systems downstream.</p><h3>Thoughts on Debezium and Kafka</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/0*ope-93aZKvplBVeG.jpeg" /></figure><h4>Think about optimizing Kafka usage in advance</h4><p>There are few decisions you can take at the beginning of your development cycle that can have an enormous impact on your Kafka usage. For example, when using several Debezium connectors in the same Kafka cluster with large tables, it is not uncommon that a given Kafka node runs out of disk space or that the initial Debezium snapshot takes days to finish. Here is some advice:</p><ul><li><strong><em>Create Kafka topics manually with the right retention policy: </em></strong>Debezium can auto create Kafka topics if they don’t exist. This is better avoided. Creating Kafka topics manually before initializing Debezium gives you control over the configuration of each of these topics. For example, using the retention policy “Compact+Delete” is particularly useful for topics used by Debezium as it allows Kafka to keep only the latest message for a given Kafka Key, which leads to less storage usage. You can automate the creation of these topics with a script or using Terraform if you have a managed Kafka service. This will also allow you to fine-tune the number of partitions of each topic based on the size of the corresponding table (see next point).</li><li><strong><em>Increase the number of partitions of Kafka topics corresponding to your largest tables: </em></strong>this will increase the parallelism of write operations in Kafka and massively expedite the initial snapshot of large tables.</li><li><strong><em>Use AVRO serialization with a Schema registry:</em></strong> instead of the default JSON serialization. This will reduce the message sizes and save on storage usage. It’s worth mentioning here that some Kafka Sink connectors only support AVRO with a Schema registry, like BigQuery sink connector. So better to prepare in advance.</li></ul><h4>Create an independent Debezium connector for each downstream system</h4><p>Let’s imagine that you have certain tables in MySQL that are needed both by your data scientists (who use Snowflake to train their models), and your data analysts (who use BigQuery to build some internal reporting).</p><p>Since it is the same source tables that are replicated to 2 different destinations, you might be tempted to define a single Debezium source connector whose data is consumed by 2 different Sink Connectors (Snowflake + BQ in this case). This creates coupling between data consumers, and it’s a bad idea for a few reasons, including:</p><ul><li>If one consumer needs to reinitialize a snapshot of a certain table, the other consumer will be forced to consume this snapshot as well.</li><li>If one consumer needs only few columns of a certain large table while the other consumer needs all the columns, the first consumer will be forced to consume the payload containing all the columns anyway.</li></ul><p>This breaks the isolation between different workloads, complexifies the maintenance of data pipelines and incurs unnecessary compute costs.</p><h4>Watch out using Debezium with PostgreSQL</h4><p>Setting up Debezium on PostgreSQL for the first time can be tricky. Debezium uses PostgreSQL’s <a href="https://www.postgresql.org/docs/current/wal-intro.html">WAL</a> (Write-Ahead Logging) through “Publications” and “Replication Slots”. All these need to be correctly configured for Debezium to function. While Debezium can create them automatically (as explained in the docs), it requires giving Debezium’s DB user ownership privilege over the replicated tables. This is not only an anti-pattern, but it might even be impossible from an organizational perspective as these tables are most probably owned by other software development teams in your company, not by the data engineers.</p><p>The better alternative is to create these replication slots manually by the table owners or by a superuser. This is explained nicely in <a href="https://elephanttamer.net/?p=50">this article</a>. Another good read on Debezium with PostgreSQL is <a href="https://debezium.io/blog/2020/02/25/lessons-learned-running-debezium-with-postgresql-on-rds/">the article</a> of Ashhar Hasan.</p><p>Be extremely cautious of WAL size growing out of control. This can happen in high traffic/low traffic situations explained <a href="https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-wal-disk-space">here</a>, or if a replication slot is left unused after some manual testing. For instance, if your PostgreSQL is on GCP’s Cloud SQL with auto-scaling enabled, growing WAL will scale up the DB’s storage (and cost), and as of now it’s not possible to scale it down again. The only remedy when you discover the problem would be to drop the database and recreate it again.</p><h4>Be aware of the major bug <a href="https://issues.apache.org/jira/browse/KAFKA-8713">KAFKA-8713</a></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/1*ImQhYHGK93peOGGRlK3TRQ.png" /></figure><p>Long story short, if there is a nullable database column with a default value defined, then KafkaConnect would incorrectly replace any null value of this column with the default value during serialization. This bug impacts Debezium downstream as it relies on KafkaConnect serialization.</p><p>Even though this bug compromises data integrity for such columns, it still didn’t seem to be getting a fix any time soon <em>-at the time of writing- </em>as it requires a KIP <em>(Kafka Improvement Proposal)</em>.</p><h3>Thoughts on Snowflake</h3><p>The nice thing about Snowflake is that it just works reliably. There are not many things you need to be worried about (besides your bill for credit usage). And to make this bill less painful, you can optimize the buffer configuration in Snowflake Kafka connector in order to use Snowpipe more wisely.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*vl8Hz90rTHpOFfsvEqvYsg.png" /><figcaption>Snowflake Kafka Sink Connector buffer configuration with default values</figcaption></figure><p>It is worth asking the question of how fast you need to flush data out of Kafka (and thus trigger Snowpipe). Every data pipeline can have different data freshness requirements. Some experimentation should be done on the buffer configuration to find the right trade-off between cost and data freshness.</p><blockquote>One inconvenience of Snowflake Kafka Sink connector is that it pushes data to Snowflake semi-structured (as JSON), so it needs to be transformed if the tables schema is needed. <a href="https://medium.com/u/76f0c8379c68">Obed</a> explained this elegantly in <a href="https://medium.com/vimeo-engineering-blog/debezium-event-flattening-with-sql-in-snowflake-b0e8397cfac2">this blog</a>.</blockquote><h3>Thoughts on Monitoring</h3><p>Debezium and KafkaConnect collectively expose an exhaustive list of metrics that can provide excellent visibility on the data pipeline. But among all these metrics, there’s one single metric that stands out: <strong>kafka_consumergroup_lag.</strong></p><p>I cannot stress enough how much this single metric can tell us about the health of the data replication pipeline end-to-end.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*n-Wk5IeATw4vt0E0u9oAbw.png" /><figcaption>Healthy oscillating trends of kafka_consumergroup_lag</figcaption></figure><p>This <a href="https://github.com/danielqsj/kafka_exporter/">Kafka Exporter</a> metric reflects the number of messages queuing in a partition of a Kafka topic unconsumed by the consumer. Ideally it should oscillate between zero and a given threshold (based on your data pipeline SLAs).</p><ul><li><strong>If it stops going up: </strong>Debezium has stopped producing messages.</li><li><strong>If it stops going down: </strong>Snowflake has stopped consuming messages.</li><li><strong>If it goes too high: </strong>Snowflake is consuming messages too slowly. Buffer flush config should be tightened.</li><li><strong>If it doesn’t go high enough: </strong>Snowflake is consuming messages too fast. There’s a potential on saving Snowpipe costs by increasing buffer flush limits.</li><li><strong>If it is absent: </strong>Kafka might be unstable.</li></ul><p><em>Here we come to the end of this self reflection on replicating transactional data to Snowflake using Debezium and Kafka. Should you have any comments, corrections or further insights; please do not hesitate to reach out!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a5430a9fe85b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/shippeo-tech-blog/debezium-to-snowflake-lessons-learned-building-data-replication-in-production-a5430a9fe85b">Debezium to Snowflake: Lessons learned building data replication in production</a> was originally published in <a href="https://medium.com/shippeo-tech-blog">Shippeo Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>