<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Adrien LEMAIRE on Medium]]></title>
        <description><![CDATA[Stories by Adrien LEMAIRE on Medium]]></description>
        <link>https://medium.com/@Fandekasp?source=rss-3c0024ab70ca------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*pVBdhpjGvqyTQOHuSdEhyg.png</url>
            <title>Stories by Adrien LEMAIRE on Medium</title>
            <link>https://medium.com/@Fandekasp?source=rss-3c0024ab70ca------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 03:13:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@Fandekasp/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A journey in Continuous Integration Testing]]></title>
            <link>https://medium.com/design-and-tech-co/a-journey-in-continuous-integration-testing-e33ca79078f4?source=rss-3c0024ab70ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/e33ca79078f4</guid>
            <category><![CDATA[nginx]]></category>
            <category><![CDATA[integration-testing]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[django]]></category>
            <category><![CDATA[seleniu]]></category>
            <dc:creator><![CDATA[Adrien LEMAIRE]]></dc:creator>
            <pubDate>Wed, 27 Feb 2019 02:14:26 GMT</pubDate>
            <atom:updated>2019-03-01T08:34:44.719Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>tl;dr: </strong>Writing integration tests for headless browsers and integrating them in your Continuous integration flow can be tedious, and won’t be a bulletproof solution.<br>Before trying something like in this blog post, consider the following options:</p><ul><li>Money is not a problem ? Use a third-part service that’ll run your tests on real devices, browsers versions and resolutions that you want supported.</li><li>No time and don’t trust machines to do a good job ? You could crowd-source user testing on services like <a href="https://www.mturk.com/">Amazon Mechanical Turk</a>.</li></ul><p>If your budget is restricted, and you’re not afraid with hacking tools to get stuff working in a reasonable amount of time, keep on reading. We’ll talk about Integration testing using <a href="https://behave.readthedocs.io/">Behave</a>, <a href="https://behave-django.readthedocs.io/">Behave-django</a> &amp; <a href="https://splinter.readthedocs.io/en/latest/">Splinter</a>, and about Continuous Integration using <a href="https://github.com/SeleniumHQ/docker-selenium">docker-selenium</a>, <a href="https://docs.docker.com/compose/">docker-compose</a> &amp; <a href="https://travis-ci.com/">TravisCI</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dFJNV9HmaAsRkzbjA6jTlQ@2x.png" /></figure><p>Meet Bran, a valiant software engineer who enjoys a work well done, loves developing new features but not so much maintaining them and even less fixing bugs.</p><p>So far, Bran has built an <a href="https://en.wikipedia.org/wiki/Single-page_application">SPA</a> using <a href="https://reactjs.org/">ReactJS</a> and <a href="https://redux.js.org/">Redux</a>.</p><p>His friends <a href="http://www.djangopony.com/">Pony</a> (who provides a nice Python Django CMS and REST API ), <a href="https://octodex.github.com/">Octocat</a> (who helps with project management, versioning and reviewing the code ) and <a href="https://blog.docker.com/2013/10/call-me-moby-dock/">Moby </a>(who sails the <a href="https://kubernetes.io/">infrastructure boat </a>) invited Bran to grab a drink and celebrate their latest production release.</p><p>After a few pints, all the pagers started ringing like crazy: “Production error!”. Bran took a long breath, and started asking his friends:</p><blockquote>“We got unit tests, right ?</blockquote><p>Yep, Jest for the frontend and pytest for the backend.</p><blockquote>We also have linters &amp; type checkers right ?</blockquote><p>Indeed, <a href="https://black.readthedocs.io/en/stable/">Black</a>, <a href="https://prettier.io/">Prettier</a> &amp; <a href="https://www.typescriptlang.org/">Typescript</a></p><blockquote>We even went as far as writing a CSS Regression test, isn’t it ?</blockquote><p>Certainly, <a href="https://medium.com/@Fandekasp/overview-of-backstopjs-a-tool-to-test-a-web-applications-ui-99234dc6c4f2">BackstopJS gave us quite the headache</a> but it was worth it.</p><blockquote>Don’t we also have API specifications ?</blockquote><p>We have <a href="https://swagger.io/">Swagger</a> setup, but our specs need a refresher..</p><blockquote>What about third-party services like Firebase, are we testing their integration ?</blockquote><p><em>Silence around the table…</em></p><blockquote>What’s the production error ?</blockquote><p>Cannot signup/login with Facebook anymore. The related unit test uses a stub, and we only take screenshots of the signup and login pages without clicking on the buttons.</p><blockquote>Alright, Moby please revert the production release right now. Everyone else, go back home and sleep tight. Tomorrow, we’ll fix the bug and re-deploy. Then, let’s write an integration test suite to avoid this kind of issue to re-happen in the future.</blockquote><p>Yes, sir!</p><p>This situation happens frequently in the world of startups and ventures, where resources and time are rarefied. It is not a good idea to rely too much on tests, but it relieves us from an otherwise time consuming user testing.</p><p>A few rules of thumb — for software engineering, not prototyping — are:</p><ul><li>Always write <strong>unit tests</strong>: blazingly fast tests, testing a single unit itself supposed to have limited operations (avoid <a href="https://en.wikipedia.org/wiki/God_object">God objects</a> like the plague!).</li><li>For branding driven projects, where design is critical, write a <strong>CSS regression test</strong>: Take screenshots of your pages or part of pages on devices, browsers and versions that you want to support.</li><li>For projects using several micro-services connected via network operations, write <strong>integration tests</strong> to insure that services really send and receive data as expected by other services.</li><li>For more mature projects with high traffic and large userbases, more specialized test suites can be written like <a href="https://en.wikipedia.org/wiki/Load_testing"><strong>load testing</strong></a>, <a href="https://en.wikipedia.org/wiki/Security_testing"><strong>security testing</strong></a> and <a href="https://en.wikipedia.org/wiki/A/B_testing"><strong>A/B Testing</strong></a> (the latter can arguably be used from earlier stages depending on the level of researches and assumptions done to that point).</li><li><a href="https://en.wikipedia.org/wiki/Usability_testing#Expert_review"><strong>User test</strong></a> (Manual QA) everything that isn’t covered via one of these testing suites.</li></ul><p>Here, our friends probably weren’t as rigorous as they should have been during the user testing phase, and they hadn’t written a comprehensive integration test suite to automate this process.</p><p>But integration tests are <strong>slow</strong>! Let’s consider the following <a href="https://en.wikipedia.org/wiki/Behavior-driven_development">BDD</a> scenario we use at <a href="https://www.gounite.com/">GOunite</a> — a japanese platform to bridge the gap between students and companies — for example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/995ddf8685e49d33a2427b62d8554ab0/href">https://medium.com/media/995ddf8685e49d33a2427b62d8554ab0/href</a></iframe><p>Although an integration test will be finished hundreds if not thousands of times faster than a manually done user test, it still has to start a browser, go to the website, wait for all resources to be downloaded and JavaScript code to be executed, and complete all the scenario operations involving several more network and database transactions.</p><p>Since Bran and company are already using Docker, Travis and Github, they could find a win-win situation: Have slow integration tests be executed automatically by TravisCI so that developers can keep working as fast as usual only relying on unit tests! We can even connect the integration test results to Github, to approve or refuse a <a href="https://help.github.com/articles/about-pull-requests/">Pull request</a>!</p><p>Hopefully by now, you can see the value of writing integration tests to make your product more robust.</p><p>Let’s now dive in our test integration using <a href="https://behave.readthedocs.io/">Behave</a> (to write user stories and share them with non-technical staff ), <a href="https://behave-django.readthedocs.io/">Behave-django</a>, <a href="https://splinter.readthedocs.io/en/latest/">Splinter</a> (to interact with a browser), <a href="https://github.com/SeleniumHQ/docker-selenium">docker-selenium </a>(to integrate our test suite in our <a href="https://docs.docker.com/compose/">docker-compose</a> development environment ) and TravisCI.</p><p>An integration test suite is supposed to test the user stories for your service. If your service is composed of several micro-services, you do not need to tie your test suite to any of these micro-service.</p><p>Yet, we chose to write our test suites close to the people using them, ie CSS regression suite written with javascript in the frontend repository, and Integration suite written with python in the backend repository.</p><p>I like the <a href="https://behave.readthedocs.io/en/latest/philosophy.html">Behave</a> Python framework, because it allows us to write features and scenarios using regular English sentences (or any language, really). It can be used before the development of a feature to agree with non-technical people on expected behavior, and re-used afterwards to validate the implementation.</p><p><a href="https://behave-django.readthedocs.io/">Behave-django</a> empowers us by allowing us to create a test database and custom fixtures using Django’s <a href="https://docs.djangoproject.com/en/2.1/ref/django-admin/">manage.py</a> tool, and running the integration test suite in the same fashion as our unit test suite.</p><p><a href="https://splinter.readthedocs.io/en/latest/browser.html">Splinter</a> is like browser testing on steroids. It’s a nice wrapper around Selenium allowing us to write succint and easy to read tests related to browser operations.</p><p>Let’s have another look at the Contact scenario from above, matched with its steps implementations:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d744c7f19cc1834557ddb60ebf95f98b/href">https://medium.com/media/d744c7f19cc1834557ddb60ebf95f98b/href</a></iframe><blockquote>Jon the reader: “That’s it ? 25 lines of code to visit a page, fill a form, submit it, verify the redirection page and even check for emails sent ?”</blockquote><p>Hey Jon, nice to meet you! Yes, it’s worth it installing a couple more libraries such as behave-django and splinter, right ?</p><blockquote>Jon the reader: “Why did you add the @fixture.browser.chrome decorator to the feature?</blockquote><p>Splinter and Selenium offer several browser integrations, so you could decide to run several browser if you’d like. Since we already have another CSS regression suite and do not care about design in our integration tests, a single browser is fine for us.</p><p>Here’s how we defined our decorator:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/73dbebb92f9077f951c7f1a2af8ec9c7/href">https://medium.com/media/73dbebb92f9077f951c7f1a2af8ec9c7/href</a></iframe><blockquote>Jon the reader: “I just noticed that Splinter has a <a href="https://splinter.readthedocs.io/en/latest/drivers/chrome.html">Chrome webdriver</a>. Why are you using the <a href="https://splinter.readthedocs.io/en/latest/drivers/remote.html">remote</a> one ? And what is this selenium-hub ?</blockquote><p>Here’s the thing: From our tests, we’re connecting to a domain name <a href="https://xpc.gounite.test/">https://xpc.gounite.test/</a> which doesn’t exist!</p><p>A django test runner would normally send queries to <a href="http://localhost:43093/">http://localhost:43093</a> or other random port. The thing is, we have several micro-services, and our React project needs js/css files to be downloaded, then once the SPA started, queries are sent to the API server.</p><p>While we use Kubernetes on production, we simplified the developer’s local infrastructure using docker-compose, and we use a nginx reverse proxy docker container to map and rewrite requests to the appropriate docker container.</p><p>Let’s look at a couple interesting configuration points.</p><pre>reverse-proxy:<br>  networks:<br>    default:<br>      aliases:<br>        - api.gounite.test<br>        - xpc.gounite.test</pre><p>Aliases are a <a href="https://docs.docker.com/compose/compose-file/compose-file-v2/#aliases">docker-compose</a> feature that map said aliased domain name to the container. Any container from the docker network will be able to query the reverse-proxy container using api.gounite.test.</p><p>In addition, our requests are not sent from the python container which runs our integration tests, but from a chrome container which contains the browser we’re using to test the user stories.</p><pre>server {<br> listen 8080;<br> resolver 127.0.0.11;<br> set $proxy_pass <a href="http://backend:8001/">http://backend:8001</a>;<br>}</pre><p>The resolver here tells nginx where to query to get a DNS resolution for the domain name api.gounite.test . Docker DNS resolver always use the same IP in any docker network, so it looks like we can safely hardcode it in the configuration. By querying the Docker DNS resolver, nginx will know that backend corresponds to the ip of the backend container within the same network, and it will be able to forward requests from the chrome container to the django test runner.</p><p>Here’s how it looks like in the end</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/54491dde7359ce658d9ab44823b73d01/href">https://medium.com/media/54491dde7359ce658d9ab44823b73d01/href</a></iframe><p><strong>One final tip</strong> regarding debugging within such environment:</p><p>Python developers must be familiar with <a href="https://github.com/gotcha/ipdb">ipdb</a> to set breakpoints in their code and easily investigate any frame. For our docker container to stop the execution on a breakpoint and give us access to a python shell, we need to restart our container with the --service-ports flag. Since this flag can only be used with docker-compose run (not exec), and we absolutely want to keep the backend container name (versus backend_run_1 or other), we need to remove the currently running container first as to avoid a naming conflict.<br>We also need the feature we’re debugging to be decorated with @wip for Behave to not capture stdout or logging output and stop at the first failure.</p><pre>$ docker-compose up -d  # Regular dev env<br>$ docker-compose rm -sf backend &amp;&amp; docker-compose run --service-ports --name backend backend</pre><p>We can also debug the frontend since we have a chrome browser ! Within your python shell, send your browser actions like context.browser.visit() , and see what’s happening within your chrome instance using a vnc viewer (Note: you need to use image: selenium/node-chrome-debug instead of image: selenium-node-chrome in your docker-compose config).</p><pre>$ vncviewer 127.0.0.1:5900  # password &#39;secret&#39;</pre><p>Here’s how it looks like:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FwduhM5UL8uo%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DwduhM5UL8uo&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FwduhM5UL8uo%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/3e0b947a2836171e72c35c5c9bd194af/href">https://medium.com/media/3e0b947a2836171e72c35c5c9bd194af/href</a></iframe><p>When Bran, Pony and Moby suddenly shouted in joy, Octocat and Travis got startled.</p><blockquote>Octocat: “I suppose you guys successfully setup our integration test suite ?”</blockquote><p>Bran giggled while he demoed it to his friends.</p><blockquote>Travis: “Great! Now, we’ve got to setup the CI as to not slow down the devs…</blockquote><p>Bran stopped smiling… he knew that wouldn’t be a walk in the park.</p><p>Running integration tests is great. But your teammates productivity will hardly increase if you ask them to run very slow tests several times a day.<br>Of course, you could find a middle ground like only running the tests before a release or so, but it’ll be much harder to debug issues as a result. Seeing errors as early as possible is good practice.</p><p>At GOunite, the most important part of our process is the Pull Request. Everything in master should just work, and master can normally be deployed to production anytime (“normally” isn’t always guaranteed, which is why a staging server is always useful).<br>Therefore, running our integration test suite at the end of each pull requests sound like a good solution. Leaving it to TravisCI is even better, freeing up local resources to focus on the next task.</p><p>Now, we do not want to build a fully fledged docker environment for all our micro-services travis builds.</p><p>So we used the infra repo to build the whole thing and run both continuous integration and css regression tests.</p><p>Then we could trigger the infra travis build from the backend or frontend repositories.</p><blockquote>Travis: “Sorry to interrupt, but… we only paid for a single concurrent job on Travis.com”</blockquote><p>Oh. right. If we try to trigger a build, then wait for it to finish, we’ll wait forever, because the next build is waiting for us to finish as well…</p><blockquote>Octocat: “I might have an idea. How about adding another PR check? When your backend PR build is finished, it triggered the slow tests on an infra build, then let github knows that the build is in progress, and when the latter is completed, it also let github knows whether the build was a success or failure.”</blockquote><p>Ok, let’s do it.</p><p>Beware the ugly script due to time constraints:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3a1714ddf98edf6b56db780e48505b02/href">https://medium.com/media/3a1714ddf98edf6b56db780e48505b02/href</a></iframe><p>We use <a href="https://github.com/plume-lib/trigger-travis">trigger-travis.sh</a> to send a request for Travis to trigger a build of the infrastructure repository. But since we needed to give it an environment variable TRIGGER_PATH for the infra build to know what github repo to send the build result to, we had to hack the query a little bit.</p><p>You’ll also notice the condition if [[ $TRAVIS_COMMIT_MESSAGE == *&quot;[slow]&quot;* ]] . This means that regular commits will not trigger a build on the infra repo. One needs to write a commit message such as</p><pre>$ git commit -am &quot;feat: PR #1 now completed! [slow]&quot;</pre><p>One important issue to keep in mind, this configuration uses <a href="https://developer.github.com/v3/repos/statuses/">Github Statuses checks</a>. Statuses are specific to commits, while <a href="https://developer.github.com/v3/checks/runs/">Github Checks Runs</a> are tied to the pull request.</p><p>This means that if we add a commit without [slow] after our commit, the check visible at the bottom of the PR is removed. Whenever we get a bit of time, we’ll use <a href="https://github.com/features/actions">Github Actions</a> to improve upon this solution.</p><h4>Bonus: configure the BackstopJS CSS regression test suite with docker-compose and Travis</h4><p>This is a follow-up to the previous article called “<a href="https://medium.com/@Fandekasp/overview-of-backstopjs-a-tool-to-test-a-web-applications-ui-99234dc6c4f2"><strong>Overview of BackstopJS, a tool to test a web application’s UI</strong></a><strong>”.</strong></p><p>In the previous blog post, we ran Backstopjs directly from the host. Half of the team runs on Linux, but the other half runs on Mac OS, and as expected these guys had all screenshots differing because the fonts had very slight variations.</p><p>We had specified some yarn shortcuts like css:test and css:reference, so let’s modify this setup to support docker:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6288455ad0f3621e2e6ed52928b11ecd/href">https://medium.com/media/6288455ad0f3621e2e6ed52928b11ecd/href</a></iframe><p>All we did was replace the backstop calls with a cd into the docker-compose configuration directory then exec docker-compose.</p><p>There are 3 important tricks to know here:</p><ul><li>The --no-deps docker-compose parameter won’t start linked services. You already have a docker fleet running and you only want to start the backstop container.</li><li>The shm_size: 512m config for the backstop container. Docker allocates 64MB to /dev/shm by default, and the chrome instance started by Backstop ends up having <a href="https://github.com/garris/BackstopJS/issues/537#issuecomment-339710797">bus errors</a>.</li><li>Backstop cannot auto-open the html report in your browser from within a docker container. This is why we added a open (mac) and xdg-open (linux) condition to auto-open the file after a test run.</li></ul><p>That’s pretty much it. If you read this post from the beginning, you know how to have Travis CI build and run the docker-compose environment.</p><p>One nice to have feature is uploading screenshots and reports somewhere (<a href="https://cloud.google.com/storage/">Google Cloud Storage</a> in our case).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b51be30c77c53c5bb7df5ac3e92d86b2/href">https://medium.com/media/b51be30c77c53c5bb7df5ac3e92d86b2/href</a></iframe><p>Any question ?</p><blockquote>Jon the reader: “How many integration tests do you conduct before validating a page ? For example, how do you handle signup and login/password validation, and re-login with the same credentials ?”</blockquote><p>Hmm… we do not validate pages, but user stories since we write integration tests in a Behavior Driven Development fashion.</p><p>Also, login/password validation should be part of the unit test suite (presumably on both backend unit tests and frontend unit tests).</p><p>Please check again the “rules of thumb” I shared earlier.</p><p>Unit tests should be simple and fast (such as testing the password validation function).</p><p>We use BackstopJS to test our design, which includes text.</p><p>Therefore, all we want to verify with our integration test suite is that pages exist and required elements are actionable. It should also be easy enough to be readable by anyone from the service and not only tech members.</p><p>Here’s a part of our authentication feature:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d7a2bb74558294f2654cbf06d0ebd153/href">https://medium.com/media/d7a2bb74558294f2654cbf06d0ebd153/href</a></iframe><p>This wraps up this blog post on Continuous Integration and Integration Testing. Feel free to post a comment if you have any question!</p><p>Many thanks to Vincent Prouillet and R. G. for helping with reviewing this blog post, and Ash Curkpatrick for the illustrations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*13yjC9hlYnho9Qcj9kuLuw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tCpdd7CYIu-KdtnXu9BWuQ.png" /><figcaption><a href="https://medium.com/design-and-tech-co"><strong>Follow Here for More Awesome Content</strong></a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e33ca79078f4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/design-and-tech-co/a-journey-in-continuous-integration-testing-e33ca79078f4">A journey in Continuous Integration Testing</a> was originally published in <a href="https://medium.com/design-and-tech-co">Design and Tech.Co</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Overview of BackstopJS, a tool to test a web application’s UI]]></title>
            <link>https://medium.com/@Fandekasp/overview-of-backstopjs-a-tool-to-test-a-web-applications-ui-99234dc6c4f2?source=rss-3c0024ab70ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/99234dc6c4f2</guid>
            <category><![CDATA[regression-testing]]></category>
            <category><![CDATA[backstopjs]]></category>
            <category><![CDATA[screenshot-testing]]></category>
            <category><![CDATA[css]]></category>
            <category><![CDATA[single-page-applications]]></category>
            <dc:creator><![CDATA[Adrien LEMAIRE]]></dc:creator>
            <pubDate>Wed, 16 Jan 2019 13:54:01 GMT</pubDate>
            <atom:updated>2019-02-27T05:45:11.825Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article, we’ll talk about the <a href="https://github.com/garris/BackstopJS">BackstopJS</a> CSS regression testing framework, and how we integrated it in our business application to answer the following questions:</p><ul><li>How can we iterate faster on UI changes without inadvertently breaking some design ?</li><li>Can we increase confidence in a stable UI design while reducing the user testing effort ?</li><li>Can we do that without considerably slowing down the daily work of our frontend developers ?</li><li>How can we test a highly dynamic application ?</li></ul><p>To save some people time, let’s start with the conclusion and problems encountered.</p><p>Our UI development flow has certainly improved and we were able to spot and fix issues that we wouldn’t have been aware of without BackstopJS. Therefore we’ll keep using it from now on and improve upon our integration as time goes.</p><p>But the integration hasn’t been easy, and some issues stay unsolved:</p><ul><li>This library can only use your machine’s resources to run the tests. While some other libraries like the Headless Chrome Node API <strong>puppeteer</strong> offer ways to simulate other devices, you’re still taking screenshots from your browser using different resolutions and not from a real device. We sometimes encounter UI issues for iOS devices only that do not appear from the Chrome’s device toolbar (eg trying to display an icon on top of another element’s padding area), and BackstopJS won’t help identifying these errors early.</li><li>Although it seems like BackstopJS did a good job at optimizing things, taking screenshots remains a slow and time consuming operation. At the time of writing, it takes us about 10 minutes to take 71 screenshots.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zuoLe2syn3cgJrx9bGFFHQ.png" /></figure><ul><li>It is sometimes difficult to stabilize a screenshot. Our application being dynamically generated, and containing animations, ensuring that a screenshot is taken at the right time from every developer’s machine is tedious.</li><li>Since some time is required to load a page, if your machine is busy running another software, chances are high that your screenshot will differ thus your test fail. We currently prepare some low-computation work to do before we start running tests to avoid this issue.</li><li>At this time, only Linux developers are able to run the test suite. That’s because other Operating systems will display fonts in slightly different manners which will result in screenshots differing by up to 5%. We’ll resolve this issue when updating our integration to take screenshots from a Docker container.</li></ul><p>Now that we got the demerits out of the bag, let’s go back to the article.</p><p>At <a href="https://www.gounite.com/">GOunite</a>, we take design seriously. The product being branding design driven, every little detail matters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Etjy0h7e78j6PqK_VVSXw.jpeg" /></figure><p>But when you develop a new service with countless new features in the backlog, and try to follow software engineering standards such as writing DRY code, it is REALLY difficult to not break some part of the design at some point.</p><p>It’s even easier to break something without noticing it when using highly volatile libraries such as Material-UI.</p><p>If you have limited resources — almost always the case for startups and new ventures — or limited skills, this can result in months being lost fixing bugs that weren’t caught during reviews, or got detected in the staging server but blocked further production releases until they got addressed.</p><p>Having built a strong infrastructure (Thanks <a href="https://console.cloud.google.com/">GCP</a> and <a href="https://kubernetes.io/">Kubernetes</a>), a good development flow (Thanks <a href="https://github.com/">Github</a>) and somewhat okay codebase (Thanks linters, code formatters, type checkers and unit test/integration test runners), the UI integration became our biggest weakness, curbing the development of the application.</p><p>How can we iterate faster while preserving a high quality level of design, supported on various viewports, browsers and devices?</p><p>Various solutions exist and often involve paying a 3rd-party service, with all benefits and disadvantages that come with it. Services aren’t cheap, and we weren’t interested in a host of options that came included in the price. I then found <a href="https://github.com/garris/BackstopJS"><strong>BackstopJS</strong></a>, which appears to be used by most of these 3rd-party services in the backend, and looked easy enough to integrate at first glance.</p><p>I’ll discuss how we quickly integrated it to reduce the user testing effort and increase UI reliability.</p><p>Backstop is a simple-to-use tool to take screenshots of pages through various viewports.</p><blockquote><strong>viewport</strong>: a framed area on a display screen for viewing information.</blockquote><p>The most common names used to describe viewports are device categories: mobile, tablet and desktop.<br>Another interesting option is to use <a href="https://material.io/design/layout/responsive-layout-grid.html#breakpoints">breakpoints</a> like those defined by Material Design. Material-UI, a React library on top of Material Design, implemented a <a href="https://material-ui.com/layout/breakpoints/">subset of them</a> which are quite intuitive.</p><h3>Overview</h3><p>Let’s first have a look at BackstopJS’s workflow (once it’s setup)</p><ol><li>Generate references with backstop reference(create screenshots that will be compared against) and save them (if using a versioning tool like git, you could commit them, but we’ll see later a better alternative).</li><li>Modify some code. Run the tests with backstop test to generate screenshots for the current version</li><li>If a test screenshot differs from the reference, an error will be show ! To fix it, either fix your code because the reference is correct, or…</li><li>Approve the new screenshot with backstop approve because the reference is inaccurate (for example because you added a new feature to your design). This will replace the reference so you need to save it again.</li></ol><p>It is recommended to only version the references and ignore the test screenshot.</p><p>The workflow is very straightforward, let’s setup BackstopJS in our project and try it out.</p><h3><strong>Setup</strong></h3><p>First run backstop init to setup your project.</p><p>backstop creates by default a backstop.json file which contains all the configuration, including all scenarios you’ll want to run.</p><p>A very minimal configuration with a single scenario would look like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/96404da86c454152bc31ddba7d0fbae4/href">https://medium.com/media/96404da86c454152bc31ddba7d0fbae4/href</a></iframe><p>Whaaat ? It is very long already. I could already see a bottleneck appear in the near future when we’ll have hundreds of scenarios (very common for a business application). It will be very hard to maintain such a file.</p><p>Moreover, don’t we need several of these configuration files ? One to run all scenarios locally one time on the developer’s machine, one for continuous testing on file changes that will only run a subset of scenarios/viewports, and one for a CI engine like Travis</p><p><a href="https://stedolan.github.io/jq/"><strong>jq</strong></a><strong> to the rescue !</strong></p><blockquote>jq is a lightweight and flexible command-line JSON processor. It is like sed for JSON data</blockquote><p>json is a great format for APIs and such, but this is no configuration file material. You always want to document the non-obvious parts, let messages for other developers, and keep the sizes small.</p><p>That’s what makes <a href="https://yaml.org/">YAML</a> more suitable. So I decided to split off the config in several files</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a3c1d62648a4d8182712e1c1e1afc773/href">https://medium.com/media/a3c1d62648a4d8182712e1c1e1afc773/href</a></iframe><p>Since backstop still needs a json file, let’s auto-generate it and gitignore the generated file. It’s a good rule of thumb to never commit auto-generated files unless there’s a good reason for it, like lock files to insure all developers use the same packages versions.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ff956576bde1685120de1d8de0517b6e/href">https://medium.com/media/ff956576bde1685120de1d8de0517b6e/href</a></iframe><p>let’s dive in the css:config command.</p><p>First, we use <a href="https://yq.readthedocs.io/en/latest/">yq</a>, which is a wrapper around jq, to parse the yaml files and return json content. All the scenarios are slurped together to form a single array of scenarios. Then we pipe the json content to jq which will merge all the scenarios (input) in the `scenarios` key of the configuration object.</p><pre>(<br>  cat backstop.yaml backstop_local.yaml # read the config file<br>    | yq .;                             # overrided if needed<br>  cat backstop_data/scenarios/*  # read the scenarios files<br>    | yq &#39;.scenarios[]&#39;          # extract the scenarios<br>    | jq -s .                    # slurp all scenarios<br>)<br>|                                # pipe the config and scenarios<br>jq &#39;.scenarios=input&#39;            # merge the scenarios in the config<br>&gt; backstop.json                  # Save the json result</pre><blockquote>Hey, your yq command fails when I try it, do you know why ?</blockquote><p>There are different softwares called yq, so my guess is you installed the wrong one. At GOunite, we use <a href="https://yq.readthedocs.io/en/latest/">kislyuk/yq</a>.</p><blockquote>In some cases you may be able to avoid writing a config file by using the node invocation e.g. <a href="https://github.com/garris/BackstopJS/blob/master/README.md#pass-a-config-object-to-the-command">Pass a config object to the command</a>.</blockquote><h3>References</h3><p>Once your setup is ready, execute backstop reference to generate your reference screenshots. It can be either a staging or production application, but we’ll see later why a staging instance makes more sense for dynamic applications.<br>Since we created a custom configuration, and to insure that a generated json file is present, we can instead run yarn css:reference .</p><p>This has created as many images in the directory backstop_data/bitmaps_reference/ as there are viewports times scenarios.</p><p>While you want to version these files in order for your team to keep testing new UI screenshots against these references, do not commit them in git! Your git repository’s size would quickly explode, making it very inconvenient for everybody.</p><p>A good solution for versioning large files is <a href="https://git-lfs.github.com/"><strong>git lfs</strong></a>. This way, instead of saving full blobs of data in git, you’ll only save the hash to the remotely stored file (github integrates it seamlessly).</p><pre>$ git lfs install<br>$ git lfs track &quot;backstop_data/bitmaps_reference/*&quot;<br>$ git add .gitattributes<br>$ git commit -am &quot;feat: save backstop references with LFS&quot;<br>$ git push</pre><p>Then ask your colleagues to install git lfs and run git lfs pull to be up to date.</p><h3>Tests</h3><p>Run backstop test to create your test screenshots that will be compared with the references. Similarly to the references, we’ve wrapped it in a new command yarn css:test to insure the config file is there.</p><p>As a developer, if you run an instance of the application locally, you can quickly spot how different your local application and the staging one are.</p><p>This is what makes CSS regression testing suites so powerful. Imagine you added a 10px padding to an element to fix a design issue. Many non-designers wouldn’t even be able to point out the issue (I certainly wouldn’t !), but if a screenshot has a long purple band showing the difference between screenshots, you’ll notice it immediately.</p><p>Let’s have a look at one screenshot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GgIEgd4n2SL8Ck20BMAzPw.png" /><figcaption>reference | test | diff</figcaption></figure><p>The screenshot above shows a fix for an address field in the mobile viewport which wasn’t aligned with other fields. While a trained eye could spot the error, this isn’t something obvious, especially when the page containing this information is very long and when it only breaks under special conditions like a viewport.</p><h3>Approve</h3><p>Finally run backstop approve whenever an error is spotted but your local version is correct. That will replace the reference with the new screenshot.</p><p>Now we can call yarn css:approve --filter=”host.*mobile”and only the mobile viewport reference for the host scenario will be created.</p><p>Great, now BackstopJS is integrated to our project. Compared to other tools, the integration was rather easy :)</p><h3>Git hook Backstop on files changed</h3><p>Running the test suite with 70 tests already take us 10 minutes, and we’ve only gotten started covering the application. Running the full suite is expected to take more than an hour, which would critically impact the speed and efficiency of our frontend developers.</p><p>We want to reduce the number of scenarios executed on the developer’s side as much as possible.</p><p>Insuring that every scenario run as expected across all viewports is the job of the <a href="https://en.wikipedia.org/wiki/Continuous_integration">CI</a> component (we use <a href="https://travis-ci.org/">Travis CI</a>), which often comes at the code review phase (during a Github <a href="https://help.github.com/articles/about-pull-requests/">Pull Request</a> in our case).</p><p>How can we make the frontend developer’s experience as painless as possible ?</p><p>One solution would be to detect what files were modified by the developer, and via labels only execute the related scenarios.</p><p>Since we use <a href="https://github.com/typicode/husky">Husky</a> to handle git hooks, let’s update the git pre-commit one. Note that <a href="https://github.com/garris/BackstopJS#reporting-workflow-tips">BackstopJS recommends integrating this in the build process</a>, so we might update our webpack setup to support it in the near future.</p><blockquote>One testing approach to consider is incorporating BackstopJS into your build process and just let the CLI report run on each build or before each deploy.</blockquote><p>Back to our pre-commit hook:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/79631e11d8a1a9dd273ab1362cc491fc/href">https://medium.com/media/79631e11d8a1a9dd273ab1362cc491fc/href</a></iframe><p>Whenever we try to commit something, it’ll figure out what css/js files have been modified, parse their path for a match to a label, then use the matched list of filters to only run backstop with said labels.</p><p><strong>Tip:</strong> As convention, always write lowercase labels, else it won’t match against the related file path.</p><p>Let’s say we modified the Header component of the Home page</p><pre>$ git status</pre><pre>Changes not staged for commit:<br>  src/components/pages/Home/Header.js</pre><p>Since our home scenario has for label “home”, which matches the lowercase path src/components/pages/home/header.js, it will be executed when committing this change.</p><p>Let’s get to the meat of the topic:</p><p>How can we take screenshots of pages which are mostly composed of dynamic data ?</p><h3>Backstop Fixtures</h3><p>If you’re testing a page with dynamic content, BackstopJS recommends either to hide/remove the related content from the screenshot, or use fixtures</p><blockquote>The best way to test a dynamic app would be to use a known static content data stub — or ideally many content stubs of varying lengths which, regardless of input length, should produce certain specific bitmap output.</blockquote><p>This is why using the production application as reference for your screenshots is not a good idea.</p><p>A staging environment is supposed to be identical to the production, albeit using different data. That makes it very suitable as a reference candidate.</p><p>I’ll mention two ways of updating a staging server’s database:</p><ul><li>Use a fixtures generator. Fixtures could be data stored in a yaml file, that could be easily loaded in the database. We often use fixtures in unit tests to bypass the database altogether and return fixtures data through mocks.</li><li>Copy production data —from old enough backups — and process it (like replacing non-staff members emails with @example.com mocked emails)</li></ul><p>Once you have a data file (often .sql or .json), you’ll need to concatenate it with another set of data dedicated to BackstopJS screenshots.</p><p>Using a django server in a kubernetes cluster, we could approach this staging database update as follow:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7a4a6f5edb76740fb3198e7aa0fb4ed4/href">https://medium.com/media/7a4a6f5edb76740fb3198e7aa0fb4ed4/href</a></iframe><ul><li>Download json data with the <a href="https://www.djangoproject.com/">Django</a> manage.py dumpdata command</li><li>Replace emails then concatenate backstop js fixtures with jq</li><li>Remove all data from the staging db, and load the new fixtures with manage.py flush &amp;&amp; manage.py loaddata</li></ul><p>Repeat the database update locally and your test and staging environment will be using the same set of backstopjs fixtures.</p><blockquote>Why do we need to use Django and Kubernetes for BackstopJS ?</blockquote><p>You don’t. I’m just sharing our configuration, but you will need to write your own script to adapt to your application’s environment.</p><p>We can now take screenshots of backstop-related pages being confident that dynamic content will not cause screenshots to break.</p><p>In some situations where it cannot be avoided — like randomly generated feeds in a list page — we should tweak the scenarios to hide or remove the placeholders using the appropriate selectors.</p><pre># HOME PAGE<br>scenarios:<br>- label: &quot;home&quot;<br>  url: &quot;<a href="https://xpc.gounite.dev/">https://gounite.local/</a>&quot;<br>  referenceUrl: &quot;<a href="https://xpc.stg.gounite.com/">https://staging.gounite.com/</a>&quot;<br>  removeSelectors:<br>  - &quot;#djDebug&quot;<br>  hideSelectors:<br>  - &quot;.card-wrap&quot;<br>  delay: 1000</pre><pre>---<br># BASIC HOST PAGE<br>scenarios:<br>- label: &quot;host-basic-sm&quot;<br>  url: &quot;<a href="https://xpc.gounite.dev/host/backstopjs">https://gounite.local/host/backstop</a>-basic&quot;<br>  referenceUrl: &quot;<a href="https://xpc.stg.gounite.com/host/backstopjs">https://staging.gounite.com/host/backstop</a>-basic&quot;<br>  removeSelectors:<br>  - &quot;#djDebug&quot;<br>  delay: 500<br>  onReadyScript: &quot;puppet/sortFeed.js&quot;<br>  scrollToSelector: &quot;#sticky-end&quot;<br>  postInteractionWait: 1000</pre><ul><li>Since we use <a href="https://django-debug-toolbar.readthedocs.io/en/latest/">Django Debug Toolbar</a>, we need to remove it from all our tests via the selector #djDebug.</li><li>Our home page has feeds of randomly selected cards, so we added a script to auto-sort them before taking a screenshot.</li></ul><p>Use the following template to create a new onReadyScript script:</p><pre>module.exports = async (page, scenario, vp) =&gt; {<br>  console.log(`SCENARIO &gt; ${scenario.label}`);<br>  await require(&quot;./clickAndHoverHelper&quot;)(page, scenario);</pre><pre>// add more ready handlers here...<br>};</pre><p>Here’s our sortFeed script:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a3d12e01653cd6181cc87eab2732f872/href">https://medium.com/media/a3d12e01653cd6181cc87eab2732f872/href</a></iframe><p>Awesome, our development environment is ready, and although there is still plenty of room for improvement, we’ll be able to iterate upon it over time when necessary.</p><p>Since we started using this tool, re-occurring design bugs have dramatically fallen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ALGj01mFShkRQRVoi92TCQ.png" /></figure><h3>Summary</h3><ul><li>Only 3 commands to remember: backstop generate (create reference screenshots), backstop test (take new screenshots and compare) and backstop approve (accept new screenshots to become new references)</li><li>The BackstopJS configuration can quickly become huge. I shared a tip to split the configuration in several yaml files for easy maintenance.</li><li>BackstopJS will spawn a headless browser, wait for the page to load, execute javascript scripts then take a screenshot. Beware time load and adjust your scenario using <a href="https://github.com/garris/BackstopJS#advanced-scenarios">appropriate properties</a> like readyselector , delay and postInteractionWait.</li><li>Comparing screenshots means that you need predictable output. To do so, make sure that your reference and test environment use the same data (share the same fixtures). You can also modify the page behavior to remove unpredictable behaviors like randomized lists via JS scripts.</li><li>Running screenshots takes a long time. Leave this job to your continuous integration tool, and only take screenshots of staged files to iterate faster on the development side. git hooks or dist builds are a good time to execute the CSS regression testing.</li></ul><p>In the next post, I’ll talk about running a Docker testing environment from Travis CI and how to run our full CSS regression testing suite from it, to validate/refuse a Pull request on Github.</p><p>Update 2019/02/27: Fixed approve part following <a href="https://github.com/garris/BackstopJS/issues/946#issuecomment-467725994">Garris’s comment</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99234dc6c4f2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automatizing AWS, Pivotal Tracker, Slack and Github]]></title>
            <link>https://medium.com/@Fandekasp/automatizing-aws-pivotal-tracker-slack-and-github-http-blog-share-wis-com-sharewis-act-43e961a50f33?source=rss-3c0024ab70ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/43e961a50f33</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[slack]]></category>
            <category><![CDATA[github]]></category>
            <dc:creator><![CDATA[Adrien LEMAIRE]]></dc:creator>
            <pubDate>Tue, 19 Apr 2016 02:38:24 GMT</pubDate>
            <atom:updated>2016-04-19T02:44:47.211Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*maFwhIFyQZD2iBOuxem1-g.jpeg" /></figure><p>Automatizing AWS, Pivotal Tracker, Slack and Github</p><p><a href="http://blog.share-wis.com/sharewis-act-automation">ShareWis ACT Staging Automation</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=43e961a50f33" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>