<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ville Lahdenvuo on Medium]]></title>
        <description><![CDATA[Stories by Ville Lahdenvuo on Medium]]></description>
        <link>https://medium.com/@tuhoojabotti?source=rss-ce1ad733d7cd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*T3adppXiym-z4MnL_tCAfg.jpeg</url>
            <title>Stories by Ville Lahdenvuo on Medium</title>
            <link>https://medium.com/@tuhoojabotti?source=rss-ce1ad733d7cd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 01:52:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tuhoojabotti/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Optimizing Angular application load performance]]></title>
            <link>https://medium.com/grano/optimizing-angular-application-load-performance-6bb796b1a7c1?source=rss-ce1ad733d7cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/6bb796b1a7c1</guid>
            <category><![CDATA[angular]]></category>
            <category><![CDATA[web-performance]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[lighthouse]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Ville Lahdenvuo]]></dc:creator>
            <pubDate>Thu, 12 Jul 2018 09:42:23 GMT</pubDate>
            <atom:updated>2018-07-31T10:28:59.338Z</atom:updated>
            <content:encoded><![CDATA[<p>Optimizing performance is a complex issue. You are taught that premature optimization is bad and it’s also something hard to sell to your Product Owner since many have an “<em>if it works, don’t fix it”</em> mentality. Here’s how we are tackling the issue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-nfwrj5XqD0fxGT1rhrfvQ.png" /><figcaption>Using automated tools to measure the impact of your changes is crucial</figcaption></figure><p>We’re currently working on a new <a href="https://granoshop.fi/en/">E-Commerce solution</a> for <a href="https://www.grano.fi/en/">Grano</a>, aiming to provide a better customer experience by modernizing our platform, improving automation and integration to our production services. We’re working with an enterprise-level product and environment: Node.js, Angular, Redux, AWS accompanied with high code quality and modern development standards.</p><p>Luckily our Product Owner understands the importance of performance and many big companies are making great points about performance and have done actual research to measure its importance:</p><blockquote>We found that <strong>53% of mobile site visits are abandoned</strong> if pages take longer than 3 seconds to load. — <a href="https://www.doubleclickbygoogle.com/articles/mobile-speed-matters/">DoubleClick by Google</a></blockquote><p>So armed with these results and the wish to improve we have been working on including performance tickets to our sprints instead of leaving them at the end of the backlog to rot.</p><p>To understand our challenges and issues you should know a bit about our application and how it works. It’s not the same to run Apache or Nginx vs. using a distributed CDN to deliver your application. You could be using server-side rendering or doing everything in the browser.</p><p>Our service runs on Amazon Web Services and we take advantage of many tools that AWS offers. The Grano Shop application consists of a Node.js API running on EC2, Angular storefront application hosted on S3 and CloudFront, Angular admin interface hosted the same way, several AWS Lambda functions and bunch of other services.</p><h3>Measuring</h3><p>You cannot improve something if you don’t measure it first. Sure you can make something <em>feel </em>faster, but to actually know you’re making the right changes is crucial.</p><p>For a long time we were running Google’s <a href="https://developers.google.com/web/tools/lighthouse/">Lighthouse</a> tool manually every now and then, but to get proper measurements the best thing is to automate it. To do this we utilised our existing continuous integration infrastructure and added a step called “audit” that runs every time we deploy to production and generate a pretty report like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pAyoRe8raHeCUzsJLuWkxg.png" /><figcaption>Lighthouse report summary for Grano Shop</figcaption></figure><p>An important note though is that this report is using mobile device emulation and network throttling. We chose to track the mobile performance to not forget about that small percentage of users and ensure the desktop performance will be even better.</p><blockquote>Our desktop performance score is 92 that counts for 75 % of our visits.</blockquote><p>Another tool we use to fight the bloat is <a href="https://github.com/siddharthkp/bundlesize">Bundlesize</a>. <em>Also check out </em><a href="https://github.com/bundlewatch/bundlewatch"><em>BundleWatch</em></a><em> which seems more active now.</em> It allows us to track the size of our code bundles and warn us if we add unexpected bloat.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ezXmF_lC8tVwZb04NgQ7cQ.png" /><figcaption>PR check telling that bundle size is under control. (The “-null” is supposed to be the difference to master, but it doesn’t work for some reason.)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u5Lty2hFgCaUdbCH4h2Bcg.png" /><figcaption>Webpack Bundle Analyzer report</figcaption></figure><p>Last but not least is the <a href="https://github.com/webpack-contrib/webpack-bundle-analyzer">Webpack Bundle Analyzer</a>, which gives a helpful breakdown about the anatomy of your bundles. It allows you to easily see what is taking up all those precious bytes in a visual manner. It really helps to find out what to remove or lazy load to decrease the initial page load time.</p><p>This tool helped us notice an embarrassing mistake — for some reason we were including our polyfills in both the main and polyfill bundles and therefore downloading them twice for no reason!</p><h3>Optimization</h3><p>This section is divided into different optimization techniques and tricks we have found and exploited. Some are hopefully useful to you but some are very specific to our application so they might not directly help you, but it might give you some ideas that you can apply to your problems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VQqMHnnABA9WPcBictdR_w.png" /><figcaption>The history of our Lighthouse scores</figcaption></figure><p>Once we started optimizing a funny thing happened. <strong>Our performance score actually went down at first!</strong> In the first reports we saw that some audits failed to complete because they timed out! That is, once we started making it faster the audits started working and lowered our score.</p><h4>TypeScript imports</h4><p>We noticed that some of our dependencies were pulling in modules and code that we didn’t need. Here’s some examples:</p><ul><li>A validation utility was pulling in <em>libphonenumber-js</em> that weighs in about 110 KB and we weren’t even using it</li><li><em>Moment.js</em> is quite big and by default includes all locales</li><li>A dependency was pulling in <em>crypto-js</em> but using only 2 functions</li><li><em>Lodash</em> was importing everything even though we only imported the functions we were using</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/98c536529d170e4a184e2eade78e79f2/href">https://medium.com/media/98c536529d170e4a184e2eade78e79f2/href</a></iframe><p>Our application is written with Angular and TypeScript. This means we have a <em>tsconfig.json</em> file that allows us to tell TypeScript how to compile our code. One immensely useful option is <em>paths</em>. It allows you to add nice shortcuts for imports.</p><p>But how did this help us reduce the bundle size? Well, what we did was create proxy files for the big dependencies and load those instead. Also using <em>lodash-es</em> to allow tree shaking to work properly and replacing Moment with <em>moment.min.js</em> which doesn’t include the locale files.</p><h4>Lambda@Edge</h4><p>We use a AWS Lambda@Edge service to run code at the CloudFront CDN edge locations. This allows us to modify the HTTP requests but keep latency down. Talking about CloudFront, make sure you have enabled HTTP2 to allow the browser to request multiple resources using the same TCP connection and prioritise them better.</p><p>For example we render the index.html with a Lambda function and embed a JSON payload to the HTML to avoid a request to the Grano Shop API before being able to render anything.</p><h4>Cache Headers</h4><p>As mentioned earlier we use AWS S3 and CloudFront to host our application. By default CloudFront caches resources, but it doesn’t send HTTP cache headers to the browser.</p><p>We have another Lambda function to add a HSTS security header to the requests telling the browser to always load the site over a secure connection. We modified the function to look at the request and if the file name contains a hash before the file extension it will add a cache header with a long expiry time. Another option would have been to set the headers in S3 as metadata, but that would have required more changes to our deployment scripts.</p><p>With Angular the hashing is a simple configuration option, but keep in mind that when using the Angular CLI the assets folder is copied as-is so keep your assets in the src folder preferably next to the component that requires it to get it hashed and therefore cached properly. For example we had some vector images that were referenced in the components and once we moved them in the component folders they were hashed properly during the build process.</p><h4>Lazy Loading</h4><p>Talking about Angular and the CLI make sure to enable lazy loading and lazy load your routes, that way the initial payload stays smaller as your application grows more and more complex and full of features. Since the generated bundles are hashed they are cached properly thanks to the cache headers from the earlier section.</p><h4>Resource Hints</h4><p>Another useful browser feature is the <a href="https://www.keycdn.com/blog/resource-hints/">Resource Hints</a>. It allows you to use HTTP headers or HTML link tags to tell the browser ahead of time that it will need a specific resource. The most common example is web fonts.</p><p>In the index.html renderer Lambda we load a list of web font files from S3 and add a preload tag for each to tell the browser to download the fonts as soon as it can before downloading and parsing the CSS. This allows the browser to render the text as soon as the CSS has been processed instead of waiting for the fonts to load. <strong>Note that web fonts always require the crossorigin attribute!</strong></p><blockquote>One interesting case in which this applies even if the fetch is not cross-origin is font files. Because of various reasons, these have to be fetched using anonymous mode CORS (see <a href="https://drafts.csswg.org/css-fonts/#font-fetching-requirements">Font fetching requirements</a> if you are interested in all the details). — <a href="https://developer.mozilla.org/en-US/docs/Web/HTML/Preloading_content">MDN</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tfeusBcayYxUhCJS6UGVcw.png" /><figcaption>Here you can see that the fonts are downloaded immediately and the hashed resources are cached.</figcaption></figure><h4>Icons</h4><p>We use Font Awesome icons on our site and we were using the version 4 using a web font. To reduce the load time we upgraded to Font Awesome 5 and started using the library provided for Angular to load the icons. This way our build system only embeds the required icons into our JavaScript bundle instead of making it download all icons like we did earlier by using the web font.</p><h3>Future ideas</h3><p>As you saw in the Lighthouse report we still have plenty of work to do and we have some ideas we want to look into.</p><h4>Better lazy loading</h4><p>We can lazy load more. We added lazy loading afterwards , that is, we didn’t have it when we started building the application and now we have to organise the application and move things around to be more lazy.</p><h4>Using a Service worker</h4><p>A servicer worker won’t help with the initial page load, but with it you can control the browser cache totally and make sure the application works offline, in our case we could allow you to browse products and prepare your order even if you’re offline.</p><h4>Smaller Polyfills</h4><p>We are currently loading polyfills to support IE11. However most of our users use modern browsers that have built-in support for the things we need. There are ways to download only necessary polyfills for example checking the browser user agent string. There are even services like <a href="https://polyfill.io/v2/docs/">Polyfill.io</a> that do it for you.</p><h3>The takeaway</h3><p>Performance is an uphill battle, but one that is always worth it. Investing in performance and using the right tools can make a big difference in the user experience. Please 👏 and 💬 if you found this interesting and would like a second part!</p><ul><li>Keep performance in mind when developing — perhaps you don’t need all those 5000 npm modules</li><li>Use the right tooling and measure <strong>everything</strong></li><li>Take advantage of new browser features such as HTTP2 and Resource Hints</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*MLNzvYEXfSkSOpJNIEqOSg.png" /><figcaption>P.S. <a href="https://www.grano.fi/rekry">We’re hiring</a>!</figcaption></figure><p>Ville works as a Lead Developer at <a href="https://www.grano.fi/en/">Grano</a>, the most versatile content service provider in Finland and the leading graphic industry company in the Nordics. Currently he is working on Grano’s E-Commerce solutions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6bb796b1a7c1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/grano/optimizing-angular-application-load-performance-6bb796b1a7c1">Optimizing Angular application load performance</a> was originally published in <a href="https://medium.com/grano">Grano</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Custom Async Validators with Angular]]></title>
            <link>https://medium.com/grano/using-custom-async-validators-with-angular-b85a9fe9e298?source=rss-ce1ad733d7cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/b85a9fe9e298</guid>
            <category><![CDATA[rxjs]]></category>
            <category><![CDATA[validation]]></category>
            <category><![CDATA[async]]></category>
            <category><![CDATA[angular]]></category>
            <category><![CDATA[typescript]]></category>
            <dc:creator><![CDATA[Ville Lahdenvuo]]></dc:creator>
            <pubDate>Tue, 30 Jan 2018 15:36:08 GMT</pubDate>
            <atom:updated>2018-01-30T15:36:08.276Z</atom:updated>
            <content:encoded><![CDATA[<p>In this post I explain what went wrong when we needed to add some backend validation to our admin panel because we wanted to get quick feedback for the operators filling the forms — and how we got it working.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*eONb3_1LypSIC5iVB2Jb4g.png" /></figure><p>We’re currently working on a new <a href="https://granoshop.fi/en/">E-Commerce solution</a> for <a href="https://www.grano.fi/en/">Grano</a>, aiming to provide a better customer experience by modernizing our platform, improving automation and integration to our production services. Our tech stack is reasonably cutting edge, considering that we’re working with an enterprise-level product and environment: Node.js, Angular, Redux, AWS accompanied with high code quality and modern development standards.</p><h3>A little bit about debounce</h3><p>If the term debounce doesn’t sound familiar to you, don’t worry. It simply means that the action is being deferred until a certain time has passed since the last call. It is used to reduce the amount of useless processing while keeping the application seem real-time.</p><p>When it comes to forms and validation, it means that we wait a certain time after every time the form is updated (a key is pressed) before running the validation. That way the validation doesn’t necessarily run on every single key press, but once the user has finished typing the validation kicks in.</p><h3>How custom validation works in Angular</h3><p>In Angular a custom validator is just a function that has the signature of (control: AbstractControl) =&gt; null | { [key: string]: any } . The function is called every time the form is updated and if it returns null the control is valid and if it returns an object the control is considered invalid and the returned keys are added to it’s errors object. <strong>An async validator works exactly the same</strong>, except the return value is a Promise or an Observable — so you can take advantage of all the RxJS magic.</p><p>So how do we debounce? Well, the simple answer would be to use the RxJS <a href="https://www.learnrxjs.io/operators/filtering/debouncetime.html">debounceTime</a> operator:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/32a4505a286a4fb567dc809122baf0a1/href">https://medium.com/media/32a4505a286a4fb567dc809122baf0a1/href</a></iframe><p>But you will find out that this approach does not work. There are a couple of reasons for this: One is that the returned Observable is <em>hot</em>. To know the difference you really should read <a href="https://medium.com/@benlesh/hot-vs-cold-observables-f8094ed53339">this article</a>. Now let’s imagine we use this code. Now every time the user presses a key the validator is called, an observable is created and the timer is started. The user types “foo”, myServiceCall is called three times and after 500 ms the validation status is updated.</p><p>You see, Angular is smart: When you update the form, it unsubscribes from the validation observable, but since it’s a hot observable we have already sent the request to the backend. Oops! That is because the debounceTime operator is meant to debounce an Observable, for example an Observable that emits every time a key is pressed, but in our case we create a new observable that emits once every time, there is nothing to debounce. The way to fix this is to create a <em>cold observable</em>. That is, defer the myServiceCall until a certain time has passed. This can be achieved using the RxJS <a href="https://www.learnrxjs.io/operators/creation/timer.html">timer</a> and <a href="https://www.learnrxjs.io/operators/transformation/switchmap.html">switchMap</a> operators:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2ab082d54f890ec23e3fcd0e71a26c3b/href">https://medium.com/media/2ab082d54f890ec23e3fcd0e71a26c3b/href</a></iframe><p>That works, great! Now when the user types “foo”, we get three observables like before, however, all but the last observable are unsubscribed right away and they never actually call myServiceCall. Once the 500 ms delay has passed we switch to a new observable using switchMap and after our service call, the form errors are updated.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fi.imgur.com%2F2JDVb3W.mp4&amp;src_secure=1&amp;url=https%3A%2F%2Fi.imgur.com%2F2JDVb3W.gif%3Fnoredirect&amp;image=https%3A%2F%2Fi.imgur.com%2F2JDVb3W.gif%3Fnoredirect&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=video%2Fmp4&amp;schema=imgur" width="350" height="197" frameborder="0" scrolling="no"><a href="https://medium.com/media/61dc103bf194574357ed2c0a0867936d/href">https://medium.com/media/61dc103bf194574357ed2c0a0867936d/href</a></iframe><p>There is only one problem. If you check myForm.invalid, it says <strong>false </strong>— great, it must be working, right? Then you check myForm.valid and it’s also <strong>false</strong>! Why is that? To see what’s going on, we must check myForm.status, and lo and behold, it says PENDING . Obviously our service has been called and the errors object has been updated, so why is Angular still waiting? The answer is in the documentation of RxJS:</p><blockquote>timer: After given duration, emit numbers in sequence every specified duration.</blockquote><p>The RxJS timer operator is actually similar to setInterval, and it will keep emitting values until we tell it to stop. To fix this, we just tell it to take the first value and stop. We can do this by using the RxJS <a href="https://www.learnrxjs.io/operators/filtering/first.html">first</a> operator:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cdb52c84c3601dbe22f0b1e176ae8ea8/href">https://medium.com/media/cdb52c84c3601dbe22f0b1e176ae8ea8/href</a></iframe><p>Now, our service is only being called once and the form status doesn’t get stuck to PENDING. It works like a charm! 🎉</p><h3>That extra little bug 🐛</h3><p>While we got the async validation working like a charm, we had another problem: The error messages weren’t updating in the view. At first, we thought there was a bug in Angular, but after creating a minimal reproduction in a jsFiddle, we realized the problem was somewhere in our code.</p><p>We found out that our little helper that we use to aggregate form errors to simplify templates was actually listening to myForm.valueChanges, and it had a debounce of 300 ms. When you would type something, it would check the errors, and after 500 ms our async validation would kick in and see that there are actually errors, but the error messages were updated before this happened.</p><p>To fix this, we simply change the helper to use the myForm.statusChanges Observable, which is called every time the form’s validation status changes, therefore it is called after our async validators finish. Phew!</p><h3>The takeaway</h3><p>First of all, credit where credit is due. I found this solution while desperately browsing Github issues, and <a href="https://github.com/n00dl3">n00dl3</a> had <a href="https://github.com/angular/angular/issues/6895#issuecomment-329464982">commented his solution</a>. To summarize:</p><ul><li>Learn RxJS properly and don’t just copy &amp; paste from tutorials — it’s awesome</li><li>If you think there is a bug in a library, create a minimal repro case</li><li>Someone else is probably struggling with the same problem—if you benefit from the community, it’s nice to contribute back</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/285/1*MLNzvYEXfSkSOpJNIEqOSg.png" /><figcaption>P.S. <a href="https://www.grano.fi/rekry">We’re hiring</a>!</figcaption></figure><p>Ville works as a Lead Developer at <a href="https://www.grano.fi/en/">Grano</a>, the most versatile content service provider in Finland and the leading graphic industry company in the Nordics. Currently he is working on Grano’s E-Commerce solutions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b85a9fe9e298" width="1" height="1" alt=""><hr><p><a href="https://medium.com/grano/using-custom-async-validators-with-angular-b85a9fe9e298">Using Custom Async Validators with Angular</a> was originally published in <a href="https://medium.com/grano">Grano</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How We Used IoT to Improve Our E‑Commerce Experience]]></title>
            <link>https://medium.com/grano/how-we-used-iot-to-improve-our-e-commerce-experience-897e93efa89c?source=rss-ce1ad733d7cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/897e93efa89c</guid>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[websocket]]></category>
            <category><![CDATA[pub-sub]]></category>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Ville Lahdenvuo]]></dc:creator>
            <pubDate>Mon, 27 Nov 2017 09:48:55 GMT</pubDate>
            <atom:updated>2017-11-27T09:48:55.183Z</atom:updated>
            <content:encoded><![CDATA[<p>I admit it, the title is kind of clickbait-y. I’m going to tell you how we used <a href="https://aws.amazon.com/iot/">AWS IoT service</a> to improve our e-commerce experience. Earlier <a href="https://medium.com/grano/the-fault-in-our-responses-bytes-remaining-to-read-91efe0553676">I wrote about a nasty bug</a> we had to hunt down before releasing our new service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-E678cJHSrAUk3AeiX86yw.png" /><figcaption>Our storefront with Christmas decorations</figcaption></figure><p>We’re currently working on a new <a href="https://granoshop.fi/en/">e-commerce solution</a> for <a href="https://grano.fi/">Grano</a>, aiming to provide a better customer experience by modernizing our platform, improving automation and integration to our production services. Our tech stack is reasonably cutting edge, considering that we’re working with an enterprise-level product and environment: Node.js, Angular, Redux, AWS accompanied with high code quality and modern development standards.</p><h3>A little bit of background</h3><p>Grano does mainly print and the webstore is not an exception. We sell anything from <a href="https://granoshop.fi/en/category/3/business-cards">business cards</a> to big <a href="https://granoshop.fi/en/category/7/roll-ups">roll-ups</a> for exhibitions and trade shows. The one thing in common with almost all print products is that you need something to print.</p><p>In our store the customer uploads a PDF file as the print material. A simple file upload to S3 you might think. We want to do more, though. In order to reduce manual work in the order flow we want to check the PDF for compatibility with the selected product before allowing the customer to place the order.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HJymBoJz2ZAExl32ZZPMYA.png" /><figcaption>Waiting for the PDF report…</figcaption></figure><p>This checking or “preflight” step is done using a third party software called <a href="https://www.enfocus.com/en/products/pitstop-server">Enfocus PitStop Server ©</a>. It runs on a Windows instance and that’s a good reason if any to also run a virus check on the uploaded PDF files before allowing further processing and downloading of the files. This is achieved by using an <a href="https://github.com/widdix/aws-s3-virusscan">AWS Lambda function</a> to check the file and move it to the right S3 bucket.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R-i78J49E4e905PomdKzyA.png" /><figcaption>How it looks if you uploaded a file with the wrong size. (Localization improvement is on the backlog.)</figcaption></figure><h3>What does IoT have to do with all this?</h3><p>All that checking and processing takes time, PitStop is known to take up to minutes if checking a complicated PDF, so we needed a way to tell the customer when the process has been completed and the checkout can be finished without telling the customer to refresh or use polling — I mean come on, it’s 2017.</p><p>Our first idea for the problem used a nifty little library called <a href="http://gun.js.org/">Gun.js</a>. It’s basically Firebase real-time database, but you can integrate it easily to your existing Node.js application and host it yourself. It worked during our tests, however we had a couple of problems with it.</p><p>First of all we obviously expect our project to be a great success and therefore we want to design it with high availability, performance and fault tolerance. To achieve this we use the AWS Application Load Balancer to start up API instances when the load increases. This is a problem because Gun.js keeps the database in-memory using an optional storage adapter such as S3, but to keep the real-time part of it working we would need to make the API instances peer dynamically when instances are created or terminated.</p><p>Secondly we didn’t find a smart way to limit read/write access to the Gun.js database. So anyone could modify PDF reports and the changes would be updated to any connected clients. On top of that we took a look at the source code and — albeit well tested — it made us <a href="https://github.com/amark/gun/blob/95280b89f8bb575b2d25331d488ec9ce0ff0842b/lib/radisk.js#L33">feel uncomfortable</a>.</p><h3>The solution</h3><p>We set out to find a better solution that we could easily deploy to AWS and that wouldn’t increase maintenance work. With some searching online we ended up reading about AWS IoT.</p><blockquote>AWS IoT is a managed cloud platform that lets connected devices easily and securely interact with cloud applications and other devices. AWS IoT can support billions of devices and trillions of messages, and can process and route those messages to AWS endpoints and to other devices reliably and securely. With AWS IoT, your applications can keep track of and communicate with all your devices, all the time, even when they aren’t connected.</blockquote><p>We found out that <strong>AWS IoT provides a nice JavaScript SDK that supports real-time pub-sub messaging via WebSockets</strong> and the cost is $5 per a million messages, not bad for a hosted solution especially since for our needs we only need to send one message per file upload.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/13d01d5b744915bd1f61f42dccc8a740/href">https://medium.com/media/13d01d5b744915bd1f61f42dccc8a740/href</a></iframe><p>We use the generated file key, which consists of a random uuid plus the filename, to scope the report messages. That should be sufficiently secure and in any case the reports don’t include any personally identifiable information.</p><p>The changes to the frontend code were a bit more involved since we now had to fetch the report status from the API and subscribe to the IoT messages. This is because the customer can close/reload the tab and later return to finish the checkout and have missed the IoT message.</p><p>We made a nifty service that takes a file key and returns an observable that simply gives you the initial status and any updates from the IoT service that we link to our Redux store to update the view.</p><p>The biggest problem we had was with authorization in AWS. We had to somehow link AWS Cognito and our API with our production and development AWS accounts so that our API instances from both environments could publish messages and our clients from both environments could subscribe to those messages. But after some policy wrangling we got it working and the messages were flowing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L_mOpGuTMHhWhCUCZGNPhw.png" /><figcaption>The final file upload flow</figcaption></figure><h3>The Takeaway</h3><p>The lesson of the story is that there is always a better solution — you just have to find it. You also have to avoid getting attached to your code, always thinking how to improve it. To summarize:</p><ol><li>Don’t be afraid to throw away code, even if it works!</li><li>Prototype ALL THE THINGS</li><li>The solution might seem strange at first — IoT, seriously?</li></ol><p><em>Don’t forget to clap and follow this publication if you enjoyed this piece and want to read more.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/285/1*MLNzvYEXfSkSOpJNIEqOSg.png" /><figcaption>P.S. <a href="https://www.grano.fi/rekry">We’re hiring</a>!</figcaption></figure><p>Ville works as a Lead Developer at <a href="https://grano.fi/">Grano</a>, the most versatile content service provider in Finland and the leading graphic industry company in the Nordics. Currently he is working on Grano’s E-Commerce solutions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=897e93efa89c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/grano/how-we-used-iot-to-improve-our-e-commerce-experience-897e93efa89c">How We Used IoT to Improve Our E‑Commerce Experience</a> was originally published in <a href="https://medium.com/grano">Grano</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Export Class SoftwareDeveloper: My Expat Stories From Spain]]></title>
            <link>https://medium.com/series/export-class-softwaredeveloper-my-expat-stories-from-spain-60d3481f7075?source=rss-ce1ad733d7cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/60d3481f7075</guid>
            <dc:creator><![CDATA[Ville Lahdenvuo]]></dc:creator>
            <pubDate>Sun, 26 Nov 2017 17:59:07 GMT</pubDate>
            <atom:updated>2017-11-26T19:11:04.074Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>I am a Finnish software developer who currently lives in Barcelona, Spain. I moved here when I was 22 years old with a fresh Bachelor’s degree of Computer Science from the University of Helsinki in my hand.</em></blockquote><h4>this.getContentTypes()</h4><p>This Series will be containing personal stories, places of interest, programming stuff, jokes and maybe some photos and profiles of people I have met.</p><blockquote>If you like photos you should definitely follow me on <a href="https://instagram.com/ville.lahdenvuo">Instagram</a> or <a href="https://facebook.com/destroyingpics">Facebook</a>.</blockquote><h4>this.getContext()</h4><blockquote>Why Spain tho</blockquote><p>I’m not a fan of the hot summers, I didn’t have a burning passion for the language(s).</p><blockquote>So why did I move to Spain?</blockquote><p>The simple answer: my fiancée. I met her online and after three years of long distance relationship I could finally move to live with her since I had a stable job and a finished bachelor’s degree in computer science. I’m going to worry about the master’s later.</p><h4>this.log(13-2–16, The arrival)</h4><p>The cool view you saw isn’t from our apartment. It was from the place where I rented a room for a month in order to find a nice apartment for us. It was nice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Np49tWIDVwpKOObxUEHwg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60d3481f7075" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Fault In Our Responses: Bytes Remaining To Read]]></title>
            <link>https://medium.com/grano/the-fault-in-our-responses-bytes-remaining-to-read-91efe0553676?source=rss-ce1ad733d7cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/91efe0553676</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[expressjs]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Ville Lahdenvuo]]></dc:creator>
            <pubDate>Fri, 28 Jul 2017 14:58:13 GMT</pubDate>
            <atom:updated>2017-09-26T08:36:29.342Z</atom:updated>
            <content:encoded><![CDATA[<p>This is the story on how a single line of code caused a lot of grey hairs and moments of frustration. I never thought I would be writing such a post, but I guess it’s my time to shine… the light on a hairy bug that halted developer productivity for three full workdays.</p><p>We’re currently working on a new e-commerce solution for <a href="https://grano.fi/">Grano</a>, aiming to provide a better customer experience by modernizing our platform, improving automation and integration to our production services. Our tech stack is reasonably cutting edge, considering that we’re working with an enterprise-level product and environment: Node.js, Angular, Redux, AWS accompanied with high code quality and modern development standards.</p><h3>Day One: The Descent</h3><p>It all started when we started noticing weird errors in the <a href="https://developer.chrome.com/devtools">Chrome DevTools</a> and soon realized that this problem occurred on every error response from our API server.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZkJVYgPlq-ks3gur." /><figcaption><em>POST /user/register net::ERR_SPDY_PROTOCOL_ERROR</em></figcaption></figure><p>We had recently switched to HTTPS on our development environment to more closely match the staging and future production environments and as we can see from the screenshot, AWS <a href="https://aws.amazon.com/elasticloadbalancing/applicationloadbalancer/">Application Load Balancer</a> (ALB) automatically uses <a href="https://en.wikipedia.org/wiki/HTTP/2">HTTP/2</a> when HTTPS is used. And as you might know, Google pioneered HTTP/2 with a custom protocol in Chrome called <a href="https://en.wikipedia.org/wiki/SPDY">SPDY</a>, which explains the protocol error we were seeing. That lead us to believe that the problem had to be in the ALB.</p><p>As we were trying to find more information on this issue we found out that people had had problems on AWS related to the MTU or <a href="https://en.wikipedia.org/wiki/Maximum_transmission_unit">Maximum Transmission Unit</a> size. So we connected to the API instance to conduct some tests and noticed no problems with the MTU. While connected to the API instance we decided to test the API server locally.</p><pre>ubuntu@aws:~$ curl -H “Content-Type: application/json” \<br>  -d ‘{”email”:”asd@example.com”,”password”:”foo&quot;}’ \<br>  <a href="http://localhost:8888/user/register">http://localhost:8888/user/register</a><br>curl: (18) <strong>transfer closed with 1 bytes remaining to read</strong><br>{“message”:”Invalid password.”%</pre><p>That ruled out any issues with the load balancer or HTTP/2. However we could not replicate the issue locally! So we thought there must be something wrong in the VPS environment, but it was time to log off and rest our eyes.</p><h3>Day Two: The Despair</h3><p>With fresh eyes and a mind free of clutter we began installing a new VPS and copied the built artefacts and copied the environment configuration and started the server. But the hopeful mood was soon swept away as we were greeted with the same message: “<em>transfer closed with 1 bytes remaining to read”</em>.</p><p>My mind starting to feel blank I realized I had not tried running the built version of the backend locally. You see we use <a href="https://babeljs.io/">Babel</a> to enjoy the latest and greatest JavaScript features without worrying about runtime support. We only install Babel plugins that are needed by modern Node versions and let everything else run natively so that they will be automatically optimized once V8 developers optimize these new language features. For development we use a handy tool called <a href="https://babeljs.io/docs/usage/cli/">babel-node</a>. It works exactly like your node binary, except it automatically transpiles your code in-memory.</p><p>So we opened the command line in the build directory and ran the transpiled code via regular node and boom, we were able to replicate the issue locally. Progress at last! So the problem must be caused by Babel, right? That sounds very strange, especially since there should be no difference transpiling in-memory or to file. We aren’t even using many bleeding-edge features in the backend. So we started removing Babel plugins and seeing it the problem persisted. One by one we removed all the plugins and we still saw the dreaded message: “<em>transfer closed with 1 bytes remaining to read</em>”.</p><p>We sank into deep despair until we realized that I still had the production environment configuration in the build directory, which meant that I was running the API server with different configuration as the local development environment. So we copied the local environment, restarted the server and the problem was gone, just like that. Now all we had to do was to copy the production configuration, one variable at a time, until we found which option broke it. Finally after 61 variables we had found it:</p><pre>SESSION_REDIS_URL=dev-session.xxxx.yyyy.euw1.cache.amazonaws.com</pre><p>Locally we store sessions in-memory, since we are only running one process there is no need to have a shared session store, but as soon as we configured a session database using <a href="https://redis.io/">Redis</a> we started to get broken error responses. But that was only half of the story…</p><h3>Day Three: The Demystifying</h3><p>We had pinpointed exactly how to reproduce the problem, it should be an easy fix. We just have to figure out why the session store breaks our responses. At first I noticed that <a href="https://www.npmjs.com/package/express-session">express-session</a> option called saveUninitialized caused the issue. So we started putting together a small Express server using the same configuration as our API server, but we couldn’t get the problem to appear reliably.</p><p>First we noticed that if the Redis host was invalid or a working Redis instance, the problem didn’t occur at all. Then we saw that if the host was valid, but did not host a Redis instance the HTTP request timed out and our problem appeared. That did not match what we saw in our app. In our case the error occurred straight way without any timeout and only on error responses while in the example the error only occurred as a result of a timeout and on all requests.</p><p>At that point I opened a <a href="https://github.com/tj/connect-redis/issues/211">bug ticket</a> on the <a href="https://github.com/tj/connect-redis">connect-redis</a> module, asking if anyone had experienced similar behavior. Marc Harter (<a href="https://github.com/wavded">@wavded</a>) responded saying that he was only able to reproduce the problem with a timeout or killing the server before the timeout.</p><p>That’s when I remembered that we had some custom error handling logic in a module we use internally. We setup a custom error handler for the Express server to return errors as JSON instead of the default HTML representation. We also filter out almost all error details in production for added security, for those we have logging in place. We added an error handler and boom, responses started breaking just like they did in our API server.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9c9603c08b8b7732c7a603ac17b4ad88/href">https://medium.com/media/9c9603c08b8b7732c7a603ac17b4ad88/href</a></iframe><p>We noticed right away that the error handler looks a bit strange, why would we call <em>next(err)</em> after responding to the client? Removing that unnecessary call solved our problem, finally! We opened a <a href="https://github.com/expressjs/session/issues/492">new issue</a> on the <a href="https://www.npmjs.com/package/express-session">express-session</a> project to try and figure out why this happens and if it could be detected so that a better error message could be displayed.</p><p>Douglas Wilson (<a href="https://github.com/dougwilson">@dougwilson</a>) from the Express team immediately responded explaining that nothing can be done, because the next proceeds to the default Express error handler, which checks if the response has been started, and it has because of the res.send, immediately disconnecting the socket. This is because we might be streaming a long response to the client and we don’t want to leave the connection open wasting our resources when an error occurs.</p><p>The problem is that the Redis store is still in process of saving the session and the error handler disconnects the socket before the response can be ended. There we had it, the fault in our responses.</p><h3>The Takeaway</h3><p>The lesson of the story is that even the most mysterious bug can be found and fixed. What you need is determination (not wanting to delay the launch), caffeine and a method to pinpoint and replicate the bug. To summarize:</p><ol><li>Replicate the bug in a development environment with all your tools available.</li><li>If you can’t replicate the bug compare the environment configurations first, before thinking about problems in the code or operating system / hardware.</li><li>The bug is most likely in your own code and not in third party code, check your own code first.</li><li>A minimal example to reproduce the bug will help tremendously when testing and is very nice when submitting an issue to an open source project.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/285/1*MLNzvYEXfSkSOpJNIEqOSg.png" /><figcaption>P.S. <a href="https://www.grano.fi/rekry">We’re hiring</a>.</figcaption></figure><p>Ville works as a Lead Developer at <a href="https://grano.fi/">Grano</a>, the most versatile content service provider in Finland and the leading graphic industry company in the Nordics. Currently he is working on Grano’s E-Commerce solutions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=91efe0553676" width="1" height="1" alt=""><hr><p><a href="https://medium.com/grano/the-fault-in-our-responses-bytes-remaining-to-read-91efe0553676">The Fault In Our Responses: Bytes Remaining To Read</a> was originally published in <a href="https://medium.com/grano">Grano</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>