<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Maciej Głowacki on Medium]]></title>
        <description><![CDATA[Stories by Maciej Głowacki on Medium]]></description>
        <link>https://medium.com/@glowacki.it?source=rss-aff6071fbf8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*XQsBcsAaKoKkTu4G4Tukiw.jpeg</url>
            <title>Stories by Maciej Głowacki on Medium</title>
            <link>https://medium.com/@glowacki.it?source=rss-aff6071fbf8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 18:35:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@glowacki.it/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Become a Master of Git Tags]]></title>
            <link>https://medium.com/daftcode-blog/how-to-become-a-master-of-git-tags-b70fbd9609d9?source=rss-aff6071fbf8------2</link>
            <guid isPermaLink="false">https://medium.com/p/b70fbd9609d9</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[git]]></category>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Maciej Głowacki]]></dc:creator>
            <pubDate>Wed, 12 Sep 2018 10:32:14 GMT</pubDate>
            <atom:updated>2018-09-12T10:34:33.813Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aDg50K5nhsveNVJ_EH3PJA.png" /><figcaption>Illustration by <a href="https://dribbble.com/ju-mieleszko">Justyna Mieleszko</a></figcaption></figure><p><strong>When creating a public library or some kind of API we have to keep in mind that there will be other people using our product. This forces us to keep the project backward compatible so it doesn’t break with every release.</strong></p><p>But at some point, we may want to release a big update that has to break some things 😳</p><p>To make the life of other developers easier we should start properly versioning our project. With good versioning, it would be possible to lock our project version and always have it compatible.</p><p>And it’s easy to do with git tags and semantic versioning.</p><h3>What are git tags?</h3><p>We can think of tags in git as “bookmarks” pointing to a specific commit. They kind of behave like branches (for example you can checkout to them), but we won’t delete them once we finish using them. They are often used to mark specific version releases.</p><p>Let’s imagine that we’re creating some kind of a library for other people. At some point we decide to add some new amazing changes to it 🙌 In order to do this we create a new branch and start adding new code there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/236/1*ecACfi4_u3aIszd5Pvhrxg.png" /></figure><p>After working for some time we decide that it’s ready and we want to merge it to master. Since we won’t be working on this branch anymore we delete it from the repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/1*e9fFG30cRocVKVwCoeDYsQ.png" /></figure><p>It turns out that our changes were all good and everything runs without any errors. We would like to somehow <strong>remember this specific point</strong> and be able to run our project from this specific commit. One way of doing this is obviously remembering a commit hash, which is f855792 (but it sounds like a terrible idea, doesn’t it? 🙈)</p><h4>Let’s add the first tag</h4><p>A better way of doing things is to add a tag and give it some name that is easy to remember. Since it’s our first working version, let’s go with v1. It should be obvious that to add tags we will use git tag command 🤓 This command expects a tag name and a hash of a commit where the tag should be added. So we execute git tag -a v1 f855792, provide a short description and that’s all. Our commit can now be referenced by v1 tag name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/129/1*8GcnsuVSVbXNh4Q4-w6gRQ.png" /></figure><p>Now if we want to go back to this commit we can do git checkout v1 (just like branches)</p><h4>Let’s look at all the magic</h4><p>You may be wondering what’s up with -a in the command we used to add tag 🔮 This tells git to create an “annotated tag”, which apart from name also stores a description and some metadata. This metadata contains a date and a person adding a tag. Since this information may be useful for other people, we almost exclusively use annotated tags.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/1*tN2bru2fX26XFkmRLuwD9g.png" /><figcaption>That’s how annotated tag looks in git</figcaption></figure><p>To see the difference we can add another tag and compare the results. This time we don’t use -a option, so we execute git tag v1-test f855792. We can use git show command to see what git knows about out tags. This new tag, called “lightweight”, doesn’t store any information apart from those already in the commit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/521/1*AZyCS-wzS-hn8ne-cfckXw.png" /><figcaption>You don’t know who added this lightweight tag, and it even doesn’t have a description</figcaption></figure><h4>Let’s share</h4><p>What good are our tags if we only store them locally? We have to <strong>push them to the repository</strong> for others to see. We, obviously, use git push for pushing and simply provide tag name as an argument. In our case we want to execute git push origin v1 (again, just like branches).</p><p>If we have more than one tag we can push all of them at once with git push origin --tags. However, we have to be careful as this command <strong>pushes all tags </strong>automatically (remember to clean up any mess before pushing).</p><h4><em>Let’s watch some tags</em></h4><p>We can already create and push tags. At some point, we may want to also see what tags are already present in the repository. To do this we can simply execute git tag and see all of them. If we’re looking for specific ones there is a simple search option with -l flag. To find all tags ending with “-test” we can do git tag -l *-test.</p><h3>How to tag like a pro?</h3><p>At first, our v1 tag seemed like a good idea, but in the future, we will see v2, v3, etc. In the meantime, there will be some bugs so we will get v3-fixed or maybe v3.1. At some point, it will be hard to keep things consistent 🤷‍</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*ru1psztvMIeXUf4hRZ6J-g.jpeg" /><figcaption>That’s actually how versioning work in real life. Source: <a href="http://www.commitstrip.com/en/2017/09/12/versioning-is-important/">CommitStrip</a></figcaption></figure><p>This problem can be solved with <a href="https://semver.org/">semantic versioning</a>. That’s one of the popular standards that uses a few simple rules. Every version is formatted as MAJOR.MINOR.PATCH and each part changes according to the following rules.</p><p>We increment:</p><ol><li>MAJOR when <strong>breaking backward compatibility,</strong></li><li>MINOR when adding a new feature which doesn’t break compatibility,</li><li>PATCH when fixing a bug without breaking compatibility</li></ol><h4>The three rules explained</h4><p>It may sound simple, but usually, the hardest thing when starting with semantic versioning is deciding which changes are MAJOR and which are MINOR. People tend to think <em>“I’ve only changed one small thing so it’s definitely a minor change”</em>, which may be misleading.</p><p>We’re should ask ourselves <em>“If I was using my library in some other project will everything work after the update?”</em>. If using the new version would require us to update some things in other projects than we’ve got a change that’s not backward-compatible. Such a change is usually called a <strong>breaking change</strong>. This means that it’s definitely a major change. Otherwise, go with the minor.</p><p>When creating a library, such breaking change will be removing a public method or changing its parameters. If someone was using this method their application will stop working 🔥</p><p>If you’re working on an API then <strong>removing an endpoint</strong> or changing the response format will be your typical breaking changes.</p><p>Adding new things or modifying internals doesn’t break anything and is considered a minor update.</p><h4>It’s still too much hassle</h4><p>Good thing that we’re living in the 21st century. There are tools that do all this stuff automatically 🤖 And they also give us changelogs and standardized commit messages.</p><h4>Those commit messages sound like fun</h4><p>One tool that helps with committing is <a href="http://commitizen.github.io/cz-cli/">commitizen</a>. Once we install it with npm install -g commitizen our git gets superpowers. We can now use git cz which will guide us through the process of creating a good commit message.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*JsuidVad1m7CaxwKuYLloA.png" /></figure><p>Apart from choosing the type of change, we can also list breaking changes or reference issue that we were working on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*JXsHcSsNlACTRv4H0jVlWw.png" /></figure><p>In the end, we get a commit message that follows <a href="https://conventionalcommits.org/">another common standard</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*6qkqa5OqnsZsFiIZ1mxVww.png" /></figure><h4>Moving on to changelogs</h4><p>The second tool we want to use is <a href="https://github.com/conventional-changelog/standard-version">Standard Version</a>. After adding it to the project we can execute npm run release to automatically generate a changelog, bump project version, create a release commit <strong>and add git tags.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/503/1*_HS5QC7tK68iHuXMi9tJXQ.png" /></figure><p>This tool is smart enough to understand commit messages generated by commitizen and figure everything out. If we only had bugfixes it will bump patch version, adding new features will give us a new minor release, and introducing breaking changes will result in a major version bump.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*KtnDoZhaAkXhmLy0-y_JoA.png" /></figure><p>If you enjoyed this post, please hit the clap button below 👏👏👏</p><p>You can also follow us on <a href="https://www.facebook.com/Daftcode/">Facebook</a>, <a href="https://twitter.com/Daftcode_pl">Twitter</a> and <a href="https://www.linkedin.com/company/daftcode-sp-z-o-o-">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b70fbd9609d9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/daftcode-blog/how-to-become-a-master-of-git-tags-b70fbd9609d9">How to Become a Master of Git Tags</a> was originally published in <a href="https://medium.com/daftcode-blog">Daftcode Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why you should visit the next RubyKaigi]]></title>
            <link>https://medium.com/daftcode-blog/why-you-should-visit-the-next-rubykaigi-501317c60d6d?source=rss-aff6071fbf8------2</link>
            <guid isPermaLink="false">https://medium.com/p/501317c60d6d</guid>
            <category><![CDATA[conference]]></category>
            <category><![CDATA[japan]]></category>
            <category><![CDATA[ruby]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Maciej Głowacki]]></dc:creator>
            <pubDate>Thu, 21 Jun 2018 15:44:00 GMT</pubDate>
            <atom:updated>2018-06-22T08:05:07.581Z</atom:updated>
            <content:encoded><![CDATA[<p>RubyKaigi is the annual Ruby conference organized in Japan — Ruby’s country of origin. This year, hundreds of Rubyists gathered in Sendai to spend 3 days expanding their knowledge and discussing future development path of the language.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6a4p0GcIhwwvkU-rIJrAKQ.png" /><figcaption>This work by <a href="https://rubykaigi.org/2018">RubyKaigi 2018</a> Team is licensed under a Creative Commons Attribution 3.0 Unported License</figcaption></figure><p>Have you been there as well? If you’ve missed the opportunity, let me tell you why you should consider going to RubyKaigi in 2019.</p><p>I guess most people would now start writing about all presentations one by one. But let’s all agree that it’s not the most exciting stuff to read. How about, instead of focusing on single talks, we take a look at a whole conference at once? <strong>Do you want to know what made it so special?</strong></p><h4>Giving the choice</h4><p>If you’re a regular conference attendant you know the struggle of listening to stuff that isn’t interesting for you. Some presentations are just too easy, others are too hard. And the worst ones are those out of the scope of your expertise 😩</p><p>That was not an issue in Sendai, because there were three parallel blocks of talks to choose from. And with three available topics you’re more than likely to find something that’s just right for you. This one thing made the whole conference much more enjoyable 👏</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*rkH1IkuDuLiN8p5WgbsfQQ.gif" /><figcaption>When you choose the best one out of three available talks</figcaption></figure><p>For example, if you didn’t have enough experience with machine learning you could have skipped all talks related to it and choose ones related to memory usage or code quality (like I did 🙋).</p><h4>Keeping things technical</h4><p>Let me ask you a question. What are the most important things you look for at technical conference? I personally always look for the talks that require and share some deep technical knowledge. If I wanted to listen about ways of achieving the success I’d have chosen some kind of coaching event, not a technical one.</p><p>Thankfully, RubyKaigi had this one figured out really well. It was a technical conference and there was not a single non-technical talk there. And among those talks you had a wide variety of topics to choose from. Some of them, like <a href="https://github.com/Bombasarkadian/ruby-kaigi-notes/tree/master/day-2#julian-nadeau---scaling-teams-using-tests-for-productivity-and-education"><em>Scaling Teams using Tests for Productivity and Education</em></a><em> </em>or<em> </em><a href="https://github.com/Bombasarkadian/ruby-kaigi-notes/tree/master/day-3#eric-hodel-ezekiel-templin---devly-a-multi-service-development-environment"><em>Devly, a multi-service development environment</em></a>, focused on utilizing technology to improve the way you work. On the other hand, you had talks like <a href="https://github.com/Bombasarkadian/ruby-kaigi-notes/blob/master/day-3#yoh-osaki---how-to-get-the-dark-power-from-iseq"><em>How to get the dark power from ISeq</em></a> and <a href="https://github.com/Bombasarkadian/ruby-kaigi-notes/blob/master/day-3#keita-sugiyama-martin-j-d%C3%BCrst---grow-and-shrink---dynamically-extending-the-ruby-vm-stack"><em>Grow and Shrink — Dynamically Extending the Ruby VM Stack</em></a>, that required a deep understanding of some intricate low-level problems 😳</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*nFhFD8l0btZX3rmFNuteBg.gif" /><figcaption>When you go to Ruby conference and they talk about stack allocation and stuff</figcaption></figure><p>But if something was too hard for you, there were always two other talks to choose from. In the end it was possible to find interesting talks that were neither too easy not too hard. It made the experience much better than during typical conference where you get coaching talks mixed with technical ones 😃</p><h4>Focusing on Ruby</h4><p>There is one other thing that RubyKaigi did well. It focused on a single piece of technology — the Ruby language.</p><p>Even though there were talks about all kinds of topics like machine learning, cryptocurrencies or memory management, you only had to know one language to be able to follow them. And it made everything much easier to understand 🤓 After all, you already knew the language so you could have focused on what was the clue of each presentation.</p><p>Another advantage of being Ruby-exclusive conference was that a lot of talks focused on the future development of the language. As expected JIT, type checking and improving performance and memory usage were really hot topics there 🔥🔥🔥</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*Q5D4Oamqb95xSugJak1uTw.gif" /><figcaption>When it’s early 2018 but you’re already talking about features coming to Ruby 3</figcaption></figure><p>Being at RubyKaigi gave you an early insight into what will be available in Ruby in the future.</p><h4>Building a community</h4><p>But let’s be honest here. Listening to the talks is not the only reason to go to the conference, right? It’s also about the people. Networking, after parties, sharing ideas and opinions… That’s as important as presentations. And it seemed to be important during RubyKaigi as well.</p><p>Just starting your day? Why not go to the venue early and grab a free breakfast with other attendants? And that was not the only food-related networking opportunity. There was also a lunch break and some kind of party every day.</p><p>Apart from that, there were short breaks between talks. A perfect opportunity to quickly exchange thoughts with other attendants, prepare some notes or ask a follow-up question to the person giving a presentation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Ff0VB9Mg4a56IoyHctXrwA.png" /><figcaption>RubyKaigi was a really international conference. There were attendants from all around the World 🌏🚀</figcaption></figure><p>And people who were there all knew Ruby, so you had an instant connection with them. And the most important thing is that everybody was friendly and willing to talk with others. It was like a one, big Ruby family. This made the 3 days in Sendai really nice and productive.</p><h3>Next year, then?</h3><p>Those were all small details, but they contributed to something bigger. They turned RubyKaigi into an exceptional conference. You had 3 days filled with high-quality presentation and many opportunities to talk with like-minded people.</p><p>I will be definitely going back to Japan for RubyKaigi 2019 and I hope that you feel convinced about it as well.</p><p>Oh, and if you actually wanted to read about the talks, you can check out my <a href="https://github.com/Bombasarkadian/ruby-kaigi-notes">notes from the conference on GitHub</a> — you won’t be disappointed 🙆</p><p><a href="https://github.com/Bombasarkadian/ruby-kaigi-notes">Bombasarkadian/ruby-kaigi-notes</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*KtnDoZhaAkXhmLy0-y_JoA.png" /></figure><p>If you enjoyed this post, please hit the clap button below 👏👏👏</p><p>You can also follow us on <a href="https://www.facebook.com/Daftcode/">Facebook</a>, <a href="https://twitter.com/Daftcode_pl">Twitter</a> and <a href="https://www.linkedin.com/company/daftcode-sp-z-o-o-">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=501317c60d6d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/daftcode-blog/why-you-should-visit-the-next-rubykaigi-501317c60d6d">Why you should visit the next RubyKaigi</a> was originally published in <a href="https://medium.com/daftcode-blog">Daftcode Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Make your CI pipeline fast and awesome with Gitlab Container Registry]]></title>
            <link>https://medium.com/daftcode-blog/make-your-ci-pipeline-fast-and-awesome-with-gitlab-container-registry-41ef244d1ce6?source=rss-aff6071fbf8------2</link>
            <guid isPermaLink="false">https://medium.com/p/41ef244d1ce6</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[tech]]></category>
            <dc:creator><![CDATA[Maciej Głowacki]]></dc:creator>
            <pubDate>Wed, 04 Apr 2018 14:09:37 GMT</pubDate>
            <atom:updated>2020-09-03T07:33:11.602Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kG_1CkapZ414RwnSvcOn2Q.png" /><figcaption>Illustration by <a href="https://dribbble.com/magdalenatomczyk">Magdalena Tomczyk</a></figcaption></figure><p><strong>You must have heard of </strong><a href="https://en.wikipedia.org/wiki/Continuous_integration"><strong>Continuous Integration</strong></a><strong>. The chances are, you’re already using it in your projects (or at least plan to do so). There are a lot of different tools that let you get the job done, but today we’ll focus on doing cool things with Rails project using GitLab’s built-in CI.</strong></p><h3>Getting started</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*XbyHX7QDUEO_zuLvqK9ZlQ.png" /></figure><p>Starting with GitLab CI is straightforward. When creating a new project we can choose a “Ruby on Rails” template which contains a very basic CI setup file.</p><p>If you don’t trust a machine to setup your project (or have one that’s already been setup) the good place to start is <a href="https://docs.gitlab.com/ce/ci/quick_start/README.html">GitLab’s Getting started with GitLab CI/CD</a>. It will tell you to create .gitlab-ci.yml file in your repository and even give an example for Ruby on Rails. How cool is that?</p><p>Whichever way you chose, you’ll have to do some simple changes. Choose proper ruby version, select different testing framework or setup the database. In the end, you should have a .gitlab-ci.yml that looks similar to this one:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7d4d184075ecbc76514d62f053521fc9/href">https://medium.com/media/7d4d184075ecbc76514d62f053521fc9/href</a></iframe><p>After you push this file to the repository and wait for a while you will see that all tests have passed. Your commit will get a green checkmark to show this 💚</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*ke-HBp-UVUc5aZ5IECKJKQ.png" /></figure><h3>Adding awesomeness</h3><p>Now that you’ve got the basic script running you may want to make it do all sort of cool things.</p><h4>Code coverage</h4><p>First, add code coverage support for tests. Adding <a href="https://github.com/colszowka/simplecov">simplecov</a> to your application is easy, but how do you make GitLab use it? Open your repository in GitLab and go to Settings &gt; CI/CD &gt; General pipelines settings. Look for Test coverage parsing section and use the example they show you there. It’s that easy 👏</p><p>And if you scroll the settings page a bit further you’ll find badges for pipeline status and code coverage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*q_13fty7kWnQtl2YGecbJQ.png" /><figcaption>Remember to add them to your project’s readme, because it’s been scientifically confirmed that the more badges you add the better your project becomes. True story</figcaption></figure><p>Apart from coverage badge, you can now see coverage in Merge Requests. This lets you find any missing tests at a glance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*zdRCjfnfdDLOnbQVeq070A.png" /></figure><h4>Code quality</h4><p>Another thing that you can improve is <a href="https://github.com/bbatsov/rubocop">rubocop</a> support. Right now it finds offenses in your code and causes the pipeline to fail. But to see what’s wrong you have to open the job details and read through the logs. You can definitely improve this experience somehow.</p><p>And that’s when <a href="https://github.com/prontolabs/pronto">pronto</a> comest to help. This nifty gem lets you integrate all kinds of different tools with GitLab. It makes their output appear as comments in Merge Requests. The first step is to add pronto and some runners to your Gemfile:</p><pre><em>group </em>:ci <strong>do<br>  </strong><em># Quick automated code review of your changes<br>  </em>gem &#39;pronto&#39;<br>  gem &#39;pronto-brakeman&#39;, require: <strong>false<br>  </strong>gem &#39;pronto-flay&#39;, require: <strong>false<br>  </strong>gem &#39;pronto-reek&#39;, require: <strong>false<br>  </strong>gem &#39;pronto-rubocop&#39;, require: <strong>false<br>end</strong></pre><p>Now, instead of doing bundle exec rubocop, you can execute bundle exec pronto run -f gitlab -c origin/master. To make it work in CI, you have to give pronto an API key that will let it write comments. You can generate this key in Access Tokens tab in profile settings (you may want to have a separate account dedicated to writing automated comments).</p><p>Now you can configure PRONTO_GITLAB_API_PRIVATE_TOKEN and PRONTO_GITLAB_API_ENDPOINT in Settings &gt; CI/CD &gt; Secret variables.</p><p>You’ll also have to tell apt-get command to install cmake. Pronto needs it to build some required libraries.</p><p>From now on your code will get checked with every push, and pronto will report all offenses as comments. You’ll see them appear under the relevant line of code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*GiX4qpGdSjwPbioJ3MKvYg.png" /><figcaption>You can of course ignore those comments as long as there is no AI sentient enough to stop you from doing this</figcaption></figure><h4>Updates</h4><p>Another thing that is easy to track with GitLab CI are updates for your project dependencies. Regular updates are important for two reasons: security and future maintainability.</p><p>You can handle the first one with a gem called <a href="https://github.com/rubysec/bundler-audit">bundler-audit</a>, It has a list of vulnerable gems, and uses it to check your Gemfile. Edit ci group in your Gemfile and add bundler-audit there:</p><pre>group :ci <strong>do</strong><br>  ...<br>  <em># Patch-level verification for Bundler<br>  </em>gem &#39;bundler-audit&#39;, require: <strong>false<br>end</strong></pre><p>To make it run in your CI pipeline add bundle audit check --update the same way you’ve added pronto before.</p><p>Now, every time you push some changes it will check if any of your gems have known vulnerabilities. If it does, your pipeline will fail so you’ll know that there is an urgent update required.</p><p>But you don’t need to have a vulnerable gems to update dependencies, do you? 😅 Luckily, Bundler has a built-in tool that helps you check for gems updates. All you have to do is execute bundle outdated --strict. The strict flag tells Bundler not to report gems that you’ve version-locked in Gemfile. Since most gems updates are minor fixes that can wait you don’t want your pipeline to fail on this command. To make it less annoying, you can allow a CI job to fail. Or in a language of GitLab config:</p><pre><strong>check_gemfile:</strong><br>  <strong>script:<br>    </strong>- bundle outdated --strict<br>  <strong>allow_failure: </strong>true</pre><p>When it finds some new gems versions available, you’ll get a warning, which looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*ohG22l8ES7SCbQnycM1TBg.png" /><figcaption>Clicking on this orange exclamation mark will tell you that it’s the check_gemfile job that didn’t pass</figcaption></figure><h3>Making it faster 🚀</h3><p>It all works, but as you will notice it’s not as fast as you might expect. Your basic pipeline will spend around 10 minutes setting up. The worker spends this time on installing dependencies and extracting cache. Can you skip those steps somehow?</p><h4>Container Registry</h4><p>As it turns out, GitLab has a cool feature called Container Registry. It lets you upload custom docker image and use it to execute CI jobs. Open your project and look for Registry button which looks like this one:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*tE5RqHT2dlvjGaL8LCIS2Q.png" /><figcaption>If you don’t see it, you may need to update GitLab or enable Container Registry in configuration</figcaption></figure><p>You’ll find a short step-by-step instruction there. It explains how to configure Docker for use with Container Registry. To create Docker images you must create a Dockerfile describing your setup. It should do the same thing that you’ve configured inside .gitlab-ci.yml:</p><ul><li>choose Ruby version,</li><li>install build dependencies,</li><li>install gems.</li></ul><p>If you use such image for CI execution, all gems will be already installed. It won’t have to spend time setting up dependencies and building libraries. Your basic Dockerfile may look like this one:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/25ae2fcd8ad0b14f48f1897597644a52/href">https://medium.com/media/25ae2fcd8ad0b14f48f1897597644a52/href</a></iframe><p>First, it installs node (required by Rails) and cmake (required by pronto). Then, it uses local Gemfile to install all gems required for testing.</p><p>Now, you’re ready to build and push according to the guide from Container Registry. Choose some tag for your image, so you can have different ones in the future and know which one is which. Let’s name this one master.</p><h4>Configuration</h4><p>The last step is changing .gitlab-ci.yml to make it use your custom image. All you have to do is change image value, and get rid of cache and most build related commands. You only need bundle install, so it can install all new gems that might be missing your image. Final setup, with all awesomeness added, looks like this one:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f9fa2d59543b91e8805a7a63f33a5b33/href">https://medium.com/media/f9fa2d59543b91e8805a7a63f33a5b33/href</a></iframe><p>It should be much faster now, because it doesn’t have to install most of the dependencies. They are already there. Your pipeline will finish in under a minute now. There are ways to optimize this (you can skip bundle install in updates job), but it’s already good enough.</p><p>And if you happen to change your Gemfile you only have to update your Dockerfile, and push a new image. You can even overwrite your master image so you won’t have to change anything inside .gitlab-ci.yml.</p><p>How cool is that? 💆</p><h3>Wrapping it up</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/1*iqJAV4DPhRGIhMoIFviznQ.gif" /><figcaption>The dog symbolizes “it” and wrapping symbolizes wrapping — obviously</figcaption></figure><p>And that’s all you have to know to make your GitLab CI faster. If you followed the steps I’ve described, you should now have a setup that:</p><ul><li>measures your code quality,</li><li>runs tests and checks code coverage,</li><li>watches for outdated dependencies.</li></ul><p>Apart from thath, thanks to Container Registry, you should have reduced pipelines runtime down to a minute.</p><p>It should be enough to get you started or help you make your existing pipelines faster. If you have some other tips for awesome things to add to CI go ahead and share it in the comments below 🙌</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*KtnDoZhaAkXhmLy0-y_JoA.png" /></figure><p>If you enjoyed this post, please hit the clap button below 👏👏👏</p><p>You can also follow us on <a href="https://www.facebook.com/Daftcode/">Facebook</a>, <a href="https://twitter.com/Daftcode_pl">Twitter</a> and <a href="https://www.linkedin.com/company/daftcode-sp-z-o-o-">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41ef244d1ce6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/daftcode-blog/make-your-ci-pipeline-fast-and-awesome-with-gitlab-container-registry-41ef244d1ce6">Make your CI pipeline fast and awesome with Gitlab Container Registry</a> was originally published in <a href="https://medium.com/daftcode-blog">Daftcode Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is Crystal production-ready yet?]]></title>
            <link>https://medium.com/daftcode-blog/is-crystal-production-ready-yet-2a3f197e2d6e?source=rss-aff6071fbf8------2</link>
            <guid isPermaLink="false">https://medium.com/p/2a3f197e2d6e</guid>
            <category><![CDATA[programming-languages]]></category>
            <category><![CDATA[crystal-lang]]></category>
            <category><![CDATA[ruby]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Maciej Głowacki]]></dc:creator>
            <pubDate>Wed, 17 Jan 2018 18:01:02 GMT</pubDate>
            <atom:updated>2018-03-02T10:38:18.436Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6IRb-KWK8ptaTyIanSiVQw.png" /><figcaption>Illustration by <a href="https://dribbble.com/magdalenatomczyk">Magdalena Tomczyk</a></figcaption></figure><p><strong>Have you heard about </strong><a href="https://crystal-lang.org/"><strong>Crystal Programming Language</strong></a><strong>? It’s been over 3 years since we saw its first version, but if you’re not familiar with it, it’s a statically typed, compiled language with a very Ruby-esque syntax.</strong></p><p><strong>This has at least three implications:</strong></p><ol><li>With static types, we are more likely to catch some errors before shipping our code (yes, I’m looking at you NoMethodError for nil:NilClass 🙄)</li><li>Compiled applications should run almost as fast and efficient as those written in low-level languages like C</li><li>Writing Crystal is supposed to be as easy as writing Ruby</li></ol><p>And it actually seems to deliver all of those features. Types interference usually works really smoothly, so you often don’t even have to worry about explicitly specifying what you need. The compiler catches all uninitialized objects before you even have a chance to run your code. Execution time is within the same order of magnitude as low-level languages, and memory usage is much lower than Ruby.</p><p>So<strong> should we just drop whatever we’re using and rewrite everything to Crystal?</strong> Well, probably not (yet), but let’s talk about it now.</p><h3>What does it mean to be production-ready?</h3><p>When creating a new application there are a lot of things to consider before choosing a technology stack:</p><ul><li>We want our language to be fast to write, and easy to maintain.</li><li>We’d also like it to have a big ecosystem with mature frameworks which won’t break after the first update.</li><li>Finally, we can’t forget about tools that surround the language and make development easier. Syntax highlighting, linters, deployment tools etc.</li></ul><p>Once we have all of the above, we can call a technology “production-ready”. So <strong>is Crystal production-ready yet?</strong></p><p>Let’s go through that list one-by-one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*yyd-FDQPmuGK-P3Res3D8w.png" /></figure><h4>Crystal is fast to write, but it keeps on changing</h4><p>High development speed is basically guaranteed here. With Ruby-inspired syntax it’s obviously really easy to write new code. Basically, if you love Ruby you’ll love Crystal as well. But what about its maintainability?</p><p>Right now, the latest released version of Crystal is 0.24.1. Whether we like it or not, it’s officially still in a pre-release phase. The language is still evolving and there are some breaking changes from time to time — most of them, however, are quite easy to introduce into an existing project. Those changes are usually a simple class renaming or new method parameters, but even those may become cumbersome in complex applications.</p><p>Since it’s still a beta release we should expect some bugs, which would require us to update. To avoid spending weeks on rewriting your code, it may be wise to keep your Crystal projects small for now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*42VaFA9d-d7AqW8TtUcUtA.jpeg" /><figcaption>Let’s start with something small, then move on to bigger things. Like maybe a death star or something? But you wouldn’t build a death star with pre-release tools, will you? — Photo by <a href="https://unsplash.com/photos/ZqqlOZyGG7g?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Daniel Cheung</a> on <a href="https://unsplash.com/search/photos/lego?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h4>There are many shards, but is the ultimate one already there?</h4><p>Shards are to Crystal what gems are to Ruby. Little (or not so little) libraries which you can easily reuse in your code. Do you want to build a web application? You’ve already got <a href="https://github.com/kemalcr/kemal">kemal</a>, <a href="https://github.com/luckyframework/lucky">lucky</a> or <a href="https://github.com/amberframework/amber">amber</a> to help you. Maybe you’re writing tests and need to do some extra stubbing? You can already find some standard stuff like <a href="https://github.com/manastech/webmock.cr">webmock.cr</a> or <a href="https://github.com/TobiasGSmollett/timecop.cr">timecop.cr</a>. Want some background processing? Worry not, <a href="https://github.com/mperham/sidekiq.cr">sidekiq.cr</a> got you covered.</p><p>All the Shard above have one thing in common. They are suspiciously similar to what we have in Ruby. If you noticed the cr in some names, it usually indicates Ruby-inspired shards, so it’s really straightforward to use them when coming from Ruby.</p><p>We can probably agree that one of the most important gems was Rails. It helped with community growth and made Ruby a lot more popular in production use. And if you are a Rubyist you may try to apply duck typing to Crystal — since it looks and behaves like Ruby, therefore it is Ruby, and it needs its own Rails. Or does it?</p><p>Since Rails already does its job quite well do we really need something similar? What if instead of trying to mimic Ruby we went a step further and looked for something that Crystal can do what Ruby couldn’t? Since it’s faster there may be possible applications in machine learning, low-level system tooling, high-performance computation, or graphics processing. Have you seen all the “Crystal intros” which do a basic Web API and compare the code with <a href="https://github.com/sinatra/sinatra">Sinatra</a>? 🙈 Maybe, instead of solving what‘s already been solved let’s create something new, something we’ve always wanted to try, but Ruby just couldn’t cut it.</p><p>If you need some inspiration look at <a href="http://crystalshards.xyz/">the existing shards</a> and browse through an <a href="https://github.com/veelenga/awesome-crystal">Awesome Crystal list</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*yyd-FDQPmuGK-P3Res3D8w.png" /></figure><h4>You can write with your favorite tools, but then it gets rough</h4><p>If we look at the Awesome Crystal list I’ve linked above, we can see plugins for any reasonable editor we may use. There are syntax highlighters and linters for Vim, Emacs, Atom, Visual Studio Code or Sublime. We can easily manage Crystal versions with <a href="https://github.com/marciogm/asdf-crystal">asdf</a> or <a href="https://github.com/pine/crenv">crenv</a>, and there is an <a href="https://github.com/CorbanR/ansible-crystal">ansible playbook</a> and a <a href="https://github.com/crystal-lang/heroku-buildpack-crystal">heroku buildpack</a> for your remote setup.</p><p>But what if we actually want to move past web applications and create some desktop app or a command line tool? In that case we’d like to distribute our application as a single, self-contained executable. And Crystal compiler can statically link required libraries, but it has problems doing this on macOS. And since Crystal is not yet Windows-compatible, our potential reach is quite limited. But Windows support has been <a href="https://github.com/crystal-lang/crystal/issues/26">already discussed</a> for a long time and seems to be a heavily worked-on feature <a href="https://github.com/crystal-lang/crystal/issues/5430">recently</a>, so those issues may soon become a history.</p><h3>So… is Crystal production-ready or not?</h3><p>Even though we’ve looked at different aspects of what makes software production-ready it’s still hard to give a specific answer 😳</p><p>Crystal might not have a clearly defined future yet, but it’s definitely a solid project with good development pace. While using it as a “new Ruby” to build web applications may be a dead end, there are a lot of other possible projects you can create with it. And yes, we’re even talking about building production grade software now.</p><p>Maybe just try to keep it small enough to account for possible syntax changes until Crystal 1.0 is out. Now is a great time to contribute to the community, create a new shard for others or help with language development itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*KtnDoZhaAkXhmLy0-y_JoA.png" /></figure><p>If you enjoyed this post, please hit the clap button below 👏👏👏</p><p>You can also follow us on <a href="https://www.facebook.com/Daftcode/">Facebook</a>, <a href="https://twitter.com/Daftcode_pl">Twitter</a> and <a href="https://www.linkedin.com/company/daftcode-sp-z-o-o-">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a3f197e2d6e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/daftcode-blog/is-crystal-production-ready-yet-2a3f197e2d6e">Is Crystal production-ready yet?</a> was originally published in <a href="https://medium.com/daftcode-blog">Daftcode Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Fastest Way of Importing Data with Ruby?]]></title>
            <link>https://medium.com/daftcode-blog/the-fastest-way-of-importing-data-with-ruby-80bd9ba6274b?source=rss-aff6071fbf8------2</link>
            <guid isPermaLink="false">https://medium.com/p/80bd9ba6274b</guid>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[ruby]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Maciej Głowacki]]></dc:creator>
            <pubDate>Wed, 11 Oct 2017 09:16:47 GMT</pubDate>
            <atom:updated>2020-09-03T07:27:57.165Z</atom:updated>
            <content:encoded><![CDATA[<p>Importing huge amount of data is not something most of us have to worry about on a daily basis. But once in a while, you may be presented with tons of data that you have to put inside your application. Why? There are many reasons: maybe you’re creating some kind of a knowledge database, or you’ve just bought your direct competitor and want to merge their users’ data with your own. Pick your favorite, it doesn’t matter right now. What matters is that we have a task waiting to be done. So let’s try to deal with it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-AX7E5VX4lAHAFyvVAgOvQ.gif" /><figcaption>Animation by <a href="https://dribbble.com/mjalbrzykowski">Michał Jałbrzykowski</a></figcaption></figure><h3>Backstory</h3><p>Let’s imagine we’re running Instagram-esque application with Ruby on Rails. Users can upload photos and videos there, and other Users can comment on them. Apart from commenting on Posts, Users can also leave comments on other Users’ profiles. Our database model may look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CIljL_Q2bGUwQcZjW-u4UQ.png" /><figcaption>Look at those polymorphic relations, aren’t they nice?</figcaption></figure><p>While we were busy designing database, our main competitor focused on actually getting people to use their application. They have 100 000 Users, each of them having 10 Images, 10 Videos and 10 Texts. Each of those posts has been commented 10 times. Each profile has been commented 10 times as well. But their database looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pK0t5HfsrXwbU8XDjZ2nBA.png" /><figcaption>🙈</figcaption></figure><p>Somehow we bought their application and now have to merge their data into ours (yup, we had some Users as well). To make things harder, they’re running MySQL while we’re using PostgreSQL. Since those two database schemas seem to differ quite a lot, and our database already has some data inside, we make a guess that scripts like <a href="https://github.com/dimitri/pgloader">pgloader</a> just won’t cut it.</p><p>We might try writing a simple task for data import. While there are hundreds of ways to do this (most of them probably not too good 🤓) we will look at just two of them which are making it all run really fast.</p><h3>Basic approach</h3><p>To migrate data we’ll probably want to create a rake task, but to do this we must first access “foreign” data. Can we simply create classes for those models? Well, actually we can. First, we have to configure the second database in config/database.yml by doing something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/256057ba509f4c0bb9940bf8f59a3923/href">https://medium.com/media/256057ba509f4c0bb9940bf8f59a3923/href</a></iframe><p>Then we can create some classes to use this connection:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e22f6542550830d18140cbc346acbc8f/href">https://medium.com/media/e22f6542550830d18140cbc346acbc8f/href</a></iframe><p>Now, we can proceed to write a base of our rake task:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ea3da6d76381de23d1327084988df02/href">https://medium.com/media/7ea3da6d76381de23d1327084988df02/href</a></iframe><p>And that’s when we stumble upon the first problem. Since Comment has both User and a Post, how can we add it from within this loop? It seems like our Comment could either have a User and possibly lack Post or have a Post but not the User. This can be easily solved by keeping original IDs and offsetting them, so they don’t collide with existing records. Think about something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6b5346cc56c798f9c50863278710b2f5/href">https://medium.com/media/6b5346cc56c798f9c50863278710b2f5/href</a></iframe><blockquote><strong>Warning!<br></strong>To do this, we have to stop the application. Any record created while our import script is running would result in a catastrophic failure <em>😳</em></blockquote><p>Now, when creating Comments we don’t have to actually pass User and Post there — we can just use their IDs, which either already exist or eventually will (when creating Users we must set their IDs the same way).</p><p>But this will still fail because Rails try to keep our data safe and add database constraints. We can disable it at the beginning of the script with:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/09c5346416d82516f94acb01bed01aa6/href">https://medium.com/media/09c5346416d82516f94acb01bed01aa6/href</a></iframe><blockquote><strong>Second warning!</strong><br>Don’t forget to enable it once the import is over.</blockquote><p>Now we can run our rake task and it should finish without errors. But it will take about 60 hours to complete. Stopping the application for so long doesn’t sound good, right? Let’s now fast forward a few iterations and see what we may end up with after some work.</p><h3>Parallel batch inserts with eager loading and Sequel</h3><p>A lot of stuff going on in this header, so how about we tackle it piece by piece? It will be best if we look at them from last to first:</p><ul><li><a href="https://github.com/jeremyevans/sequel"><strong>Sequel</strong></a><br>Sequel is one of available <a href="https://en.wikipedia.org/wiki/Object-relational_mapping">ORMs</a> for Ruby (think ActiveRecord but with 3x less memory footprint). With ActiveRecord we’d have to allocate close to 100GB during our import process, while with Sequel we can keep it under 35GB. Less objects to build = faster processing.</li><li><strong>Eager Loading</strong><br>If we are not careful enough we may run into <a href="https://stackoverflow.com/a/97253">N+1 query problem</a>. Luckily it can be easily avoided with eager loading. In ActiveRecord we do in with .includes and in Sequel we have .eager. But they both have the same result, which is reducing the number of selects we make.</li><li><strong>Batch Insert</strong><br>Since we’ve reduced the number of selects we should also try to bring down the number of inserts. The easiest way is to avoid saving every object separately, but rather saving them in bigger batches. There’s a gem for this, which is called <a href="https://github.com/zdennis/activerecord-import">activerecord-import</a>. Under the hood it creates SQL queries like this one:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6869a87d7f4dfa83059c55f4b1679b2a/href">https://medium.com/media/6869a87d7f4dfa83059c55f4b1679b2a/href</a></iframe><ul><li><strong>Parallel</strong><br>After doing above optimisations we may look at the logfile and see that we spend some time processing data between reading and writing them to the database. We could probably run our task in a few parallel processes to make full use of our database (when one process is writing others have time to prepare data). You might have never done parallel processing in Ruby, but it’s actually quite easy (if we skip the hard parts 😅):</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9b8e8c6fff3305e4bd233cb7224d6910/href">https://medium.com/media/9b8e8c6fff3305e4bd233cb7224d6910/href</a></iframe><p>We can now run our new import script and it finishes in <strong>just 3.6 hours</strong>. With only a few easy changes we’ve achieved <strong>15x speedup</strong>. Do you think we can do it even faster?</p><h3>Skipping Ruby</h3><p>The previous method was really useful as it allowed us to do any kind of data processing — we could have done stuff like generating passwords for Users or extracting something from JSON before saving to the database. But if we take a closer look at what we’re doing now, we can see that the only processing we need is offsetting IDs. It sounds like something we can do with SQL alone.</p><p>So what if we could skip ActiveRecord, Sequel and all this slow stuff entirely, and try to migrate our data using SQL? For loading data into Postgres we can use COPY command, which is even faster than batch insert and allows reading data from a text file. On the other end, we have MySQL which can save select results into CSV file. What if we try to make them work together?</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b06bcc3a6460b9d7819f779832318116/href">https://medium.com/media/b06bcc3a6460b9d7819f779832318116/href</a></iframe><p>Now we only have to write similar queries for every resource and we’re pretty much done. It couldn’t get much easier. And the results are just amazing. Importing all data (which took us 60 hours in the beginning) finishes in <strong>just 8 minutes</strong>.</p><p>That’s <strong>430x faster!</strong></p><p>This is a solution we can definitely accept 👍</p><h3>Wrapping up</h3><p>We’ve taken a look at two possible ways of fast data import with Ruby (with some help from raw SQL queries of course).</p><p>First of them is really useful when we need to do some complex data processing before we can save it to the database. The second one only allows for very basic processing but gives an incredible speed boost.</p><p>I have yet to find any faster method of importing data, but if you happen to know one, don’t hesitate to share it in the comments below this article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*KtnDoZhaAkXhmLy0-y_JoA.png" /></figure><p>If you enjoyed this post, please hit the clap button below 👏👏👏</p><p>You can also follow us on <a href="https://www.facebook.com/Daftcode/">Facebook</a>, <a href="https://twitter.com/Daftcode_pl">Twitter</a> and <a href="https://www.linkedin.com/company/daftcode-sp-z-o-o-">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80bd9ba6274b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/daftcode-blog/the-fastest-way-of-importing-data-with-ruby-80bd9ba6274b">The Fastest Way of Importing Data with Ruby?</a> was originally published in <a href="https://medium.com/daftcode-blog">Daftcode Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How We Made Writing Tests Fun and Easy]]></title>
            <link>https://medium.com/daftcode-blog/how-we-made-writing-tests-fun-and-easy-2d7e1fac6d16?source=rss-aff6071fbf8------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d7e1fac6d16</guid>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[ruby]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[rspec]]></category>
            <dc:creator><![CDATA[Maciej Głowacki]]></dc:creator>
            <pubDate>Tue, 29 Aug 2017 13:11:46 GMT</pubDate>
            <atom:updated>2020-09-03T07:21:45.630Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1yQdnB7Bn4sTuuBnzu5Ahw.png" /><figcaption>Illustration by <a href="https://dribbble.com/SoniaBudner">Sonia Budner</a></figcaption></figure><p><strong>Testing, right? You may love it, you may hate it, but you should agree that good tests can be beneficial to you, your team, and people who might work on your project in the future. Testing might not be the most exciting part of your work, but it’s definitely quite useful. Well tested code brings peace to your mind when refactoring and creating new features.</strong></p><p>Or does it? What if those tests weren’t written by you? Are you sure they cover all cases you think they do? Do they actually test anything or just mock pretty much the whole application? Ouch. Now apart from what you were supposed to do, you also have to spend time making sure that existing tests are useful and well written.</p><p>That’s how it usually goes if there are no rules for testing within a project.</p><h3>Let’s create some rules</h3><p>You may now be tempted to come up with rules regarding testing. Yeah, let’s do this! You say:</p><blockquote>From now on, let’s write good tests and aim for 100% code coverage</blockquote><p>You then pass that idea to the rest of your team and wait for it to work.</p><p>But will it work? Well, you’ll probably end up with a bunch of “tests” that make a few requests here and there, resulting in high coverage without doing much. What about the “good tests” part? Who knows what it even means. I bet you’re not satisfied with such an outcome. Let’s change something about it!</p><p>But do you actually know what’s wrong with this approach? To begin with, it does nothing to make development faster or easier. It actually does quite the opposite — forcing developers to write at least twice the amount of code, without even giving them a hope for some reward. If you ask others to write tests they will probably do it, but should you expect them to put heart and soul into it?</p><h4>Developers want carrots, not sticks</h4><p>Since the stick is not a good way to enforce writing tests, <a href="https://en.wikipedia.org/wiki/Carrot_and_stick">let’s try with a carrot</a>. What if writing tests gave an instant reward? How about having an API documentation generated for you without any extra work? It’s great if you ask me, and this specific “reward” was exactly what we needed to start writing better tests.</p><p>(Don’t get me wrong here. Good tests are great by themselves, and they’ll always pay off in a longer term. However, some instant gratification can be a real productivity booster, especially when doing tasks which are a bit mundane.)</p><p>At this point, <strong>you have to make a choice</strong>.</p><p>You can either keep on reading to discover how much fun testing can become (which I recommend you do), or you can jump straight into an <a href="https://github.com/Bombasarkadian/testifier"><strong>example application</strong></a> (but you may miss out on stories which are about to unfold).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/460/1*Phj0Wu5HBz2v-dkBMHuVMA.png" /></figure><p>So you’ve chosen to stay here? Very good! Then, if you like the idea of killing two birds with one stone let’s see how can we make writing acceptance tests easier. Since we’re using rspec we can make things easier by using the <a href="https://github.com/zipmark/rspec_api_documentation">rspec_api_documentation</a> gem. Add it to your project according to the instructions so you can create the very first test:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1a1258fdb5e174edcf769ab596218062/href">https://medium.com/media/1a1258fdb5e174edcf769ab596218062/href</a></iframe><p>Woah, just looking at this test lets you get a grasp of what this application can do. We can instantly see what are the parameters, what’s the response and what headers should be sent. But it gets even better after you run rake docs:generate and wait for tests to finish because you get this as a result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*OiwSCHkAWFY2Hca5fauoiA.png" /><figcaption>This looks like something we can just show to people who will use our API. All the styling is already done for us. There are two documentation browsers available out of the box for rspec_api_documentation: <a href="https://github.com/jejacks0n/apitome">apitome</a> and <a href="https://github.com/smartlogic/raddocs">raddocs</a></figcaption></figure><p>That was fast and easy, wasn’t it? Now if we want to add any more examples to this documentation, we have to actually write tests for it. This gives us a push to instantly cover all 3 of the following cases:</p><ul><li>valid request</li><li>invalid parameters</li><li>missing parameters</li></ul><p>And now we’re talking! Our tests are beginning to get useful. Have we accidentally broken something in an endpoint for creating new posts? Worry not — there is a test for this case and we will know about it. Maybe we’ve disabled some validation that seemed unnecessary, but actually it wasn’t? Since we wanted documentation for invalid parameters, there’s a test for this as well.</p><p>We may have just solved one issue with testing. It’s now more fun than before and yields something that is instantly visible. But our tests are neither easier to write nor prettier yet.</p><h3>The ugly side of testing</h3><p>So we have this API that allows creating posts. Wouldn’t it be great if users could choose to publish those posts on, say, Twitter or Facebook? Aww yeah, sounds awesome! But we don’t want to hit 3rd party API every time we run tests, do we? At the same time, it would be great to check whether or not there will be a request made there.</p><p>Sounds like something <a href="https://github.com/bblimke/webmock">webmock</a> can do. We add it to the Gemfile and install it according to instructions. From now on we can’t talk to the internets from within our tests. We have to explicitly tell webmock that we’ll be making a specific request and remember to set an expectation with rspec:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/67c1c4f8333b8070fa509eab9f22899b/href">https://medium.com/media/67c1c4f8333b8070fa509eab9f22899b/href</a></iframe><p>This doesn’t look too bad. But we’re just looking at one test written by one person. If we ask 10 people to write the same test we may end up with 10 different solutions. What if someone wants to get over stubbing quickly and doesn’t even check parameters that they send? What if somebody else forgets to actually check if they’ve made a request? Something can get broken later and nobody will know about it until it’s too late.</p><p>It seems that we’re back to square one — we have to make sure that other people’s tests behave the way we expect them to. But how can we make sure all people will write tests in the same way?</p><h3>Make it easier</h3><p>The problem is, writing bad tests is way easier than writing good ones. Why would people care about stubbing requests properly and setting good expectations, when they can just “make it green” with much lower effort? After all, they will have a passing test and a generated documentation.</p><p>We must somehow outsmart their inner lazy-developer and make writing good tests easier than writing bad ones. What if we could give them a way to write good tests without them actually feeling like they write tests? What if those tests just wrote themselves? Sound like some sort of dark magic? Well, maybe. But it doesn’t matter it’s impossible.</p><p>The idea here is to create some sort of internal <a href="https://en.wikipedia.org/wiki/Domain-specific_language">DSL (Domain-specific language)</a> to describe our test cases. We don’t want it to be too fancy — just a simple way to extract common test logic. We also want it to look familiar for people who already know rspec, so we’ll build it around existing syntax.</p><p>Extracting common logic sounds like a task for <a href="https://relishapp.com/rspec/rspec-core/docs/example-groups/shared-examples">shared examples</a>. Let’s create our shared_examples_for_api_request and initialize it with everything that describes our endpoint:</p><ul><li>Name</li><li>Explanation</li><li>Headers</li><li>Request example</li></ul><p>It will look like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6a381fc9ab2f3deffb27af4b24156dd6/href">https://medium.com/media/6a381fc9ab2f3deffb27af4b24156dd6/href</a></iframe><p>To use this we will just call:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0555d0513fff223d65e22e9c8c4251e2/href">https://medium.com/media/0555d0513fff223d65e22e9c8c4251e2/href</a></iframe><p>Now we can start working on the most interesting part. Our own DSL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qwmdr-XQyYkSFEFp_MlKaA.jpeg" /><figcaption>Those are hands, and we’re about to get them dirty. Do you know how hard it is to find pictures for technical articles? 🙉 (ph. universe.com)</figcaption></figure><h3>Getting your hands dirty</h3><p>Our goal here is to create an object that will be used to automatically setup stubs and expectations for our tests. We should begin with a new class then:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e908380759ab701034e0f05886d4ab31/href">https://medium.com/media/e908380759ab701034e0f05886d4ab31/href</a></iframe><p>To make things easy and readable we use it as rspec subject like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5263da21603e649395a8cac3efd89ccb/href">https://medium.com/media/5263da21603e649395a8cac3efd89ccb/href</a></iframe><p>We now have it available inside our shared example, but it’s not really useful yet. The first thing we’d like to check is whether or not the request was successful. What do you think about specifying it by calling .success or .failure on our ApiRequest?</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6fe4f2d02e0b18b64544b50d6806580a/href">https://medium.com/media/6fe4f2d02e0b18b64544b50d6806580a/href</a></iframe><p>Those are simply ApiRequest methods, which can change its inner state to specify expected response code. They should return the object we’re working on, so we can chain more stuff later on:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c7daaf6b49373c93f3e33c69bd228c5e/href">https://medium.com/media/c7daaf6b49373c93f3e33c69bd228c5e/href</a></iframe><p>Now we can add our first expect to the shared example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fb00b1ace2ebeb9ef2c5b04ae3961c52/href">https://medium.com/media/fb00b1ace2ebeb9ef2c5b04ae3961c52/href</a></iframe><p>Woohoo! It’s starting to get useful. But just checking the status code is not enough. We’d also like to check the response. And maybe we’ll also change this .new into something more English-like?</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/57a69556d466836b47bec731d3d8779c/href">https://medium.com/media/57a69556d466836b47bec731d3d8779c/href</a></iframe><p>Much nicer. Now, it’s the time for the implementation. Using .test to initialize object is as easy as delegating it to .new, but when implementing .response we have to remember that we want it to accept both keywords and key-value pairs. We have to store them separately, as they will be tested in different ways:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ea2ccebac57f107888cc59c91aa6ce55/href">https://medium.com/media/ea2ccebac57f107888cc59c91aa6ce55/href</a></iframe><p>Finally we can write expects:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cb4edfb730fa39e86ce7846bd601c2de/href">https://medium.com/media/cb4edfb730fa39e86ce7846bd601c2de/href</a></iframe><p>We’ve already got some solid base for testing our requests. But it’s often necessary to check some object state after the request. This can, however, differ from test to test, so we can’t describe it as a part of our DSL. But we can make it possible to pass some custom tests like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/38201815bbc419d717c24ce30f346aae/href">https://medium.com/media/38201815bbc419d717c24ce30f346aae/href</a></iframe><p>We’re passing a block here that will be evaluated in an appropriate context. You can probably guess how the implementation looks:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dff1fed760570c78d884d0369090b58f/href">https://medium.com/media/dff1fed760570c78d884d0369090b58f/href</a></iframe><p>And in the shared example we can call it like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/acf8f73ce3efaaef2b7d41bbca60f869/href">https://medium.com/media/acf8f73ce3efaaef2b7d41bbca60f869/href</a></iframe><p>Our DSL is already starting to look pretty nice, and we haven’t even implemented its killer feature. Get ready for requests stubbing. Hold tight as it’s going to get a bit more difficult.</p><h3>Let’s roll the big guns</h3><p>Before we dive into stubbing API calls, there is one more thing that we should look at. Let’s assume that apart from posting to Twitter and Facebook, our application also sends an email (I don’t know whom to — probably to CIA). This sounds like something we should handle in acceptance tests as well.</p><p>Let’s say we want to check if we send notification email after the new post is created. I suggest we do it like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f7eb91475b46f4e5fe59675aacb08d4b/href">https://medium.com/media/f7eb91475b46f4e5fe59675aacb08d4b/href</a></iframe><p>Now if we think about it, then .to and .with probably don’t belong to ApiRequest itself. It looks like .email should create an object of some other class, which is bound to the request we’re working on. We can see it as a way to tell ApiRequest that we’re be describing its Mail now. Sounds reasonable? Let’s code this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b7cc3830571653adb4b15ef81f0b3d28/href">https://medium.com/media/b7cc3830571653adb4b15ef81f0b3d28/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ed05a0e95e93bcdf212be4f1c9e11f1a/href">https://medium.com/media/ed05a0e95e93bcdf212be4f1c9e11f1a/href</a></iframe><p>Before we move on we still have to figure out how to get out of Mail and return back to the ApiRequest we’ve been working on. We can assume that if Mail object receives something other than .to, .from or .with, then we want it to be handled by ApiRequest, not the Mail. Equipped with such knowledge we can do this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0474a2e872356b04d558294170a752c1/href">https://medium.com/media/0474a2e872356b04d558294170a752c1/href</a></iframe><p>Now we can safely expand our shared example with email related expects:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b5a6fe4b411a1dd2e2d74621cc8ab7ad/href">https://medium.com/media/b5a6fe4b411a1dd2e2d74621cc8ab7ad/href</a></iframe><p>We’re doing a little hack here by exploiting the fact that most of mail fields are either an array or a string, and both have include? method.</p><p>And now it’s time for the most difficult part. We want to create stubs and setup expects on them using syntax like this one:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0a23ce7a97b455f29bf78a9fd8a3630d/href">https://medium.com/media/0a23ce7a97b455f29bf78a9fd8a3630d/href</a></iframe><p>This time we will make it a bit different than with Mail and won’t use method_missing to decide when we’re done stubbing. What we’ll do instead, is have two methods: .success and .failure (not shown in the example above) that will be responsible for choosing an appropriate response and going back to ApiRequest we were working on. This way we can use method_missing to read parameters passed using .with. But we’ll get there one step at a time. Let’s start with adding required methods and classes. We already know how to do this for the .request method:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c75db2b04fe2d432654a57b57ca20ce8/href">https://medium.com/media/c75db2b04fe2d432654a57b57ca20ce8/href</a></iframe><p>We also need to create ApiRequest::Stub:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aabd46c44812c38bd42a80895fc10139/href">https://medium.com/media/aabd46c44812c38bd42a80895fc10139/href</a></iframe><p>Now, inside ApiRequest::Stub we can add request specific methods such as .facebook, .twitter etc.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a21c93e61c045147d33ec2a7c2244f64/href">https://medium.com/media/a21c93e61c045147d33ec2a7c2244f64/href</a></iframe><p>To keep everything nice and clean those methods should return some kind of specialised class. All those classes are sure to have something in common (trust me, I’ve been through this 😅), so we will also create a superclass for them.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2d7006cca1b3bb1ee87aa1b27ccd6b9f/href">https://medium.com/media/2d7006cca1b3bb1ee87aa1b27ccd6b9f/href</a></iframe><p>We’re passing self to those classes because we need a way to go back there later. The next method in stub creation is .with. We can pass parameters there and they will be used inside our stub. Keep in mind, that right now we’ll be working on stubber object, so we add this method to our AbstractStubber like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4f0d73e942da4b354290d7de447a7461/href">https://medium.com/media/4f0d73e942da4b354290d7de447a7461/href</a></iframe><p>Take a look at this code. We’ve added .with method as a way to store parameters for later, and we’ll be using method_missing to retrieve them later. Why? We’re about to find out now, as we implement handling of specific request types. To properly stub a request we need to have its URL, request body and something to return later:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c1a44c004175e625920b99d35f739da9/href">https://medium.com/media/c1a44c004175e625920b99d35f739da9/href</a></iframe><p>Do you see it now? If we want to have parametrized requests, we can just call a method named like a parameter and our method_missing will handle the rest of it. This way we make adding new stubs much easier — we just have to specify its address and write some response. This also makes sure that people stub requests consciously, as those parameters are required (take a look at the method_missing implementation — if the parameter is not found, we will raise an error).</p><p>If you ever need to stub a request to a different Twitter API endpoint you’d only have to define one new method here and use it in tests. It’s actually that easy.</p><p>And now that we’re done setting up our stub, there is one last method call we make. We can make this request be a .success or a .failure. Those two methods will be responsible for preparing final Stub that we’ll use in our tests. They will also handle going back to our original request object.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/73e1d8cfbf3691fe4b9006edaf85def2/href">https://medium.com/media/73e1d8cfbf3691fe4b9006edaf85def2/href</a></iframe><p>We’ve added all stuff required for stubbing with webmock. We still have to implement .finish_stubbing method on ApiRequest::Stub:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/681ad838d38d5d4321f3ab7a1b65a850/href">https://medium.com/media/681ad838d38d5d4321f3ab7a1b65a850/href</a></iframe><p>Now we’re ready to use those stubs in our shared example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f5fcf9f6804b6da33430d9471596f05a/href">https://medium.com/media/f5fcf9f6804b6da33430d9471596f05a/href</a></iframe><p>And we’re pretty much done here. There are just a few small things we can add to make our ApiRequest even more fun.</p><h3>Finishing touches</h3><p>The gem we use for generating documentation allows you to make a few requests within the single example, but our implementation is limited to just one. To get around this we can accept an array of ApiRequests instead of just one instance. To keep compatibility with existing code, we will wrap our subject in array (it won’t do anything if it’s already an array):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/635dd8a36c1660cdbc1c69b3f4f2d286/href">https://medium.com/media/635dd8a36c1660cdbc1c69b3f4f2d286/href</a></iframe><p>Ok, so we can execute many requests in one example, but it’s not very usable yet. We have to add a way to easily override some parameters. Let’s add one last method to our ApiRequest class:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0a815f9c38ceec526da61951359d6a62/href">https://medium.com/media/0a815f9c38ceec526da61951359d6a62/href</a></iframe><p>Now when making a request we pass those parameters and they will neatly override default ones:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b11750a4eb66686d46905200cb71ad09/href">https://medium.com/media/b11750a4eb66686d46905200cb71ad09/href</a></iframe><p>With this in place, we can now do multiple requests in each example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/74010339dd2cbce15eb8ee07fe2ccc3c/href">https://medium.com/media/74010339dd2cbce15eb8ee07fe2ccc3c/href</a></iframe><p>And the best thing is that we instantly have them in our documentation (watch out, long image ahead 🙈)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*O19upzYhVnz5Xq7Kw4d6NQ.png" /><figcaption>002 is our validation message for body text shorter than 10 letters</figcaption></figure><p>And we’re finally done. We’ve just created an easy to use DSL which allows us to change this long and error-prone test:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fb0c73fe676593158a2928d38aed3262/href">https://medium.com/media/fb0c73fe676593158a2928d38aed3262/href</a></iframe><p>Into a hopefully more readable and easier to use form:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4423dbe5f4a92d2281c07c8264678f0d/href">https://medium.com/media/4423dbe5f4a92d2281c07c8264678f0d/href</a></iframe><p>It’s now faster to use ApiRequest than to write tests manually, so we can easily convince the rest of our team to use it for acceptance testing. As a result, we get tests that can be trusted, and a documentation for our API. Goal achieved!</p><h3>Final words</h3><p>With the help of ApiRequest class, we can write tests for new endpoints in a matter of minutes. It’s easy to specify both business requirements and the rough idea of a feature using this method, so further development gets easier as well. But keep in mind that those are only acceptance tests. You should still unit test your code to catch any implementation errors.</p><p>To show you how to use this method in a real application, I’ve prepared a GitHub repository with a complete (but very basic) use case. Go ahead, <strong>clone it and try it out yourself:</strong></p><blockquote><a href="https://github.com/glowacki-dev/testifier">https://github.com/glowacki-dev/testifier</a></blockquote><p>That’s as far as this article goes. If you’re still here I guess that you liked this idea of making tests easy to write while keeping a high level of their usability. There is still a lot of things that can be done here. For example, we could partially generate endpoint descriptions based on stubbing we make. Or we could think of a way to extract some common logic to a gem to share with the World (you have my blessing if you want to do this 🙌).</p><p>If you have any questions or remarks regarding this approach (or anything else) feel free to ask them in the comments below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*KtnDoZhaAkXhmLy0-y_JoA.png" /></figure><p>If you enjoyed this post, please hit the clap button below 👏👏👏</p><p>You can also follow us on <a href="https://www.facebook.com/Daftcode/">Facebook</a>, <a href="https://twitter.com/Daftcode_pl">Twitter</a> and <a href="https://www.linkedin.com/company/daftcode-sp-z-o-o-">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d7e1fac6d16" width="1" height="1" alt=""><hr><p><a href="https://medium.com/daftcode-blog/how-we-made-writing-tests-fun-and-easy-2d7e1fac6d16">How We Made Writing Tests Fun and Easy</a> was originally published in <a href="https://medium.com/daftcode-blog">Daftcode Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fixing Unicode for Ruby Developers]]></title>
            <link>https://medium.com/daftcode-blog/fixing-unicode-for-ruby-developers-60d7f6377388?source=rss-aff6071fbf8------2</link>
            <guid isPermaLink="false">https://medium.com/p/60d7f6377388</guid>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[ruby]]></category>
            <category><![CDATA[unicode]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Maciej Głowacki]]></dc:creator>
            <pubDate>Wed, 10 May 2017 09:15:41 GMT</pubDate>
            <atom:updated>2018-03-02T10:47:51.746Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xzmTRco0RcFr5Yn47GST8Q.png" /><figcaption>Illustration by <a href="https://dribbble.com/SoniaBudner">Sonia Budner</a></figcaption></figure><p><strong>We’ve already learned about </strong><a href="https://blog.daftcode.pl/a-brief-introduction-to-unicode-for-everybody-bc26f5761548"><strong>the history of Unicode</strong></a><strong> and </strong><a href="https://blog.daftcode.pl/unicode-details-for-developers-300dc6f6e11e"><strong>how it works under the hood</strong></a><strong>. In this final part, we’ll learn the theory and practice of handling Unicode properly.</strong></p><p>Before we get into practice I’d like to talk a bit about Unicode support in Ruby language. If you’re not a Ruby developer, feel free to scroll past the first chapter.</p><h3><strong>Unicode + Ruby = </strong>😢</h3><p>This header suggests, that Ruby and Unicode don’t go well together. And it might seem surprising because after all Ruby allows you to write some declarations like this one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/969/1*wiOEleQ10jV-FUcspacs3g.png" /></figure><p>Which then can be used for method calls.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o59kXyGipzEi8RL-Xg0C2Q.png" /><figcaption>“From new moon up to waning crescent cycle five times and with each slice of five moons print those moons” — obvious</figcaption></figure><p>And it gives an expected output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GwMbY5XuBbUiEXXKQ_pbdA.png" /></figure><p>Looks like pretty good Unicode support, doesn’t it? After all, we’re writing code with emojis, which are very Unicode’y. But there is something more about supporting Unicode than being able to just write it.</p><h4><strong>The beginnings</strong></h4><p>As usual, let’s start with history. How about going back to the times when Ruby was not even in version 1.9? Have any of you ever used Ruby pre-1.9? Honestly, I haven’t, but from what the elders of the Internet have told me, Strings didn’t have any special powers there. They were just simple arrays of bytes. And an array of bytes doesn’t care about character encoding at all. You give it some data and it will hold this data for you, but ask it to do something with it… and things can get strange.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*MMzFXMk8jWoijin-p2I4QA.jpeg" /></figure><p>Let’s start with Ruby 1.8.7 (the latest one before 1.9) and check how Strings work there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XuAnW642krIMyioi57D4rA.png" /></figure><p>Nothing surprising here. We’ve created a String without any problems. But what if you’re a developer from Poland, and you wanted to use Polish variable names (please don’t ever do this) and Polish strings? Let’s call our variable “treść” (which means content) and assign local version of Hello World to it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4UomIi91cMdsUjPWNACe1w.png" /><figcaption>Seriously, don’t name variables this way</figcaption></figure><blockquote>(╯°□°)╯︵ ┻━┻</blockquote><p>We can see that Ruby didn’t even manage to read the name of our variable. Actually, it didn’t even manage to properly recover from this Error and just shut down the irb. It’s a pretty obvious sign that we shouldn’t expect much Unicode support here.</p><p>But just to be sure, let’s remove special characters from variable name and try again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c8uXN5Efx4-K_KY_7S0CFQ.png" /><figcaption>It didn’t blow up. That’s a progress</figcaption></figure><p>We can see that instead of the letter “Ś” (U+015A) we got “\305\232”. Those numbers are <a href="https://en.wikipedia.org/wiki/Octal">octal form</a> of UTF-8 representation of this letter (“U+015A in UTF-8 is C59A). No support for Unicode this time.</p><p>Well, actually there was a way to ask Ruby to use UTF-8 as the default encoding. Will it work for us?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4uKDA2HS-CwQz4UHhwv7XQ.png" /><figcaption>“żółw” is Polish for turtle 🐢, our spirit animal in this article</figcaption></figure><p>Woah, it looks like everything is working now. Can you guess how long is this obviously-4-letters-long-text?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LqdVSqNWAv4O4U1-_7P9Zg.png" /></figure><p>Obviously… 4 letters long text is 7 letters long. How about trying to make it all uppercase?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rAKHeK-KnVg43wd-0GAUrg.png" /><figcaption>Hey, it could have been even worse. Right? Right…?</figcaption></figure><blockquote>(╯°□°)╯︵ ┻━┻</blockquote><h4><strong>But there were changes</strong></h4><p>That Ruby version should be actually considered a prehistory. What followed next, brought quite a few good changes. Everybody welcome Ruby 1.9.</p><p>We saw it’s release on 25th Dec 2007 — exactly 11 years after Ruby 1.0. One of the biggest changes was the improved String class. From now on, it would support many different encodings and support UTF-8 as a default one. Are you as excited as I am to check it out?</p><p>Let’s fire up irb for Ruby 1.9.3 and check if its default encoding really is UTF-8.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dd80AANUS1XERqGWHhq17g.png" /></figure><p>My Ruby build was somehow broken and decided not to show version number in prompt. But it seems that it indeed supports UTF-8 and we can use it out of the box.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*ugjC-4Snyj4ykXZ5BIQmKg.jpeg" /></figure><p>Let’s do the turtle test once more. Can you guess how long will the turtle be this time?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pCotYqKPHdqVwegLJMJ-bA.png" /></figure><p>Wooohoo 👏🎉🍾 Our turtle is 4 characters long. But wait. There was a second test. Can we make the turtle bigger?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MM2vxW0OBV2d3QARL7PyYA.png" /><figcaption>Come on. It’s not that hard, is it?</figcaption></figure><blockquote>(╯°□°)╯︵ ┻━┻</blockquote><h4><strong>But it was all a long time ago</strong></h4><p>Ruby 1.9 is actually also a prehistory. After all, we’re using ruby 2.x right now. It’s not 1, it’s 2. The whole new version of a language. Let’s just open the prompt for Ruby 2.3.3, which I happened to have set as a default one.</p><p>Enough talking, let’s make a big turtle!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XdpGaqlb00SHH6atb8b8cw.png" /></figure><p>Actually, I’m not sure what I’ve expected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qObABRh1dkCwrXWHN1EhQw.jpeg" /></figure><h4><strong>But there are even more changes</strong></h4><p>As you may know, Ruby 2.3 is not the latest one. Not so long ago, on 25th Dec 2016, Ruby 2.4 had its release. Exactly 20 years after the first release, there was one important change:</p><p>“String supports Unicode case mappings”</p><p>Is this what we’ve all been waiting for? Can we finally make our turtle bigger?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TsLjUEHtCRoKgMKHhRB9Hw.png" /></figure><h3>Yes we can!</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NC3HZtCJo3z6S9BFZ0bg0w.png" /><figcaption>Let’s all take a moment to contemplate on this leap for humanity</figcaption></figure><p>We can all happily go home now. Our work is done for today. But in the very last moment, a new wild turtle appears.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JuTFDGtjEoyP9g2zMaNoBw.png" /></figure><p>It looks just like the previous one, but we have the feeling that something is not quite right with it. But just to be sure, let’s check it’s length. Can you guess how long can it be?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sl6nF4t6RGd6Ls_nmDjWHA.png" /><figcaption>No comments 😒</figcaption></figure><p>What happened here? If it was 7 characters long it would have been obvious. But it’s not. Let’s inspect what’s inside of it (sounds quite brutal when talking about turtles 😨).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*faBsyuUa8FC2S1ZSPTn4eQ.png" /></figure><p>For some reason, our “ż” and “ó” got separated into two characters. If you remember things I’ve talked about previously, you may recognise Combining Diacritical Marks here. But just to make things clear. In Unicode, there are two ways to write regional characters. The first one is to use precomposed letter, which is represented as one character. The other method is to use the “base” letter and follow it with a special combining character (such as dot, wave, slash etc.) which is displayed together with the preceding letter. But as we can see displaying is not the same as keeping in memory.</p><p>We can now verify whether or not Ruby handles such combined characters properly. Can we find a precomposed “ż” inside our “żółw”?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h0utcQCwJE4YPKV1yoMU6A.png" /><figcaption>Both forms of “ż” look exactly the same, so you have to believe me which one is which</figcaption></figure><p>To be honest there was no reason at all for this to work.</p><p>We’re almost ready to talk about ways of fixing this. There is only one more thing left for us to check. What if we tried to reverse our turtle? Or rather have it reversed and try to make it “right” again?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l1ILtkNJJ7nN5udQZgjVDA.png" /><figcaption>“włóż” means “to insert” in Polish, and there is a joke about reversing this word, but it makes no sense in English</figcaption></figure><p>We’ve just created a monster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*SUsrYsFJ5AYyLD1uYUCAtw.gif" /></figure><p>Look at how our “ó” lost its upper thing (there is definitely some smart name for it) and became a plain boring “o”. And our “ż” became a “ź”. Not to mention the opening quotation mark of our string having a dot above. A dot above a quotation mark… 🙃</p><p>The reason for this is that deep inside, Strings are still like arrays. If you reverse it, you get what you ask for.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gu5ab7lyM2JXxYJckJ2Dzw.png" /><figcaption>Read it backwards, it would totally compose into “żółw”</figcaption></figure><p>It’s high time we try to solve all those issues.</p><h3><strong>What can we do about it?</strong></h3><p>At this point, everyone skipping the Ruby part should join us. Hello 👋 And I’ve lied to you because the first solution I’ll show is Ruby only.</p><h4><strong>Let ActiveSupport help us</strong></h4><p>If you’re working with Ruby on Rails, you’d already have ActiveSupport installed. It extends String with mb_chars method. What this method does is wrap the string in “ActiveSupport::Multibyte::Chars” class. This class makes all String methods such as “reverse”, “upcase” etc. work as expected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h3mQ444rgfuQg1QbpG3sBA.png" /></figure><p>And indeed it does work. And just to make sure we’ve been reversing a string with composing characters let’s see all the characters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r6IrzPxEKI68nBrdP9KTnw.png" /></figure><p>As we can see, mb_chars made our string behave less like an array and more like an actual text.</p><p>But some of us would probably want to avoid using Ruby on Rails (or loading huge gems) just to work with text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*ywpi_b2MqLNPw6maxU5G0A.jpeg" /></figure><p>And that’s when we should talk about built-in ways of handling this.</p><h4><strong>Let’s normalize</strong></h4><p>Normalization is a way of turning many forms of data into a unified one. In the case of text strings, it means encoding them into one before-agreed form, called a normal form. The Ruby way of normalizing strings is “unicode_normalize” method. Other languages should have something similar (provided that they don’t do this by default).</p><p>To see it in action, first create two different turtles: one with precomposed letters, other using composable characters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BUtT8ES4X6y8oJFYlJKrWQ.png" /></figure><p>We can see that those two strings are indeed different. But when we normalize them before comparing…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wbp0yQXG2_8QcFCpU7gBcQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/275/1*mPUc2fU1VPbW6gjbw1DjeQ.gif" /></figure><p>To see what’s going on here, take a look at the string before and after normalization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1ZIxAHB7xEPN6tzSB3sR5A.png" /></figure><p>It looks like “unicode_normalize” changes all Unicode-magical characters into normal ones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*EY8V-uJa_2Nw8Oz2D2VBuw.png" /><figcaption>I’m not saying that composed characters aren’t normal or anything like this 😅</figcaption></figure><p>Sounds cool, right? Let’s do this in our production environment. We’re now ready for everything Unicode has for us. For example, someone sends us some text which looks like “a a a a…” but has all sorts of funky a-like characters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hDYNuFGmJ7v--QFgFyP5PA.png" /><figcaption>“a”, italic “a”, fancy “a”, encircled “a”, “a” slightly above… All those characters are written with the same font</figcaption></figure><p>But we’ve got nothing to worry about. We’re normalizing our strings, so it should all get changed into some normal “a”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tn8lPfZ8ZqQUQX5KLtYVcQ.png" /></figure><p>But it was not.</p><p>The reason for this is, that there are many different normal forms you can use. Unicode Standard describes 4 ways to normalize a string. To see all of them let’s use a string which has all kinds of funky Unicode stuff inside.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7LLA3wN8_pJAUxBqvzp2cg.png" /><figcaption>“¼ węża” means “one fourth of a snake” 🐍</figcaption></figure><p>We have a “¼” character which everybody knows has 3 different characters inside: “1”, “/”, and “4”. Then is a space, or rather a nonbreaking space which looks like a space but cannot be broken across lines. “w” is quite normal here, but then you have a Polish precomposed letter “ę” and “z” with a composable dot above. The “a” is once again just a boring letter.</p><p>The first (default) normal form is called NFC. NF stands for Normal Form, while C means Compose. You can probably guess how it works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JDr6dtFzEOwjJqJoxUTSAg.png" /></figure><p>All it does is taking letters with composable characters and turning it into one, precomposed letter. It composes “z” and a dot into “ż”.</p><p>The second normal form is NFD, with D meaning Decompose. This is opposite to what we just saw above and is usually used only for some specialised internal processing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tbLkR9G8XW-pnh-2-RTwHg.png" /></figure><p>As expected, it turned the precomposed “ę” into “e” and a composable hook which is called “<a href="https://en.wikipedia.org/wiki/Ogonek">ogonek</a>”.</p><p>The next form is the most common when turning strings into identifiers (such as database keys). It’s called NFKC, with K standing for Compatibility.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7qCQQiNxO5_-kzqfW3aCbw.png" /><figcaption>The C was already needed for Compose ¯\_(ツ)_/¯</figcaption></figure><p>Apart from doing the compose part, it additionally takes care of all those special characters such as “¼” and the nonbreaking space. It changes them into their compatible counterparts, which are “1”, “/”, “4” and a standard space.</p><p>As you might have guessed, the last form is NFKD.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*op9ttUrwHm5eF61tPG7WKw.png" /></figure><p>It’s like the previous one, but decomposes characters instead of composing them.</p><p>Equipped with all this knowledge we can now try another take on normalizing our “a a a a…”. It looks like the default NFC form is missing the Compatibility part, which we need here. Let’s check if NFKC gives us better results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uoDoB-o1w_i_rJ6HQbnTgA.png" /></figure><p>It gives us an expected string of standard letters “a”. But at this point, we probably have some trust issues as far as Unicode is concerned. We’d better make sure all those “a” are the same. Can we turn them into turtles?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jt6m7itDHkjyq2FZ38ql9A.png" /><figcaption>The ultimate method of proving things — turning them into turtles</figcaption></figure><h3><strong>Things to watch out for</strong></h3><p>Normalization is a great way to handle Unicode strings and turn them into more usable form. It solves many problems, but can also create new ones if used without special care.</p><p>One example of how Unicode can create security threats in your application <a href="https://labs.spotify.com/2013/06/18/creative-usernames/">happened to Spotify in 2013</a>. There was a way to takeover any user account using Unicode-related stuff.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*VFU2CoUa8HoGx8qdpLARew.png" /><figcaption>Spotify being all suspicious about Unicode</figcaption></figure><p>Spotify has supported Unicode in usernames for a long time (seriously, it’s possible to name your account something like “¼🐍🐢”). To handle this correctly they use two forms of your username. One is a verbatim form and is used purely for display purposes. The second one, called a canonical form, is used for all sorts of processing (comparing, checking the database etc). To turn a verbatim form into canonical one, it has to undergo a process called canonicalisation. And that’s where the problem came from.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*cHl8qv3V1CwGpaxCwqBCSw.jpeg" /><figcaption>Fun fact. In Polish, the name for canonicalisation is the same as canonisation — which is when the Pope declares that someone who has died was a saint</figcaption></figure><h4>Let’s make an app</h4><p>For the sake of this article we’ll simplify what happened to Spotify but will keep the main idea and an attack vector unchanged. Let’s see how easy it is to create a serious security issue.</p><p>Imagine you have an app, where usernames aren’t case sensitive. Your users are happy as they don’t have to remember whether they used capital letters or not. They can login as “GreatUser”, “greatuser”, “greatuSER” etc. and still get access to their account. To achieve this, you’ve created a very simple canonicalising method, which makes username all uppercase.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*xTQopP1zK4Fr34ihZme7dw.png" /><figcaption>Nothing too fancy here…</figcaption></figure><p>Now when you log in using “Admin” username to access all the secret admin stuff in your app, the username will be turned into uppercase “ADMIN”. It all works as expected.</p><p>But since we’re all modern and keeping up with latest trends, we allow users to use Unicode characters when logging in. It’s all working great until some evil hacker appears with the intention to mislead users on your message board (yes, we’re making a message board type application 😐). What they do is register as an “𝖠dmin” with some hacky magical Unicode “𝖠”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LqBCRDW6wAruuokV9cKU3g.png" /><figcaption>Those evil hackers always hacking and breaking stuff 😢 (╯°□°）╯︵ ┻━┻</figcaption></figure><p>And they can do this because this “𝖠” is not the same as “A”.</p><h4>Quick fix time</h4><p>We cannot allow for such thing! Luckily, we remember how we’ve learned about Unicode normalization some time ago, and decide that it’s the way to go. We may even remember that the best normal form for things like identifiers is NFKC. So we proceed and fix out canonicalisation method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UsHbmcEimgXCW6Bl63LADQ.png" /><figcaption>Whew… That was actually quite easy 😅 ┬──┬ ノ( ゜-゜ノ)</figcaption></figure><p>Hooray! Problem solved. Now, none of those evil hackers can login as “𝖠dmin” anymore.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YBKPK-IbVaryOyL19WunMg.png" /><figcaption>Epic win</figcaption></figure><h4>One important detail</h4><p>But there is one problem with our canonicalisation method. In real life, things such as canonicalisation may be performed a lot of times on the same text. You can do this as many times as you like thanks to method <a href="https://en.wikipedia.org/wiki/Idempotence">idempotence</a> (watch out, a wild hard word appeared). The idempotent method has a nice feature, that when you call it many times it gives the same result as when you call it only once. For example “upcase” is idempotent. You can’t make a text more uppercase even when you try it a million times.</p><p>On the first sight, we might think that our canonicalisation method is idempotent as well. After all, when we try “to_can” on those evil “𝖠dmin” thing, we get exactly the same result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GkcJvCj9MCRBzWUmzkt54Q.png" /></figure><p>But keep in mind that hackers are clever. And they come up with a new attack variation. Now, they try to use “ᵃdmin” as a login. And to our dismay, it’s not the same after two method calls as it is after one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wOyOCGLdQbrdEO93Km5JpQ.png" /><figcaption>Look at this cute little floating “a” ☺️</figcaption></figure><p>Does it mean that our canonicalisation method is not idempotent? Let’s check what’s happening here step by step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qFpmlnqbM-XT0PrMmcmZkw.png" /></figure><p>Now it should get obvious. Even though we may expect it to do so, calling “upcase” on this little “ᵃ” doesn’t turn it into “ᴬ”. So after uppercasing the username, we’re left with “ᵃDMIN” which gets normalised into “aDMIN”, which is not taken yet. The hacker can sign up with such username and it will be stored in the database as “aDMIN”.</p><p>Then, they request a password reset for this account. We extract this user from the database, allow the user to set a new password and save the new password to the database. And if at this very moment, just before saving, we canonicalise the username again (not our fault, we thought our method is idempotent). We’ve just changed a password for “ADMIN” user 😱</p><p>And it’s more or less what happened to Spotify.</p><h4>Fixing this for good</h4><p>Do you have an idea for fixing it? It’s actually really easy. Our problem is in “upcase” not handling all Unicode characters as expected. To get rid of this we should normalise the string <strong>before</strong> making it uppercase.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AKhiEZE1wYw7kmedJrvIQw.png" /><figcaption>Actually, normalising things is, pretty much always, the first thing you should do with them — never the last</figcaption></figure><h3>Parting words</h3><p>And that would be all for this wild 3-part Unicode adventure. Thanks for staying with me and I hope you’ve learned something useful here. Feel free to ask any questions, and I’ll try to do my best to help you out 🤓</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*MWL0f6uXpPKlySANxDaaoQ.jpeg" /><figcaption>You’re now officially ready to go on the Unicode adventure</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*KtnDoZhaAkXhmLy0-y_JoA.png" /></figure><p>If you enjoyed this post, please don’t forget to tap ❤! You can also follow us on <a href="https://www.facebook.com/Daftcode/">Facebook</a>, <a href="https://twitter.com/Daftcode_pl">Twitter</a> and <a href="https://www.linkedin.com/company/daftcode-sp-z-o-o-">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60d7f6377388" width="1" height="1" alt=""><hr><p><a href="https://medium.com/daftcode-blog/fixing-unicode-for-ruby-developers-60d7f6377388">Fixing Unicode for Ruby Developers</a> was originally published in <a href="https://medium.com/daftcode-blog">Daftcode Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unicode Details for Developers]]></title>
            <link>https://medium.com/daftcode-blog/unicode-details-for-developers-300dc6f6e11e?source=rss-aff6071fbf8------2</link>
            <guid isPermaLink="false">https://medium.com/p/300dc6f6e11e</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[how-to]]></category>
            <category><![CDATA[unicode]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[history]]></category>
            <dc:creator><![CDATA[Maciej Głowacki]]></dc:creator>
            <pubDate>Wed, 26 Apr 2017 13:48:23 GMT</pubDate>
            <atom:updated>2018-03-02T10:48:09.136Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F38Vi6sgWMHkDjW-YvRASw.png" /><figcaption>Illustration by <a href="https://dribbble.com/SoniaBudner">Sonia Budner</a></figcaption></figure><p><strong>Last time, we went </strong><a href="https://blog.daftcode.pl/brain-dead-simple-introduction-to-unicode-for-everybody-bc26f5761548"><strong>all the way from the Morse Code to the creation of Unicode</strong></a><strong>. And I’ve left you there with a question. How can you fit millions of characters into an 8-bit encoding? Wasn’t it exactly how that wild ride began — with short code encodings?</strong></p><p>As promised, we’ll now dive into the more technical part of the Unicode. I’ll try to keep it relatively simple, but it’s going to be more complex than the previous part.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RiQMP1xSfwsKjGtmKCUU2A.jpeg" /><figcaption>Since we’re talking about programming stuff, here is a photo of some hands typing on a computer keyboard</figcaption></figure><h3>Fitting more into less</h3><p>It isn’t completely right (nor is it wrong) to say UTF-8 is an 8 bit encoding because it’s a variable-length code. As the name suggests there won’t be a single length the code uses. In the case of UTF-8, you get code length varying from 8 do 32 bits.</p><p>But wasn’t this the problem with the Morse Code? If one letter uses longer code than the other, how can you know which one is which? It’s actually not that hard.</p><h4>Make it variable-length</h4><p>First, we have to remember that we’re operating on full bytes (which are 8-bits each), so we can only use code lengths of 8, 16, 24 and 32 bits. With this knowledge, we can sacrifice some bits for the purpose of storing code length in them. Let’s first take a look at the reference table, and I’ll follow with an explanation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SZ3_xoiT0LNm2u_w-g0LHw.png" /><figcaption>If it’s all clear to you, feel free to scroll all the way to the next chapter</figcaption></figure><p>Let’s focus on “Byte 1” column. We can see that depending on a number of bytes we use, the code begins differently. 4 bytes code starts with 4 ones, 3 bytes with 3 ones, 2 bytes with 2 ones, and 1 byte is different.</p><p>Anyway, as you might have guessed, the first byte tells us how many bytes should be read as one character. If you see something beginning with “1110…” you know that you need to read two more bytes to get a whole character. And as you can see those “next” bytes will all begin with “10…”. If you start reading a character and it begins with “10” then you instantly know something is not right. But you already know how it should begin, so you can skip a few bytes until you find one starting with anything other than “10…”. And that’s a great feature of this encoding, as it allows easy and fast processing and error recovery (you know, data get lost in the Internet sometimes).</p><h4>Sacrifice some bits</h4><p>However, you may be worried, that we’re losing some precious space by sacrificing bits of information for such things. And we are indeed. The fixed-length 32-bit code could theoretically encode 4 billion characters. But as we see in the table above, UTF-8 only gives you 21 bits for code point. Anyway, it’s still over 2 million characters, enough to meet Unicode standard (which defines a maximum of 1112064 characters). The limiting factor here is backwards compatibility with UTF-16, which is the oldest of Unicode encodings.</p><p>But even those 1,1 million characters is more than enough. Unicode 9.0 uses just 11% of all available space (128237 characters to be exact), and we’ve already encoded pretty much all available languages and scripts including Egyptian Hieroglyphs, Domino and Mahjong tiles, and of course Emojis — the language of the 21st century 👌😂</p><h4>Keep it compatible</h4><p>But why is one byte code different from the others? Why didn’t they do it all nice and satisfying?</p><p>The answer is backwards compatibility.</p><p>People creating UTF-8 wanted it to be as easy to use as possible. If most encodings were based on ASCII, then it would be only reasonable to make it ASCII-compatible as well. The way they did it makes all ASCII encoded texts also UTF-8 encoded. This made switching to Unicode much easier and is one of the great things about this encoding.</p><p>Enough with the theory. Let’s see it in practice.</p><h3>Let’s encode</h3><p>The good start to lay base ground for required calculations is trying to encode something from ASCII, because it’s supposed to fit into one byte.</p><p>For example, let’s take a letter “D”. If we check the ASCII table I’ve posted in the previous post, we can check that it’s code is 68, which is ox44 in base-16 (<a href="https://en.wikipedia.org/wiki/Hexadecimal">hexadecimal codes</a> are actually used more commonly, so we’ll stick with them). Now we should <a href="https://unicode-table.com/en/0044/">check Unicode table</a> to find out that “D” is U+0044. “U+” is a prefix for Unicode codes, and the rest is a hexadecimal value. If you know how to convert from base-16 into binary, you know that 0x44 = 01000100. Otherwise just believe my words.</p><p>So, we already know the code for “D” in ASCII. And if we look at the table I’ve posted above, we can see that U+0044 is lower than U+007F, so this letter fits into one byte. So the code for “D” in Unicode is also 01000100. Just as expected.</p><p>We can now do the same for letters “o” and “g” to encode the word “Dog”. It will be the same in both ASCII and Unicode: 01000100 01101111 01100111. Notice how each letter starts with a zero — this is an easy way to tell we’re dealing with ASCII here (which is equivalent to 1-bit UTF-8).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*u0Rc06DQOykAE0FZ5375EA.jpeg" /></figure><h4>Time for something harder</h4><p>But let’s be serious. It’s 2017 and nobody writes “Dog” anymore. Nowadays we use a dog emoji “🐶” instead. Emojis didn’t exist in ASCII so we go straight to Unicode table to find it’s code: U+1F436.</p><p>Looking at the conversion table, we’ll need all 4 bits to encode it, because U+1F436 is bigger than U+10000. After converting to binary we get 000011111010000110110. There are 4 zeros in the beginning to make it have a required length of 21 bits.</p><p>We can now change “x” in the conversion table into our binary digits and get an UTF-8 code for “🐶”: 1111<strong>0000</strong> 10<strong>011111</strong> 10<strong>010000</strong> 10<strong>110110</strong>. Numbers written in bold are those, we’ve calculated above.</p><p>As we’ve expected. No single byte starts with a zero, and all bytes except for the first begin with “10”.</p><h3>What’s so hard about it?</h3><p>If we look at what we’ve already learned it all looks relatively simple and well thought. After all, we’ve just manually encoded some data. Why would there be any problems with it? There are three reasons why.</p><h4>Being compatible</h4><p>The first one goes all the way back to 1960’s when the ASCII was created. Since it became the de-facto standard base for all future encodings, there was no reason for Unicode to break it. So it was created to be compatible with ASCII. And it was a nice little thing. But why limit compatibility to ASCII alone? We can try to make it overlap with as many encodings as possible. And it usually went quite well, but there were a few exceptions.</p><p>Do you know what character had a 0xB5 (decimal 181) code in standard Extended ASCII? It was µ — a symbol for micro. And do you know what is the 12th letter of the Greek alphabet? It is of course μ — the same character. Or is it?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rU8W0m8QCOjgOeUbUQqWmg.png" /><figcaption><a href="http://stackoverflow.com/questions/20674577/how-to-compare-unicode-characters-that-look-alike">This StackOverflow post</a> is an explaination by itself</figcaption></figure><p>You see, the first character (we’ll call it “micro”) was kept as U+00B5 to make Unicode and older encodings work together nicely. It looks the same, and actually comes from the greek letter μ (which is called “mu”), but it has a different meaning. Since Unicode doesn’t group characters by their appearance, but rather divides them into groups of the same origin and context. This means that there is a letter μ again under U+03BC, where all the greek letters are.</p><p>This is of course well defined in Unicode standard, and the micro character is marked as “compatibility”. It is recommended to use small greek μ instead, but try and tell it to people using your application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/1*-N4tfZyh6UVR76mXQEoUwA.png" /><figcaption>It looks like I’ve written all those posts around xkcd comics. But it’s not my fault there is always <a href="https://xkcd.com/1726/">a relevant one</a>.</figcaption></figure><h4>Supporting human languages</h4><p>The second one goes back about 8000 years back. Yes, it’s no mistake here. Human languages and writing systems themselves are probably one of the hardest things to standardise. There are thousands of languages, using hundreds of writing systems. Through history, many languages have diverged from others making them similar but not quite the same. Writing systems often evolved one from another making them share some symbols, but not all of them.</p><p>Time for an example. Let’s say you’re going to visit the Netherlands. You board a plane, and after a few hours, you arrive in Amsterdam. We all know its name begins with the letter “A” — the first letter of an alphabet. But when you look at the map you can find many other cities. For example, Astana which is the capital city of Kazakhstan. Or if you have a map which happens to be in Russian look for “Астана”. This first Cyrillic letter looks a lot like a letter “A”, doesn’t it? But we know it can’t be the same letter — after all, it’s from a language using a different alphabet. Or you may find yourself in sunny Athens, the capital of Greece. But if you’ve bought a Greek map, you’re looking for “Αθήνα”. And again, this first letter looks like “A”, but we know it’s a greek capital letter alpha.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MDgoZC0dKIFQsHPDPpLMUw.png" /><figcaption>Or you just have one map, with every city written using different alphabet</figcaption></figure><p>As you can see, we’ve got three letters that look like “A”, but only one of them is “A” as we mean it. The reason for this is that all those languages share a common origin, but have since separated. Across Unicode there are a lot of characters that look similar (or sometimes the same) but come from different writing systems. Unicode groups characters by their origin, not their appearance, so such look-alikes can occur.</p><p>This can pose a security threat, as those characters look the same to people, but can be treated as different ones by your application. Some evil people may try to make use of this for makings scams, phishing and all sorts of tricks to confuse unsuspecting Internet users.</p><p>Thankfully, Unicode maintains the <a href="http://www.unicode.org/Public/security/latest/confusables.txt">list called confusables</a> (beware, this is a huge text file). They even prepared <a href="http://unicode.org/cldr/utility/confusables.jsp">a tool</a> which helps you find similar looking glyphs (the default value of paypal is referring to <a href="https://en.wikipedia.org/wiki/PayPaI">the famous PayPaI scam</a>).</p><h4>Handling every letter possible</h4><p>The last problem lies within Unicode trying not to omit any existing letter. Do you know how many regional variants of the letter “a” are there? You can have an “a” with a dot ȧ, with two dots ä, with a small circle å, or with a small circle below ḁ. Maybe you feel like writing an “a” with a wave above ã, a wave in the middle a̴, or a wave below a̰. With Unicode you can even have all of it at once ḁ̴̰̇̈̊̃ (this letter is a good test of Unicode support on your device). Is there any language that uses such a monstrosity? No? Probably? Well, who knows. So just in case — if you need it you can write it.</p><p>Of course, nobody sat there thinking of every possible variant you can imagine. Instead, there is a group of Unicode characters called “Combining Diacritical Marks”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Li3YrYgso8LjhfRe9mLj0w.png" /><figcaption>These are all sorts of things you can do to a letter. They’re defined in <a href="http://unicode.org/charts/PDF/U0300.pdf">Unicode range 0300–036F</a></figcaption></figure><p>And it would be alright if it was the only way to input such letters. But many of such combinations really exist in different languages, so they are included in a precomposed form as well.</p><p>For example, there is a Polish letter “ą” (which sounds more like “o” than “a”). You can find it in Unicode table as “Latin Small Letter a with Ogonek” under U+0105. If you type it from a keyboard, you’ll get exactly this character. However, you can sometimes find this encoded as a letter “a” (U+0061) followed by a “Combining Ogonek” ◌̨ (U+0328). The result is exactly the same “ą” when you look at it, but it’s completely different in terms of encoding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UVENBb_R9kSQiyNvQmwwFw.png" /><figcaption>Yeah… Those two are definitely not the same thing</figcaption></figure><p>I’m sure your developer part of mind is already getting nervous imagining what could go wrong with it.</p><h3>Do things really go wrong?</h3><p>I think it’s enough talking. How about we just take a look at a very simple example written in C#? It’s a good language to test such things, as it’s been built with Unicode support from the very beginning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yaYjwdnAaUsaauGOBKPy1w.png" /><figcaption>Why do you think this happens?</figcaption></figure><p>As we can see, even in languages with deeply-rooted Unicode support, there are still some things that can surprise us.</p><p>What are the ways to avoid such issues? And what do you think would happen if we were using a language with a much shorter history of supporting this standard?</p><p>Stay with me until the next part. We’ll learn some good practices of working with Unicode and will go through history of Ruby in context of Unicode support.</p><h3>Read next:</h3><p><a href="https://blog.daftcode.pl/fixing-unicode-for-ruby-developers-60d7f6377388">Fixing Unicode for Ruby Developers</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*KtnDoZhaAkXhmLy0-y_JoA.png" /></figure><p>If you enjoyed this post, please don’t forget to tap ❤! You can also follow us on <a href="https://www.facebook.com/Daftcode/">Facebook</a>, <a href="https://twitter.com/Daftcode_pl">Twitter</a> and <a href="https://www.linkedin.com/company/daftcode-sp-z-o-o-">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=300dc6f6e11e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/daftcode-blog/unicode-details-for-developers-300dc6f6e11e">Unicode Details for Developers</a> was originally published in <a href="https://medium.com/daftcode-blog">Daftcode Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Brief Introduction to Unicode for Everybody]]></title>
            <link>https://medium.com/daftcode-blog/a-brief-introduction-to-unicode-for-everybody-bc26f5761548?source=rss-aff6071fbf8------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc26f5761548</guid>
            <category><![CDATA[history]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[unicode]]></category>
            <category><![CDATA[copywriting]]></category>
            <category><![CDATA[tech]]></category>
            <dc:creator><![CDATA[Maciej Głowacki]]></dc:creator>
            <pubDate>Wed, 19 Apr 2017 16:38:29 GMT</pubDate>
            <atom:updated>2018-03-02T10:48:21.307Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lNMzhC5Zz1QOKLgziW6u6Q.png" /><figcaption>Illustration by <a href="https://dribbble.com/SoniaBudner">Sonia Budner</a></figcaption></figure><p><strong>Have you heard of either Unicode or UTF? I can tell you that by reading this article you’re using UTF-8 right now (well, unless you have this text printed on paper). You may even have a subconscious feeling that this whole UTF is somehow related to the Unicode. Even if nothing rings a bell — don’t worry, I’ll explain everything soon. So what is this whole Unicode?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wmVZBsj5oe4DKRRrsZzNQA.jpeg" /><figcaption>Spoilers alert: we’ll be talking about letters, so here is a photo of a typewriter — a clever device for making letters apear on paper</figcaption></figure><h3>A look at the past</h3><p>To have a good understanding of what Unicode is and why it exists, we have to look back on PCs history. Actually we have to look even further than the first computer you’ve ever used or seen. Let’s begin in the telegraph era.</p><p>For the sake of simplicity, let’s say that the telegraph works by sending letters from one machine to another (quite like the Internet). And to do this, it uses electricity. But how do you “send a letter” using something as simple as an electric signal, which either is “ON” or “OFF”? That’s what the Morse Code was for.</p><h4>The very first encoding</h4><p>I’m sure you know what the Morse Code is. A series of dots and dashes, or beeps and boops, somehow making sense to people who know it.</p><pre>-.. .- ..-. - -.-. --- -.. .</pre><p>For example, if you know the Morse Code, you can read that the sequence above is “daftcode”. Otherwise you may be wondering what’s happening here. Well, the Morse Code is nothing more then a set of rules — or a code if you want:</p><pre>a = .-<br>b = -...<br>c = -.-.<br>d = -..<br>e = .</pre><pre>and so on</pre><p>With these rules, you can change a text written with letters into text written with symbols — a process called encoding. And the good thing about the Morse Code is that it works both ways. You use the same rules (but the other way around) for decoding.</p><p>Now, it get’s clear how to send the text with a telegraph. First, you use the Morse Code to encode your message. Then you “beep” it with short and long signals, waiting between letters. This way the person on the other end can decode it using the very same Morse Code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jaomGuIUq9bFDM6OVMn6fw.jpeg" /><figcaption>How do you show the Morse Code on a photo if not with a radio? (they didn’t actually do “beeps” using the microphone)</figcaption></figure><h4>Fast-forward 100 years</h4><p>We’re moving forward to the year 1960. But in the meantime, people have figured out how to encode telegraphs automatically. Now, all you need to communicate through electricity is a special device called the teleprinter. Think of it as a typewriter that can receive and type text sent by other people.</p><p>But it was hard to make it work using the Morse Code. You see, letters encoded with the Morse Code have a different length. For example, the letter “e” only uses 1 signal, while “b” needs 4 of them.</p><p>So people have invented a new code which makes all the letters the same length (5 in this case).</p><p>Now all the letters are the same length, so you don’ have to wait between them. Count the signals and you know when a new letter begins.</p><p>So we’re left with a constant stream of long and short signals. Instead of sending long signals let’s send nothing (an “OFF” state) for the same duration we send the short one. Now, we don’t need any kind of “silences” in-between. What we have is a stream of “ONs” and “OFFs”, or 1 and 0, or “bits” as they’re usually referred to.</p><p>And that’s more or less what have happened before 1960.</p><h3>Say hello to ASCII</h3><p>Here we are — the year 1960. People are using teleprinters, but are often dissatisfied with the limited available characters. At the time you could only send capital letters, numbers and some symbols. The problem is that there are only 5 bits available. Since a single bit can be either one or zero there are only 2⁵ = 32 possible characters you can send. Well, actually people use a clever trick (which is not important for this article) to send a little above 50 characters. But English alphabet has 26 letters and 10 digits, so there aren’t many characters left out of those 50.</p><p>That’s the reason why the <em>American Standard Code for Information Interchange,</em> better known as ASCII, got created. It used the same concept as previous encodings but was 7 bits long. 7 bits mean 2⁷ = 128 available characters. With so many characters available, they were able to fit both lowercase and capital letters, numbers, and all kind of special characters in there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DdgD00dAdXggzMdWDt7GSA.png" /><figcaption>Just look at all those brand-new letters available</figcaption></figure><p>It was a great standard and became a foundation of everything that was about to come.</p><h4>Going international</h4><p>Despite having such an extensive character range, there was one serious issue with ASCII. It was designed with English alphabet in mind. And what about German <em>ä</em>, <em>ë</em> and the rest of the umlauts? Or Polish <em>ź</em>, <em>ł</em>, <em>ę</em> and all the others? Or any other European language which definitely uses some special characters?</p><h4>That’s when ISO joins the game</h4><blockquote>By the way, did you know that ISO stands for <em>International Organization for Standardization</em>, which in no way seem to shorten to ISO — pretty ironic as far as dealing with standards go.</blockquote><p>Anyway, ISO issued a group of standards named ISO/IEC 646. These standards explained how to adapt ASCII to regional needs. It indicated 10 ASCII characters that can be replaced with special, regional characters. There were also two characters dedicated for currency symbols. Now you could have used your local currency instead of a “$”. But good luck trying to fit all 18 Polish letters in there. Or imagine you are a software developer writing in C, which was a prevailing programming language back then, and trying to write some code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*EKkzQRfYQIEZGmZAmeFm4g.png" /><figcaption>That’s how C code looks like. Quite a lot unsupported characters here…</figcaption></figure><p>The thing is your language standard doesn’t support any of the following characters:</p><pre># { } [ ]</pre><p>And it only got worse from here…</p><h3>Welcome to the 80s</h3><p>Do you know what happened in the eighties? Well, probably a lot of things, but one of them was 8-bit Personal Computers getting more and more popular. From now on we can call 8 bits a byte. Anyway, this one extra bit made PCs quite different from earlier 7-bit devices. And it was pretty convenient as far as text encoding goes. Every extra bit doubles the number of available characters. There were now 128 new empty spaces to fill. And computer manufacturers didn’t wait for any standard to arise. Each of them took initiative and invented their own encoding system. We saw (well, not me personally, but I’m sure some of us did) rise of encodings such as HP Roman, Mac OS Roman, and Code Page 437 — used by IBM and in DOS.</p><p>It desperately called for help.</p><h4>ISO once again</h4><p>The call for help was once again heard by the guys from ISO. And what they did was (most likely) say:</p><blockquote>Enough of this! Here is the one and ultimate standard for character encoding called ISO-8859–1.</blockquote><p>And it was great as long as you were using Western European languages. If you were more into Central European languages there was a standard for this as well, called ISO-8859–2. Actually, ISO-8859 was a whole group of standards all the way up to the number 16.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*9nMBMt-OugnruBr_M-WuEQ.png" /><figcaption>Some people say, there’s a relevant XKCD comic for everything. <a href="https://xkcd.com/927/">927 — Standards</a> is relevant here.</figcaption></figure><p>It looks like this whole thing with creating new standards was quite fun. And Microsoft didn’t want to miss on that, so they’ve joined the bandwagon as well. They created Windows-1250, which was almost like ISO-8859–2, but not quite. And Windows-1252, which resembled ISO-8859–1, but with some additions here and there. And it didn’t stop here. More and more encoding were created, often changing and remixing existing ones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*bZGJohPUoVYAD5Dn-zlF0g.gif" /><figcaption>That looks like fun. It’s probably why Microsoft joined the bandwagon.</figcaption></figure><p>But let’s stop for a moment and ask ourselves one important question: “why?”</p><p>I mean there is an encoding which supports western languages, there is also an encoding for eastern ones. Why would you create some strange hybrids which support a few characters from here, a few from there and throw in a few entirely new ones? Imagine you have an encoding which supports French letters. You also have another one which supports Greek alphabet. Using only one of them (you can’t use two encodings in one file) write a text in French about Greek language.</p><p>Now you see why.</p><h3>And the fun didn’t quite stop here</h3><p>Up to this point, we’ve been mostly working with languages using Latin letters with some regional modifications. But as you may know, there is a bunch of languages that don’t do this —people from China, Japan and Korea were all using computers as well. But how were they supposed to encode tens of thousands of Chinese characters with only 256 characters available in an 8-bit encoding?</p><p>They all agreed (that’s something new as far as encodings go) that it’s necessary to use 16 bits (otherwise known as 2 bytes) to encode all those characters. It might not look like a big difference, but keep in mind that instead od 2⁸ = 256 possibilities, you now get 2¹⁶=65536 possible characters. 65 thousand characters were just enough for encoding Asian languages. But don’t be fooled. Agreeing on code length didn’t mean agreeing on one encoding standard. The same encoding frenzy that was happening in Europe and the USA was happening here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*NYcYBQxkzCrU92FnHUhG8w.jpeg" /><figcaption>Can we all agree to never repeat a “war of standards” like that one?</figcaption></figure><h3>Unicode to the rescue</h3><p>It was at this moment people saw the ultimate solution to all their problems… Or did they?</p><p>To resolve all encoding issues, there was an idea to create one global, universal characters map. Unlike previous ISO encodings, it would cover all existing languages. This idea resulted in creation of the Universal Character Set (created as a norm called ISO/IEC 10646). But as you might have guessed, it was not the only implementation of such idea. Anyway, the first thing they did was create a Basic Multilingual Plane. It was a map of most common Chinese characters and letters for all other languages.</p><p>But they didn’t only create a list of all those characters. They also suggested a way to encode them, which they called UCS-2. The Chinese language needs most of the mapped characters out there and they did fine with 2 bytes. For this reason (among many others) UCS-2 also used 2 bytes — 16 bit code length (hence 2 in name). But it quickly turned out that Basic Multilingual Plane was actually quite incomplete.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*_fG-RuPYu4xRPA7maj1xDA.gif" /><figcaption>And it all started with sending electrical “beeps” two centuries ago</figcaption></figure><p>If 2 bytes are not enough for you to encode all the characters, what do you do? That’s right. You create a new encoding with a longer code. And so the idea for UCS-4 was born.</p><h4>Birth and rule of the Unicode</h4><p>To turn this idea into something real, IEEE suggested a character encoding which met UCS-4 requirements. But for some reasons, which are out of the scope of this article, their encoding was not accepted as a standard. That’s when the other player took the initiative.</p><p>This other player was Unicode. Unicode is an organization (Unicode Consortium) creating the Unicode Standard. Unicode Standard is more or less the same thing as Universal Character Set with some extra rules. Unicode had their own 16-bit encoding as well, and was creating a 32-bit one. It turned out, that Unicode Standard was easier to use than Universal Character Set. More and more companies were leaning towards the simpler alternative. At this point, ISO noticed what’s happening. Once again, many competing standards were starting to exist. To prevent this, ISO negotiated with Unicode. And miraculously, they both agreed on one common solution.</p><p>This way Unicode’s encodings, known as UTF (short for Unicode Transformation Format), became a standard way to encode texts in the modern world. There are a few variants like UTF-8, UTF-16 and UTF-32 (yup, those numbers are code length) which are all used nowadays. Each of them has its advantages and disadvantages, but all of them can support millions of characters, which should be enough for many years to come…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*-9VBNLgoY5udC3vCa3XqpQ.gif" /><figcaption><em>One Encoding to rule them all, One Encoding to find them, One Encoding to bring them all and in the darkness bind them all</em></figcaption></figure><p>That was a long way humanity had to come to decide on one universal way to send a text using electricity. I had to simplify a lot of things to make it fit in one article, but the main story remained intact. The Internet nowadays is a much happier place with almost 90% of all websites using a common standard for character encoding — UTF-8.</p><p>Woah, wait! But how do they fit millions of characters into an 8-bit UTF-8? Wasn’t 8-bit ASCII only supporting 256 characters?</p><p>Well, that’s a different story, which happens to be a second part of the one you’ve just read. We’ll get into technical details of Unicode there.</p><h3>Read next:</h3><p><a href="https://blog.daftcode.pl/unicode-details-for-developers-300dc6f6e11e">Unicode Details for Developers</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*KtnDoZhaAkXhmLy0-y_JoA.png" /></figure><p>If you enjoyed this post, please don’t forget to tap ❤! You can also follow us on <a href="https://www.facebook.com/Daftcode/">Facebook</a>, <a href="https://twitter.com/Daftcode_pl">Twitter</a> and <a href="https://www.linkedin.com/company/daftcode-sp-z-o-o-">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc26f5761548" width="1" height="1" alt=""><hr><p><a href="https://medium.com/daftcode-blog/a-brief-introduction-to-unicode-for-everybody-bc26f5761548">A Brief Introduction to Unicode for Everybody</a> was originally published in <a href="https://medium.com/daftcode-blog">Daftcode Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Make Uploading 10x Faster]]></title>
            <link>https://medium.com/daftcode-blog/how-to-make-uploading-10x-faster-f5b3f9cfcd52?source=rss-aff6071fbf8------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5b3f9cfcd52</guid>
            <category><![CDATA[front-end-development]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Maciej Głowacki]]></dc:creator>
            <pubDate>Wed, 25 Jan 2017 14:20:36 GMT</pubDate>
            <atom:updated>2017-01-27T08:47:02.900Z</atom:updated>
            <content:encoded><![CDATA[<h4>Do you feel like your web application could be responding a bit faster? We had the same feeling so we tried something new…</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6e9lkdUoQQEI_AEiNB9iQw.jpeg" /><figcaption><a href="https://www.flickr.com/photos/143325470@N04/31337888094/">“Highest level of abstracton”</a> by <a href="https://www.flickr.com/photos/143325470@N04/">Tomasz Urbaszek</a></figcaption></figure><p><strong>When we’ve noticed that our app was responding slowly, we tried the usual optimisations. But the effect wasn’t really satisfying. As it turned out, the slowest part of our application was uploading files. But is it even possible to make uploading faster?</strong></p><p>How about a brief introduction before we get our hands dirty? If you’re in a web applications business, you’ve probably faced some issues with your app not being responsive enough. The bottleneck is usually caused by one of the following:</p><ul><li>Poorly optimised code</li><li>Servers and clients’ devices running slow</li><li>Bad network connection</li></ul><p>The first one, whether it’s a legacy server code, non-optimal database queries or some browser-blocking JavaScript, can usually be tracked and fixed with relative ease (easy as “I know what should be done”, not necessarily “it’s a piece of cake”).</p><p>What if your code is already top-notch quality, but your servers begin to have trouble serving all the people who use your application? It actually sounds like quite a good sign for you and your business. You’ll probably end up switching to more powerful servers to handle all the incoming traffic. Or maybe it’s just a high latency issue? Then a <a href="https://en.wikipedia.org/wiki/Content_delivery_network">Content Delivery Network</a> (CDN) might be what you need.</p><p>Client devices being too slow is usually a non-issue in case of most web applications, because almost any device capable of running a web browser should provide a decent experience to the user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J7Ve4wdRbQ14y5Iu-vMiVg.jpeg" /><figcaption>Every article about the Internet seem to have a photo of a router and network cables ¯\_(ツ)_/¯</figcaption></figure><p>We’re now left with only one possible cause: poor network connection. Thankfully download speeds are getting better, but you can still make your website load faster with assets minification and compression. Any modern web server is capable of serving <a href="https://en.wikipedia.org/wiki/Gzip">gzipped</a> files, which allows clients to download less data to see your website (and if you’re not gzipping yet, it’s probably a good time to consider doing so). But what if you want faster responses the other way around?</p><h4>Welcome to the Dark Grounds of Uploading</h4><p>What is so dark about it? For starters, if we were to compare download and upload speeds we will find out that a lot of people use asymmetric Internet connection (no source for this, but you’re also one of them, aren’t you?). And the asymmetry means upload speeds up to 10 times slower than download. So, while 60Mbps connection is not bad at all, 6Mbps upload sounds rather slow. This is actually not even 1MB/s.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/488/1*ywB514EK5yxiriS06Vs2nQ.gif" /><figcaption>This is what uploading 160MB file at 8Mbps feels like. Just looking at it is frustrating, isn’t it? ლ(ಠ_ಠლ)</figcaption></figure><p>Moreover, unlike servers, browsers have no native way to compress data before uploading. So here we are, stuck with slower network speeds and bigger files, or… are we?</p><p>We came up with a simple idea to solve this problem. While we can’t force people to get a faster connection, we can try to send less data — the same way we do it on servers. But how can we compress data in a browser before uploading?</p><h3>Let’s Meet Pako</h3><p>What is <a href="https://github.com/nodeca/pako">pako</a>? According to the description, it’s a <em>zlib port to javascript, very fast!</em> In other words we’ve just equipped browsers with a power of gzipping data — this allows us to turn an idea into a working solution. And using it is almost a no-brainer. For example, before uploading a file, you would compress it with:</p><pre><strong>var</strong> compressed_file = pako.deflate(input_file);</pre><p>That was easy, wasn’t it? But let’s stop for a second and talk about when you should actually do this.</p><h4>We Can’t Compress Everything</h4><p>OK, we can but sometimes it just doesn’t make sense. For example, you won’t make images or videos any smaller with gzip (at least most of them — standard formats like jpg or mpeg are already well compressed).</p><p>Text data, on the other side, compresses really well. Depending on a file, we may actually expect up to 90% compression ratio. And it’s not only plain text that can be packed that well. In our case it was 3D related files (obj, stl, etc). Maybe you’ve got some Excel spreadsheets or huge Photoshop files waiting to be uploaded? Go ahead, and try to compress them first. You might be surprised by how much space this can save you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o3gpTuPAS94W6BUShiemrw.png" /><figcaption>First row are uncompressed files, second row are the same files after compression.</figcaption></figure><h4>But Compression is Slow</h4><p>And so is uploading files. Remember how we’ve talked about asymmetric networks? With upload speeds that slow, there should be a point where compression time is considerably shorter than uploading. And it’s not really hard to reach this point. For most well-compressable files, you’re already there. Compressing and uploading smaller file will be faster than uploading uncompressed, bigger data.</p><p>But to make sure we don’t spend too long compressing, we will use the lowest compression level (1 out of 9). It’s the fastest one but still reduces file size significantly. In most cases you won’t gain much with higher levels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*TCA9LpzIn40-MLKaCmOMcg.png" /></figure><p>As you can see we only get 2–3% better compression ratio with much, much longer compression times. (At this point I should either show you a chart with time/size comparison or, better still, you may check it yourself.)</p><p>To change compression level in pako, just pass a level option like this:</p><pre><strong>var</strong> compressed_file = pako.deflate(input_file, {level: 1});</pre><p>Now, that we know how to do this, let’s check the effect of the above actions.</p><h3>Did We Really Make Uploading Faster?</h3><p>Well, yes we did.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/488/1*NPfa9AX8JmynlkPQAtK-2Q.gif" /><figcaption>The left file is sent uncompressed, while the right one gets compressed before uploading. The choice was obvious for us.</figcaption></figure><p>After testing this method for a while, we’ve decided to use it in our app. In our case, we were uploading some 3D model data described with <a href="https://en.wikipedia.org/wiki/JSON">JSON</a>. Before compressing, they sometimes grew to 5–6MB. For those files we were able to achieve 90% compression ratio, which means uploading just 0.6MB instead of 6MB. Now, with 6Mbps upload speed, client will only wait 0.8s for the upload to complete instead of 8. Compressing these files takes below 0.1s, so we’ve achieved <strong>close to 10x speedup</strong>.</p><p>But you don’t have to believe me. In the end I’m just some random guy from the Internet.</p><h4>Why Don’t You See for Yourself?</h4><p>I’ve prepared a <a href="http://codepen.io/Bombasarkadian/full/RKgXQx/">small demo app</a> that you can use to see whether or not this might be usable in your case. Just select upload speed and add some files. The demo will take care of compressing it and simulating upload speed. Don’t worry — it’s not really sending your files anywhere, it’s just moving the progress bars so you can compare time required to send uncompressed and compressed data.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcodepen.io%2FBombasarkadian%2Fembed%2Fpreview%2FRKgXQx%3Fheight%3D600%26amp%3Bslug-hash%3DRKgXQx%26amp%3Bdefault-tabs%3Djs%2Cresult%26amp%3Bhost%3Dhttp%253A%252F%252Fcodepen.io%26amp%3Bembed-version%3D2&amp;url=https%3A%2F%2Fcodepen.io%2FBombasarkadian%2Fpen%2FRKgXQx&amp;image=https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fi.cdpn.io%2F747171.RKgXQx.small.d766b61e-a05c-4c9b-8335-de179a0b5ac3.png&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=codepen" width="800" height="600" frameborder="0" scrolling="no"><a href="https://medium.com/media/905019fb36149e1776644475b653a1d7/href">https://medium.com/media/905019fb36149e1776644475b653a1d7/href</a></iframe><h4>What Next?</h4><p>There are a few more things to keep in mind. The most important is to remember, that your server now receives compressed data, so if you need to work with it, you’ll have to unpack it first. Unpacking is still way faster than packing, so it shouldn’t be an issue — especially considering how much extra time we’ve gained.</p><p>Another thing to keep in mind is that compression is not instantaneous. It may take a few seconds for bigger files (for some huge 450MB files, you may need to wait about 30 seconds). Be nice to browsers. To keep your website from freezing, either process files in chunks (pako has a support for this), or try throwing some <a href="https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API">Web Workers</a> in there.</p><p>Well, that’s simply it. What use do you see for this solution in your apps?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*KtnDoZhaAkXhmLy0-y_JoA.png" /></figure><p>If you enjoyed this post, please don’t forget to tap ❤! You can also follow us on <a href="https://www.facebook.com/Daftcode/">Facebook</a>, <a href="https://www.linkedin.com/company/daftcode-sp-z-o-o-">LinkedIn</a> and <a href="https://twitter.com/Daftcode_pl">Twitter</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5b3f9cfcd52" width="1" height="1" alt=""><hr><p><a href="https://medium.com/daftcode-blog/how-to-make-uploading-10x-faster-f5b3f9cfcd52">How to Make Uploading 10x Faster</a> was originally published in <a href="https://medium.com/daftcode-blog">Daftcode Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>