<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[hh.ru - Medium]]></title>
        <description><![CDATA[Крупнейшая онлайн-рекрутинговая платформа в России и один из лидеров разработки HR-tech решений и сервисов. Истории о нашей разработке от наших участников. - Medium]]></description>
        <link>https://medium.com/hh-ru?source=rss----d470d2ec65f1---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>hh.ru - Medium</title>
            <link>https://medium.com/hh-ru?source=rss----d470d2ec65f1---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 19:26:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/hh-ru" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Quality assurance for mobile development at hh.ru]]></title>
            <link>https://medium.com/hh-ru/quality-assurance-for-mobile-development-at-hh-ru-b9683e63414f?source=rss----d470d2ec65f1---4</link>
            <guid isPermaLink="false">https://medium.com/p/b9683e63414f</guid>
            <dc:creator><![CDATA[Dan Kheyfets]]></dc:creator>
            <pubDate>Tue, 23 Aug 2022 10:26:39 GMT</pubDate>
            <atom:updated>2022-08-23T10:26:39.067Z</atom:updated>
            <content:encoded><![CDATA[<p>I don’t know what developers talk about at conferences, but QA engineers talk only about processes. How testing is organised, how many autotests there are, who and when writes them, where they are run, how quality is ensured at all stages of development? My article today is just about this — about how we build quality testing in hh. There will be a little bit of theory and a lot of practice. Let’s get going!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ms7vOYNzKwZvV2REFsoRrg.jpeg" /></figure><p><strong>Who they are: a tester and QA</strong></p><p>First of all, it is worth postulating the following: quality in hh is not the area of responsibility of the tester alone. Quality is everyone’s responsibility: testers, developers, designers, and product managers. We try to deal with this at each stage of product development. We want to begin with the ancient debate about testing. Now, we can distinguish two terms: testing and quality assurance, which are often used to refer to the same thing.</p><p><strong>Testing</strong> is a verification of the correspondence between the real and expected software behaviour. <strong>Quality Assurance</strong>, aka QA, which is an acronym that is often used, is a preventive process aimed at ensuring that all necessary techniques, procedures, standards and methodologies are followed in the product development process and provide a fault-free result.</p><p>If you don’t want to be too nerdy, it would be easier to say this: testing is one of the quality assurance stages. At hh, testers are quality assurance engineers; they influence the entire development process, not just looking for bugs. Further, in the article, I will use the terms “tester” and “QA” as synonyms in order to avoid unnecessary nerdiness.</p><p>So we came to the conclusion that QA is not just about testing. If we’re talking about QA engineers, their job is not just to test all the buttons in a new feature and test them again before release to make sure nothing falls off. QA is involved in all stages of product development. Our goal is to release an excellent quality product in a reasonable amount of time. We try to ensure quality at every stage of development. We monitor the quality of code, interfaces, design, and the final quality of the entire resulting product. QA can improve any of these processes.</p><h3>Feature development stages</h3><p>Let me briefly remind you how we write code: what we had three years ago and what we are up to now. Two or three years ago, we were working on so-called trunk-based development. We weren’t happy with that, and over time we switched to GitHub Flow. I already told you about how we improved the whole process and how the transition was made in a previous <a href="https://medium.com/hh-ru/how-to-bring-off-a-gentlemans-release-9884a954e4a7">article</a>, so I won’t dwell on that now. Let’s move on to how the entire feature development process is organised.</p><p>The entire process can be divided into several stages:</p><ul><li>Idea;</li><li>Demo-design;</li><li>Technical elaboration, decomposition, and evaluation of tasks;</li><li>Development;</li><li>Testing;</li><li>Final release — putting a new feature into production.</li></ul><p>And now let’s dwell on each of the product development stages.</p><h4>Feature idea</h4><p>At the idea stage, product people generate a hypothesis, come up with a new feature or functionality, discuss the idea with the designer, and together they develop the initial layout. At this stage, ideas are never too rigidly fixed. There is always an opportunity to change something, add more detail to layouts, and think through user scenarios more thoroughly. We don’t concrete all the requirements at this stage, so that we can discuss them with the team later, and someone will suggest something else, maybe. Or we’ll realise that in this form the feature won’t work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u6y0nYdnozTMoLkPPxsP1g.jpeg" /></figure><h4>Design demo</h4><p>When the idea stage is over, it is followed by the design demo. At this stage, the whole team is directly involved — two iOS developers, two Android developers, and a QA engineer. This is how a typical dreamteam in the mobile direction works, and we have five of them. Three of them in the job seeker branch, one in the employer branch, and one in the platform branch. Other than that, the hh team is not just a batch of programmers, but a set of different competency areas, each of which is important. Each developer isn’t just cool at writing code, they can also be great at design systems or writing a network layer.</p><p>At the design demo stage, the entire team and sometimes other interested parties take part. These are product managers from other areas, testers, and so on. The testers definitely participate at the stage of presenting the layout. This is because they usually know this product better than anyone else in the team. They can highlight weak points, ill-conceived scenarios, and interactions of the new feature with existing ones.</p><p>Our feature development process is always iterative: if something goes wrong after the first discussion of a feature, the designer and product manager simply go back to work on it. They may rework the layout a bit or rethink the user scenario a bit. After that, we meet again to discuss the new layout. If everyone is happy with it, then the layout goes directly into development. If not, we can redo it and discuss it again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jvx5OQnD4ADYAY_FcqqIGg.jpeg" /></figure><p>Engaging testers and the entire development team at the earliest stages is very beneficial for the process and the business as a whole. There is a simple rule: the earlier a fault or a problem is detected, the cheaper it will be to fix it. If the fault is detected at the stage of layouts in Figma, or within an idea at all, then it costs nothing to simply redraw or rethink it. But if the bug is detected directly at release or even after development, it is much more expensive to fix it.</p><p>Of course, in the era of cascade development, it was much harder to fix a bug in production. When releases were once a year and there were no online updates, if you had already released a bug in the production, it was very difficult to fix it there without problems. In the era of iterative development and agile approaches, where we do releases on the site several times a day and mobile app releases once a week, it’s much easier to fix a bug from the production.</p><p>But the problems still remain. If the bug got in the production and was in the paid functionality, which because of the bug does not work, then we need at least return the money to the users who paid for it. And in the worst case, we can also face troubles and unpleasant reputational risks.</p><h4>Technical elaboration, decomposition, and evaluation</h4><p>The next stage is the technical elaboration of the feature, its decomposition and evaluation. Developers and technical specialists, if necessary, take part in this stage. The participation of QA engineers at this stage is mandatory. At this stage developers can discuss the implementation of a particular feature: what the problems can occur, what will have to be changed in the existing functionality, and maybe even something will have to be refactored.</p><p>The tester listens carefully to what developers are saying, discusses, and can already plan his testing around potential problems. For example, if they hear that in order to develop a feature they will have to fix some element of the design system, they will probably also have to fix it in other areas where it is invoked.</p><p>At this stage, the tester can already estimate approximately how much time he will need for testing. We never strictly record this estimate from the tester, but we need it for further planning of the team’s workload.</p><h4>Development</h4><p>The next stage is development itself. In many companies, at this point, testers leave to write some test documentation, test cases, or to update something. We, on the other hand, write autotests intensively at this moment. But not for the new functionality, but for the one that already exists, has undergone some A/B-experiments and so on. There is always something that is not covered by tests and urgently needs them. We may also have unstable tests or tests that need to be updated. These can be dealt with.</p><p>But we don’t always wait for the full development cycle of a particular feature. The tester can take over a build somewhere in the middle of the process, if there is a partially working functionality ready, and see how it is implemented. After all, even at this stage we may already encounter problems, and the sooner we notice them, the better.</p><h4>Testing</h4><p>As you can easily guess, at this stage the testing of the developed feature takes place. The main way of testing in hh is exploratory testing. This is a type of testing that does not require writing test cases, but implies that each subsequent test is selected based on the previous ones.</p><p>This does not mean that the tester randomly presses buttons and sees what happens. Unfortunately, no. This method requires a great deal of tester’s experience and a very good knowledge of the product. The main advantages of exploratory testing that we see for ourselves are:</p><ul><li>Doesn’t require support and writing test cases;</li><li>We are not subject to the so-called pesticide paradox;</li><li>Increased testing speed.</li></ul><p>What is the pesticide paradox? This analogy was introduced by Boris Beizer in 1983. He gave the example of treating fields against pests. That is, if you treat fields with a certain pesticide, not all insects may die. Conventionally, 80% of pests will die, but 20% will stay. And if you re-treat these 20% of insects with the same pesticide, they are likely to be immune to this poison.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9o-tgKdU4XJWjXYiKXQFmw.jpeg" /></figure><p>It’s the same with testing. Repeated use of the same test cases that are not regularly reviewed or updated simply won’t be effective for existing bugs. After all, within the scenario described in the test case, you have most likely already found all the bugs.</p><p>But if you take a small step sideways, you may find that there are other bugs lurking there as well. This is why it is so important to regularly update your test documentation. But since we don’t have it at hh, and everything is tied to exploratory testing, the pesticide paradox in our work is quite rare.</p><p>And due to the fact that we do not write and regularly update test documentation, our testing speed is increasing slightly, but still.</p><p>We use Jira to introduce new bugs. In the bug ticket, we enter everything necessary for the bug to be easily reproduced and localized as much as possible. This can be logs, screenshots, videos, test environment, etc. Of course, a precondition, steps of replaying, expected and actual results must be present there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dHmY5-Qhh6lXJJLNDFSUsA.png" /></figure><h4>Our testing tools</h4><p>One of our main tools is sniffer traffic. They can be different. They can be Charles, Fiddler, Mitmproxy, and so on. We use Charles, simply because we are more comfortable with it. It’s not better or worse than other tools.</p><p>First of all, we use it to view the network requests from the application and the responses to them. And also to modify those requests and responses. Sometimes instead of making some unusual request from the application, it’s easier to replace it manually. Besides, we use different features like Throttling or proxying to testbenches. In general, anything that can help us in testing.</p><p>The next important tool is Postman. It’s a bit similar to the previous tool, and we use it to send network requests to the API and get responses. It’s not always the case when the interface is implemented, but we want to see some request right away and see what the API will return for it. That’s exactly what we use Postman for. In addition, we use Postman to send pushes to our test builds if we need to check some particularly custom one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MO3so3iTExd4bW_SZmQGew.jpeg" /></figure><p>We also use test benches. This is a complete copy of the production, but with test data. The test bench has all the services that are currently in the production, they are always up to date, if you update the bench regularly. The tester can switch different services on or off, generate any test data, or even break the bench completely. But that’s okay, in that case they will just create it anew.</p><p>In addition, the bench has test data generation via fixtures. Basically, it’s a wrapper over the API that allows us to create any test data we need. It’s much easier than making them through the interface.</p><p>It is also worth mentioning the debug panel. The debug panel is a part of the application which is only available in test builds. With it testers can enable certain features, generate test data, see logs and many other useful things. If you are in a large company, and you do not have a debug panel yet, it is likely that after a while testers will come to you with a similar request. Because the debug panel really speeds up testing. And if testers don’t come to you with such requests, most likely, they don’t even know that it is possible. So it’s worthwhile to show them the value of this tool.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cxCbMUO0cKUtVtm7cyQISg.jpeg" /><figcaption>We had to divide a long picture into two</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DQ0ZuHGeQ9ZZbukgDSd4_w.jpeg" /><figcaption>This is the second one</figcaption></figure><p>Other useful tools are adb and Android studio or Xcode for iOS development. I think these tools don’t need much of an introduction, so let’s move on.</p><p>A fact that worries a lot of people is that we have no detailed documentation on the product in hh. The feature description is just a layout and the product manager’s text about the main idea of this feature: what the analytics should be and everything in that vein. We also don’t have detailed test plans, test cases or test scenarios. The only test documentation we have is a checklist. It lists all the features that we have in our application so that when we deal with regress, the tester doesn’t forget what we have and check everything.</p><p>Yes, this sometimes causes some difficulties. We may change testers, someone may forget something. Sometimes there are questions — why does it work this way and not that way? But there are advantages here as well. If such questions arise, it means there’s something wrong here, and it’s worth discussing. After all, if everything works well, these kinds of questions are unlikely to arise.</p><h4>Here our powers are over</h4><p>QA in hh has the same powers as in all large companies with an advanced engineering culture. QA can block the release if they understand that there are some critical bugs, massive user complaints or crashes in the product. Also testers determine the severity of a particular bug. There are many factors for this: the functionality criticality, reputational risks, or its massiveness.</p><p>Besides, only QA has the right to merge a new feature into the develop branch. Of course, teamleaders also have this right, but they try not to use it just for no reason. This is done to ensure that nothing gets past the tester. Anything that has gone into the develop branch is guaranteed to be tested and reviewed by the tester.</p><p>QA is also involved in the process of releasing mobile apps into the stores. You might get the impression that testers are some reluctant egocentrists who think they’re the only ones who know the right thing to do. In fact, that’s not the case. Dialogue is very important in our company. You can always discuss the criticality of a bug with someone, be it a product manager or a developer. Perhaps some bug we can just accept, and accept the risks associated with it, and release it. But then we will definitely fix it.</p><p>A key quality of QA in our company is the ability to keep a balance between speed and quality. We try to work on a fine line when we can’t guarantee that there are no bugs in our prototype at all, that everything is perfect and nothing crashes. But we try to maintain a good speed of delivering the feature to the user with optimal quality. We teach this approach to all the new testers who come to work for us so that the tester is not an unnecessary headache in the entire development. However, we still try to maintain high quality standards in our application.</p><h4>Release</h4><p>The last stage in development is the release itself. As I said before, our testers are in charge of the releases. They create release branches themselves, make builds and put them in the store. Recently we’ve automated all this routine. How we did it, I told in one of our <a href="https://medium.com/hh-ru/automating-the-release-process-af8f97edc7bc">previous articles</a>.</p><p>All of our regress is kept to a minimum: we have very good UI-test coverage. And, accordingly, before releases, testers check only the functionality that is not covered by tests, or which is better to check once again. Before a new app is released, we put it into the beta channel on Google Play. The beta usually has a limited number of users who are warned that the functionality may be unstable and with some number of issues.</p><p>With a small number of users, we can make sure that there is nothing wrong with our application. It’s not always possible to find all crashes and bugs at the testing or development stage, and here you can find a hidden crash. If there is one and it’s massive enough, we fix it and only then take it to production.</p><p>Fixing such a crash is a rather nontrivial task. And it is not always easy to reproduce it. Very often developers just look at stacktrace, see what the problem might be, fix it, and the tester has no way to check it all manually. That’s why we just re-enter the beta channel and see what happens. If everything is fine in beta, then the next step is to get it to production. In production we never go 100% at once either. First we put out the build at 20%, then we go for 50%, 70%, and finally we get to 100%.</p><p>We never put 100% in a rollout for all users at once. Google Play has a peculiarity: if you roll out an application to 100%, then the build that crashed cannot be removed from there anymore. In other words, this APK will remain there forever until it is replaced by a fresh version of the application with the fixed bug. If you roll out the build only 99%, then in the case of a massive problem, such a build can be removed from Google Play. The remaining single percentage is usually not affected. We have never had any complaints that users didn’t get the update.</p><h4>The life after the release</h4><p>After release, we make sure to monitor what’s going on with the application. First of all, we look at crashes. We have two important indicators: crash-free according to users and according to sessions.</p><p>We use tools like Firebase Crashlytics or Appmetrica to do this. We try to adhere to the figure of 99.9% of users without crashes. We just decided to set a value for ourselves and keep to it. It is counted for a certain period for a certain version.</p><p>In addition to crashes, we make sure to monitor feedback about the new version. And we definitely interact with our technical support team. We may receive complaints from users about some bugs, the tech support team handles them and, if the bug repeats, sends it to the development team. We reproduce it, look into the problem and fix it.</p><p>Also, all of our functionality is run under A/B experiments. Thanks to this, we can always just turn off something that isn’t working. This is, in essence, a feature-toggle. If there’s an experiment, we turn it off, the developers fix it, and we release the whole thing in the next version. And only then we turn the experiment back on.</p><h3>Summary</h3><p>Let’s summarize briefly. We build quality at every stage of a feature’s development, be it design-demo, development, testing, or release. The earlier the problem is found, the easier and cheaper it is to be fixed.</p><p>To make it easier for us to ensure the quality of development, we actively use autotests. We have them completely native for both iOS and Android. For writing iOS autotests we use XCUITest, for Android we use Kaspresso. We use our self-written DSL to generate the test data on Android. You can read more about it in<a href="https://habr.com/ru/company/hh/blog/455042/"> our article on Habr</a> (rus).</p><p>A little bit of numbers about our autotests. In Android, we have about 400 UI tests. There are 500 of them in iOS. Basically all of our UI-tests are written by testers, but developers are not forbidden to write them, either. For example, if UI tests are broken during the development of a new functionality or the improvement of an old one, they are fixed by the developer.</p><p>We have an inverted testing pyramid. Our main focus is not on Unit tests, but on UI tests. Yes, UI tests are more unstable than Unit tests, they take longer, but at the same time we trust them more because they repeat all user scenarios in more detail.</p><p>In addition to UI tests, we have Unit tests. Android has about 1,800 of them, and iOS has almost 2,000. Most of the tests are run at night. All of the tests are run on the branches of the features being developed, as well as in develop. In the morning, developers can see what the state of their developed functionality is. We mainly focus on develop. If a test crashed there overnight, that’s the first place the developer goes to look what happened. Perhaps there is a bug there or an unstable test. In any case, this must be fixed.</p><p>We always strive to make sure that the tests in our develop are green. This is very important for the test’s credibility. If they constantly randomly crash just because they are unstable, then at some point developers just lose trust in such tests. That’s why we routinely analyze crashes, fix our unstable tests, and develop our test infrastructure.</p><p>That is all. Feel free to write about your testing process in the comments and ask any questions.</p><p>Stay tuned.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9683e63414f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hh-ru/quality-assurance-for-mobile-development-at-hh-ru-b9683e63414f">Quality assurance for mobile development at hh.ru</a> was originally published in <a href="https://medium.com/hh-ru">hh.ru</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stored data in iOS: save, update, use]]></title>
            <link>https://medium.com/hh-ru/stored-data-in-ios-save-update-use-d0b5bf50212c?source=rss----d470d2ec65f1---4</link>
            <guid isPermaLink="false">https://medium.com/p/d0b5bf50212c</guid>
            <dc:creator><![CDATA[Alex Tsybulko]]></dc:creator>
            <pubDate>Tue, 16 Aug 2022 08:03:22 GMT</pubDate>
            <atom:updated>2022-08-16T08:03:21.415Z</atom:updated>
            <content:encoded><![CDATA[<p>Hi there! My name is Alex, and I’m an iOS developer at hh.ru.</p><p>Countries, cities, professional fields, languages, currencies are the names of directories within our mobile app. The names are changed very rarely but used everywhere, therefore they must be up-to-date and users and developers should not waste time updating them. This article will show you how to work with directories simple and convenient.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zIrxHHJ7WSJRFJMjrsgFSQ.jpeg" /></figure><p><strong>Where we started</strong></p><p>We have been applying directories in our application for a very long time. Until recently, they were just ordinary text files where we saved responses from the server in JSON-format. There were about 20 MB of such data which is really a lot for a mobile application. Loading this data into the app was very slow, and the wrapper was written in Objective-C. In short, the horror and nightmare of legacy.</p><p>A year ago, we started working on a new application for employers. One of the challenges for us was to create a new feature for working with directories. In its current form we didn’t want to drag it into a fresh and brand-new application, so we decided to change the directories.</p><p>I contacted various developers, talked to them about preferences for the new feature, and made a list of requirements:</p><ul><li>data should be available as quickly as possible, right away from the application launch;</li><li>it is necessary to provide updates of directories from the server;</li><li>take into account that the structure of information can change. For example, new directories can be added, old links can be changed, so we need to provide for the possibility of migration;</li><li>provide a quick search for information in these directories;</li><li>the whole thing should work in a separate feature, which will be shared between our applications.</li></ul><p>After looking at these requirements, I realised that the question “how to store directory information” simply does not exist — we need a database. It will provide us with the storage of directory information, quick search, the ability to migrate and update.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AizbdNrB3Ck6yyGIxZN0AA.jpeg" /></figure><p>So most of the requirements we meet are practically “out of the box”. One thing remained to understand — what kind of base we wanted to use and how we would work with it.</p><p>Eventually, we ended up with several tasks:</p><ul><li>First, we need to learn how to deliver the data along with the application;</li><li>Second, we need to make it easy to work with the data inside our application;</li><li>Third, we need to make it so that the data is updated only when needed.</li></ul><p>In a perfect world, the developer doesn’t need to update data in the bundle manually. In a perfect world, all the necessary updates happen either on schedule on the CI side, or during the creation of a release candidate for regression. Some scripts are launched and a fresh version of the database is pushed to the repository. The validity of that database is ensured by integration tests. In general, we needed to automate the delivery of data with the application as much as possible and combine the generation of directories between iOS and Android.</p><p><strong>Databases and their precious fellows</strong></p><p>Taking the main requirements as a basis, I have collected the key options for working with the database in iOS. There were three options: CoreData, Realm, SQLite. In addition, I looked at less popular options like YAP, but quickly gave them up. Now I will briefly talk about the pros and cons of each of the main options.</p><p><strong>CoreData</strong>.</p><p><strong>Pros:</strong> it is as native as possible, adding a framework does not affect the size of the application, and most of our developers have already worked with it. Plus, it’s easy enough to write migrations.</p><p><strong>Cons:</strong> The database can be only generated on Mac or iOS. You can’t just go ahead and generate a SQLite file and swap it in. You have to be sure to use the model version from your project. CoreData is only suitable for iOS — no cross-platform. It also has a rather bulky API. Apple has improved it, but CoreData still loses a lot to other solutions in terms of usability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HMq59o-QgF6_XmnT_9Pk_g.jpeg" /></figure><p><strong>Realm</strong>.</p><p><strong>Pros: </strong>cross-platform, fast data search, simple and clear code in the project.</p><p><strong>Cons: </strong>To generate, you need a project with Realm connected — you can’t do without a script here. This is a third-party library and an additional dependency, which also increases the size of the application from 5 to 14 MB.</p><p>In addition, our Android developers did not want to bring Realm into our application. Accordingly, we lose the advantage of the cross-platform.</p><p><strong>SQLite</strong>.</p><p><strong>Pros:</strong> The main advantage is that the database is easy to generate. Just write scripts that create tables and insert data. It’s the ultimate cross-platform: the database engine in Android also uses SQLite.</p><p><strong>Cons: </strong>The minuses and other pluses depend on which library you choose to work with the database. I considered three options.</p><p>One of them is native, it has a big advantage in the form of no dependencies. But there is a big disadvantage: the API is inconvenient. Apple suggests using the OpaquePointer to work with the database.</p><p><strong>GRDB.swift.</strong></p><p><strong>Pros:</strong> easy to work with data, fast speed, excellent support of the author and the community. <strong>Cons:</strong> it has no support for Carthage, and it was a principled position of the library’s author. For us this was a stop factor, because we actively work with Carthage and all our main libraries are connected with it.</p><p><strong>And finally, SQLite.swift.</strong></p><p><strong>Pros:</strong> easy and convenient, there is community support. We already had this library in our project as a dependency on another library.</p><p><strong>Cons.</strong> At the time of research, the last commit was in the repository about two years ago. But before recording the <a href="https://www.youtube.com/watch?v=A2DJm0FuKCU">video version</a> of this article, I visited the library’s page, and it was updated quite recently. Another thing to note is that a separate manager is used for migrations, but that’s a common problem with working with SQLite.</p><p>For each of the five database options, I created a small test project that demonstrated the possibilities of interaction: how to work with models, how the database itself can be generated and updated. And then my team and I held a small vote to single out the absolute favorite. It is worth noting that we were solving a specific task of working with directories, rather than choosing a single database for our application.</p><p>We evaluated the work with the database according to 4 indicators: convenience of generation, convenience of work in the code, data migration and quick search in the database. But there were only two really important criteria — the convenience of generation and the convenience of working in code, taking into account the goals we set. In the end, the 1st place in generation was deservedly taken by SQLite, the 2nd by CoreData, the 3rd by Realm. In our conditions, it was very difficult to generate a database on Realm.</p><p>But in terms of work convenience Realm took the first place, of course, taking into account the work scenarios that we developed according to the voting results. In second place was SQLite, not the native one, but with the use of some libraries. And in third place was CoreData. As for the rest items — any variant gives a good search speed, and we refused migration at all. But we will come back to it later.</p><p>After all, we chose the easiest way to generate SQLite and the SQLite.swift library, which was already in the project. SQLite database generation itself is quite simple. We wrote a script in Python that downloads data from the server and puts them into tables, with each individual directory putting them into its own table. Or as, for example, in the subway, in two tables: separately for stations and separately for lines. Our final database weighs about seven megabytes, but before adding it to the bundle, I put the whole thing in a zip-archive and compressed it to two and a half.</p><p><strong>Directories, tests, data</strong></p><p>I won’t go too deep into the work with the directories, because there’s nothing interesting here. In general terms, for each table a different provider was written. And it had two types of get-methods: to work synchronously and asynchronously through Combine.</p><p>Synchronous methods were needed to simplify the work with legacy code, because everything worked synchronously there. And in order to switch to directories gradually, I added an additional legacy wrapper for them. Synchronous calls are sometimes much easier to build, and for simple queries it works very quickly.</p><p>And perhaps the most interesting part is the tests. Tests have been written on the entire database and on each table to make sure that the model in the code matches the tables in the database, which is in the application’s bundle. This is how we make sure that our Python script has generated exactly what we need and that the code is up and running.</p><p>And now we have only one task left to solve: updating the data while the application is running. In the course of discussions and the choice of the database, to make things easier, we decided to eliminate the option of migrating from one version to another.</p><p>Now everything works as follows:</p><ul><li><strong>First</strong>, if the user downloads a new version of our application, at launch, the database from the bundle will replace the one available now. This simple mechanism allows us to completely forget about migrations and not to be afraid that the data models in the code and table will not match.</li><li><strong>Second</strong>, we decided to add a special table with metadata to the database. We store the latest update date for each directory, as well as a special ETAG number in it. Seven days after the latest update date, the application will try to download new data, taking ETAG into account, and will change the date in the table to the new one. Using ETAG means that if the data has not changed on the server, nothing will change in the database either, and there will be no unnecessary traffic or load.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Bfl9y_1Hr5AAHJC8" /></figure><p><strong>Summary</strong></p><p>By switching to SQLite for directories, we got rid of JSON files and reduced the size of our application by 15 mb. Updating the version of the directories in the bundle has become much easier, and working with them in code has become much more comfortable. And my main message for today is: “Do not be afraid to use the SQLite database in your projects. In some scenarios, it can give a tangible head start to other solutions.”</p><p>That’s it for now. Ask any questions in the comments to the article or on our <a href="https://t.me/hh_tech">Telegram</a>. See you soon!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0b5bf50212c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hh-ru/stored-data-in-ios-save-update-use-d0b5bf50212c">Stored data in iOS: save, update, use</a> was originally published in <a href="https://medium.com/hh-ru">hh.ru</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hunting for toggle: Complicating simple features as much as we can]]></title>
            <link>https://medium.com/hh-ru/hunting-for-toggle-complicating-simple-features-as-much-as-we-can-c4933c2d513a?source=rss----d470d2ec65f1---4</link>
            <guid isPermaLink="false">https://medium.com/p/c4933c2d513a</guid>
            <dc:creator><![CDATA[Ekaterina Guschina]]></dc:creator>
            <pubDate>Tue, 09 Aug 2022 15:03:06 GMT</pubDate>
            <atom:updated>2022-08-09T15:03:00.738Z</atom:updated>
            <content:encoded><![CDATA[<p>– Hey, Kate, there’s a little bug crawling, take a look, please.</p><p>– I’m on it, bro. What’s the issue?</p><p>– When I return to the screen, the toggle is reset to default. <strong>Easy peasy lemon squeezy.</strong></p><p>That innocent phrase kickstarted my adventure into the world of mind-blowing architecture, insane fixes and red eyes. It was a trap.</p><p>Hello, my name is Kate, I’m an Android developer at SimbirSoft and I help improve the product at hh.ru. In this article I’m going to tell a story revealing how the developers of two companies, an Android tech lead and even the Head of Mobile were writing a minimal feature on MVI with toggle, and still they failed to catch a bug after long hours of projecting. Together we will find out what programmers sacrifice for the sake of good UX, why the initial decision was wrong, and how it was fixed.</p><p>This is a textual version of <a href="https://www.youtube.com/watch?v=Y57rH_trNDA">our vlog</a>, so if you prefer visual content, welcome to our <a href="https://www.youtube.com/channel/UCWztYdpQS3v9Gjno8bBZcNA">Youtube channel.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ibaVytFgTR_218_zQEOFkg.jpeg" /></figure><h4>Finding the problem at the bottom of the sea</h4><p>Since the screen was created by a different developer, the first thing I did was study it thoroughly. The screen had three states: loading, error and content.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bwD-RsY1uSP4nrNNElySxg.jpeg" /><figcaption>Screen states: loading, error and content</figcaption></figure><p>The content included that very unfortunate toggle. When clicking on the toggle, a request to the server would be sent — we would deliver a new value of the configuration flag.</p><p>Important condition: we have to enable the user to switch this toggle as many times as necessary, not showing any preloaders.</p><p>At first I had only one little task: correct the process of saving the toggle’s state. A user can leave the screen before the request is finished (provided that the Internet is slow). That’s why the toggle’s state would constantly return to the initial meaning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*2U-6rk-C2tSdKey2JubPbg.gif" /><figcaption>What did it look like? Dropping toggle when leaving the screen</figcaption></figure><p>What was the bug about? If the user switched the toggle when the Internet was slow, the following would happen. The user opens the screen, clicks on the toggle, switches it to ‘unchecked’, leaves the screen, but then comes back and sees that it’s ‘checked’. What a mess!</p><h4>Screen development scheme</h4><p>Our feature is basically a black box hiding the essence of the screen logic inside. It can access with the repository, which sends requests to the server. The repository also stores the cache.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*qyu2XK0UKJnJEclbFTSb_w.png" /><figcaption>The scheme of our screen implementation</figcaption></figure><p>As a result of the communication between our feature and repository, we get State. Initially it was divided in two fields. The first showed the cached state. It was coherent with the flag state on the server. The second field — UI state, responsible for what we display to the user.</p><p><strong>Attention! </strong>All coding examples are just examples, don’t take them to prod!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e004e678967bbcd4c6b8668ecf74f275/href">https://medium.com/media/e004e678967bbcd4c6b8668ecf74f275/href</a></iframe><p>In our case both states were interconnected, kept in one model, and downloaded when entering the screen. The only difference between them was the condition, according to which we would change the value. UI state changes all the time, as long as the user clicks on the toggle. However, the cached state only changes when the server replies to our request successfully. Since both of these meanings got overwritten in feature whereas we were starting the screen, we failed to recover the UI state and to show it to the user. That’s what the problem was about.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D_C1JhJ75EZDKxZKalUexQ.jpeg" /></figure><h4>Ways of solving the problem</h4><p>The first solution that came to our mind was to separate those two stages as much as possible. In such case we would proceed with updating the cached state when entering the screen, and UI state would be kept as long as it was needed.</p><p>However, the tests unveiled numerous bugs, because a user might switch toggles multiple times. Plus, we couldn’t guarantee that the requests would be completed sequentially. To sum up, the results could come in a surprising order. It was not right.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*LpS_92P3qH5Xp5rOrcLyrQ.gif" /><figcaption>Switching toggles</figcaption></figure><p>We tried to fix it by stopping the process of getting the results from the previous request, when we were already sending a new one. I offer you several ways of implementing such a pause.</p><h4>Event bus</h4><p>The first is to use an event bus. We send there an event about wanting to interrupt date processing, and then we catch this event inside the chain of our request processing. In this case the events will no longer be processed.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e414795c911a7efaac3b5c3a803b7d65/href">https://medium.com/media/e414795c911a7efaac3b5c3a803b7d65/href</a></iframe><p>For this solution we used <a href="https://reactivex.io/documentation/subject.html">RX Subject</a> and <a href="https://reactivex.io/documentation/operators/takeuntil.html">takeUntil</a> operator. In this case, before making a new request, we send the events about interruption to Subject. Inside the chain of request processing we use <a href="https://reactivex.io/documentation/operators/takeuntil.html">takeUntil,</a> an operator that interrupts the further Rx chain execution, if the source mentioned is sending any data. As a result, if we got events from <a href="https://reactivex.io/documentation/subject.html">Subject</a>, the next steps won’t be executed. It’s a viable solution, we used it in our feature for pagination.</p><h4>Storing the list version number</h4><p>The idea of the method is the following: let’s introduce a new special AtomicInteger, which will store the global version of the list data, and the version’s local variable, which will be in accordance with the global after the request is finished.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/880d5607b05558f3068cccb2d9866f74/href">https://medium.com/media/880d5607b05558f3068cccb2d9866f74/href</a></iframe><p>When we carry out the list download again, we increment the global number of the data version in the list and memorize the updated number of this version. After having received the data to our request, we can compare the global version to the stored local one after the request, the results from which we received.</p><p>If the versions are equal, we can proceed with the results processing, if not — we just skip this phase.</p><p>Both ways have their own advantages. If we use the event bus, we won’t process the further data chain from the request. If we store the list version number, we will know for sure that a reload has happened. Plus, we’ll be aware of its sequence number. However, neither of those two ideas suited us.</p><h4>So what’s wrong?</h4><p>First, the incomprehensible nature of what’s happening. A person who has never faced the problem of ‘excessive requests’ or who sees such a screen for the first time in their life, might just freak our seeing such an implementation.</p><p>Secondly, keeping an additional flag in the repository (about the downloading process). An additional source of truth emerges, which needs to be coordinated properly.</p><p>Thirdly, storing cache in repository and in the feature. Again — several sources of truth. Sooner or later, it will lead to desynchronization.</p><p>Finally, excessive requests will be occurring anyway, no matter whether we use an event bus or store the list version number.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2DyT8JYVzKANn9KHohmbSQ.jpeg" /></figure><h4>Error correction</h4><p>As usual, the solution turned out to be much simpler. Let the feature manage its cache itself! That will allow us to get rid of the repository cache and leave it only in the feature. Consequently, we will have one and only source of truth about the screen data. So in this state implementation we now have three fields: Cached State, UI-State and Progress Flag.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*aUuFQEpKYwBllXgzPIhn3A.png" /><figcaption>An updated version of our screen implementation</figcaption></figure><p>Okay, the user has launched the screen and we have to display something for them. The data will be downloaded from the server only when the cache is empty or the information there is no longer valid. Data validation can be arranged as you like, we use cache validation by date.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*C5nt942513OGZKfaBXMGcA.png" /><figcaption>A scheme of data download</figcaption></figure><p>When the user switches a toggle, we keep its value in State and check whether the request is being sent. If the request is absent, we send a new one with the toggle’s value.</p><p>On having completed the request successfully, we store the data in State (flag’s cached state). After that we need to check if UIState corresponds with the cached state. If they differ, we send a new request to the server.</p><p>Following this scheme, we benefit from two advantages: the cache remains consistent, and the number of requests is kept minimal. Stonks.</p><h4>Implementation</h4><p>That’s how we came to the general scheme. Now it is time we talked about implementation. First of all, we used <a href="https://badoo.github.io/MVICore/">MVICore</a> library by Badoo. <a href="https://badoo.github.io/MVICore/">MVICore</a> is a library for MVI pattern implementation on Android apps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*R4qxZXdKPQOjBwRBV-cUAQ.png" /><figcaption>Feature’s work scheme</figcaption></figure><p>Let’s say, our user changed the meaning of the toggle, so we write down a new meaning in UIState. Having done that, we check whether we can send a request. You can do it in <a href="https://badoo.github.io/MVICore/features/actorreducerfeature/">Actor</a>.</p><p><a href="https://badoo.github.io/MVICore/features/actorreducerfeature/">Actor</a> represents the overall logic of feature work. For example, it can send requests to the server or delegate event about State changes.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0c4766a6c8f3e4ed9463d6aad230d646/href">https://medium.com/media/0c4766a6c8f3e4ed9463d6aad230d646/href</a></iframe><p>We now understand that we can send the request, so we do it. After the successful completion, we save the new value of the flag in State.</p><p>Having updated State, we can now track its change in <a href="https://badoo.github.io/MVICore/features/fullfeatured/">PostProcessor</a>. We go there and check UIState and cached State inside State. If they are not equal, we repeat the whole chain of checks-requests-updates again.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dbf12ad11cced792a4fc94fbcc3c91f9/href">https://medium.com/media/dbf12ad11cced792a4fc94fbcc3c91f9/href</a></iframe><p>As a result of such work, our user can switch toggle with no worries, and we minimize the number of requests. That’s all.</p><h4>Well, not really</h4><p>Our scheme is not flawless. Soon after implementing it, we caught one more small bug: when the user logged out, we were still trying to send a request to the server, despite the fact that the unlogged user can’t really receive such a flag. In cached state the flag’s value disappeared, and the feature would start ‘knocking’ on the server’s door endlessly in order to get updates.</p><p>However, fixing it didn’t present any problem: we had to save the default value in the cached state.</p><h4>Hunt managed!</h4><p>Hallelujah, we fixed that bug, but at what price! A huge amount of time was spent, but that experience taught us a lot of things:</p><ul><li>First, before starting a feature development, look at it closely and create an approximate scheme or plan of what to do. In this case we can, for example, project State correctly and avoid any other problems.</li><li>Secondly, to prevent our colleagues from wasting time on similar cases, we decided to create a special architectural ‘cookbook’ on MVI. The first case for it we’ve just explored with you here. When you have some confirmed solutions at hand, it’s much easier to avoid similar bugs.</li></ul><p>Write in the comments, what do you think about the solution that we suggested and how would you solve this problem. We are planning to dive into more stories from our practice in the future. Buckle up!</p><p>Wishing you a productive and prosperous development!</p><h4>P.S.</h4><p>It would seem that the story is finished, the vlog is filmed, the article on Habr is published, but still…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Urqt8Oh0KM2AQF51ZIr-_A.jpeg" /></figure><p>Speaking of rocket science =)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4933c2d513a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hh-ru/hunting-for-toggle-complicating-simple-features-as-much-as-we-can-c4933c2d513a">Hunting for toggle: Complicating simple features as much as we can</a> was originally published in <a href="https://medium.com/hh-ru">hh.ru</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we moved to Java 15 or the story of jvm bug that lived six years long]]></title>
            <link>https://medium.com/hh-ru/how-we-moved-to-java-15-or-the-story-of-jvm-bug-that-lived-six-years-long-a8422f10cfd0?source=rss----d470d2ec65f1---4</link>
            <guid isPermaLink="false">https://medium.com/p/a8422f10cfd0</guid>
            <category><![CDATA[jvm]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[java15]]></category>
            <category><![CDATA[jdk-15]]></category>
            <dc:creator><![CDATA[Nikolai Gribanov]]></dc:creator>
            <pubDate>Wed, 03 Aug 2022 10:03:43 GMT</pubDate>
            <atom:updated>2022-08-03T10:03:40.506Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NOa5QQczrWyoloWZmrKqwg.jpeg" /></figure><p>We have been preparing for the release of Java 15 due to some of its new features. In particular, text blocks. Yes, they were introduced in Java 14 (you can read about new features in Java 14 <a href="https://habr.com/ru/post/491546/">here</a>), but only as a preview feature. It became available as a complete feature in Java 15.</p><p>At hh.ru, we are used to introducing and using the latest technologies in software development. Trying something new is one of the key tasks of the Architecture team. While many people write in Java 8, we are already close to sending Java 11 to the dustbin of history.</p><h4>How we moved to Java 15 or the story of one bug</h4><p>As you know, Java is releasing versions a lot more frequently, so we have more work on updating the versions. On the one hand, we had to adapt to the new realities and change our habits, which is not always comfortable. On the other hand, we can aim at the features which will appear in a new version of the language in advance and not wait for the release for 3 years.</p><h4>Migration from Java 14 to Java 15. Something went wrong</h4><p>Having waited for the new Java to come out, we began the transition. Without any second thought, we chose one of the busy services that was already running on Java 14. In theory there shouldn’t be any difficulties with the migration, in practice it turned out to be so. Updating Java 14 to Java 15 is not the same as updating Java 8 to Java 11.</p><p>The “hh and in production” service is updated, the work is done. What’s next? And next is monitoring the work. We use <a href="http://okmeter.io/"><em>okmeter</em> </a>to collect metrics. We used it to monitor the behavior of the updated service. There were no anomalies compared to the previous version of Java, except for one — native memory. In particular, the <em>Code Cache area has almost doubled</em>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/0*BgwmYhCbaTRGyDXX" /></figure><p>Until the end of November 17 Java 14, after Java 15</p><p>For a service with a large number of instances, every megabyte of memory counts. In addition to its spike, we can see an increasing trend on the graph. It seems that we are dealing with a native memory leak in Code Cache.</p><h4>What is this Code Cache of yours all about?</h4><p><strong>Code Cache</strong> is an area of native memory where the Java bytecode interpreter, JIT compilers C1 and C2, and their optimized code are stored. The primary user is the JIT. All recompiled code will be stored in Code Cache.</p><p>Since Java 9, the Code Cache is divided into three separate segments, each of which stores a different type of optimized code (<a href="http://openjdk.java.net/jeps/197">JEP 197</a>). But in the graph above, you can see only one selected area, even though Java 14 and Java 15 are there. Why just one?</p><p>The thing is that we had fine-tuned the memory size when migrating services to Docker (you can read about it <a href="https://habr.com/ru/company/hh/blog/450954/">here</a>) and deliberately set the Code Cache size flag (<em>ReservedCodeCacheSize</em>) to 72 MB in this service.</p><p>The three segments can be obtained in two ways: leave the default <em>ReservedCodeCacheSize</em> value (256MB) or use the <em>SegmentedCodeCache</em> key. This is what these zones look like in the graph from our other service:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/0*CnXdeAXg-RBL1Tk6" /></figure><h4>Finding native memory leaks in Code Cache</h4><p>Where to start investigating? The first thing that comes to mind is to use <em>Native Memory Tracking</em>, a feature of the HotSpot virtual machine that allows you to track native memory changes by specific zones. In our case, there is no need to use Native Memory Tracking, since we’ve already figured out that the problem is in Code Cache, thanks to the metrics we’ve gathered. So we decided to do the following — <em>run the service instances with Java 14 and Java 15 together</em>. Since we’ve had the service running for three days now on 15, we add one instance on 14.</p><p>We’re deciding to continue the search for the leak using Java utilities. We start with <em>jcmd</em>. Since we know that our Code Cache is leaking, we access it. If the service is running in Docker, we can run the command this way for each instance:</p><blockquote>docker exec &lt;container_id&gt; jcmd 1 Compiler.CodeHeap_Analytics</blockquote><p>We get two very long and detailed reports on the state of the Code Cache. After painstakingly</p><p>comparing them, we noticed the following interesting fact related to Code Cache cleanup:</p><blockquote>// Java 14</blockquote><blockquote>Code cache sweeper statistics:</blockquote><blockquote>Total sweep time: 9999 ms</blockquote><blockquote>Total number of full sweeps: 17833</blockquote><blockquote>Total number of flushed methods: 10681 (thereof 1017 C2 methods)</blockquote><blockquote>Total size of flushed methods: 20180 kB</blockquote><blockquote>// Java 15</blockquote><blockquote>Code cache sweeper statistics:</blockquote><blockquote>Total sweep time: 5592 ms</blockquote><blockquote>Total number of full sweeps: 236</blockquote><blockquote>Total number of flushed methods: 11925 (thereof 1146 C2 methods)</blockquote><blockquote>Total size of flushed methods: 44598 kB</blockquote><p>Note the number of full cleanup cycles — <em>Total number of full sweeps</em>. Recall that the service in Java 15 runs for 3 days, and in Java 14 only 20 minutes. But the number of full Code Cache cleanups is strikingly different — almost 18 thousand in 20 minutes, versus 236 in three days.</p><h4>How Code Cache Sweeper works</h4><p>Now it’s time to go deeper into details. A separate jvm thread <a href="https://github.com/AdoptOpenJDK/openjdk-jdk15u/blob/61c35054bd4b320479c1764a790cfb2c82909f3e/src/hotspot/share/runtime/thread.cpp#L3500">CodeCacheSweeperThread</a>, which is called with certain heuristics, is responsible for cleaning up the Code Cache. The thread is implemented as an infinite <em>while</em> cycle, within which it is blocked until a 24-hour timeout expires or the blocking is unblocked by a call.</p><p><em>CodeSweeper_lock-&gt;notify();</em></p><p>After the lock is released, the thread checks if the timeout has expired and if at least one of the two flags that trigger Code Cache cleanup has the value <em>true</em>. Only if these conditions are met, the thread will call<a href="https://github.com/AdoptOpenJDK/openjdk-jdk15u/blob/master/src/hotspot/share/runtime/sweeper.cpp#L259"> sweep()</a> to clean up the Code Cache. Let’s take a closer look at the flags:</p><p><em>should_sweep. This flag is responsible for two strategies to clean up the Code Cache — normal and aggressive. We will talk about the strategies next.</em></p><p><em>force_</em>sweep. This flag is set to true when it is necessary to force the Code Cache cleanup without the conditions of normal and aggressive cleanup strategies being met. Used in <em>jdk</em> test classes.</p><h4>Standard cleanup</h4><ol><li>During a GC call, methods stored in the Code Cache can change cleaningtheir state according to the following scenario: <em>alive -&gt; nonentrant -&gt; zombie. Non-alive methods are marked as “must be removed from Code Cache at the next run of cleanup thread”.</em></li><li><em>At the end of its work GC passes a link to all non-alive objects in the </em><a href="https://github.com/AdoptOpenJDK/openjdk-jdk15u/blob/master/src/hotspot/share/runtime/sweeper.cpp#L483"><em>report_state_change </em></a><em>method.</em></li><li><em>Then the total size of the objects marked as non-alive in this GC pass is incremented into a special variable bytes_changed.</em></li><li>When bytes_changed reaches the limit set in the sweep_threshold_bytes variable, the should_sweep flag is set to true and the cleanup thread is unblocked.</li><li>The Code Cache cleanup algorithm is started, at the beginning of which the bytes_changed value is reset. It itself consists of two phases: scanning the stack for active methods and removing inactive ones from the Code Cache. This completes the normal cleanup process.</li></ol><p>Starting from Java 15, the limit value can be controlled with the jvm SweeperThreshold flag — it takes the value as a percentage of the total Code Cache memory set with the ReservedCodeCacheSize flag.</p><h4>Aggressive sweeping</h4><p>This type of cleanup appeared in Java 9, as one of the ways to combat Code Cache overflow. It is performed at the moment when the free space in the Code Cache memory becomes less than the predefined percentage. This percentage can be set independently using StartAggressiveSweepingAt key, by default it is 10.</p><p>Unlike standard cleanup, where we wait for the buffer to fill with “ dead “ methods, the check to start aggressive cleanup is run every time we try to allocate memory to the Code Cache. In other words, when the JIT compiler wants to put new optimized methods into the Code Cache, it runs a check to see if cleaning needs to start before allocation. This <a href="https://github.com/AdoptOpenJDK/openjdk-jdk15u/blob/master/src/hotspot/share/runtime/sweeper.cpp#L275">check </a>is quite simple; if there is less free space than specified in StartAggressiveSweepingAt, the cleanup is triggered forcibly. The cleaning algorithm is the same as in the normal strategy. And only after the cleanup is done, JIT will be able to put new methods into the Code Cache.</p><h4>What have we got?</h4><p>In our case the Code Cache size was limited to 72 MB, and we didn’t set the StartAggressiveSweepingAt flag, so by default it is 10. If you look at the Code Cache cleanup statistics, it might seem that it is the aggressive strategy that works in Java 14. We were additionally convinced by the same graph, but with a larger scale:</p><p>Java 14</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/0*KH4H_qGg-aVB4hIT" /></figure><p>It has a jagged structure, which indicates that the cleanup is frequent, and probably methods are unloaded from the Code Cache in circles, in the next iteration of the JIT compilation are put back in, then removed again, etc.</p><p>But how is this possible? Why does the aggressive cleanup strategy work? By default it should run when the free space in Code Cache is less than 10%, in our case only when it reaches 65 megabytes, but we see that it also happens at 30–35 megabytes of used memory.</p><p>In comparison, the graph with Java 15 running looks different:</p><p>Java 15</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/0*dNNSkfcrqoP5Ds9x" /></figure><p>There is no jagged structure, there is a smooth growth, then purification and growth again. The clue is somewhere nearby.</p><h4>A leak is not a leak</h4><p>Since the Code Cache is managed by jvm, we went looking for answers in the openJDK sources, comparing <a href="https://github.com/AdoptOpenJDK/openjdk-jdk14u">Java 14</a> and <a href="https://github.com/AdoptOpenJDK/openjdk-jdk15u">Java 15</a>. While searching, we found an interesting <a href="https://bugs.openjdk.java.net/browse/JDK-8244278">bug</a>. It said that the aggressive Code Cache cleanup has not worked correctly since it was introduced in Java 9. Instead of starting aggressive cleanup at 10% free space, it was called at 90% free space, which is almost always. In other words, leaving the StartAggressiveSweepingAt = 10 option would actually leave StartAggressiveSweepingAt = 90. The bug was fixed on July 3, 2020. And it was all about one <a href="https://github.com/AdoptOpenJDK/openjdk-jdk15u/commit/f7cb0f760f0460123b1a2a33ec48793eccf9c83a">line</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/0*4wrhRQzvgTFOirEV" /></figure><p>This fix is included in all versions of Java after Java 9. But then why isn’t it in our Java 14? It turns out that our Java 14 docker image was built on April 15, 2020, and then it becomes clear why the fix is not included there:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/0*P0_OR_1E9DBI09Ae" /></figure><p>So there is no native memory leak in Code Cache either? It’s just that all the time the cleanup wasn’t working properly, wasting cpu resources. After observing the service in Java 15 for a few more days, we concluded that this was the case. The overall native memory graph had plateaued and stopped showing an upward tendency:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/0*rRRXbdEuD5JSMvO1" /></figure><p>the spike in the graph is a migration to java 15</p><h4>Summary</h4><ol><li>Update your Java as often as possible. This applies not only to major versions, but also to patch versions. It may contain important fixes</li><li>Wise use of metrics helps detect potential problems and anomalies</li><li>Switch to Java 15, it’s worth it. <a href="https://openjdk.java.net/projects/jdk/15/">Here’s </a>a list of all the new features that came out in 15</li><li>If you are using Java 8, you do not have the problem of aggressively cleaning the Code Cache, for lack of this functionality as such. However, there is a risk that the Code Cache could overflow and the JIT compilation will be forcibly disabled</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8422f10cfd0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hh-ru/how-we-moved-to-java-15-or-the-story-of-jvm-bug-that-lived-six-years-long-a8422f10cfd0">How we moved to Java 15 or the story of jvm bug that lived six years long</a> was originally published in <a href="https://medium.com/hh-ru">hh.ru</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating the release process]]></title>
            <link>https://medium.com/hh-ru/automating-the-release-process-af8f97edc7bc?source=rss----d470d2ec65f1---4</link>
            <guid isPermaLink="false">https://medium.com/p/af8f97edc7bc</guid>
            <category><![CDATA[qa]]></category>
            <category><![CDATA[releases]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[process]]></category>
            <category><![CDATA[mobile]]></category>
            <dc:creator><![CDATA[Dan Kheyfets]]></dc:creator>
            <pubDate>Tue, 26 Jul 2022 06:02:59 GMT</pubDate>
            <atom:updated>2022-07-26T06:02:56.305Z</atom:updated>
            <content:encoded><![CDATA[<p>Hi there! My name is Daniel Kheyfets and I am a QA Lead for mobile at hh.ru. In this article I will tell you how we automated the flow of mobile application release, how we divided it into stages, what tools we used and what problems we faced.</p><p>This is a text transcript of<a href="https://www.youtube.com/watch?v=T2O3uzFZQOU"> the video on our channel</a>, so if you prefer to watch it instead of reading it, welcome to <a href="https://www.youtube.com/channel/UCWztYdpQS3v9Gjno8bBZcNA">our Youtube channel</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*zaCpMOxwZocgVxIKrcnghQ.png" /></figure><p>In <a href="https://medium.com/hh-ru/how-to-bring-off-a-gentlemans-release-9884a954e4a7">previous episodes</a>, I’ve talked about how we’ve improved our mobile app release processes, what practices we’ve introduced, what problems we’ve encountered, and what we’ve come up with in the end.</p><blockquote>In the previous episode</blockquote><blockquote>We switched to GitHub flow, wrote autotests and ran them on CI, launched Release Train, and introduced the practice of Duty. Thus, we achieved stable, regular, weekly releases. But there were still a lot of routines we wanted to get rid of.</blockquote><h4>Routine</h4><p>All the routine for mobile app release can be divided into two parts: technical and managerial.</p><p>Technical routine:</p><ul><li>creating a release branch</li><li>version increment</li><li>upload to the app stores</li><li>autotest run</li><li>build launch</li></ul><p>Managerial routine:</p><ul><li>release description</li><li>what to write about in the “What’s New” section of the Stores</li><li>setting a task for marketing to write “What’s New” texts</li></ul><p>Although all these operations are not complicated, they require human participation. Accordingly, the most common human mistakes are not excluded: someone forgot something or didn’t do something, and so on. Of course, we had checklists and instructions for such situations. However, even this did not save us from small but unpleasant problems.</p><p>To eliminate the human factor, we decided to automate the process, and we found the task interesting.</p><h4>Work planning</h4><p>Before automating, we decided to visualize the entire process of our release: how it looks, what statuses it takes, what causes the statuses to change.</p><p>The stages that make up our release flow:</p><ul><li>Creating a release</li><li>Running autotests</li><li>Launching the release build</li><li>Uploading to Google Play / App Gallery / Apple Store</li><li>Closing release</li></ul><p>There are transitions and events between stages that cause the release statuses to change. This is where relatively harmless events can happen, as well as real trouble. For example, if we find a bug/crash in the production, we need to fix it, run autotests again, rebuild the app, and upload it again. And we can also get rejected at a Store review, and then we have to go through exactly the same cycle of checks and rebuilds again.</p><p>Our plan turned out to be universal enough that we were able to describe it at once for all our apps (jobseeker /employer), platforms (Android, iOS) and Stores (Google Play/App Store/Huawei App Gallery).</p><h4>Work detalization</h4><p>After describing the release process, we decided to analyze in detail what specific steps would be taken at each stage. We dug around in our checklists, instructions, invited a ̶s̶h̶a̶m̶a̶n̶ ̶t̶o̶ ̶s̶h̶a̶m̶a̶n̶i̶z̶e̶ ̶w̶i̶t̶h̶ ̶a̶ ̶t̶a̶m̶b̶o̶u̶r̶i̶n̶e̶ and described it all in a large and detailed diagram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pVAxsogkZBuoaeJnLCzd0g.jpeg" /><figcaption>Detailed release process</figcaption></figure><p>After taking a closer look at the scheme, we realized that at each stage there is a certain set of atomic actions that are, for the most part, independent of each other.</p><p>For example, at the stage of creating the release we:</p><ul><li>create a separate branch for a future release</li><li>increment the application version</li><li>look for a list of tasks completed since the last release</li><li>ask marketing to write text for a future release based on the list of completed tasks.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9N2f15mhVSruL-xMAeAZ4g.png" /></figure><p>We prioritized each step: something we wanted to automate first, something we could wait on, and something we might like to automate, but we haven’t had the opportunity yet.</p><p>Excellent: the release stages are clear, the actions at each stage are clear, it’s time to make a plan for how to implement it all.</p><p>We started by explaining to management why we needed this task. If we do this task, it will save us a lot of time and help us to avoid mistakes in manual release management. We’ve clearly outlined deadlines for automating each step. This all helped us prioritize the task for automation, and we finally got the go-ahead.</p><p>Let’s get the work done!</p><h3>Implementation</h3><h4>Data structure</h4><p>First of all, we decided how we were going to store all the data. We decided that we would describe a simple PostgreSQL database of two tables: the main one storing all release data (release number, branch name, creation date, Jira ticket link, release status, etc.), and an auxiliary one for information about marketing tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/993/1*AINBFm_HrzVFg1ZNyofxww.png" /><figcaption>A piece if the main table</figcaption></figure><p>Okay, we’ve figured out how we’re going to store the data, now we’re going to decide how to interact with it.</p><h4>Data visualization</h4><p>A typical Tuesday of the hh team before the release automations:</p><ul><li>“Has the release branch been created?”</li><li>“Did you run the autotests?”</li><li>“What’s the current version, help me!”</li><li>“Where are the texts for the stores?!”</li></ul><p>As you can see, the problem we had was that no one ever knew what the status of a release was. Had the apps been uploaded to the store? Has the old release been rolled out? Where are the actual texts? So many questions, so few answers.</p><p>Thanks to the database we created, we have a source of truth about the releases, but going to the database every time you want to see what happens there wasn’t comfortable.</p><p>So we decided that to visualize this process we would use the tool we usually work with — Jira. We made a board, where each column symbolized a particular stage in our release process. Such as: launching autotests, what autotests passed, uploading to the stores, or rolling out to users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rqH-6lz8rHwOCd6pWeEcPw.png" /><figcaption>Jira board with releases</figcaption></figure><p>And we also created a special type of task for this board — “release”. In this task we attached all the necessary information that we wanted to know about the release: what its status was, whether it passed autotests, a link to run these autotests, as well as the tasks that were included in it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*riFdYezEJAhacVPDLJ03tA.png" /><figcaption>A release ticket</figcaption></figure><p>In addition, the task displays the dates of each stage. So we can easily see on what date and at what time each stage was passed.</p><p>Thus, we managed to visualize all the most important information about our release and we don’t need to make manual queries to the database to do it. Anyone can now simply open the board in Jira and see what the status of a particular stage is.</p><h4>How do you manage a release?</h4><p>But displaying the database status in Jira is just half of the task. You also have to modify it, and to do that you have to come up with a scheme to manage it all.</p><p>We decided to manage our releases from the same Jira. By moving the ticket there, we can start one or another stage of our automation. When the status of the release task changes, we’ll start a special plan on CI, which will start doing all the necessary work.</p><p>Since our CI server is Bamboo from Atlassian, it was pretty easy to make it all work.</p><p>There was only one question: what to write automation scripts in?</p><p>We investigated existing tools and came to the conclusion that the most convenient one for us would be <a href="https://fastlane.tools/">fastlane</a>. There were several reasons for this choice.</p><p>First, fastlane is probably the only convenient way to release and automate anything related to iOS apps. In addition, the iOS team has used this tool in their existing automations.</p><p>Secondly, fastlane has many<a href="https://docs.fastlane.tools/plugins/available-plugins/"> implemented plugins</a>, <a href="https://docs.fastlane.tools/actions/">actions </a>that we need for our tasks, which means we don’t have to write everything from scratch.</p><p>Third, fastlane can work with both iOS and Android applications. This will allow you not to write automation for each platform, but just use a single tool, without duplicating code and basically make everything much faster. You can see <a href="https://www.youtube.com/watch?v=deQg6xRTc60">our effing story</a> about using fastlane for Android.</p><h4>What’s under the hood of the release train</h4><p>In our automation system we got four main parts: the PostgreSQL database, the Bamboo CI server, fastlane itself, and Jira.</p><p>As I wrote above, thanks to the easy integration of Jira and Bamboo, we made it so that when we move a Jira release ticket from one status to another, we run a special plan on Bamboo. And it is triggered by a special lane fastlane.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*46amFgIrUgCaOZmH1ZeYEg.png" /></figure><p>Each stage of release in the code looks like a separate lane fastlane, which defines everything that needs to be done at this release stage. The script goes to the database and determines what status the last existing release is in. There are only two statuses: EXIST or RELEASED. If the status of the most recent release we have is RELEASED, then there is no release, and we need to create a new one. If the status of the most recent release is EXIST, then there is a release in progress right now. And we need to go to Jira to clarify the stage.</p><p>The actions description for each release step in the code looks like a separate lane fastlane.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vhgYrdmoEk1Pa0hm4ou2eg.png" /></figure><p>All of the actions during the release can be broken down into four large groups:</p><ul><li>Work with DB;</li><li>Work with Jira;</li><li>Work with GitHub;</li><li>Some kind of auxiliary utilities.</li></ul><p>For each task type, we have described Ruby wrapper classes in the code. We use the <a href="https://github.com/sumoheavy/jira-ruby">Jira-ruby</a> library to work with Jira and the standard Postgres library to work with the database. We also wrote a small client to send queries to the database and handle errors. We also described the necessary commands for the git and other utilities. For example, the code for sending notifications to Slack.</p><p>We got such a universal set of functions, which we put in a common repository, and use both in iOS and Android. We created such a<a href="https://docs.fastlane.tools/plugins/create-plugin/"> plugin to be connected</a> that we use when developing in the iOS repository or in Android to avoid code duplication.</p><h3>What’s the bottom line</h3><p>We did quite an extensive job of automating our release, and we got a lot done.</p><ol><li>We eliminated all the manual steps to create a release branch, increase the version number, run autotests, build and upload to the stores.</li><li>We got rid of the routine of describing the release composition: our scripts automatically analyze merge commits to the development branch and generate a list of tasks that make it into the release. All necessary tasks in Jira related to the work of other departments are also created automatically.</li><li>We have added many notifications of changes in release status for all concerned: for managers, we send notifications of releases rolled out in the production, for testers, about the readiness of builds for regression, about successful (and not really) runs of autotests.</li><li>And, most importantly, we’ve visualized our entire release process. Jira is now a single source of truth about the release, now no one has any questions about what’s going on with the release.</li></ol><p>The solution, of course, is far from perfect — somewhere there are technical errors, but step by step we get rid of the human factor in important stages of our releases.</p><h4>Has it gotten better?</h4><p>Has it gotten better since we implemented our automations? Definitely yes: it’s much easier and more convenient for us to drop releases now. But there’s always room for growth.</p><p>We’ve built a foundation on which we can continue to thread many other automations that may appear in the future. We already have several tasks waiting to be done:</p><ol><li>We want to learn how to change the status of different product tasks, depending on the release status. Now we occasionally forget to put different labels to tasks in Jira, or move them to the final workflow status.</li><li>We want to automatically roll out apps not only to App Distribution, but also to the stores themselves, including Huawei App Gallery for Android. At the same time, we want to roll out apps to internal, beta, and release tracks as well.</li><li>It would be great to learn how to automatically start Jira tasks for new crashes and bugs after they appear in production, as well as some mass user complaints.</li></ol><p>That’s it. Automate the routine, comment, share your findings. Bye!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af8f97edc7bc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hh-ru/automating-the-release-process-af8f97edc7bc">Automating the release process</a> was originally published in <a href="https://medium.com/hh-ru">hh.ru</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Top 5 Junior Automation Tester’s Questions About Automated Testing]]></title>
            <link>https://medium.com/hh-ru/top-5-junior-automation-testers-questions-about-automated-testing-c7c03c674eeb?source=rss----d470d2ec65f1---4</link>
            <guid isPermaLink="false">https://medium.com/p/c7c03c674eeb</guid>
            <category><![CDATA[automated-testing]]></category>
            <category><![CDATA[automation-testing]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[testing]]></category>
            <dc:creator><![CDATA[Olya Kabanova]]></dc:creator>
            <pubDate>Tue, 19 Jul 2022 06:03:29 GMT</pubDate>
            <atom:updated>2022-07-19T06:03:27.726Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*Rc3klMp1SMHje6-_PkPSgw.jpeg" /></figure><p>Hey everyone! I’m Olya, mobile app tester at hh.ru. Today we’re finishing the series of answers to the most popular questions on testing automation. We have already answered the questions from a <a href="https://medium.com/hh-ru/top-5-manual-testers-questions-about-automated-testing-aae16a3402bc">manual tester</a>, a <a href="https://medium.com/hh-ru/top-5-manager-questions-about-automated-testing-4f194bc02375">manager</a>, and a <a href="https://medium.com/hh-ru/top-5-technical-directors-questions-about-automated-testing-ca2fe4619aab">chief technology officer</a>. Now it’s time to answer the five most interesting questions that beginner test automation engineers ask — about flaky tests and production bugs, our fight for stability and how to live up to the expectations regarding autotests.</p><p>The video version of my Q&amp;A is available <a href="https://www.youtube.com/watch?v=jVATgT-IOEY&amp;feature=youtu.be">here</a>.</p><h4>Question #1<strong>. We write and run autotests, but production bugs are still haunting us</strong></h4><p>Let’s start with the idea that production bugs will keep occurring anyway, it’s absolutely normal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/1*vNz3uEPMCxAIF_ekkz3Rjg.png" /></figure><p>Users may perform unbelievable operations with your app and go through fascinating and unpredictable scenarios: open a tab, switch to another app, block the phone, receive push notifications, surf the feed, and then come back to our app. That’s where the bug is waiting. You can partially automatize such cases — with the help of overload, interruptions and other operations. However, you shouldn’t cover all possible unique scenarios with autotests. At least because no CI will ever survive such a load, and the time wasted on test running is not worth it.</p><p>Apart from that, beware of the device-dependent bugs, which you can’t catch with autotests, if you were put to test this very troubled machine. Of course, you can try to configure CI so that it creates various emulators, or run tests on various real devices, but it will significantly complicate the process of supporting such autotests and most likely reduce the stability. And again, it won’t be worth the time wasted on support and runnings, especially due to the fact that the bugs you find will be minor.</p><h4><strong>Question #2. How to make autotests less painful for everybody?</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*vGALbq6BPEv4986ckEBv3g.png" /></figure><p>In order for autotests not to make everybody suffer all the time, it’s enough to follow several rules:</p><p><strong>Rule 1. Write a clear and coherent code, set the reports.</strong></p><p>The tests should be comprehensible not only for you, but for other testers and developers. If your tests failed, the person in charge shouldn’t sit with it half a day or pester the author crying “help me identify the crash”. They should be able to understand the location and reason of it. It’s even better if you arrange transparent reports, where it’s possible to trace the steps and the exact spot of test failure.</p><p>Nobody wants to waste time on checking how the test works and what it does in general — all of this should be clear.</p><p><strong>Rule 2. A review is required.</strong></p><p>In order to make sure that your test is actually helpful and perfect, somebody else needs to confirm it. Only then you can sleep well at night.</p><p>Here at hh.ru we require a necessary review for all tasks. For all tasks with autotests it’s obligatory to have a review from QA. It’s extremely important, as the reviewer helps find not only an incorrect indent, but also mistakes in logics. They can also conduct additional testings or identify an error that you might have missed.</p><p><strong>Rule 3. Implement autotests in your CI/CD right away.</strong></p><p>It’s essential that you benefit from autotests here and now. Don’t put them on the back burner. The sooner your people start to get used to the fact that their code is now tested, the easier it will be for you to live. Meanwhile, it’s important to remember that autotest releases should be as stable as possible. Before merging your tests, it’s better to run them on CI 100 times (I’m not joking). Only after becoming 100% sure that your test doesn’t flake, doesn’t crash at random spots, and works properly — only then move forward!</p><h4><strong>Question #3. Why do tests constantly crash? How can I trust them?</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*dtUjEftotNz5tmd9dsLHmw.png" /></figure><p>Let’s start with the fact that such autotests will be crashing anyway. It’s absolutely normal, so if they crash all the time, it’s worth pondering whether they actually check anything.</p><p>Our duty is to make them fail less. In order to do so, we need to learn to detect and solve the problems which cause the crash, plus, improve the weak points.</p><p>While working with autotests, I detected my personal TOP-4 problems with autotests:</p><ol><li><strong>Somebody released something and everything crashed.</strong></li></ol><p>Here everything is simple. The functional changes should be made simultaneously with autotest correction. Such corrections take some time, which we consider during task evaluation and decomposition. When a developer changes somethings in the functions that had already been covered with autotests, before merging the branch, he needs to edit autotests, so that they keep checking these functions. It’s essential to stabilize this process and not to ignore the tests. Always keep them relevant.</p><p><strong>2. Tests started crashing, although we hadn’t changed anything</strong></p><p>It’s even more exciting. The developer hadn’t changed anything, but for some reason the tests started to fail. It could be that they changed something on backend.</p><p>Why do our UI-autotests depend on backends?</p><p>Our autotests are run on test stands — it’s a copy of production backend (without prod database). We update their relevance — every day autotests are run on relevant backends. Therefore, if somebody releases anything new, our autotests correspond to that and may crash. It’s an emergency for us, so we instantly check if it’s a bug or our new reality, which should be updated. Then we contact the team or correct the tests.</p><p>It’s important to mention that it doesn’t happen with every backend change. Most of the time we synchronize our teams. However, the reality dictates different rules: sometimes we omit tiny details (e.g. somebody didn’t consider affecting mobile apps). Our tests are very sensitive to this.</p><p>Every time it happens, we think of mocks. Mocks are good, if you don’t use them too much. But I think that such autotest sensitivity to changes — it’s ok, because this way we’re in the loop of new features, which affect our app from other teams’ perspective. Plus, mocks require time for support and updating. Here it’s up to each person what they want to support more.</p><p><strong>3. Bugs</strong></p><p>My favourite reason of autotest crashes. No comments here. If your autotest found a bug, it’s a very good autotest. And you’ve done a great job.</p><p><strong>4. Flaking</strong></p><p>My least favourite reason. That’s what we’re going to discuss next.</p><h4><strong>Question #4. Why do my tests flake?</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*Vo2YzI--hzSuJWs3GBhO7Q.png" /></figure><p>Flaking is like emotional swings — your tests are green, then red, everybody is sad, then happy, and nobody knows how to live with it. In order to make your app’s and test’s relationship healthy, we suggest that you follow several rules about how to avoid flakes and create more stable autotests:</p><p><strong>1. Unique test data</strong></p><p>Each autotest should be accompanied with new unique test data, which only this test works with. You can’t just hope that something would pop up on the screen and that’s ok. Predictability is key.</p><p><strong>2. Wait till all elements appear on the screen</strong></p><p>You made a logical test, step-by-step, but it still crashes at random spots. It must be that it doesn’t wait till the necessary element and taps in the wrong place. Animations, loaders, alerts only worsen the situation.</p><p>To prevent that from happening, before any action (tapping, swiping) or assert (checking) you need to wait until this element is shown on the screen and becomes available.</p><p><a href="https://github.com/KasperskyLab/Kaspresso">Kaspresso</a> for Android has an embedded mechanism, which waits for the necessary element, (meanwhile doing retries on isVisible), and only then it commits the action. Thanks to that, your autotests will have no problems with wrong or early tapping. Also you can set the timer for this waiter, because apparently waiting for an element during 10 seconds may be a bug too.</p><p>XCUITest on iOS has waiters too, but you have to set them by yourself before any each action. Or it’s worth to write a mechanism similar to Kaspresso.</p><p>The main thing is not to leave waits in code for a particular amount of seconds, because it might prolong the test’s time. And make it a rule: before messing with the element, make sure that it’s present on screen.</p><p><strong>3. Configure stable infrastructure</strong></p><p>If the test infrastructure keeps bringing you down, it’s needed to solve this problem ASAP. Freezing and slow emulators, crashing test stands — this atmosphere is not beneficial for your tests. Plus, everyone will soon get fed up with investigating the reasons of another fall. That’s why it’s necessary to dedicate the team’s time and resources to infrastructure stabilization. If everything is optimized, but there’re still problems, maybe you need to update your hardware. Time to empty the budget.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*uDp_XALHGbu5QBT9jGjBsw.png" /></figure><h4><strong>Question 5. What should I do, if there are too many tests now?</strong></h4><p>At first you coded small autotests — open a screen on some app and check whether it launches or not. Then you start automatizing more complicated scenarios. The number of tests rises, the test time increases. When something breaks down, it’s not one test that crashes, but ten, all with the same mistake, whereas the time of the crashed build increases, because the failed tests were retried.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/978/1*S-YhmZjvfSaXE5Gym34XlQ.png" /></figure><p>In this situation it’s important to let go of your old yet stable tests and delete them, if the same checkupsappeared in the new ones. The hardware is not immortal, plus spending double time on app launch for each new test is antiproductive and expensive.</p><p>In order to avoid diving into huge future refactoring, use the following rule — update the old things before starting to write the new ones. If you start applying this principle right away, you won’t have to deal with such a problem.</p><h3><strong>Instead of conclusion:</strong></h3><p>This article is the final article from the Q&amp;A series on automation, but we will continue posting about testing and automation at hh.ru, sharing how our releases go and a lot of other helpful and interesting things.</p><p>Follow <a href="https://t.me/hh_tech_news">our Telegram news channel</a> and <a href="https://www.youtube.com/c/hhtech/">Youtube channel with “HHella cool stories</a>” to keep up with the new videos, articles and other news. You can also ask any questions to our engineers in the comments or <a href="https://t.me/hh_tech">contact the developers in our Telegram chat</a> or privately.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c7c03c674eeb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hh-ru/top-5-junior-automation-testers-questions-about-automated-testing-c7c03c674eeb">Top 5 Junior Automation Tester’s Questions About Automated Testing</a> was originally published in <a href="https://medium.com/hh-ru">hh.ru</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sharing a socket connection between tabs]]></title>
            <link>https://medium.com/hh-ru/sharing-a-socket-connection-between-tabs-c93add6cbbe8?source=rss----d470d2ec65f1---4</link>
            <guid isPermaLink="false">https://medium.com/p/c93add6cbbe8</guid>
            <category><![CDATA[chat]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[websocket]]></category>
            <category><![CDATA[frontend]]></category>
            <dc:creator><![CDATA[Brian Connoly]]></dc:creator>
            <pubDate>Wed, 13 Jul 2022 08:02:58 GMT</pubDate>
            <atom:updated>2022-07-13T08:02:57.239Z</atom:updated>
            <content:encoded><![CDATA[<p>Every modern service should have stories and a chat. We decided to start by making a messenger in <a href="http://hh.ru/">hh.ru</a>. My name is Vladislove Korotun, I am a frontend developer. In this article, I will tell you how an unconventional approach to using Web Workers helped us to complete this task.</p><p>For those of you who prefer watching and listening, check <a href="https://www.youtube.com/watch?v=PUL2fN8Ry5k">this youtube link</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uV-pAnu-mJ7zHWljZIWywg.jpeg" /></figure><h3>Preparation</h3><p>For the chat to be as responsive as possible and not to spam the server with empty timer requests, we decided to make it on <a href="https://javascript.info/websocket">WebSocket</a>. Also, we really wanted to have a dynamic new message counter on each page.</p><p>Here we faced our first problem. Our site users often open a huge number of tabs, and having an active socket connection on each of them is a very costly approach. In addition, simply updating the counter via http by timer on each tab created a large load on the server, even when the chat itself was populated by only 20% of the audience.</p><p>We went for search for new solutions and look how others deal with it. Some services open the chat in an entire new tab and set up a socket connection only there. But we wanted to have access to the server messages on all our tabs, so this approach didn’t work for us.</p><p>We also found an <a href="https://ayushgp.xyz/scaling-websockets-using-sharedworkers/">interesting article</a> on how you can use <a href="https://developer.mozilla.org/en-US/docs/Web/API/SharedWorker/SharedWorker">SharedWorker</a> for sharing of socket connections between the tabs available to the worker. But unfortunately, Safari has now dropped support for SharedWorker. And that’s a large percentage of our audience.</p><p>In our search for a suitable solution, we turned our attention to <a href="https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API">ServiceWorker</a>. These workers are available in all tabs and frameworks of their Origin and support working with WebSocket. Our goal was to develop a system that would allow us to use a single socket connection in all of our web applications, including those with a different origin than the server itself.</p><p>The scheme is as follows: the socket server itself is located on the <a href="http://websocket.hh.ru/">websocket.hh.ru</a> subdomain. There we also have a Proxy page that installs and activates the worker. The page can be embedded into any of our sites using IFrame and proxy socket events to the parent window via PostMessage.</p><p>Thus, each of our applications that need sockets can simply embed this IFrame and use a special wrapper to access the message channel from the server. With this approach, there will only be one socket connection, and it will live in the ServiceWorker run by the Proxy page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a1KEsSlNGxdQo1ZZhrinRQ.jpeg" /></figure><h3>First difficulties</h3><p>During development, everything worked like a charm. But when we submitted the task to testing, we found that 10–15 minutes after the page was launched, the counter stopped responding to updates.</p><p>It turned out that connection to the server was lost and was not restored until the page was reloaded. Here we had to go a little deeper and read the documentation more attentively.</p><p>It turns out that the main difference between ServiceWorkers and SharedWorkers is that the former doesn’t run permanently. They are launched to perform some specific tasks, and after some time the browser silently shuts them down until the next requests appear.</p><p>In our scheme, socket connections are only used in a one-way format. We catch server events, but we go for data via http. So the browser would decide that the worker was done and needed to be shut down. We decided to try to get around this limitation and tried sending empty messages to the worker using a timer. The experiment was a success.</p><p>As long as the worker was receiving requests from its clients, it continued to live and maintain an active socket connection. Even if the browser at some point decided that the worker was running too long and shut it down, the next ping message woke it up, and the socket connection was restored.</p><p>At this point, a second problem awaited us. Also during testing, we found that in a tab opened in a new incognito mode session, when the worker registers for the first time, the connection is established, but the event does not reach the tab. However, after refreshing the page, everything works as we expected.</p><p>The problem is hidden again in the ServiceWorker implementation. Since their main purpose is to create a caching layer and intercept http requests, for consistent page behavior when registering the worker, the current tab does not become its client, because it has already started working without using the worker. After refreshing the page, it loads already with the registered worker, and then the connection works.</p><p>To make the worker immediately catch the current tab, we have resorted to the built-in method self.clients.claim(), which we also read about in the documentation. This method allows us to make all tabs in the worker’s scope to become its clients.</p><p>If there is no support for workers in the browser, our proxy page switches to fallback mode and establishes a direct socket connection without a proxy. Currently, 3% of our active connections works in fallback mode.</p><h3>Optimizing counter requests</h3><p>One socket connection is fine, but requests for a new counter still go via http. Since in our socket implementation, information exchange takes place in a one-way format, the counter itself must be requested via http. And at some point each tab will try to update it via http. Here again, ServiceWorker comes to the rescue.</p><p>We have developed a separate worker, linked to the chat domain this time, and a proxy-page to access it. The counter proxy page builds in connection proxy-page and catches its updates. When the counter needs to be updated, a request gets into worker, where debounce takes place. A request for a new counter will happen once and for all tabs at a certain time. The received value will be proxied to all clients — proxy pages of this Worker, and they will transfer the new value into parent app, where we can write a new digit.</p><p>As a result, the load on the unread message counter has decreased by 40 times. But this solution has its disadvantages as well. Because we’re cheating the browser in some way, one day that loophole might be sealed, and we’ll have to look for a new solution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0NU1NaXsXtgYDc0accxdJQ.jpeg" /></figure><h3>Summary</h3><p>At the moment, this system has been running in production for almost a year and does not create any problems either for users or developers. Each of our applications on any domain has the ability to access a single socket connection and receive events from the server in real time.</p><p>I’m sure we’ll find even more uses for this technology in the future, and we’ll delight you with new features that work at blazing speed.</p><p>That’s all I have to say. Let us know how you’ve used the Workers and what approaches you’ve used to solve similar problems. It will be interesting for us to learn new things from you.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c93add6cbbe8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hh-ru/sharing-a-socket-connection-between-tabs-c93add6cbbe8">Sharing a socket connection between tabs</a> was originally published in <a href="https://medium.com/hh-ru">hh.ru</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to bring off a gentleman’s release]]></title>
            <link>https://medium.com/hh-ru/how-to-bring-off-a-gentlemans-release-9884a954e4a7?source=rss----d470d2ec65f1---4</link>
            <guid isPermaLink="false">https://medium.com/p/9884a954e4a7</guid>
            <dc:creator><![CDATA[Dan Kheyfets]]></dc:creator>
            <pubDate>Tue, 05 Jul 2022 06:03:21 GMT</pubDate>
            <atom:updated>2022-07-05T06:03:17.120Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G6d5yGXsBewVIjmKeUrgpQ.jpeg" /></figure><p>Releases don’t come out exactly on time? Lots of bugs on regress? Does regress itself take several days? Lots of bugs complaints? Is the release being handled by everyone at once and product development completely halted? Can you relate? Do you feel the same way? I think a lot of people have faced this, including us.</p><p>Hey, everybody! My name is Daniel, and I’m QA lead of the mobile department at hh.ru. Today, I will share how we moved away from unstable and irregular releases to a cool and fast mobile app release process. Our releases have started to come out once a week, and the number of complaints has dropped manifold. I’ll also tell you what problems we encountered and how we solved them.</p><p>This is a text transcript of <a href="https://www.youtube.com/watch?v=8siVB28i3ck&amp;ab_channel=hhtech">our vlog</a>, so if you prefer to watch it rather than read it, welcome to <a href="https://www.youtube.com/channel/UCWztYdpQS3v9Gjno8bBZcNA">our Youtube channel.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OYu-qxr86kqL9UBhHa5KIA.png" /></figure><h3>Houston, we have a problem</h3><p>Let’s start with what problems we’ve had with releases:</p><ul><li>Huge and unpredictable feature set</li><li>No exact release date known in advance</li><li>Long and completely manual regress</li><li>Unpredictable release date of new features</li><li>Interface changed too much from release to release</li></ul><p>Between long releases, the app changed a lot, both visually and internally, which really made it hard for our users. If three months ago they had one interface and a certain feature set, then in the new release users could get a completely new application, with new features, new interface, but without any of the old and familiar things. Such a drastic change caused nothing but negativity from users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VYBoUOEy10ORvTlrXK8sFw.jpeg" /></figure><h3>The solution we wanted</h3><p>Based on the problems we identified, we decided what we wanted to end up with. We need to make stable, regular and small releases. The more frequent the release, the less functionality goes into it, and therefore updates will be easier for the user. With releases once a week, we can drop new functionality often and in small amounts. Users will not feel like they have a new app every time we update it. Because of the small number of changes, the release itself will be as stable and tested as possible. It will be much easier for the testing team to regress such a release, and the fewer changes, the less chance of a bug hitting the production.</p><p>It will also give us a fast time-to-market and allow us to schedule the delivery of this or that functionality to users.</p><blockquote>Prerequisites</blockquote><blockquote><em>Three years ago, we had 2 small platform teams: iOS and Android + a few QA, but also we were planning to expand the number of developers and testers. If we had not begun to solve the problems listed above, things could have gotten much worse. As long as there are few of you, and you are all around, you can just ask each other what is going on with the release now, what its status is, and what got into it. But if a number of teams is growing, and they get distributed, life with chaos in releases gets much more troublesome.</em></blockquote><h3>Time to solve problems</h3><p>The problems are clear, the goals are set, and now it’s possible to try to solve them.</p><p>To begin with, we identified the main technical problem in development processes — unstable develop branch. In our understanding, “unstable” means that there is untested or even non-working functionality, and the appearance of new functionality can spoil the existing one.</p><h4>Duty</h4><p>The first thing we decided to do was just fix it with processes! We will allocate people who will be responsible for the develop stability. They will fix bugs, crashes from the production, create a release and make sure nothing crashes. So we had a duty: for a week or two allocated a few developers and a tester, who only engaged in plugging holes in our endless haze of bugs and crashes. And it seemed like an OK idea, but not really. This approach turned the team off from product development almost completely, the focus on current tasks was lost, and employees were demotivated, as all work was concentrated on fixing other people’s bugs.</p><h4>Github-flow</h4><p>Then we decided to figure out why we were always getting something unfinished or broken in the develop branch. We figured it out. We worked on so-called “trunk-based development,” but in our case it was dirty trunk-based. We never tested anything that got into development, but rather tested some intermediate variant, for example when a feature is completely made or when several bugs are fixed.</p><p>We didn’t like this approach, so we decided to change the workflow. We switched to github-flow: we decided that each new feature will be done in its own branch. That means that all small problems, into which the feature is decomposed, will merge into this branch. When all the features are done, a tester will take this branch, build the application and test it. And if all goes well, then you can merge the whole thing into the develop. So we got rid of the situation when in develop branch we have something untested or unfinished.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QEYy7rizDz825zds9o_BwA.png" /></figure><p>Of course, this approach has its downsides. While development is performed in a separate branch, new functionality can appear in the develop, and the developer periodically synchronizes his branch with the develop branch. Because of this it turns out that the tester has to recheck some functionality, if something affecting it comes from the development. But we’ve put up with it, and to be honest, we don’t struggle with it much.</p><p>This approach certainly helped us a lot in stabilizing the development, but it also put a lot of stress on the testers. The whole process was locked in on the tester: until they test and approve the merge of a new feature into the develop branch, nothing could be merged. And in addition to testing the feature itself, we had to make sure that nothing had fallen apart from what had already been done. We realized it was time to save our QAs.</p><h4>Autotests</h4><p>Perhaps the easiest way to save the tester from the routine of endless regression checks is autotests. No problem, let’s start writing them.</p><blockquote>Thoughts Out Loud</blockquote><blockquote><em>I would like to make a small digression here: you may get the impression that all these process changes and the introduction of autotests took place instantly, quickly and painlessly. Of course not. The transition to github-flow was a long process that was constantly improving, the scope of duty responsibilities is constantly being reviewed and changed. And writing autotests in general is a big, complex and long task. Everything was gradual and steady.</em></blockquote><p>We started exploring approaches and frameworks for writing tests. We decided on <a href="https://developer.apple.com/documentation/xctest">XCUITests</a> for iOS, and <a href="https://github.com/KasperskyLab/Kaspresso">Kaspresso</a> for Android. To learn more about how to choose a framework for autotests we told in our series of <a href="https://www.youtube.com/watch?v=jVATgT-IOEY">videos on questions about automation</a>.</p><p>Along with writing autotests, we immediately started integrating them into our CI. We use build-server Bamboo, which is a tool from Atlassian stack. We use <a href="https://github.com/MarathonLabs/marathon">marathon utility</a> to run android tests in parallel, and native solutions from xcode for iOS.</p><p>Actually, autotests that are only run locally and are not launched regularly on CI are a very bad practice. Tests will be useful only when any interested person can easily run them and see the results of the run.</p><p>When the first autotests were written and the launch was set up, we immediately integrated them into our development process. We set up a test run plan (a plan is a pool of tasks to be done) on Bamboo, where every night we ran tests on all branches of features and develop. Now at the beginning of each work day developers can see how things are going.</p><p>Besides, green tests are now a mandatory condition for a new feature to get into develop. After a tester has tested the feature, a developer opens a pull request to develop branchand runs tests on this PR. If all goes well, merge, if not, then please fix it.</p><p>This way we greatly relieved the testers, and added another degree of protection for our developer from instability.</p><h3>Release train, that could</h3><p>So, we achieved a stable development, but there was still one problem: the unstable release date. Even if we agreed on a certain date, it didn’t mean that the release would be exactly on time… We kept getting requests like “ wait for us”, “well, while there’s regression going on, can we add some more?” and so on. One of the reasons for these requests was the lack of a clear understanding of the next release deadline. If you didn’t get into this release, there was a risk of waiting too long for the next one.</p><p>We didn’t like it that way, so we decided to guarantee a date for each release to our colleagues, so that they wouldn’t have the fear of hanging around in the void for too long. We introduced such a practice as “release train”. Furthermore, we agreed that once every two weeks, on a certain day and hour, we would do a release thread and not wait for anyone. If you didn’t make it on time, sorry, wait for the next train. The developer who didn’t make the release didn’t get too upset, because he clearly knew when the next release would be, and that he would definitely make it.</p><p>Of course, it was still just a verbal agreement. We tried to be flexible: to go into individual situations and even wait for someone. But on the whole, things were headed for a serious clamping down. At some point we began to strictly follow the release schedule, and the releases began to arrive weekly, without any delays. Eventually, we stopped being asked to wait for some unfinished functionality.</p><h3>Brief summary</h3><p>What we had:</p><ul><li>Large and unstable releases</li><li>No clear release date</li><li>Long regress</li><li>Buffled release content</li></ul><p>What practices were introduced:</p><ul><li>Github-flow</li><li>Autotests</li><li>Duty</li><li>Release train</li></ul><p>With this we have tamed the chaos of releases. The releases have become regular and small. The time-to-market has also been significantly reduced.</p><p>A lot of people wonder if things have really gotten that good. Mostly, yes. You can go about your life in peace with this approach. We’ve been functioning this way for more than a year now, and we don’t have any problems at all. However, there are a lot of routines in the release process that we would like to get rid of.</p><p>All the release routines can be divided into two parts: technical and manager’s</p><p>Technical part includes the following:</p><ul><li>Manually create release branch</li><li>Upgrade the version there</li><li>Complete the build</li><li>Run autotests</li><li>Drop in a store</li><li>Notify everyone</li></ul><p>During the release, the manager would create a page in Confluence that would have stated who was responsible for the release, the release contents, release dates, test runs, etc. In addition, we had to order texts for the “What’s new” section, and each platform has its own section.</p><p>Even though these are not complicated operations, but they take time and require human involvement, so there is always a risk of making a mistake or forgetting something. You don’t have to worry about this, but at hh.ru we like to automate everything. If some routine can be automated in a reasonable amount of time, then we will do it.</p><p>In the next article we’ll share a story about how we automated everything and what we got out of it.</p><p>Stay tuned!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9884a954e4a7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hh-ru/how-to-bring-off-a-gentlemans-release-9884a954e4a7">How to bring off a gentleman’s release</a> was originally published in <a href="https://medium.com/hh-ru">hh.ru</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we made a shift from kotlinx.synthetics to Android View Binding]]></title>
            <link>https://medium.com/hh-ru/how-we-made-a-shift-from-kotlinx-synthetics-to-android-view-binding-2d8b39ae667b?source=rss----d470d2ec65f1---4</link>
            <guid isPermaLink="false">https://medium.com/p/2d8b39ae667b</guid>
            <dc:creator><![CDATA[Ivan Antsiferov]]></dc:creator>
            <pubDate>Tue, 28 Jun 2022 06:01:19 GMT</pubDate>
            <atom:updated>2022-06-28T06:01:18.615Z</atom:updated>
            <content:encoded><![CDATA[<p>Hi! I’m Vanya, I’m Android-developer from hh.ru’s product team. In this article, I will speak on our experience of migration to <em>ViewBinding</em>.</p><p>In late 2020, the official Android Developers blog announced that the android-kotlin-extensions plugin for Gradle is no longer Koltin friendly starting September 2021 and will be declared e̶x̶c̶o̶m̶m̶u̶n̶i̶c̶a̶d̶o̶ deprecated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Kul5txXQ0X5fEp8CAFkwg.jpeg" /></figure><p>Our Android app has a rather large code base, and this plugin was used everywhere and for every screen. We didn’t want to make a code freeze and send all the developers’ efforts for migration, so we decided to try to automate the refactoring process and make it iterative at the same time.</p><p>For those who aren’t the fans of reading, we made a <a href="https://www.youtube.com/watch?v=mGfg8M1Bi_g&amp;feature=youtu.be">video version</a>.</p><h4>Why is migration needed?</h4><p>In late 2020, there was a <a href="https://android-developers.googleblog.com/2020/11/the-future-of-kotlin-android-extensions.html">post</a> warning that the android-kotlin-extensions plugin for Gradle would no longer come with Koltin. Even though as of February 2022 it still comes with Kotlin, migration was still inevitable. After all, sooner or later it will be removed from there, and the price of using synthetics in the project now is the potential inability to update to new Kotlin versions in the future. Of course, this will be one of the other problems that arise when using kotlinx.synthetics.</p><p>Google immediately offered <a href="https://developer.android.com/topic/libraries/view-binding">ViewBinding</a> as a replacement, so in this article we will not look at alternatives, but focus on the features and pitfalls of switching from one method of working with <em>View</em> to another.</p><h4>What we had</h4><p>Before we started the work, we decided to estimate the problem scale. We had 570 files using kotlinx.android.synthetic and about 5 thousand requests to <em>View</em> using synthetics. There were also 3 types of classes that used this way of working with <em>View: Fragment, Custom View </em>and<em> Cell</em> — our abstraction over list elements, saving us from writing a boilerplate. And all of them were permeated by it through and through.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fm9akqmfN7dCZeSyU5TLfw.jpeg" /></figure><p>We had 570 files using kotlinx.android.synthetic and about 5 thousand calls to View using synthetics. There were also 3 types of classes that used this way of working with View…</p><p>The task was shaped in the following way:</p><ol><li>Add a <em>Binding</em> object declaration to each class with Synthetics. Different for each of the 3 types of classes</li><li>All call to <em>View</em> via Synthetics are to changed for requests via binding</li><li>Delete synthetics imports, add imports for <em>ViewBinding</em></li><li>In the build module files, remove the kotlin-android-extensions plugin and add android.buildFeatures.viewBinding = true</li></ol><h4>What should be used to migrate to ViewBinding?</h4><p>Surprisingly, no ready-made migration solutions suitable for our project could be found.</p><p>The first thing that came up in our heads was to write a script, for example in Python, which:</p><ul><li>Will parse all .xml files in the<em> src/main/res/layout</em> folders of all modules in the project for the “<em>android:id=”@+id/whatever</em>” attribute</li><li>With a lot of IDs, will go through Fragment.kt and similar files, replacing the calls corresponding to the parsed IDs with <em>binding.someViewId</em></li></ul><p>It seems that with such an approach, it would not be difficult to perform all the other task stages as well. But we rejected it because of the unreliability of the solution: with .kt files the script would work as with plain text, so there is a great risk of getting completely invalid code, for example, when accessing within <em>ViewHolder</em> via <em>itemView</em>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b75b3c236aa08699bd2387d9287a7295/href">https://medium.com/media/b75b3c236aa08699bd2387d9287a7295/href</a></iframe><h4>Why we chose IntelliJ Plugins</h4><p>The main reason is that IDEA plugins allow you to work with project code as<a href="https://ru.wikipedia.org/wiki/%D0%90%D0%B1%D1%81%D1%82%D1%80%D0%B0%D0%BA%D1%82%D0%BD%D0%BE%D0%B5_%D1%81%D0%B8%D0%BD%D1%82%D0%B0%D0%BA%D1%81%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%BE%D0%B5_%D0%B4%D0%B5%D1%80%D0%B5%D0%B2%D0%BE"> Abstract Syntax Tree</a>, presenting each element of our code as a typed tree element object. Such an object contains all the necessary metadata about the code element it represents. So we were confident that this way we could define calls to the <em>View </em>through Kotlinx Synthetics.</p><p>This solution seemed more reliable than working with code as with text. We didn’t expect false responses from it, either. In addition, plugins have access to the project’s indexes, which allows us to collect data about the connection between the code and the <em>xml layout</em>.</p><p>The second reason is that the plugin is more flexible than the script. Accordingly, it will be easier to adapt it for other projects.</p><p>The third reason is the team’s expertise growth. One more person’s dive into the subject has improved the bus factor.</p><h4>Contiguous migration or single shots</h4><p>IntelliJ Plugins allow you to write custom actions for the context menu that are applied to a user-selected IDE file, so we decided to use this method to implement our refactoring.</p><p>But why not do refactoring for the whole project at once? You can run the plugin, it goes through the whole project — profit. But thanks to the experience of our previous global refactoring, we realized that this is a completely uncontrollable process, which will also break the building of the entire project, until the developers fix each file manually. Therefore, step by step.</p><h4>What we have achieved</h4><ul><li>Find ~95% of synthetics usages and replace them with <em>ViewBinding</em></li><li>Replace file’s imports</li><li>Add a declaration of the <em>View Binding</em> property to the class (<em>Fragment, Cell, View</em>)</li></ul><p>The last point from the initial task: change the <em>build.gradle</em> in the module — was not implemented, but through plugins for IDEA it is quite possible. You can see an example in <a href="https://github.com/hhru/android-multimodule-plugin">other plugins</a> from our team.</p><p>Import replacements and property declaration is not that difficult and interesting task, you can see the detailed implementation of these items in the <a href="https://github.com/hhru/hh-histories-view-binding-migration-plugin">plugin repository</a>. But I will tell you about implementation of the funniest part — finding invocations through synthetics and replacing them with <em>ViewBinding</em>.</p><h3>Synthetics -&gt; View Binding</h3><h4>Searching for files for refactoring</h4><p>That was the first time I encountered plugin development for IDEA, and in many ways I was pleasantly surprised. For example, the following code is enough to check that the selected file is suitable for our plugin:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e79517584a7da5e143a0dddd04b87116/href">https://medium.com/media/e79517584a7da5e143a0dddd04b87116/href</a></iframe><p>The code of <em>hasSyntheticImports</em> extension is also quite declarative:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6d2e1399ac74441f06c20d4d2454c095/href">https://medium.com/media/6d2e1399ac74441f06c20d4d2454c095/href</a></iframe><p>This is all the code that was needed for our <em>Action</em> to recognize the correct files.</p><h4><strong>Searching for synthetics usage</strong></h4><p>The basic unit in IntelliJ’s Abstract Syntax Tree is the <em>PsiElement</em>.</p><p>All the possible object types in our code are inherited from it: operators, expressions, arguments, classes, and so on:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/0*scan-IwSug90CR9d" /></figure><p><em>A small explanation of PsiElement</em></p><p>It is a basic interface for any element that represents an object in code. Functions (e.g., KtReferenceExpression), annotations (KtAnnotiatonEntry), classes (KtClass), and whatever else is inherited from it: from variables to binary operators.</p><p>We needed to find <em>PsiElements</em> inside this tree, which represented the following expressions:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2e1a754d84309b6dd255f84d89cf1655/href">https://medium.com/media/2e1a754d84309b6dd255f84d89cf1655/href</a></iframe><p>To find and collect information to later replace them with <em>ViewBinding</em>, we decided to bypass the PSI tree of Kotlin files using inheritance from <em>KotlinRecursiveElementVisitor</em>. The <em>Visitor</em> pattern is a standard way to bypass code in IntelliJ Plugins:</p><p>By trial and error, we figured out that we need to redefine two callbacks:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3438583d4d5abb1e0c6a4ab7d293ec05/href">https://medium.com/media/3438583d4d5abb1e0c6a4ab7d293ec05/href</a></iframe><p>We get <em>KtReferenceExpression</em> and <em>KtCallExpression</em>, both inherited from <em>PsiElement</em>, as a function parameter in callbacks. From that interface we can get a lot of useful data about the element, for example: the text representation, all of its branches and links to all of the related objects. This is exactly the list of references we need.</p><p>The link is represented by the <em>PsiReference</em> interface, and contains the resolve() method, which allows you to get the <em>PsiElement</em> at which the link points to. It can be used to determine if the received element is a <em>View</em> ID:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/980378cfc0264f2c692d22fd7a9b0c0b/href">https://medium.com/media/980378cfc0264f2c692d22fd7a9b0c0b/href</a></iframe><h4><strong>Modifying the code</strong></h4><p>The last step is to modify the code. We need to do the following:</p><ul><li>Generate the text for a new code</li><li>Create a <em>PsiElement</em> heir object of the type we want</li><li>Replace a previously found call via synthetic with a new object</li></ul><p>The first action is pretty straightforward. We already got an <em>XmlAttributeValue</em> object, where the value field is the text from the <em>android:id</em> field of our <em>xml</em> file. For example “<em>@+id/fragment_about_description_text_view”</em>. We need to get a line like “<em>binding.fragmentAboutDescriptionTextView</em>” in any convenient way.</p><p><em>KtPsiFactory</em> will help us in creating a new piece of code and from this object we can create any element, such as arguments for functions and expressions:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f78e80f3f697f84087239ab5723dca41/href">https://medium.com/media/f78e80f3f697f84087239ab5723dca41/href</a></iframe><p>For now, this is just an object in our IDE’s memory, not written to the codebase. The last step is to replace the previously found <em>PsiElement</em> with it:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ca36bc2c3bacfec2af8b59216f9f1f04/href">https://medium.com/media/ca36bc2c3bacfec2af8b59216f9f1f04/href</a></iframe><p>As a result, our code has changed like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/984781cb6083c01a79f9abe7161ee3fc/href">https://medium.com/media/984781cb6083c01a79f9abe7161ee3fc/href</a></iframe><p>That’s all the basic part of our plugin’s algorithm. For more details on the synthetic search engine and code modifications, you can visit the<a href="https://github.com/hhru/hh-histories-view-binding-migration-plugin"> plugin repository.</a></p><h4>How plugin usage looked like</h4><p>We divided the codebase into 10 parts with roughly equal number of uses of kotlinx.android.synthetic. We applied the plugin sequentially to files from each part, then went through those files manually, fixing highlighted IDE errors. These were mostly nullable <em>View</em> calls that were supported by synthetic, but not supported by <em>ViewBinding</em>, which sometimes required a little local refactoring.</p><p>Much rarer were files that used more than one <em>.xml layout</em>. We didn’t teach the plugin to handle such cases, so we had to intervene here too. More details on how to handle such cases in the <em>ViewBinding</em> in general will be described in the “Pitfalls” section.</p><p>Another frustrating thing was the <em>with</em>, <em>apply</em> and <em>let</em> constructions with large attachment. The plugin couldn’t get very deep, and the calls to the View remained there in the old form of <em>some_view_id</em>.</p><p>The total amount of changed code is about +6800 and — 6000 lines. There is more code, but mostly due to the need to declare <em>ViewBinding</em> in every file where it is used.</p><h3>Pitfalls</h3><h4><strong>ID from android space</strong></h4><p>Since the refactoring didn’t create any new scenarios, but affected all existing ones, the UI tests helped to discover the vast majority of the problems resulting from the refactoring. Their run-through after the infusion of some refactoring usually looked like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/0*ndvxIcgb0TwEQvvF" /></figure><p>After a few hours or so of searching for the problem and its solution, it was getting better:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/0*4WGStGsNqJ_xNIz1" /></figure><p>Then the typical “right, I forgot to fix it still in the horizontal layout,” and finally:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/0*0YUu-UexfJkTqMTq" /></figure><p>Tests caught such a kind of rare case as a crash due to using <em>View</em> <em>“android:id=android:id/text1”</em> deep in the bowels of the old screens for <em>ArrayAdapter </em>with a drop-down list. The solution was to add:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/43ed96a40fb0980a9547eb6ed96281f4/href">https://medium.com/media/43ed96a40fb0980a9547eb6ed96281f4/href</a></iframe><p>in the <em>layout </em>file, which was used as an item in the <em>ArrayAdapter</em>. The <em>ViewBinding</em> cannot generate a wrapper for a <em>View </em>with an ID from android space.</p><h4><strong>The problem of layout with qualifiers</strong></h4><p><em>ViewBinding </em>cannot wisely generate a wrapper for <em>xml layout</em>, if different versions of it, for example for phones and tablets, have different <em>View</em> sets. It will only work fine if the base <em>layout</em> has a <em>View</em> that is not present in the other configurations.</p><p>The easiest solution was to write your own <em>Binding</em> class, which has a proxy to all <em>View</em> from all configurations, making them nullable:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bd897d6221ed45febe52e039032e3e64/href">https://medium.com/media/bd897d6221ed45febe52e039032e3e64/href</a></iframe><h4><strong>&lt;include&gt;</strong></h4><p>We often use <em>include</em> to break up a large <em>layout</em> into separate parts or to reuse its fragments. There are a couple of pitfalls hidden here, too.</p><p>Each<em> included layout </em>will have its own <em>ViewBinding</em> class, which can be obtained in different ways, depending on the following conditions:</p><ol><li>If the root <em>layout</em> tag we specified in the include is &lt;<em>merge</em>&gt;, then we need to manually create a <em>Binding </em>object for it:</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2e45e3e5bbda3a5f92f5b61932cfd764/href">https://medium.com/media/2e45e3e5bbda3a5f92f5b61932cfd764/href</a></iframe><p>So, besides the main <em>ViewBinding</em> object, we will have auxiliary ones for each <em>included</em> <em>merge</em> <em>layout</em> of our fragment. And one more important thing — in <em>xml</em> for such a tag you can’t specify the <em>ID</em> attribute, it will result in a compilation error.</p><p>2. If the root tag is an any <em>View</em>, the situation becomes easier. Be sure to specify an <em>ID</em> attribute for our <em>included layout</em>, and then we can get it from the main <em>binding</em> object:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8158b5cc981b4e2ff2ed72903c20682f/href">https://medium.com/media/8158b5cc981b4e2ff2ed72903c20682f/href</a></iframe><p>Where <em>fragmentNotificationSettingsMainLoadingContainer</em> is the ID of<em> the included layout</em>, through which we get an attached object <em>ViewBinding</em>.</p><h3><strong>Highlights from Intellij Plugins development</strong></h3><h4><strong>Debugger</strong></h4><p>Another point worth mentioning is the debugger for IDEA plugins:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/0*_8rbFAJa4nSJHMaQ" /></figure><p>It allows you to run code in runtime and see the result, having all the objects of the scope where the current breakpoint is. The debugger helped me a lot when searching for a way to determine that the current code element is a <em>View</em> reference.</p><h4><strong>Internal Actions Viewer</strong></h4><p>The last killer feature for plugin developers, which helped me enormously, is <em>Viewer</em> for any interface elements. Click control + option + lmb on any button, for example, in the file context menu, and you see this window:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/0*47_bhs4SyJm-31Mm" /></figure><p>In the <em>property</em> column, pay attention to the <em>Action</em> value — there is written the specific class that performs the action for which we called this window. This way, you can see how JetBrains themselves have implemented various IDE functions, such as code formatting or renaming a class. A lot of things can be seen and adopted for your plugins.</p><p>To enable this debugging option, you need to write a flag in the IDEA according to <a href="https://plugins.jetbrains.com/docs/intellij/enabling-internal.html">these instructions</a>.</p><h4><strong>MISSING_DEPENDENCY_CLASS</strong></h4><p>The most troublesome problem is that it slowed me down a lot in the beginning. It seemed to connect everything correctly, but the IDE highlighted an error in the code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/0*_gvU9RILRux9bPZ4" /></figure><p>At the same time, the project was getting built, but the previously mentioned debugger functions refused to work, which greatly slowed down the process of learning the API of working with Abstract Syntax Tree.</p><p>After some time of research, I managed to find out that the problem was not in my build file, but in the Kotlin plugin for Gradle. You can read about it in more detail in the relevant issues <a href="https://youtrack.jetbrains.com/issue/KTIJ-19485">here</a> and <a href="https://youtrack.jetbrains.com/issue/KTIJ-10861">here</a>.</p><p>The problem is solved very simply (and weirdly):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef2c3fbbddccdf03e85e04641b514c18/href">https://medium.com/media/ef2c3fbbddccdf03e85e04641b514c18/href</a></iframe><p>Simply put, we add manual cast to the necessary type <em>file.getClasses()</em> as <em>Array&lt;PsiClass&gt;</em>. The IDE will highlight it as redundant, but the debugger will start working and <em>MISSING_DEPENDENCY_CLASS</em> will stop bothering you.</p><h3><strong>Instead of summary</strong></h3><p>Developing autorefactoring through a plugin was an interesting experience. I definitely recommend you to consider this method if your code refactoring requires some metadata about the objects, for example, to check the type of variable or the return type of function. If it is not specified in the declaration, then working with the code as text, it will be impossible to do this, while PSI will have this data.</p><p>On the other hand, for simple cases, such as replacing the package in all the files, Python looks like the fastest solution, and in many cases the built-in IDEA refactoring tools will be enough.</p><p>In general, as always, it is necessary to choose a solution tool individually for each problem, but I can definitely recommend taking Intellij Plugins into your arsenal.</p><h4><strong>Useful links</strong></h4><p>Issue on the tracker with a warning to declare deprecated kotlin-android-extensions plugin: <a href="https://youtrack.jetbrains.com/issue/KT-42121">https://youtrack.jetbrains.com/issue/KT-42121</a></p><p>If you want to dive into developing your own IDE plugins, here are some useful links:</p><ul><li>You can start with a workshop by my colleague, where he develops a small plugin from scratch and shares the handy lifehacks for quick immersion onto the topic: <a href="https://www.youtube.com/watch?v=3jVkNsWPJ20">part 1</a> and <a href="https://www.youtube.com/watch?v=ohMEwwmb_X8">part 2</a>.</li><li><a href="https://github.com/JetBrains/intellij-platform-plugin-template">Template of plugin from Jetbrains on Github</a>. It is already configured CI and all sorts of basic stuff. I recommend cloning it. This will save some time on writing a boilerplate and setting up the project.</li><li><a href="https://plugins.jetbrains.com/docs/intellij/github-template.html">The official Intellij Plugins documentation</a> was unfortunately much less helpful because of the almost complete lack of examples, but there is plenty of theory.</li></ul><p>And, of course,<a href="https://github.com/hhru/hh-histories-view-binding-migration-plugin"> the repository with our plugin</a>. This solution is by no means universal and will definitely require modification if you want to use it, but it will certainly give you a glimpse of something useful.</p><p>Thank you for your attention. Have fast migrations and less problems with automation, see you on our blog!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d8b39ae667b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hh-ru/how-we-made-a-shift-from-kotlinx-synthetics-to-android-view-binding-2d8b39ae667b">How we made a shift from kotlinx.synthetics to Android View Binding</a> was originally published in <a href="https://medium.com/hh-ru">hh.ru</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Modularity lifehacks on iOS]]></title>
            <link>https://medium.com/hh-ru/modularity-lifehacks-on-ios-f52ec54f452e?source=rss----d470d2ec65f1---4</link>
            <guid isPermaLink="false">https://medium.com/p/f52ec54f452e</guid>
            <category><![CDATA[ios-development]]></category>
            <category><![CDATA[tuist]]></category>
            <category><![CDATA[ios]]></category>
            <category><![CDATA[swift]]></category>
            <category><![CDATA[xcode]]></category>
            <dc:creator><![CDATA[Yura Privezentsev]]></dc:creator>
            <pubDate>Tue, 21 Jun 2022 06:03:29 GMT</pubDate>
            <atom:updated>2022-06-21T06:03:28.516Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello, everyone! My name is Yura and I’m an iOS-developer in core-team at hh.ru. In this article I’m going to talk about how we work with multiple modules in our iOS-apps. We’ll discuss the environment and structure of our project, touch the topic of build time and dive into code generation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WBC7jvEFSV2FgGgeB1SnjA.jpeg" /></figure><h4><strong>Our team and projects</strong></h4><p>Our core-team is responsible for making it convenient to work on the app for other iOS teams. We create automations, improve codebase and refactoring and so on.</p><p>Currently we’re working on two applications: for applicants and employers. 10 iOS developers are working on both projects, and we’re distributed between 5 teams in the mobile department. In order to make simultaneous work comfortable, we split our app into multiple modules or features.</p><p>In our terminology feature is a separate Xcode project with its own isolated code and everything needed like resources or texts and which is connected to other features and the main app. Thanks to such a feature division we provide possibility for independent work in each team.</p><p>In 2018 our application was a huge monolith piece of code, poorly divided into four barely independent, but still interrelated parts. Lately we’ve been cracking and dragging these parts, so now the monolith looks much smaller. As for new features, we make them all isolated. Nowadays there exists 75 independent features in our app, it has become some kind of a container ship which carries a lot of container features.</p><p>But with the division came the problems. In the article I’m going to thoroughly describe the following:</p><p><strong>Firstly,</strong> such a huge number of features is difficult to support in the app: you need to follow the coherent structure and networks between them.</p><p><strong>Secondly</strong>, our project is really big. As it consists of a great amount of lines of code, the compilation takes a lot of time.</p><p><strong>Thirdly, </strong>a big project is hard to<strong> </strong>maintain due to the<strong> </strong>tremendous number of agreements and relations between features. In a nutshell, the size of the project complicates the development.</p><h4><strong>Problem 1: Project structure support</strong></h4><p>As it naturally comes, a huge number of xcode projects are difficult to manage. You must have faced the conflicts of xcodeproj files while merging. Most of the time they are solved pretty easily. Nevertheless, if they appear quite often, it becomes annoying to waste time on them.</p><p>As the app consists of many independent features, their support can be difficult. While managing projects in Xcode interface you can change some projects, but easily miss or forget to add something to others. That’s why we started looking for the source of the problem right away.</p><p>At first we used <a href="https://cocoapods.org/">CocoaPods</a> — popular dependency manager. We realized that we could build our multi-module architecture via applying development pods to our features. Inner projects for CocoaPods are written as a special podspec file. It’s smaller than the Xcode project description and contains only the main information.</p><p>However, with growing project we started to encounter more and more problems and restrictions from CocoaPods.</p><p>For example, CocoaPods forbids its simultaneous work with Swift Package Manager and Carthage for external dependencies. At some point it became a serious problem for us. Plus, CocoaPods is quite tricky about cashing dependencies.</p><p>We started contemplating leaving Cocoapods, but we need to decide where to go and what other tools we could use for external dependencies. As a result of our thorough search, we came across <a href="https://tuist.io/">Tuist</a>. It’s a utility which generates and supports Xcode-project.</p><p>Project description with Tuist looks something similar to package description for SPM but supports nearly all settings of xcodeproj. Thanks to Tuist, our project description has become even easier for comprehension.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8a7c2a43a1dffc0bba0ea82a8124f350/href">https://medium.com/media/8a7c2a43a1dffc0bba0ea82a8124f350/href</a></iframe><p>We hid all the meta information which could be found in other sources, and left only the most important things. It comes down to mentioning the name of the feature, which resources should be generated for it with <a href="https://github.com/SwiftGen/SwiftGen">SwiftGen</a> and what targets it has. The lists of dependencies are formed automatically for each target. It is done with the help of the script that looks for all import in the target’s files. The writing of the project got easier, and the process became more comfortable.</p><p>While we were moving from CocoaPods to Tuist, we noticed that the location of our project’s files was quite peculiar due to some historical reasons and God-knows-what else. To avoid repeating the action twice, we cleared the structure of the project and arranged everything, so now it lies in the right folders. That was what the developers thanked us for.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OvfoQj_hrI2JSu6A4uWeig.jpeg" /></figure><h4><strong>Problem 2: Slow compilation</strong></h4><p>At some point the developers noticed that project build took around five minutes. That became a huge problem, as they got distracted during that time and fell out of the loop, switching to other tasks. That’s why a lot of time was wasted on diving into the work again.</p><p>We started thinking how we could fix that and decided to begin with gathering honest statistics on how much time is spent on our projects’ compilation. That’s where <a href="https://github.com/MobileNativeFoundation/XCLogParser">XCLogParser</a> came to our rescue. Xcode presents reports after build in xcactivitylog format. These reports contain all the information about the build, including each file. At the same time, XCLogParser can parse all these reports into a format which is easier to understand for an ordinary human. For example, reflect it visually or in json format.</p><p>Here’s an example of json, which is presented by XCLogParser after report analysis. The line that interests us is the time spent on the project compilation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2I0uE2gSnr2wNHNnOG38Xg.png" /></figure><p>Nevertheless, this json is actually enormous and it contains the most detailed information possible about each step of project build. This is what really excites us! We take all this information and send it to our database. There we use it to create beautiful graphics about the time spent on project build in <a href="https://grafana.com/">Grafana</a>. That’s what graphics for apps look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*chIfav-plylJr07C7dB3ZA.png" /></figure><p>They’re going up and down, but it’s ok, because currently we have two different types of machines on CI: processors Intel and M1. The machines on M1 build a project faster, of course. We don’t just look at these graphics; they help us find problems in the configuration of our projects. You can find more details on changing the speed of build in <a href="https://medium.com/hh-ru/build-time-in-ios-56bbc15c9c8f">this article</a>.</p><h4><strong>Environment setup</strong></h4><p>An obvious point. If a buggy commit got into the repository, this bug should be reproduced similarly: both in the developers’ mac and on CI. Therefore, if the bug is fixed, the fix should work on all devices too, the developers’ and CI.</p><p>This leads us to the idea that the project’s environment should be reproducible and unified, so that there are no true stories, when somebody complains in the chat about having a bug and the app build is screwed, whereas the guy on the other end replies: “WORKSFORME”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ei5dWP57634lp_g6S6xDsQ.jpeg" /></figure><p>In order to avoid such situations, we have a little lifehack. For example, apart from the swift code and the app itself, our project consists of various tools like Ruby, Fastlane, Danger, app dependencies which we install via Carthage, and swift utilities installed via Mint.</p><p>If you install, set up and launch these things by yourself, you can die a little. It means that the developer has to keep in mind a bunch of tools and remember the sequences of their installation, so that nothing crashes. And what if you have to update something? A whole different story.</p><p>We’d come to the conclusion that we needed a unified script for setting up the whole environment. And we wrote it:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3d42298173759279369925858aec1050/href">https://medium.com/media/3d42298173759279369925858aec1050/href</a></iframe><p>It installs all the necessary dependencies step by step, configures the project’s environment and does everything possible so that the developers don’t waste their time on thinking about it. Now it’s possible to download the project’s repository, launch our Bootstrap-script, make some tea, and come back later to find that everything is open and ready for productive work. The true wonder of this tool is that all dependencies and tools of the project are written in the repository. Simply put, the repository becomes the one and only source of truth. And this is very convenient — you can install and set up all this environment by using just one command.</p><h4><strong>Code generation</strong></h4><p>The next story is about code generation. One of the tools that we use — Xcode templates. I assume you’re already familiar with them. There are lots of useful articles on Xcode templates on the Internet, and they do their thing pretty well. The only drawback is a sophisticated configuration process. If you add different conditions when generating the template, it becomes twice as more complicated. And it has a disgusting syntax. Well, you know.</p><p>Nevertheless, Xcode templates are pretty good for some small groups and separate files. They’re convenient and available right from the standard Xcode menu. This allows it to stay in one app, not to fall out from the context and create small files quickly. Our project has some templates for such purposes, like for MVVM modules. Also there is a set of small templates for generating models, analytics and view files, design systems and so on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*suwtb1TPyvSNHOp11szOYA.png" /></figure><p>The next tool is again Tuist. It offers a <a href="https://docs.tuist.io/commands/scaffold">command</a> for file generation — scaffold. It’s rather convenient for creating big file groups and folder hierarchy. We use it for generating new features. A new blank feature actually contains a lot of code. It arranges the necessary folder hierarchy right away, so that the developer can generate everything necessary and move on to completing their task.</p><p>Here you can both create separate files via stencil templates, which should be in the content of this file. All in all, we have everything needed for creating a new feature.</p><p>Apart from that, we use SwiftGen in our project for generating code and access to different resources, for example to images and localization strings. For all the rest we use Sourcery which enables us to generate mocks for Unit-tests and other details on the project. So it saves us for real.</p><h4><strong>Conclusions</strong></h4><p>Tuist makes working with the project a lot easier: both in generation, and in new features’ creation. We constantly monitor the speed of the build, which allows us to spot the problems as soon as they appear. We try not to build the whole project at once: we assemble only the most necessary parts, thanks to the idea of splitting the project into features. Last but not least, we automate the development process and feature configuration, so that developers don’t waste time on such routines.</p><p>That’s all for now! Share with us your experience in working with big projects in the comments and feel free to ask any questions!</p><p><em>Sounds fun? Subscribe to our </em><a href="https://t.me/hh_tech_news"><em>news channel on telegram</em></a><em> and “</em><a href="https://www.youtube.com/channel/UCWztYdpQS3v9Gjno8bBZcNA"><em>HHella cool stories</em></a><em>” channel so you won’t miss new videos, articles, and other news. And you can ask our engineers any questions on any topic in the comments or in </em><a href="https://t.me/hh_tech"><em>hh developers’ telegram-chat</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f52ec54f452e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hh-ru/modularity-lifehacks-on-ios-f52ec54f452e">Modularity lifehacks on iOS</a> was originally published in <a href="https://medium.com/hh-ru">hh.ru</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>